No, scaling an object will not have adverse effects as described by yourself. Physics will still function. And like @gjocking mentions in his answer, floating point errors might creep in on a way bigger size.
The only adverse effect scaling has is the extra draw call if you are using dynamic batching
also in case of scaled meshes GPU re-normalization state must be set
(otherwise your lighting will go wonky), therefore scaled and
non-scaled meshes can not be batched together.
Also if a non uniform scale is used the mesh will be copied. ONLY if non-uniform scale is used. In case of uniform scale (both X, Y and Z scale are equal) Unity will NOT make a copy of a mesh and will be completely handled by the GPU.