When you nest objects in the Hierarchy, all the transformations applied to the parent get applied to its children.
That means that if a parent has a non-uniform scale applied, its child objects will be distorted non-uniformly - getting stretched or squashed more in one direction than another.
When you add a child to a parent, Unity (by default) automatically re-calculates its local transformations to compensate for the parent's, so the net transformation on it remains as close as possible to what you had before parenting it. This can mask the effect of the parent's transformation at first.
As you rotate the child object relative to the parent's scaled axes, you change which way it's getting squashed or stretched by the parent's non-uniform scale.
eg. if I have a narrow wall perpendicular to a highly stretched axis of the parent, and gradually turn the wall until it's parallel to that axis, the wall will get longer and longer.
As a result, it's usually most predictable & controllable if you try to keep non-uniform scales only at the leaf level of your transform hierarchy (objects with no children of their own). You can use it to stretch your cubes into skinny beams and flat walls. Then when you need to combine those objects into larger constructs, put them inside container objects that have only uniform scaling applied (ideally 1, 1, 1), and keep your scales uniform at all parent levels.