I have created a rectangle, then converted it into an editable poly. Then I created a circle and converted it to editable poly too. How to make it so that that the circle becomes a hole in the rectangle?
That's called a Boolean operation. You can find a video tutorial here for almost the exact situation you're asking about.
Essentially, it looks like you create a new object with your two objects selected. You'll likely need to make your circle a cylinder first. The new object is a Compound Object of type Boolean. Then in the operations you select A-B (where A is your rectangle and B is your cylinder).
It's still a boolean operation. It's just done with splines that share a common plane. Check the Boolean heading on this page.
If you don't want to use booleans...
Obviously this creates a square hole. However, if you add more verts around the edges of the cube before using INSET, you could later manually shape the hole.
this video shows how to create a circle in your square using the 'loop tools' in the graphite editor. Works perfect, maintains quads everywhere. https://www.youtube.com/watch?v=nwzqn3K7RxE
I know, the question was asked about 3 years ago, but never mind. I don't think the boolean operation is a good solution ant it is not required here. It is desined for other cases, to my mind.
So, you created 2 splines. You should convert one of them, for instance rectangle, to editable spline, than Attach the second. Than apply Extrude modifier. The nested object automatically becomes a hole in the parent object.
So the recipe is:
Therefore extruding a circle inside a circle (in other words a flat a ring) creates a tube.
The cap created automatically will be without any edges in it. If you are bothering about topology - you may:
If you're creating game assets - that Quadify Mesh may be not desirable. But if you're going only to have smoothed model for one render - it's a quick way.
Your question seem to be more general than creating a cirular hole in a box but...
In case you want to create a perfectly round circle in a square (you may want to insert it into other objects) using TurboSmooth, you do
Here we are, having low poly model.
And that is is. Other steps are necessary only for smoothing: