I have 2 polygons. I know the vertex coordinates of both polygons. What's the best way to check if one is completely inside the other? For example, the algorithm should only recognize the black trapezoid below as being contained:
There are tons of source snippets for a method that performs a test for "point inside polygon". The principle comes from Jordan's curve theorem for polygons (http://www-cgrl.cs.mcgill.ca/~godfried/teaching/cg-projects/97/Octavian/compgeom.html).
The naive way would be: having that method, call it
Theoretically, it shouldn't miss any scenario for your polygons, but it's not the optimal solution.
LATER EDIT: without edge vs edge intersection checks, as pointed out in the comments, the approach might return false positives for some concave polygons (e.g. a V shaped quad and a rectangle - the rectangle might have all of its vertices inside the V shape, but intersect it, thus having at least some areas outside).
UPDATE: after one checks for at least one of the vertices of the inner polygon to be inside the outer one, and if there are no intersecting edges, it means the sought after condition is satisfied.
Try doing a line intersection with each red line. In pseudocode:
However, as you can see, this solution will get slower as you add more polygons to check. A different solution could be:
This solution is very fast, but it depends on your implementation (and what you want to do with the result of your check) what solution works best for you.