If you can guarantee that the 8 points form a prism with pairs of opposite faces parallel (even if the corners aren't perpendicular), then we can simplify this significantly.
We'll construct a transformation matrix that maps a standardized cube to this prism, then invert that matrix, and check if the test point transformed by that matrix lies inside the standard cube.
We'll make a 3x3 matrix M whose columns are:
- P4 - P1 (width)
- P5 - P1 (height)
- P2 - P1 (depth)
You can confirm that taking any point in the cube formed by three coordinates in the range 0...1 and multiplying it by this matrix then adding P1 gives you a point in your original prism.
Then we can invert the matrix (use the matrix inverse function of whatever C# vector library you're using) and multiply your test point minus P1 by this inverse. If the test point were inside the prism, the output will have all three coordinates in the range 0...1 (with 0 and 1 being the outer faces). Otherwise, the point is outside the prism.
The advantage of this route is you can compute the inverse just once each time the prism is transformed, then re-use it to test many points against it cheaply. Each test is just one vector subtraction, three dot products, and six float comparisons.
Here's a C#-style example, assuming a sufficiently friendly vector math and matrix API:
public struct Cuboid {
Matrix3x3 _matrix;
Matrix3x3 _inverse;
Vector3 _corner;
// You can use 8 named arguments instead of an array,
// I just got tired of typing on my phone.
public Cuboid(Vector3[] points) {
_corner = points[0];
_matrix = new Matrix3x3(
points[3] - points[0], // left column
points[4] - points[0], // middle column
points[1] - points[0] // right column
);
_inverse = _matrix.inverse;
}
public bool Contains(Vector3 point) {
// Remap point into "standard cube space".
Vector3 standard = _inverse * (point - _corner);
// Check if the remapped point lands inside
// the 0...1 range of the standard cube.
return (standard.x >= 0) & (standard.x <= 1)
& (standard.y >= 0) & (standard.y <= 1)
& (standard.z >= 0) & (standard.z <= 1);
}
}
Note that even storing the matrix and inverse (for scale/rotation/skew) plus a corner (for translation), this data structure is still more compact (21 floats) than storing all 8 corner points (24 floats) and you can still extract all 8 corner points on demand. E.g. point[1] = _corner + _matrix.columns[1]
This method does not work if your 8-vertex object is not an affine transformation of a cube (i.e. if one of its faces is not a plane quadrilateral / not parallel to its opposite face).
