I have a parameter in HLSL shader in a constant buffer which is a
float3x3. Is there is 3x3 matrix struct in Directx 11 or should I use 4x4 matrix instead, and if so, how?
In general, you can store a 3x3 matrix in the upper-left of a 4x4 matrix (setting the bottom-right-most element to 1) if that 3x3 is only storing scale and/or rotation in 3D. If you were storing a 2D translation in there or anything else, you'd have to rebuild the 4x4 matrix appropriately based on whatever data you were putting into the 3x3 matrix.