I ended up solving this with a little of guesswork. I suppose this can be optimized further.
First, some mental health. I declared a convert function:
vec3 convert(mat4 matrix, vec3 vector)
return (matrix * vec4(vector, 1.0)).xyz;
And then in the fragment shader I have my matrices like this:
mat4 ModelToWorld = Model;