Yes, in instanced rendering, you typically supply transformation matrices.
And this is how you do that in OpenGL:
In the vertex shader, you supply vertex position and normal as you usually do, but now you also have a vertex attribute for the transformation like this:
in mediump vec4 position;
in mediump vec4 surfacenormal;
in mediump mat4 trf;
And then you multiply your position and surfacenormal with the transformation matrix like so:
mediump vec4 transformedpos = trf * position;
mediump vec4 sn = surfacenormal;
sn.w = 0.0;
mediump vec4 transformedsurfacenormal = trf * sn;
Next, use the transformed version in your vertex shader as you usually do.
The matrix on the application side, is treated as 4 consecutive vertex attributes of 4 floats each.
You need to tell OpenGL that these vertex attributes need to advance to the next value, not for each vertex, but once for each instance. You do this with glVertexAttribDivisor() calls like so:
glVertexAttribDivisor( ATTRIB_TRF+0, 1 );
glVertexAttribDivisor( ATTRIB_TRF+1, 1 );
glVertexAttribDivisor( ATTRIB_TRF+2, 1 );
glVertexAttribDivisor( ATTRIB_TRF+3, 1 );
And tell OpenGL where to find these 4 consecutive attributes in the VBO:
glVertexAttribPointer( ATTRIB_TRF+0, 4, GL_FLOAT, 0, 16 * sizeof(float), (void*) ( off + 0*sizeof(float) ) );
glVertexAttribPointer( ATTRIB_TRF+1, 4, GL_FLOAT, 0, 16 * sizeof(float), (void*) ( off + 4*sizeof(float) ) );
glVertexAttribPointer( ATTRIB_TRF+2, 4, GL_FLOAT, 0, 16 * sizeof(float), (void*) ( off + 8*sizeof(float) ) );
glVertexAttribPointer( ATTRIB_TRF+3, 4, GL_FLOAT, 0, 16 * sizeof(float), (void*) ( off + 12*sizeof(float) ) );
Note that you will want to put the per-instance data in the VBO, after all the normal model (per-vertex) data, which in my code sample has size 'off'.
Typically, you will want to change the values for these transformation each frame. This means you need to update a part of your VBO and put in the new versions of the matrices. This is done with glBufferSubData()
const size_t sz = 16 * sizeof( float ) * objCnt;
glBufferSubData( GL_ARRAY_BUFFER, off, sz, object_matrices );
And finally draw all instances with a call to glDrawInstancedArrays()
glDrawArraysInstanced( GL_TRIANGLES, 0, numVerts, objCnt );
Here, numverts is the number of vertices in the model itself, not the total nr of verts drawn for all instances.
To sum it all up: the instance transformation matrix is treated as a per vertex attribute in your GLSL vertex shader. Yet, the data is fed not as a value per vertex, but only a new value per (typically 1) instance using glVertexAttribDivisor() which you need to call four times, once for each row in the matrix.