The three columns of a 3x3 rotation matrix (or the upper 3x3 block of a 4x4 rotation-and-translation matrix) can be thought of as the images of unit vectors in the x+, y+, and z+ directions, respectively, after rotation.
That means we can find the first column by rotating the vector \$(1, 0, 0)\$ by the quaternion \$q\$:
$$\begin{align}
q (1i + 0j + 0k + 0) q^{-1} &= (q_xi + q_yj + q_zk + q_w) (i) (-q_xi - q_yj - q_zk + q_w)\\
& = (q_x^2 - q_y^2 -q_z^2 + q_w^2)i + 2(q_xq_y + 2q_zq_w)j + 2(q_xq_z - 2q_yq_w)k\\
\end{align}$$
And we can repeat this for each of the other columns, and use the fact that \$\|q\| = 1\$ to simplify the terms with the squares into the equivalent \$q_x^2 - q_y^2 -q_z^2 + q_w^2 = 1 - 2(q_y^2 + q_z^2)\$ and similar for the other axes. That gives us this 3x3 rotation matrix \$R\$ that does the same job as the quaternion \$q\$:
$$R = 2 \begin{bmatrix}
\frac 1 2 - q_y^2 - q_z^2 & q_xq_y -q_zq_w & q_xq_z + q_yq_w\\
q_xq_y +q_zq_w & \frac 1 2 - q_x^2 - q_z^2 & q_yq_z -q_xq_w\\
q_xq_z -q_yq_w & q_yq_z + q_xq_w & \frac 1 2 - q_x^2 - q_y^2
\end{bmatrix}$$
But of course for a view matrix, we want the inverse of this rotation. But for a pure rotation matrix where all columns are orthogonal unit vectors, that's just the transpose:
$$R^T = 2 \begin{bmatrix}
\frac 1 2 - q_y^2 - q_z^2 & q_xq_y +q_zq_w & q_xq_z - q_yq_w\\
q_xq_y - q_zq_w & \frac 1 2 - q_x^2 - q_z^2 & q_yq_z +q_xq_w\\
q_xq_z + q_yq_w & q_yq_z - q_xq_w & \frac 1 2 - q_x^2 - q_y^2
\end{bmatrix}$$
Now to translate this back so that some position vector \$\vec p = (p_x, p_y, p_z)\$ gets mapped to the origin, we can take the image of \$\vec p\$ after rotation by \$R^T\$:
$$\vec p^\prime = R^T \vec p = 2\begin{bmatrix}
p_x(\frac 1 2 - q_y^2 - q_z^2) + p_y(q_xq_y + q_zq_w) + p_z(q_xq_z - q_yq_w)\\
p_x(q_xq_y - q_zq_w) + p_y(\frac 1 2 - q_x^2 - q_z^2) + p_z(q_yq_z +q_xq_w)\\
p_x(q_xq_z + q_yq_w) + p_y(q_yq_z - q_xq_w) + p_z(\frac 1 2 - q_x^2 - q_y^2)
\end{bmatrix}$$
And subtract it to get the fourth column of our 4x4 view matrix \$V\$:
$$V = \begin{bmatrix}R^T &-\vec p^\prime\\
0 \, 0 \, 0 & 1\end{bmatrix}$$