I've come across games that are 3D that are playable in a browser. They require a plugin to be installed to work. I guess the plugin creates a D3D window inside the browser.
How would I go about implementing something similar?
Unity web player (for windows) can use both DirectX and OpenGL in a window (however it abstracts your access to these APIs).
Another approach would be to implement a DirectX plugin using ActiveX (for Internet Explorer), or NPAPI (for other browsers).
If you use NPAPI, when your plugin starts up you'll receive a window handle where you can draw whatever you want (including a D3D window). If you go this route you're essentially running native code on the clients machines.
The game you are playing might use WebGL http://en.wikipedia.org/wiki/WebGL, or something more exotic like a firefox plugin, but I don't think you can embed machine code in a firefox plugin (I'm not sure, that would be a little security flaw).
Unity implements a web plugin version of their engine, and it seems they do so with ActiveX. I don't know if it implements on Mac..., I wonder though.
The best way to find out how this game you are playing use a 3D API or any other kind of plugin stuff, is to look at the HTML source of the page, and look for object or activex or else.
Check out Firebreath for Cross-broswer support.