There isn't a single term for it (other than "Quake-like" or similar), because rendering engines can differ or not in a variety of ways.
That said, the Quake rendering engine does has several identifying features:
it's a true 3D engine, in that it could render actual 3D geometry and not the sort of extruded and offset 2D maps of some of its predecessors.
it is a brush engine, in that it uses oriented convex 3D geometry to define interior spaces of the game world.
it is a BSP engine, because after pre-processing the brushes that define a map, a BSP tree is built for dealing with polygon visibility.
it used scanline rasterization, as noted in the comments, to produce the final scene image.
...and so on. There are several other ways in which you could classify the engine (such as how it rendered the environment front-to-back but still wrote to a depth buffer for handling the rendering of mobile objects, like characters). But you can already see how calling it a "true-3D-brush-and-BSP-using-scanline-rasterization-engine" starts to get a little cumbersome. So we just call it a "Quake" engine.
You can see examples of how the same basic engine can still handle some things quite differently in exploring how the old Thief games rendered the world.
And of course, if you really want the nitty gritty details of Quake's engine, you can peruse Michael Abrash's extremely in-depth tome on the matter.