I think a lot of the answers miss an important point: you can write apps that access hardware directly, but not on modern operating systems. It's not just a time problem, but a "you don't have a choice" problem.
Windows, Linux, OSX, etc. all ban direct hardware access to arbitrary applications. This is important for security reasons: you don't want any random app to be able to read arbitrary GPU memory for the same reason you don't want any random app to be able to read system memory. Things like the framebuffer for your bank account or whatnot live in GPU memory. You want that stuff isolated and protected and access controlled by your OS.
Only drivers can talk directly to most hardware, and the only way to talk to the driver is via the OS' exposed HAL and each driver's proprietary and incompatible interface. That driver interface will not only be different for each vendor but will even differ between versions of the driver itself, making it near impossible to talk directly to the interface in a consumer application. These layers are often covered by access controls that further restrict the ability for an application to access them.
So no, your game cannot just use the hardware directly, unless of course you're only targeting insecure operating systems like DOS, and your only feasible option for a game on modern consumer operating systems is to target a public API like DirectX, OpenGL or Vulkan.