The two are not mutually exclusive. It's a reasonable approach as long as your comfortable working that way. I assume when you say pure C# you're still using MeshRenderer components, gameObjects, etc.
I've worked this way before in Unity, especially early on as I came from Objective C and had written a bespoke 2D Sprite system, and instinctively wanted to work the same way and have a lot of manual control.
I realised though, and this applies to your situation, that there is a certain fundamental design in Unity that can be difficult to work outside of without constantly wrestling with those paradigms. Then add to that the fact that there's a lot of functionally there that can speed up a workflow that you might be ignoring.
User input you mentioned is a good example. If you're not using camera-sourced raycasts and component lookups on the hit results, you might be instead doing it an unnecessarily harder way (I can only speculate).
I'm still code heavy in approach myself, and slightly more OOP than the recommended dosage. I use quite a feature-full boilerplate framework filled with convenience methods and shortcuts, but I also use and respect all of Unity's component based approaches, and editor features. I still get annoyed how so much "magic" process goes on behind a curtain, but I live with it.
Bottom line is I have benefitted greatly from adapting to and appreciating Unity's flow, design, and systems.