I want to know the difference in purely hardware level if there is any. For example the most simple question: Is there a chip for DirectX and another chip for OpenGL? What do hardware producers do to make their cards DirectX or OpenGL compliant?
No, OpenGL and Direct3D are both APIs that abstract the functionality of the actual hardware. Direct3D is a Microsoft application and standard while OpenGL is an open standard.
Any GPU will work with OpenGL and Direct3D as long as there's a driver for it. The driver is what takes Direct3D/OpenGL calls and does all the really low-level interfacing with the GPU. There is no "DirectX only" GPU and no "OpenGL only" GPU, unless for some strange reason the drivers only support one of the APIs.
From the graphics card's perspective, it's only communicating with the driver. The driver then communicates with all the APIs it supports.
Hardware vendors must write drivers against the Direct3D and OpenGL specifications in order to make their cards compliant with that API.
If you're looking for some differences in the two APIs, DirectX is more geared towards games while OpenGL is a more general 3d rendering platform. More on that in the Wikipedia article comparing the two.
Direct3D and OpenGL are APIs, they're implemented in the drivers by the manufacturers. For OpenGL, the manufacturer has to implement a set of functions that act in a way described by the specification provided by the Khronos Group. Then the programmer can retrieve the addresses to those functions and use them in the program, the details are OS dependent, it's usually done in C or C++ sine that's what OS APIs mostly use. OpenGL doesn't care how much RAM your graphics card has, so to speak. Direct3D has a more manufacturer friendly philosophy behind it, Microsoft gives more leeway, but it's still just a matter of drivers.