The Neo Geo's distinct sound is due to its sound chip, the Yamaha YM2610 a.k.a. OPNB, which is part of Yamaha's OPN family of sound chips. In terms of "instruments", it has the following:
Each has a distinct sound:
FM synthesis: this type of instrument produces sounds by generating sine waves and applying various operations on them, such as filters and envelopes. You may be familiar with platforms like the SEGA Megadrive / Genesis, or the NEC PC-98 (home of Japanese games like Touhou). These also used OPN sound chips, and have the same distinct sound coming from FM synthesis instruments. Although FM synthesis is capable of creating a wide variety of sounds, it is difficult to achieve a specific sound with them, such as emulating real instruments, so they can give off an "electronic" or "tinny" sound.
SSG square wave: the square wave is what most people associate with chiptune music and 8-bit consoles like the NES. The OPN SSG is rather limited in comparison however, because it only generates square waves, whereas other types of waves are possible in typical PSGs like the sawtooth or triangle waves.
SSG noise: also found on many PSGs, this is typically used for simple percussion.
PCM: this is a more typical instrument, which plays an audio sample. The fact that there is only one PCM channel that allows variable pitch means that the PCM channels are mostly used for drums (fixed pitch) and the occasional, monophonic instrument using realistic samples, such as voices, wind/brass instruments or a lead melody.
In essense, the OPNB's sound is mostly FM synth instruments, supplemented by PCM drum samples and the occasional PCM instrument or SSG noise instrument.
To replicate this distinct sound, you have to learn about FM synth instruments. Making them from scratch can be an art unto itself, but you can somewhat get around it by taking instruments from existing tracks. From here you can go one of two routes:
Emulate the OPNB sound chip
There are some music trackers that emulate the OPNB, including its limitations such as limited number of channels. Example: DefleMask. You will be able to authentically recreate the sound, but the learning curve can be high, because music trackers work quite differently to typical DAWs.
Use plugins that recreate OPNB instruments in your favorite DAW
Example: ADLplug. Depending on your experience, DAWs tend to be easier to use and more flexible. Without the hard limitations of the OPNB chip, your music can sound less authentic, but you'll have the flexibility in using more channels, for instance.