This should be very straightforward, since you are already loading the TGA yourself, and assuming you only want to deal with true-color TGA files (the TGA file format allows for alternative color map types; true-color is the one where pixel colors are stored directly, without a palette or anything).
TGA files have an 18-byte header, after which is an optional image ID string (usually not present). Palette data follows the ID string, and may exists even in a true-color image. After the palette data is the RGB pixel data you are interested in.
All of this should be old-hat if you have actually written the loader, because you'd need to account for all of this to get to the pixel data that you're loading into your textures.
The rest of the file is either raw or RLE RGB pixel data. The TGA format stores the blue component in the least-significant byte of the pixel, followed by the green byte, the red byte and finally the alpha byte (if present).
Consequently, you can read the RGB color data by simply modifying or reusing the portion of your RGA loader code that finds the offset of the pixel data, and then reading the bytes linearly from the file (using the information in the header to know how many bytes are in the file total).
More information about the TGA file format is available here.