First make a white-on-black mask of your logo/text and blur it.
Then create a repeating (tileable) solid noise texture (GIMP used here)
Use the Map->Tile... filter to create a 3x3 tiled pattern (in this example, 128x128 x 3 = 384x384) for the next step to ensure our texture is still repeatable - we'll keep only the center part.
Use Blur->Motion Blur... to blur the texture upward and keep only the center 1/3rd (back to 128x128)
Multiply both textures together on the GPU and use this for opacity.
Then animate it by moving the pattern texture upward over the mask texture:
Done for the animation part.
Then you can apply a gradient map (black -> red -> yellow -> white) to give it fire colours:
Other colours can be used create a ghastly blue fire, a light yellow aura field, a more smoky effect, etc.
Now if you combine this as additive over your logo and the 3D render you get the wanted effect:
+ + =
The effect can darkened by adjusting the mask and/or pattern brightness, and/or vertex color, and/or gradient color map to the designed level.
You can even use two textured patterns together (Mask * Fire Pattern * Fire Pattern) at different speeds and directions to create a more complex fire effect.
Technically on the N64 they may have created an approximation of the mask using a triangle mesh and vertex color instead of the mask texture due to the N64's hardware limitation regarding textures but the end result is the same ((vertex color * pattern texture) vs (mask texture * pattern texture)).
We can still use vertex colour but 20 years later we can make our lives easier and just use 2 textures or more, even today's mobile GPUs can handle an extra 256x128 gray texture without problem.