Is there some way I can directly access texture memory from fragment shader in
GLSL? I don't need the sampler to be involved since I am just using it as a look-up table.
texelFetch is used for lockups without performing mip-mappnig, filtering..etc. and even with this you still need to pass a sampler.
Otherwise your out of luck, as far as I am aware, making such operation available restricts driver writers from doing certain optimization, for instance they need to keep textures in a certain layout in memory.