First, to my setup. I'm using LWJGL for a 2D project, rendering in immediate mode (yes, I know, FBOs, but I feel like it's too late to change it now). Textures can be scaled, rotated tinted and cropped in any way and are read from packed spritesheets that are multiples of 2 (the one in question is 512x1024 for example).
Because there is no smoothing applied to the textures, scaled and rotated textures look quite ugly, showing jagged lines and black outlines. I tried improving it with using
glEnable(GL_POLYGON_SMOOTH) (even though I know now that this is apparantly a bad practice, but I have no idea how to add multisampling to immediate mode rendering, other sites even state that multisampling is probably overkill for 2D games, which leaves me even more confused). This is what it looks like, with and without
Quite clearly, there are multiple issues visible:
POLYGON_SMOOTHdisabled, there are horribly jagged lines and dark edges (that aren't there in the texture)
POLYGON_SMOOTHenabled, the edges are no longer jagged, but the dark edges are still there and now there are those diagonal lines. After reading up on that a bit, I found out that it has something to do with how graphics cards render quads, which is as two triangles, and when smoothing is applied, this happens. To test that out, instead of drawing a single quad I draw two triangles which indeed moves the diagonal lines around.
The dark edges seem to be caused by something like like pre-multiplied alpha (according to this), but when I check the raw pixel data, it doesn't look pre-multiplied to me. A completely white texture with 0.5f alpha for example is represented as (in ARGB)
0x80FFFFFF. So applying the "fix" (using
glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA) instead of the default
glBlendFunc(GL_SRC, GL_ONE_MINUS_SRC_ALPHA)suggested in the article doesn't really work, because my textures aren't premultiplied alpha, so alpha components don't work anymore (same texture, black background):
Texture without alpha:
I do use custom shaders, so I guess I probably could make alpha work in some way by passing it to the shader and then multiplying the color by it, would that fix the dark edges (which would enable me to use
GL_ONE, the "better" blend function)? That would still leave me with the issue of the diagonal lines though, which I seem to be able to move around, but not remove.