How to fix texture edge artefacts?

Question

This issue has been annoying me for a long time now and even after reading a lot of articles about it, I am still unable to fix the issue.

First, to my setup. I'm using LWJGL for a 2D project, rendering in immediate mode (yes, I know, FBOs, but I feel like it's too late to change it now). Textures can be scaled, rotated tinted and cropped in any way and are read from packed spritesheets that are multiples of 2 (the one in question is 512x1024 for example).

The Problem

Because there is no smoothing applied to the textures, scaled and rotated textures look quite ugly, showing jagged lines and black outlines. I tried improving it with using glEnable(GL_POLYGON_SMOOTH) (even though I know now that this is apparantly a bad practice, but I have no idea how to add multisampling to immediate mode rendering, other sites even state that multisampling is probably overkill for 2D games, which leaves me even more confused). This is what it looks like, with and without POLYGON_SMOOTH:

Quite clearly, there are multiple issues visible:

1. With POLYGON_SMOOTH disabled, there are horribly jagged lines and dark edges (that aren't there in the texture)
2. With POLYGON_SMOOTH enabled, the edges are no longer jagged, but the dark edges are still there and now there are those diagonal lines. After reading up on that a bit, I found out that it has something to do with how graphics cards render quads, which is as two triangles, and when smoothing is applied, this happens. To test that out, instead of drawing a single quad I draw two triangles which indeed moves the diagonal lines around.

The dark edges seem to be caused by something like like pre-multiplied alpha (according to this), but when I check the raw pixel data, it doesn't look pre-multiplied to me. A completely white texture with 0.5f alpha for example is represented as (in ARGB) 0x80FFFFFF. So applying the "fix" (using glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA) instead of the default glBlendFunc(GL_SRC, GL_ONE_MINUS_SRC_ALPHA)suggested in the article doesn't really work, because my textures aren't premultiplied alpha, so alpha components don't work anymore (same texture, black background):

Texture without alpha:

I do use custom shaders, so I guess I probably could make alpha work in some way by passing it to the shader and then multiplying the color by it, would that fix the dark edges (which would enable me to use GL_ONE, the "better" blend function)? That would still leave me with the issue of the diagonal lines though, which I seem to be able to move around, but not remove.

Answer 1

To get rid of the jagged lines add 1 line of transparent pixels around the rectangle texture.

The issue with the dark lines is that you are most likely using black transparent pixels (color #000000). The issue is most likely to be gone if you use colored transparency instead. - Repeating the border pixels of your sprite - but simply with alpha=0