3
\$\begingroup\$

This issue has been annoying me for a long time now and even after reading a lot of articles about it, I am still unable to fix the issue.

First, to my setup. I'm using LWJGL for a 2D project, rendering in immediate mode (yes, I know, FBOs, but I feel like it's too late to change it now). Textures can be scaled, rotated tinted and cropped in any way and are read from packed spritesheets that are multiples of 2 (the one in question is 512x1024 for example).

The Problem

Because there is no smoothing applied to the textures, scaled and rotated textures look quite ugly, showing jagged lines and black outlines. I tried improving it with using glEnable(GL_POLYGON_SMOOTH) (even though I know now that this is apparantly a bad practice, but I have no idea how to add multisampling to immediate mode rendering, other sites even state that multisampling is probably overkill for 2D games, which leaves me even more confused). This is what it looks like, with and without POLYGON_SMOOTH:

Texture artefacts, with and without POLYGON_SMOOTH enabled

Quite clearly, there are multiple issues visible:

  1. With POLYGON_SMOOTH disabled, there are horribly jagged lines and dark edges (that aren't there in the texture)
  2. With POLYGON_SMOOTH enabled, the edges are no longer jagged, but the dark edges are still there and now there are those diagonal lines. After reading up on that a bit, I found out that it has something to do with how graphics cards render quads, which is as two triangles, and when smoothing is applied, this happens. To test that out, instead of drawing a single quad I draw two triangles which indeed moves the diagonal lines around.

The dark edges seem to be caused by something like like pre-multiplied alpha (according to this), but when I check the raw pixel data, it doesn't look pre-multiplied to me. A completely white texture with 0.5f alpha for example is represented as (in ARGB) 0x80FFFFFF. So applying the "fix" (using glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA) instead of the default glBlendFunc(GL_SRC, GL_ONE_MINUS_SRC_ALPHA)suggested in the article doesn't really work, because my textures aren't premultiplied alpha, so alpha components don't work anymore (same texture, black background):

Texture without alpha:

Texture without alpha with glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA)

Texture with alpha components with the two glBlendFunc's

I do use custom shaders, so I guess I probably could make alpha work in some way by passing it to the shader and then multiplying the color by it, would that fix the dark edges (which would enable me to use GL_ONE, the "better" blend function)? That would still leave me with the issue of the diagonal lines though, which I seem to be able to move around, but not remove.

\$\endgroup\$
  • 1
    \$\begingroup\$ Don't use POLYGON_SMOOTH as you just can't get proper antialiasing with that. Instead enable antialiasing for the surface. Check LWJGL documentation on how to enable that. \$\endgroup\$ – msell Oct 25 '15 at 12:45
  • \$\begingroup\$ @msell Thanks for your comment. What exactly is the problem with POLGYON_SMOOTH? Why is it still there if it doesn't work properly? What do you mean with surface anti aliasing? I never heard of that before, and a simply googling for it doesn't seem to give me proper results, just general threads about antialiasing. \$\endgroup\$ – flotothemoon Oct 25 '15 at 12:48
  • \$\begingroup\$ POLYGON_SMOOTH is there for backwards compatibility and those rare cases when someone knows where and how to apply it \$\endgroup\$ – Kromster Oct 25 '15 at 15:53
  • 1
    \$\begingroup\$ I mean MSAA. You didn't specify the version of LWJGL you are using. In LWJGL 2 you enable it by providing number of samples for PixelFormat when calling Display.create(...),. LWJGL 3 uses GLFW for window management, where you enable multisampling with glfwWindowHint(GLFW_SAMPLES, ...). \$\endgroup\$ – msell Oct 26 '15 at 5:57
  • \$\begingroup\$ @msell Sorry, forgot to add that. I'm using LWJGL 2.9.0. \$\endgroup\$ – flotothemoon Oct 26 '15 at 9:33
3
\$\begingroup\$

To get rid of the jagged lines add 1 line of transparent pixels around the rectangle texture.

The issue with the dark lines is that you are most likely using black transparent pixels (color #000000). The issue is most likely to be gone if you use colored transparency instead. - Repeating the border pixels of your sprite - but simply with alpha=0

\$\endgroup\$
  • \$\begingroup\$ I'm kind of already doing the first thing. This is how the spritesheet looks like: imgur.com/WsIM6rd (look at the 1 pixel transparent border). I don't quite understand how that would help with the jagged lines though, isn't some kind of anti-aliasing still required? As for the black transparent pixels, that seems very plausible, I read about that in multiple places. How would I go about doing that? What do you mean with the border of the sprite? Where the transparancy starts (the visible shape) or the literal border of the image (the rectangle)? How could I apply that? Thanks :) \$\endgroup\$ – flotothemoon Oct 25 '15 at 18:03
  • \$\begingroup\$ Update: Wow, you were right, that actually fixed both issues, I'm impressed. Is that actually common practice? I can't imagine that everybody doing this kind of thing does it to be honest. But hey, it works. So now I just need to find a way to do that for textures that aren't rectangularm (well the image is, but the visible area isn't) and apply that to all the assets. Is there any nice way for doing that? \$\endgroup\$ – flotothemoon Oct 25 '15 at 19:05
  • \$\begingroup\$ You can use TexturePacker. It contains the features you need. See codeandweb.com/blog/2015/09/16/… Make sure to enable "Reduce border artifacts". \$\endgroup\$ – Andreas Löw Oct 25 '15 at 20:59
  • \$\begingroup\$ That's quite an amazing piece of software. Had I only known about it two months ago... unfortunately I already wrote my own version of a texture packer, which is obviously far less advanced. So, I realize that this is probably kind of the point of selling the software and I don't need to know exactly how it was written, but a basic idea of how it works might help :) \$\endgroup\$ – flotothemoon Oct 25 '15 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.