Getting a pixel of Sprite at the cursor position in Unity

Question

Summary: I've been trying to figure out a way to use simple box colliders as a nice and efficient "first pass" collider. For the "second pass", I want to check that the area clicked has an alpha value > 10%. I know this is generally possible with graphics programming, but I'm struggling to find a way to do this in Unity2D with packed sprites (packed via the native Sprite Packer tool).

What I was hoping to do (but Unity will not allow) would've been something like...

// Check to see if a point intersects with a solid color
private bool CheckPoint()
{
    Vector2 cursorWorld = Camera.main.ScreenToWorldPoint(Input.mousePosition);

    //foreach (Collider2D col2D in Physics2D.OverlapPointAll(cursorWorld))
    foreach (RaycastHit2D hit in Physics2D.RaycastAll(cursorWorld, Vector2.zero))
    {
        if (hit.collider.gameObject != gameObject) continue;

        foreach (SpriteRenderer sprRend in hit.collider.gameObject.GetComponentsInChildren<SpriteRenderer>())
        {
            Texture2D tex = (Texture2D)sprRend.sprite.texture;

            // this is the missing link; easy with 3d, but impossible with 2d?
            Vector2 pixelUV = hit.??????; 

            int uvX = (int)(pixelUV.x * tex.width);
            int uvY = (int)(pixelUV.y * tex.height);

            Color hitColor = tex.GetPixel(uvX, uvY);

            if (hitColor.a > 0.1) return true;
        }
    }

    return false;
}

---

So far: I've been able to find some solutions for doing this with (3D) meshes and 3D colliders. However, the 2D colliders do not include the UV data that would be needed to make this work.

Someone was also able to get this working for their specific game using SpriteManager2 (which we're not using since Unity 4.3 makes SM2 mostly unnecessary) and posted their solution here... http://answers.unity3d.com/questions/17903/detecting-mouseclick-on-pixels-in-sprite-using-sm2.html and I think the same approach might work with minor adaptations for Unity2D, but I'm thinking there might be a more robust approach.

I'm also looking at trying to do something like http://answers.unity3d.com/questions/687116/what-is-the-easiest-way-to-convert-raycasthit2d-to.html but even if I can calculate the local UV, that wont help me with the real UV for the packed sprite unless I can somehow get the sprite's UV offset into the sheet.

Answer 1

I can't give exact code because I've not done this myself yet (may make a good exercise with the new 2D functionality) but I can see the outlines of how you would implement this.

First off, detect which sprite was clicked either using good ol' OnMouseDown. Use Input.mousePosition along with the sprite's transform to determine where the cursor is on the sprite.

With that, use properties of the sprite to figure out the exact pixel; I'm not sure off the top of my head which ones you'd use, but the script reference for Sprite mentions stuff like 'rect' and 'textureRectOffset'

Finally get the Texture2D with the sprite's 'texture' property and call GetPixel()

Answer 2

Below I've posted something I mocked up going for deformable pixel terrain. You may be able to get something from it... You may not... Either way, It can obtain the pixels of an image around the mouse position. The names are pretty self explanatory but if you don't understand something, post a comment here.

using UnityEngine;
using System.Collections;
using System.Collections.Generic;

[RequireComponent (typeof (SpriteRenderer))]
public class BitmapMask : MonoBehaviour 
{
    public Texture2D spriteTexture;
    public Texture2D maskedTexture;
    public int diameter = 50;

    public List <bool> mask;

    private float _width;
    private float _height;
    private Sprite _image;
    private SpriteRenderer _spriteRenderer;

    private Color[] tempColors;
    private Color[] maskColors;

    void Start () 
    {
        _spriteRenderer = GetComponent <SpriteRenderer> ();
        maskColors = maskedTexture.GetPixels ();

        _image = GetMaskedSprite (spriteTexture, maskedTexture);

        _spriteRenderer.sprite = _image;

        _width = _image.bounds.size.x * _image.pixelsPerUnit;
        _height = _image.bounds.size.y * _image.pixelsPerUnit;
    }

    void Update () 
    {
        Vector2 mousePos = Camera.main.ScreenToWorldPoint (Input.mousePosition);

        mousePos += (Vector2)_spriteRenderer.sprite.bounds.extents;
        mousePos *= _spriteRenderer.sprite.pixelsPerUnit;

        if (Input.GetMouseButton (0))
        {
            ChangePixels ((int)mousePos.x, (int)mousePos.y, diameter, Color.black);
        }

        if (Input.GetMouseButtonDown (1))
        {
            ChangePixels ((int)mousePos.x, (int)mousePos.y, 50, Color.white);

        }
    }


    Sprite GetMaskedSprite (Texture2D spriteTexture, Texture2D maskTexture)
    {
        Texture2D texture = new Texture2D (spriteTexture.width, spriteTexture.height);
        Color[] spriteColors = spriteTexture.GetPixels ();

        tempColors = new Color[spriteTexture.width * spriteTexture.height];

        for (int y = 0; y < maskTexture.height; y++)
        {
            for (int x = 0; x < maskTexture.width; x++)
            {
                bool masked = (maskColors[x + y * maskTexture.width]) == Color.white; // change to white and change actual image

                if (masked)
                {
                    tempColors[x + y * maskTexture.width] = new Color (0, 0, 0, 0);
                }
                else
                {
                    tempColors[x + y * maskTexture.width] = spriteColors[x + y * maskTexture.width];
                }
            }
        }
        texture.SetPixels (tempColors);
        texture.Apply ();

        return Sprite.Create (texture, new Rect (0, 0, texture.width, texture.height), new Vector2 (0.5f, 0.5f));
    }

    void ChangePixels (int centreX, int centreY, int diameter, Color color)
    {
        int radius = diameter / 2;
        int a;
        int b;

        for (int y = centreY - (diameter/2); y < centreY + (diameter/2); y++)
        {
            for (int x = centreX - (diameter/2); x < centreX + (diameter/2); x++)
            {
                if (x < 0 || y < 0 || x >= _width || y >= _height) continue;

                a = x - centreX;
                b = y - centreY;

                if ((a * a) + (b * b) <= (radius * radius))
                    maskColors[x + y * maskedTexture.width] = color;
            }
        }

        ChangeSprite (GetMaskedSprite (spriteTexture, maskedTexture));
    }

    void ChangeSprite (Sprite updatedSprite)
    {
        _spriteRenderer.sprite = updatedSprite;
    }

}

Basically it takes in a spriteTexture and a maskedTexture. The masked texture gets altered to either black or white around the mouse position when a click is detected. The masked texture is then used to determine whether pixels from the sprite Texture should be shown, obviously based on the colour.

It may not exactly be what you're looking for but its a start in the right direction. Oh yeh and its not optimized and I have no idea about the bugs...

Goodluck.