I have created a texture in Blender in texture paint mode. 
Then I have started to add some details for side and corner texture. They are primarily height maps.
I had some minor pixels being a different colour on the edges (don't know why they appeared). So I painted them out with colour picker. If you look at the texture (the one that I cleaned up is third) there shouldn't be any artifacts or missing pixels. Though, data from them was somehow stored inside the texture on different layer or what I don't understand that. As a result it created weird pixels that appear only on mip maps.
Notice that original texture doesn't have these problems.
If I use image in texture on a displacement modifier in blender it uses that data to displace the mesh which is totally weird for me because I have painted out those pixels. Did I do something wrong? I have tried all the blending modes in Blender. I have tried to do everything I thought of in Krita, Paint, Paint3D.. I have changed the colours, erased the alpha, erased the colour and painted over and over again. Nothing helps. Even if the colour is set to bump the mesh, it uses the data normally but as soon as I change it back to neutral colour it uses wrong data again.
Result in Unity.
Original texture result
All the settings are the same.