There are a couple of points you need to take into account to solve this.
First you'll want to switch your sprite's Mesh Mode in its texture import settings to Full Rect so you can enable 9-slice scaling on it as described in the Unity docs.
This lets you select either the Sliced or Tiled modes on your Sprite or Image components, which will ensure borders of your sprite stay the same thickness in the game world/canvas when the sprite is enlarged - they'll be spaced out or tiled instead of stretched to fit.
For a non-UI sprite, the size of the border in actual display pixels will depend on how your camera sees it. This includes your pixels per unit setting, the size of your orthographic camera, and the resolution of the display viewport. You can work out the result using these formulas:
world thickness = (1 pixel of texture) ÷ (pixels per unit)
vertical screen fraction = (world thickness) ÷ (camera height * 2)
display pixels = (vertical screen fraction) * (pixel height of display viewport)
(For perspective cameras you also need to factor in the depth of the sprite from the camera)
You'll want to carefully choose your pixel per unit and camera settings to keep this at an integer value. Otherwise you can get one border drawn thicker than the other if one border rounds the fraction differently.
For an Image component displayed as part of your UI, this is a bit easier to manage. Set up your Canvas with a CanvasScaler set to Constant Pixel Size, with the Reference Pixels per Unit value the same as the one you used to import your sprites, or an integer multiple of it if you want to upscale the art (Don't forget to switch your textures to Point filtering if you're upscaling pixel art).
Then enable the Pixel Perfect box in the Canvas component, which turns off anti-aliasing on UI so you get exactly the pixels of your sprite.