So I'm making this topdown game and I'm trying to implement player control movement so that it feels more natural. Right now I have an update method like this:

void update(float delta) {
    velocity = sf::Vector2f();

    if right:
        velocity.x = 1;
    if left:
        velocity.x = -1;
    if up:
        velocity.y = -1;
    if down:
        velocity.y = 1;

    velocity *= sf::Vector2f(delta, delta);
    position += velocity;
}

I want to implement acceleration, so that the player doesn't reach top speed as soon as they hit a key, as well as de-acceleration so that they slow down (but quicker than they accelerate) as they release the key.

So I remove the velocity being reset to zero, and if they go right I increment the x component of the velocity by the speed and then if they still hold right I increment the speed by acceleration. Then when they release the key I slow down this acceleration, but the way I've done it you move right it works, then you move left and the player goes right again but really fast.

Note that I'm only doing left + right before I move on to up and down since its the same principle, so the code is unfinished:

void Player::update(double delta) {
    if (sf::Keyboard::isKeyPressed(sf::Keyboard::D)) {
        vel.x += speed;
        speed += accel;
        if (speed > max_accel) {
            speed = max_accel;
        }
    }
    else if (vel.x > 0) {
        speed -= accel;
        if (vel.x <= 0) {
            vel.x = 0;
            speed = accel;
        }
    }

    if (sf::Keyboard::isKeyPressed(sf::Keyboard::A)) {
        vel.x -= speed;
        speed -= accel;
        if (speed < -max_accel) {
            speed = -max_accel;
        }
    }
    else if (vel.x < 0) {
        vel.x += speed;
        if (vel.x >= 0) {
            vel.x = 0;
            speed = accel;
        }
    }

    pos.x += vel.x * delta;
    pos.y += vel.y * delta;
}

What am I doing wrong here? I can't seem to get the logic right for when you release a key or not. I figured that if they aren't holding A, but their velocity is in the direction of A then we apply the slowdown but I can't seem to figure out how to handle when we actually reach the velocity of zero.