I don't think your issue is necessarily with sf::Text -- I wrote up a small program to draw multiple sf::Texts and record the frames per second for 40 seconds, and got these results:
Drawing 1 sf::Text: Average 738.7 FPS
Drawing 4 sf::Texts: Average 735 FPS
That's a slight difference, but nowhere near your 40 FPS drop; you may want to double-check that nothing was changed other than the number of Text items drawn.
My program doesn't do anything special with the text items -- other than FPS counting, all it does is draw either 1 or all 4 of the sf::Text objects declared outside of the program loop (I could show the code if you want) -- which leads me to believe that the issue likely has something to do with how your program handles drawing & manipulating the Text objects.
I don't see anything particularly suspicious in the extract of code your post shows (assuming there's a reason you're creating the Text with new). It might help to be able to see where you actually draw them.
So, without knowledge of your code, there are a couple of potential causes that I can think of:
For some reason, your text objects are being recreated each frame, or perhaps are resetting some of their data which needs to be drawn. The former could cause slight performance loss on its own, but both would require new vertex data to be sent to the GPU each frame, which can (potentially) be a performance loss more noticeable with more things to be drawn.
Your draw code alternates between drawing various different textures (drawing text between drawing other objects?). Changing the active texture when drawing is not a cheap OpenGL operation. SFML uses OpenGL, and thus is partially vulnerable to this.
Using multiple sf::Fonts alongside multiple sf::Texts. The sf::Font class is more computationally expensive than sf::Texts to use.
Your program is simple enough that quadrupling the number of drawn texts is actually a pretty large increase in computational load.
Edit: How many frames per second do you get if you don't draw any of the sf::Text objects?