Is there any book or article that describe interaction of RPG elements and human psychology?
For example: what will be more popular and attracting for people - linear function of experience per level or exponential function of experience per level?
Don't design like this.
It is definitely possible to figure out the "optimal" way to design each part in your game to attract some hypothetical average player. Sometimes you'll find books about the thing you're trying to design, sometimes papers.
But the average person is a fiction. No one actually is the average person; everyone deviates greatly in at least some preferences and attitudes. Some what paradoxically, by designing a game for the average person you are designing a game for no one.
Design a game you'd like to play. Design a game your friends, or parents, or partner(s), or kids want to play. Design one your neighbors or church or people from your hometown or people who also like your favorite book want to play. Root the design in a real locus for real people.
Don't design a game that the average person wants to play, because then no one will want to play it. And even if someone does, it will be a thoroughly average game.
You don't strictly need to read a psychology book for that, actually, but The Art of Game Design has what you need. With examples by real developers.