What decrease in polycount should occur as a function of distance. I make a terrain and I wounder what is the right formula to keep the "poly size"(the screen size) constant. Or is there some mental flaw in this approach?
As @Adam said, one or two pixels is as small as you want to go. In fact, 1 pixel by 2 pixels is the correct size (unless multisampling, which decouples screen pixels from geometry pixels).
The reason it's correct is that 1x2 yields one pixel for each triangle. If your polygons are smaller than that, then multiple color outputs must be combined into a single pixel. The final result will vary depending on subtle changes like orientation. On screen, that looks like jittering textures. The pixel will be green one second, then brown, then black, when really is should be somewhat constant or smoothly changing. There are ways to fight it, but if you have exact control of the polygon count through some form of shader tessellation, aim for 1 pixel per polygon.
In terms of math and the "right formula, rendering is always an application of signal processing. This article is an interesting high-level discussion of what happens to a 1-d signal when sampling varies slightly. As that article describes, the math is almost a century old. You would probably drive yourself insane trying to apply this theory directly to your rendering, but it does yield a good rule of thumb: one or two pixels.
And obviously, the problem with bigger polygons:
In general when polygons get smaller than one or two pixels in either width or height you can get aliasing from that. This is generally called Geometry Aliasing.
It is most noticeable when the small polygons have significantly different colours to the ones next to them. Different normals can also be a problem because specular highlights will change the colour based on the normal.
The aliasing can be reduced significantly by MSAA as it effectively makes the pixels smaller, but screen space techniques like FXAA aren't very good at fixing it.
Of course the size in pixels on screen of a polygon varies with the camera position, orientation and field of view, so the ideal place to select the poly count is in real time with a shader. Failing that you can estimate it based on standard camera settings and a target screen resolution.
From my litte experience with terrain rendering, one of the visually most offensive things that happens when naively reducing the polycount with increasing distance is that -for example- the peaks of mountains can get "cut off" or valleys vanish. This obviously will happen when removing vertices uniformly from the mesh and these landscape features simply aren't positioned conveniently enough to be represented by a vertex in the mesh with reduced detail.
So generally I think you will want to retain the shape or silhouette of the terrain as much as possible. One way to achieve this might be to favor local extreme points in your heightmap. That means when reducing the polycount, you first remove vertices that aren't located in the vicinity of a local extreme point or ridge-line, as those should usually represent the peaks of mountains/hill, the bottom of valleys or any other features that greatly define the overall look of the landscape.
I don't think there is a right formula, it's more important to keep your frame rate smooth rather than keeping the number of polygons fixed. Most games have some polygons budget that they strive to keep their polygons count below. So I would say change how aggressively your application simplify the terrain based on how good the game performs and how good it looks.
It's important to measure and decide what is needed. polycount is a tangible number just like FPS both are not particularly accurate at measuring your game's performance, and there are other things to measure that govern the GPU time such as fillrate, draw batches.
There are famous terrain rendering algorithms such as ROAM that uses a heuristic approach to determine the best detail/performance ratio.