I had rotate an object with openGL (JOGL). Is there a simply way how I can get the new vertices?
There is a way, but it is not easy: set up an transform feedback buffer, and have it gather all the points emitted by the geometry shader. Perform the matrix calculation on the vertex shader, pass the calculated vertices to the geometry stage and then emit them so that the transform feedback buffer catches them. It's hardly worth it though, I'd go with what the other dude said: calculate it on the CPU.
Why do you want the vertices back? Do you want to apply additional transformations on them? If so, you can just multiply the model matrix you used to draw the last frame with an matrix that corresponds to whatever transformations you want to do this frame.
For example, say that you want to rotate an object around the Y axis, making one loop every 10 seconds. Then you just create an identity matrix and save this matrix, as "modelmatrix". For every frame, assuming that delta miliseconds has passed since the last frame, you create a new identity matrix, and rotate it around ((delta/10000.0f) * 3.14f * 2.0f) radians around the Y axis. Then you multiply your previously saved modelmatrix with this matrix, and make sure that this operation mutates modelmatrix, eg: modelmatrix = modelmatrix * rotationmatrix.
Then you throw your rotation matrix away (or better yet, save it, and mutate it to the identity matrix so you don't need to allocate/deallocate the memory for 4*4 floats all the time).
The easiest way to do this is to keep a copy of the vertices on the CPU-side, and perform the same transformations to your "local" copy. This can usually be done by accumulating the rotations and translations in a transformation matrix, then applying the transformation matrix to each vertex whenever you access them locally.