Edited Answer
Before writing to a Buffer Object, you must have some space allocated in GPU's memory, you can grow or shrink the Buffer Objects via:
glBufferData(int target, long size, int usage);
When the Buffer Object already have size bytes, you can call glMapBuffer passing the full size, or some smaller number (you dont have to actually shrink the Buffer Object if it will most likely grow again soon)
glMapBuffer(int target, int access)
Returns a brand new ByteBuffer, with the current size of Buffer Object;
glMapBuffer(int target, int access, ByteBuffer old_buffer)
Returns a new ByteBuffer, or reuses the old object (if not null), with the current size of Buffer Object (not the old_buffer object);
glMapBuffer(int target, int access, long length, ByteBuffer old_buffer)
This is like the previous one, but it is faster because it don't have to query for the Buffer Object's size.
When passing an old_buffer, it unmaps it first (and may use some optimization) before mapping it again.
Don't forget to call glUnmapBuffer(int target) at the end of it all. (After that your mapped ByteBuffer will be in an invalid state and you must not read/write to it)
Example
private int id;
private long currentSize;
public void updateBuffer(...) {
long calculatedSize = ...;
if (currentSize < calculatedSize) {
currentSize = calculatedSize;
glBufferData(GL_ARRAY_BUFFER, currentSize, GL_DYNAMIC_DRAW);
}
ByteBuffer bb = glMapBuffer(GL_ARRAY_BUFFER, GL_WRITE_ONLY);
try {
FloatBuffer floatBuffer = ...
...
} finally {
glUnmapBuffer(GL_ARRAY_BUFFER);
}
}
