Here is the description given in paper credited as the source for N-buffers, that being “N-Buffers for efficient depth map query" by Xavier Décoret:
We introduce the N-buffer as a tool for multiresolution depth map representation. This neighborhood buffer encodes the value and position of local depth extrema at different scales in an image cube, in contrast to the image pyramid.
An N-buffer is a sequence of depth maps similar to an image
pyramid [Wil83] except that all levels have the same resolution. Level 0 is a standard depth map. A pixel at level \$i\$
stores the maximum depth of the pixels in a neighborhood of
size \$i\$ in level 0. Different definitions of neighborhood can be
used. For the moment, we define the neighborhood of size \$i\$
of pixel \$(x, y)\$ to be the \$2^i \times 2^i\$
grid of pixels whose lower left
corner is located at \$(x, y)\$.
Figure 1 shows an example of the
four first levels of an N-buffer. Because all levels have the
same resolution, an N-buffer can be built from a depth map
of any size; for this definition of neighborhood, the number
of levels is the log of the largest side of the initial depth map.
Figure 1: (a) Level 0 of an N-buffer is a depth map. (b) A pixel (boxed in red) in level 1 stores the maximum depth of the 2×2
pixels north and east of it (shown on left image). (c) At level 2 it stores the maximum depth of 4×4 pixels (boxed in purple). (d)
At level 3 it stores the maximum depth of 8×8 pixels (boxed in yellow).
So to answer your question:
If I understood correctly, N-buffers basically repeat an operation on further and further away pixels and stores the result in different levels, a bit like mipmaps and blurring, but every level is the same resolution. Except that the operation doesn't need to be blurring, it can be maxing, or any other convolution matrix. Would you agree with this definition?
I'd say there's precedent for this. Later in the N-buffer paper (in section 3.3), Décoret shows a modified N-buffer using a min operation instead of a max. In the paper you linked, Kroes, Schut, and Eisemann further generalize this to use averaging rather than a min or max.
So we might say:
"A neighbourhood buffer (N-buffer) caches the result of applying a given operation over neighbourhoods of varying sizes in a data set. Level 0 contains the original source data. The \$i^{th}\$ level of the N-buffer contains the results of applying the operation over each neighbourhood of size \$i\$ in level 0."
