In real physics, light (like many other influences) falls off as the inverse square of distance from the source.
You can visualize this by imagining a lightbulb giving off one instantaneous flash - the photons of light released form a hollow sphere expanding away from the bulb like a ripple in a pond. As the sphere expands, no new photons or energy are added to it, so the same amount of brightness is spread across a wider and wider area. If we sample the brightness at one distance (say 1 metre), then wait until the sphere has grown twice as big, its surface area is now four times as large, so the brightness at any one spot will be one quarter of what we saw before. At triple the distance, the sphere has nine times the area, so the brightness will be one ninth, etc.
(This is simplified, of course. Ask on the Physics StackExchange if you want all the messy details of waves and quantum stuff etc...)
So, by this theoretical line of thinking, attenuation = 1.0 / (dist * dist)
(Where our "reference brightness" for the light is measured one unit away, where attenuation = 1)
Okay, now we enter the practical world of cheats that make sense for computer hardware and help us do our jobs as game developers.
Firstly: the formula above has a big problem: it has a singularity when distance is zero!
Usually in physics we'd only model a light source as an infinitely small point like this if we're enough distance away that the approximation won't matter, but in games our point/spot lights are often literally infinitely small points, and if we're not careful, geometry in our scene could get quite close to them or even cross them, making the math become very ill-behaved (eg. as your firefly lands on a leaf, it makes a bright spot brighter than the Sun)
So, we fix that by adding an offset: attenuation = 1.0 / (1.0 + dist * dist)
Now even if distance approaches zero, attenuation will never exceed 1.0, and so the light will never shine brighter than the reference brightness we've set.
(Our reference brightness is now measured at the position of the light itself, rather than one unit away. If we want the same behaviour at the 1 unit mark, we'll need to double the reference brightness compared to the previous/physical version of the formula)
Great. But it's not very customizable.
When we're lighting a game, we often care more about control to be able to get the look we want, rather than strict obedience to physical laws. So we add a coefficient that lets us make the falloff a little sharper or a little shallower. In fact, let's add two coefficients so we can shape the falloff curve:
Now we get the familiar 1.0 / (1.0 + a * dist + b * dist * dist)
So, why add that linear term? This answer suggests that it was useful in older games when rendering without proper gamma correction. The display gamma plus an inverse square falloff would combine to make the light falloff look too sharp. By using an inverse linear falloff (a ~ 1, b ~ 0) we can cheat it to look more correct, letting the display gamma do the squaring. In modern games though, we'd prefer to do proper gamma correction (in addition to things like HDR tonemapping) to make sure our colours look right, so nowadays we'd usually only play with these coefficients for stylistic reasons.
tl;dr: a = 0, b = 1 is a pretty safe bet unless you want a more stylistic effect.
