Lacking any specific context, it means what the word itself means:
the practice or the doctrine of strict adherence to prescribed or external forms (as in religion or art)
(among other similar definitions)
Within the context of the games industry, and game design or game theory in particular, "formalism" doesn't hold a specialized meaning. The concepts of philosophical, artistic and/or mathematical formalism exist and may be appropriate for a particular conversation about game design or theory, since it's a domain that has a mix of all three of those things.
But there's nothing really unique about the term as it applies to game development, in general (or at least, not in a fashion that is an established or ad hoc standard).