Suppose one were to write a manual for a game, where 'game' can be either a video game or board game - that doesn't matter. Are there symbolical/analytical ways to write down a set of rules for a game, as for example a formal language?
Joris Dormans's Machinations is an attempt to create a formal diagramming language for expressing game mechanics - especially those that relate to resource flows.
By thinking of "player action" as a resource, or mechanics involving navigating connected locations in space as a series of resource pools that the "player presence" resource can move between, you can stretch this to cover more general classes of game mechanics - with a little squinting/imagination.
This language takes the form of a kind of flow chart, with tokens that can sit within each node, and well-defined rules for when and how tokens are moved from one node to another.
There's a web-based tool you can use at machinations.io to create these diagrams, as well as to "run" them as automated simulations or interactive minigames, to explore the dynamics produced.
This is suitable for documenting the dynamical engine at the heart of a game design, for an audience of fellow designers/game developers well-versed in this particular diagramming dialect. It's not so suitable for communicating to players or other less specialized audiences. It also abstracts away many aspects of game mechanics - like the means the player uses to provide input, or the perceptible affordances and feedback the game provides, so it can't serve as a complete game design document on its own.