In general computer science there are modeling "languages" (read: standardized diagramming techniques) such as UML 1,2, in databases there are things like ERD3, in business there are other types such as BPMN4,5. Is there anything like these that are Game Design specific?
There's no industry standard, but most high-profile studios do create a game design document.
Game development, after all, encompasses quite a number of fields, so there will often be a combination of storyboarding, UML for the programming side, a script for dialogue, and so on.
That being said, the number one "modelling language" I've encountered: flow charts.
But that's hardly specific to game design!
One of the actual models that I know about in game design is the MDA framework. This is the closest thing to your question that I can think of, as MDA gives you a reference to think and talk in (thus a language). And you can visualize it in a diagram if necessary.
A proper definition is given in this paper by Hunicke et al, and I'll give a short description below.
The MDA framework
In MDA mechanics are the things that you define in a game as a designer. An example of a mechanic could be - "A timer starting at 60 and counting down by one every second. If it reaches zero the game is over."
A dynamic describes the actual behaviour of a mechanic. Or in other words: the influence of a mechanic on your game. The timer mechanic would add a time pressure dynamic to your game.
The aesthetics of a game describe the overal feel and types of play in a game. The 8 basic aesthetics are:
- Sensation (pleasure of the senses)
- Fantasy (make believe)
- Narrative (drama)
- Challenge (obstacles to overcome)
- Fellowship (social interaction)
- Discovery (uncharted territory)
- Expression (self-discovery)
- Submission (pastime)
The aesthetic generated by a timer is challenge, because the time pressure is an obstacle. Note that you can add any new (sub-)aesthetics if necessary.
MDA allows you to reason about your game design both bottom-up and top-down. If there's some aesthetic in your game that you feel is not represented strongly enough, you can add more mechanics to support it. You could also model your game in a diagram using MDA by connecting the mechanics that support specific dynamics, and so on.
I learned about it in a game design course, but you can probably find some further reading on the web too. I hope this is what you were looking for, good luck!