First, a disclaimer: I'm a lawyer who practices in California, but this post is not intended as legal advice. "Is it legal" questions are inherently not a good idea for the stack exchange format. Different jurisdictions have different laws, and in many jurisdictions, offering legal advice absent a license constitutes unauthorized practice of law (i.e., it's a crime). Moreover, you should not rely on advice you find online to determine whether some conduct is legal or illegal. Finally, circumstances vary. Whether your use of architecture is lawful or unlawful may depend on specifics that are not provided by your question.
So the real answer is: consult with a competent lawyer in your jurisdiction.
Having said that, by way of background, I can tell you that there have been lawsuits related to allegedly unauthorized uses. My favorite example is E.S.S. Entertainment 20002, Inc. v. Rockstar Videos, Inc., et al., 547 F.3d 1095 (9th Cir. 2008), which involved Grand Theft Auto: San Andreas, and the allegation that the game infringed on an actual strip club:
ESS Entertainment 2000, Inc. ("ESS"), operates a strip club, which
features females dancing nude, on the eastern edge of downtown Los
Angeles under the name Play Pen Gentlemen's Club ("Play Pen"). ESS
claims that Rockstar's depiction of an East Los Santos strip club
called the Pig Pen infringes its trademark and trade dress associated
with the Play Pen.
Rockstar prevailed in that lawsuit on first amendment and fair use grounds, but that does not mean you would, nor does it mean that the matter was cheap or trivial for Rockstar to defend.
If you're considering using a real building or, for example, parodying it by tweaking the name PlayPen to PigPen, you may someday find yourself answering questions at a deposition like this:
"Q Okay. Did anything humorous about pigs in any way influence your
artistic creation of the PIGPEN? A I can't remember. . . . Q As you
sit here today can you think of anything funny about pigs that may
have inspired or influenced your artistic rendition of the PIGPEN?"