How does file encryption in game assets work, for example the unreal
engine has a .assest extension, or I also see a .dta extension in
another game, are these zip type packages which have a custom
encryption algorithm set by the developer or how does it work?
Most of the time, those are not encrypted data files.
There's a difference between encryption and a custom file format. Most games tend to use the latter, not the former. Encryption is concerned with protecting data in such a way that only certain people can read it; a custom file format is generally only about storing a particular collection of data in the most-efficient or otherwise most-useful form for the program that will consume the data.
Encryption is relatively pointless for most game data that will be directly read client-side, on the user's computer. The user's computer must perforce decrypt the data to use it, at which point somebody could intercept the data in-memory without much more effort than it took to read the data on disk, so all that extra effort and potential extra file size that went in to "protecting" the data is lost.
The games that do use encryption somewhere (in a network protocol or a server back end, or even the few that do make use of it somewhere on a client side) will generally use an established encryption protocol of some flavor (and not rely on security through obscurity). Reinventing your own is a very bad idea, because it's surprisingly easy to create something that looks secure but is open to one of the many vulnerabilities or attack vectors that have been developed over the years. It's a fascinating subject if you're into that sort of thing, and you might consider taking a few classes on the subject.