How does file encryption in game assets work, for example the unreal engine has a .assest extension, or I also see a .dta extension in another game, are these .zip type packages which have a custom encryption algorithm set by the developer or how does it work?
Is it as simple as coming up with your own encrytion algorithm then extracting all the file such as in a .pk3 file?
How does file encryption in game assets work, for example the unreal engine has a .assest extension, or I also see a .dta extension in another game, are these zip type packages which have a custom encryption algorithm set by the developer or how does it work?
Most of the time, those are not encrypted data files.
There's a difference between encryption and a custom file format. Most games tend to use the latter, not the former. Encryption is concerned with protecting data in such a way that only certain people can read it; a custom file format is generally only about storing a particular collection of data in the most-efficient or otherwise most-useful form for the program that will consume the data.
Encryption is relatively pointless for most game data that will be directly read client-side, on the user's computer. The user's computer must perforce decrypt the data to use it, at which point somebody could intercept the data in-memory without much more effort than it took to read the data on disk, so all that extra effort and potential extra file size that went in to "protecting" the data is lost.
The games that do use encryption somewhere (in a network protocol or a server back end, or even the few that do make use of it somewhere on a client side) will generally use an established encryption protocol of some flavor (and not rely on security through obscurity). Reinventing your own is a very bad idea, because it's surprisingly easy to create something that looks secure but is open to one of the many vulnerabilities or attack vectors that have been developed over the years. It's a fascinating subject if you're into that sort of thing, and you might consider taking a few classes on the subject.
Truth to be told, there's no reason to encrypt data in a program that will run client-side (that does include any flash/HTML or unity based games running in the browser). Anyone that's inclined enough can get the data anyway - be it by hijacking internal function calls, the OpenGL/DirectX calls, etc. Once it's on the PC of the user, you must understand that there's no way you can prevent a dedicated user from extracting the data.
So the only thing you can prevent is casual users from browsing the data, and honestly, just renaming your zip-files to dat will do the trick here.
I believe this might help you.
In short, there are unlimited number of ways to encrypt your game data. Anything from simply bit swapping to more complicated key based, or even using PGP are all valid ways of storing and encrypting the game data.
Compression should happen before encryption as Kornel Kisielewicz has pointed out. You can use whatever compression format you like for this, although I've had good results with huffman based compression.
xor: way more work, most likely a lot slower, both help against 99.99% of all "hackers" (= script kiddies) - and both wont do shit against someone who knows his crypto. \$\endgroup\$