Various unreproducible crashes caused by asset bundles

We recently released a game on iOS and Android, and a bunch of crash reports have started coming in on iOS through Crashlytics that appear to be due to problems with shaders in asset bundles and other asset bundle problems (such as loading models, textures, basically anything and everything in an asset bundle). We haven't been able to reproduce this problem, so we aren't sure what we can do about them other than do a new release with new builds of the asset bundles and hope that they work. We're using Unity 2108.2.12f1.

Is there anything we can do to try to resolve this problem?

For reference, here are some of the places where it has crashed. All of these are internal to Unity apparently. There's nothing specific to our game in the stack trace. I'm not sure how useful these different stack traces are.

Crashed: UnityPreload

__bzero + 28

SerializeTraits.h - Line 488
SerializeTraits<dynamic_array<unsigned char, 0ul> >::ResizeSTLStyleArray(dynamic_array<unsigned char, 0ul>&, int)

StreamedBinaryRead.h - Line 77
void StreamedBinaryRead::TransferSTLStyleArray<dynamic_array<unsigned char, 0ul> >(dynamic_array<unsigned char, 0ul>&, TransferMetaFlags)

SerializeTraits.h - Line 481

CompressedMesh.h - Line 155

SerializeTraitsBase.h - Line 54

SerializedFile.cpp - Line 2164
SerializedFile::ReadObject(long long, ObjectCreationMode, bool, TypeTree const**, bool*, Object&)

PersistentManager.cpp - Line 2330
PersistentManager::ReadAndActivateObjectThreaded(int, SerializedObjectIdentifier const&, SerializedFile*, bool, bool, PersistentManager::LockFlags)

PersistentManager.cpp - Line 1785

dynamic_array.h - Line 146

PreloadManager.cpp - Line 281

PreloadManager.cpp - Line 338

PreloadManager.cpp - Line 187

Thread.cpp - Line 42


And a dozen similar ones.

Crashed: UnityPreload

StringStorageDefault.h - Line 31

type_traits - Line 3615

vector - Line 1467

SerializeTraits.h - Line 260

iterator - Line 1190

SerializeTraits.h - Line 260

iterator - Line 1190

SerializeTraits.h - Line 260

iterator - Line 1190

SerializeTraits.h - Line 260

iterator - Line 1190

SerializeTraits.h - Line 260

SerializeTraits.h - Line 480

SerializedFile.cpp - Line 2164
SerializedFile::ReadObject(long long, ObjectCreationMode, bool, TypeTree const**, bool*, Object&)

PersistentManager.cpp - Line 2330
PersistentManager::ReadAndActivateObjectThreaded(int, SerializedObjectIdentifier const&, SerializedFile*, bool, bool, PersistentManager::LockFlags)

PersistentManager.cpp - Line 1785

dynamic_array.h - Line 146

PreloadManager.cpp - Line 281

PreloadManager.cpp - Line 338

PreloadManager.cpp - Line 187

Thread.cpp - Line 42


And a dozen similar ones

Crashed: com.apple.main-thread
libsystem_platform.dylib
_platform_memmove + 352

CachedReader.cpp - Line 98

StreamedBinaryRead.h - Line 88
void StreamedBinaryRead::TransferSTLStyleArray<core::basic_string<char, core::StringStorageDefault<char> > >(core::basic_string<char, core::StringStorageDefault<char> >&, TransferMetaFlags)

SerializeTraits.h - Line 237

SpriteFrame.cpp - Line 134

SerializedFile.cpp - Line 2164
SerializedFile::ReadObject(long long, ObjectCreationMode, bool, TypeTree const**, bool*, Object&)

PersistentManager.cpp - Line 2330
PersistentManager::ReadAndActivateObjectThreaded(int, SerializedObjectIdentifier const&, SerializedFile*, bool, bool, PersistentManager::LockFlags)

PersistentManager.cpp - Line 1663

PersistentManager.cpp - Line 1390

AssetBundleUtility.cpp - Line 143
ForcePreload(AssetBundle&, dynamic_array<int, 0ul>, bool)

dynamic_array.h - Line 114
ProcessAssetBundleEntries(AssetBundle&, std::__1::pair<std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<core::basic_string<char, core::StringStorageDefault<char> >, AssetBundle::AssetInfo>, std::__1::__tree_node<std::__1::__value_type<core::basic_string<char, core::StringStorageDefault<char> >, AssetBundle::AssetInfo>, void*>*, long> >, std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<core::basic_string<char, core::StringStorageDefault<char> >, AssetBundle::AssetInfo>, std::__1::__tree_node<std::__1::__value_type<core::basic_string<char, core::StringStorageDefault<char> >, AssetBundle::AssetInfo>, void*>*, long> > > const&, ScriptingSystemTypeObjectPtr, dynamic_array<Object*, 0ul>&, bool)

dynamic_array.h - Line 471
LoadNamedObjectFromAssetBundle(AssetBundle&, core::basic_string<char, core::StringStorageDefault<char> > const&, ScriptingSystemTypeObjectPtr)

AssetBundle.bindings.gen.cpp - Line 265

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 3860516

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 7811508

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 21918336

Il2CppInvokerTable.cpp - Line 4336374932
RuntimeInvoker_Void_t1185182177_RuntimeObject_IntPtr_t(void (*)(), MethodInfo const*, void*, void**)

Runtime.cpp - Line 521
il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**)

ScriptingApi_Il2Cpp.cpp - Line 245
scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool)

ScriptingInvocation.cpp - Line 341
ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool)

ScriptingInvocation.h - Line 71
Coroutine::InvokeMoveNext(ScriptingExceptionPtr*)

Coroutine.cpp - Line 257
Coroutine::Run(bool*)

MonoBehaviour.cpp - Line 742
MonoBehaviour::TryCreateAndRunCoroutine(ScriptingObjectPtr, ScriptingMethodPtr, Coroutine**)

MonoBehaviour.cpp - Line 760
MonoBehaviour::StartCoroutineManaged2(ScriptingObjectPtr)

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 3882300

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 26739188

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 1768320

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 1768580

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 2599928

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 2602580

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 2037840

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 21782776

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 21782776

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 26101624

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 22342724

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 22341516

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 22339732

Il2CppInvokerTable.cpp - Line 99434
RuntimeInvoker_Void_t1185182177(void (*)(), MethodInfo const*, void*, void**)

Runtime.cpp - Line 521
il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**)

ScriptingApi_Il2Cpp.cpp - Line 245
scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool)

ScriptingInvocation.cpp - Line 341
ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool)

MonoBehaviour.cpp - Line 571
MonoBehaviour::CallUpdateMethod(int)

Behaviour.cpp - Line 176
void BaseBehaviourManager::CommonUpdate<BehaviourManager>()

PlayerLoop.cpp - Line 340
ExecutePlayerLoop(NativePlayerLoopSystem*)

PlayerLoop.cpp - Line 357
ExecutePlayerLoop(NativePlayerLoopSystem*)

RecursionLimit.h - Line 45
PlayerLoop()

LibEntryPoint.mm - Line 299
UnityPlayerLoopImpl(bool)

UnityAppController+Rendering.mm - Line 280
UnityRepaint

UnityAppController+Rendering.mm - Line 77
QuartzCore
CA::Display::DisplayLink::dispatch_items(unsigned long long, unsigned long long, unsigned long long) + 632


And again, a dozen similar ones.

• Have you tested on hardware other than your development platforms? Do you have any info about how the crashing platforms differ from reliable ones? Is it possible for you to upgrade to a more recent Unity release? – Pikalek Apr 24 '19 at 17:33
• @Pikalek We have plans to eventually update Unity versions, but cannot do so in the immediate future, due to our desire to avoid unnecessary risk immediately after the 1.0 release. As for devices it's occurring on, we are testing on a variety of devices and OS versions, and the crashes occur on pretty much every combination of device and OS version we support, though not consistently (about 1% of sessions). Users are generally able to play the game but apparently crash occasionally where it would previously work fine, and works fine again later. – Ed Marty Apr 24 '19 at 18:16
• Are these bundles that you made in house? – Pikalek Apr 25 '19 at 1:34
• Indeed they are. Our build process rebuilds them automatically on each build, and uploads them to a CDN if they have changed. It then updates an asset within the game to point to the correct version. – Ed Marty Apr 25 '19 at 12:40

The first thing to do is try to narrow down why EXC_BAD_ACCESS errors are occurring. A common cause is attempting to access a deallocated object, also known as a dangling pointer error.

You can check for this in XCode by enabling zombie objects. Doing so allows deallocated objects to be kept around as 'zombies' for debugging purposes. Accessing a zombie will still result in a EXC_BAD_ACCESS. However, it will give you more information about what object you were trying to access, which makes debugging easier.

If that doesn't help, the next step is to have Xcode analyze your project. This will generate a list of issues. These are warnings about things that may need attention. One approach is to filter through the issues & their linked code looking for things that seem most relevant to the problem at hand. If time / resources permit it might be better to go through them and address/correct as many as possible.

Note: sometimes the warnings will not be applicable to your particular situation. However, keep in mind that your code is crashing; don't be quick to convince yourself that you are right & the analysis tool is wrong.

• Unfortunately, as I mentioned, we have not been able to get this to happen when testing on our own. These are crashes coming in from the wild. And since this is a Unity-generated project, there are countless warnings, and the code that is crashing is not even IL2CPP code. It's all internal to Unity's process as far as we can tell (since it's crashing on the UnityPreload thread, not the main Unity thread). – Ed Marty Apr 25 '19 at 20:11

I would recommend running with Address Sanitizer. In Xcode's scheme settings for your app, select the "Run" tab on the left, then the "Diagnostics" tab on the top. The first checkbox is "Address Sanitizer." (If it's grayed out, turn off "Malloc Guard Edges" or other settings in that group.) Check the box for Address Sanitizer and check the box for "Detect use Of stack after return". Rebuild, and run. Do whatever it is that triggers serialization. Address Sanitizer will likely stop your application and point out any address misuse you have. It's possible that even though it's not crashing on your system, you may still trigger Address Sanitizer when you read or write past the end of an array, or something similar.

Turned out this was a Unity bug that they have fixed in the latest Unity update.