Various unreproducible crashes caused by asset bundles

Question

We recently released a game on iOS and Android, and a bunch of crash reports have started coming in on iOS through Crashlytics that appear to be due to problems with shaders in asset bundles and other asset bundle problems (such as loading models, textures, basically anything and everything in an asset bundle). We haven't been able to reproduce this problem, so we aren't sure what we can do about them other than do a new release with new builds of the asset bundles and hope that they work. We're using Unity 2108.2.12f1.

Is there anything we can do to try to resolve this problem?

For reference, here are some of the places where it has crashed. All of these are internal to Unity apparently. There's nothing specific to our game in the stack trace. I'm not sure how useful these different stack traces are.

Crashed: UnityPreload
EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000000

__bzero + 28

SerializeTraits.h - Line 488
SerializeTraits<dynamic_array<unsigned char, 0ul> >::ResizeSTLStyleArray(dynamic_array<unsigned char, 0ul>&, int)

StreamedBinaryRead.h - Line 77
void StreamedBinaryRead::TransferSTLStyleArray<dynamic_array<unsigned char, 0ul> >(dynamic_array<unsigned char, 0ul>&, TransferMetaFlags)

SerializeTraits.h - Line 481
void PackedFloatVector::Transfer<StreamedBinaryRead>(StreamedBinaryRead&)

CompressedMesh.h - Line 155
void CompressedMesh::Transfer<StreamedBinaryRead>(StreamedBinaryRead&)

SerializeTraitsBase.h - Line 54
void Mesh::Transfer<StreamedBinaryRead>(StreamedBinaryRead&)

SerializedFile.cpp - Line 2164
SerializedFile::ReadObject(long long, ObjectCreationMode, bool, TypeTree const**, bool*, Object&)

PersistentManager.cpp - Line 2330
PersistentManager::ReadAndActivateObjectThreaded(int, SerializedObjectIdentifier const&, SerializedFile*, bool, bool, PersistentManager::LockFlags)

PersistentManager.cpp - Line 1785
PersistentManager::LoadObjectsThreaded(int const*, int, LoadProgress&, bool)

dynamic_array.h - Line 146
LoadSceneOperation::perform()

PreloadManager.cpp - Line 281
PreloadManager::processSingleOperation()

PreloadManager.cpp - Line 338
PreloadManager::Run()

PreloadManager.cpp - Line 187
PreloadManager::Run(void*)

Thread.cpp - Line 42
Thread::RunThreadWrapper(void*)

And a dozen similar ones.

Crashed: UnityPreload
EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000028

StringStorageDefault.h - Line 31
std::__1::vector<ShaderLab::SerializedSubProgram::StructParameter, std::__1::allocator<ShaderLab::SerializedSubProgram::StructParameter> >::__append(unsigned long)

type_traits - Line 3615
void resize_trimmed<std::__1::vector<ShaderLab::SerializedSubProgram::StructParameter, std::__1::allocator<ShaderLab::SerializedSubProgram::StructParameter> > >(std::__1::vector<ShaderLab::SerializedSubProgram::StructParameter, std::__1::allocator<ShaderLab::SerializedSubProgram::StructParameter> >&, unsigned int)

vector - Line 1467
void StreamedBinaryRead::TransferSTLStyleArray<std::__1::vector<ShaderLab::SerializedSubProgram::StructParameter, std::__1::allocator<ShaderLab::SerializedSubProgram::StructParameter> > >(std::__1::vector<ShaderLab::SerializedSubProgram::StructParameter, std::__1::allocator<ShaderLab::SerializedSubProgram::StructParameter> >&, TransferMetaFlags)

SerializeTraits.h - Line 260
void ShaderLab::SerializedSubProgram::ConstantBuffer::Transfer<StreamedBinaryRead>(StreamedBinaryRead&)

iterator - Line 1190
void StreamedBinaryRead::TransferSTLStyleArray<std::__1::vector<ShaderLab::SerializedSubProgram::ConstantBuffer, std::__1::allocator<ShaderLab::SerializedSubProgram::ConstantBuffer> > >(std::__1::vector<ShaderLab::SerializedSubProgram::ConstantBuffer, std::__1::allocator<ShaderLab::SerializedSubProgram::ConstantBuffer> >&, TransferMetaFlags)

SerializeTraits.h - Line 260
void ShaderLab::SerializedSubProgram::Transfer<StreamedBinaryRead>(StreamedBinaryRead&)

iterator - Line 1190
void StreamedBinaryRead::TransferSTLStyleArray<std::__1::vector<ShaderLab::SerializedSubProgram, std::__1::allocator<ShaderLab::SerializedSubProgram> > >(std::__1::vector<ShaderLab::SerializedSubProgram, std::__1::allocator<ShaderLab::SerializedSubProgram> >&, TransferMetaFlags)

SerializeTraits.h - Line 260
void ShaderLab::SerializedPass::Transfer<StreamedBinaryRead>(StreamedBinaryRead&)

iterator - Line 1190
void StreamedBinaryRead::TransferSTLStyleArray<std::__1::vector<ShaderLab::SerializedPass, std::__1::allocator<ShaderLab::SerializedPass> > >(std::__1::vector<ShaderLab::SerializedPass, std::__1::allocator<ShaderLab::SerializedPass> >&, TransferMetaFlags)

SerializeTraits.h - Line 260
void ShaderLab::SerializedSubShader::Transfer<StreamedBinaryRead>(StreamedBinaryRead&)

iterator - Line 1190
void StreamedBinaryRead::TransferSTLStyleArray<std::__1::vector<ShaderLab::SerializedSubShader, std::__1::allocator<ShaderLab::SerializedSubShader> > >(std::__1::vector<ShaderLab::SerializedSubShader, std::__1::allocator<ShaderLab::SerializedSubShader> >&, TransferMetaFlags)

SerializeTraits.h - Line 260
void ShaderLab::SerializedShader::Transfer<StreamedBinaryRead>(StreamedBinaryRead&)

SerializeTraits.h - Line 480
void Shader::Transfer<StreamedBinaryRead>(StreamedBinaryRead&)

SerializedFile.cpp - Line 2164
SerializedFile::ReadObject(long long, ObjectCreationMode, bool, TypeTree const**, bool*, Object&)

PersistentManager.cpp - Line 2330
PersistentManager::ReadAndActivateObjectThreaded(int, SerializedObjectIdentifier const&, SerializedFile*, bool, bool, PersistentManager::LockFlags)

PersistentManager.cpp - Line 1785
PersistentManager::LoadObjectsThreaded(int const*, int, LoadProgress&, bool)

dynamic_array.h - Line 146
LoadSceneOperation::perform()

PreloadManager.cpp - Line 281
PreloadManager::processSingleOperation()

PreloadManager.cpp - Line 338
PreloadManager::Run()

PreloadManager.cpp - Line 187
PreloadManager::Run(void*)

Thread.cpp - Line 42
Thread::RunThreadWrapper(void*)

And a dozen similar ones

Crashed: com.apple.main-thread
EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x00000000b253c034
libsystem_platform.dylib
_platform_memmove + 352

CachedReader.cpp - Line 98
CachedReader::Read(void*, unsigned long)

StreamedBinaryRead.h - Line 88
void StreamedBinaryRead::TransferSTLStyleArray<core::basic_string<char, core::StringStorageDefault<char> > >(core::basic_string<char, core::StringStorageDefault<char> >&, TransferMetaFlags)

SerializeTraits.h - Line 237
void NamedObject::Transfer<StreamedBinaryRead>(StreamedBinaryRead&)

SpriteFrame.cpp - Line 134
void Sprite::Transfer<StreamedBinaryRead>(StreamedBinaryRead&)

SerializedFile.cpp - Line 2164
SerializedFile::ReadObject(long long, ObjectCreationMode, bool, TypeTree const**, bool*, Object&)

PersistentManager.cpp - Line 2330
PersistentManager::ReadAndActivateObjectThreaded(int, SerializedObjectIdentifier const&, SerializedFile*, bool, bool, PersistentManager::LockFlags)

PersistentManager.cpp - Line 1663
PersistentManager::ReadObjectThreaded(int, PersistentManager::LockFlags)

PersistentManager.cpp - Line 1390
PersistentManager::ReadObject(int, AwakeFromLoadMode)

AssetBundleUtility.cpp - Line 143
ForcePreload(AssetBundle&, dynamic_array<int, 0ul>, bool)

dynamic_array.h - Line 114
ProcessAssetBundleEntries(AssetBundle&, std::__1::pair<std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<core::basic_string<char, core::StringStorageDefault<char> >, AssetBundle::AssetInfo>, std::__1::__tree_node<std::__1::__value_type<core::basic_string<char, core::StringStorageDefault<char> >, AssetBundle::AssetInfo>, void*>*, long> >, std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<core::basic_string<char, core::StringStorageDefault<char> >, AssetBundle::AssetInfo>, std::__1::__tree_node<std::__1::__value_type<core::basic_string<char, core::StringStorageDefault<char> >, AssetBundle::AssetInfo>, void*>*, long> > > const&, ScriptingSystemTypeObjectPtr, dynamic_array<Object*, 0ul>&, bool)

dynamic_array.h - Line 471
LoadNamedObjectFromAssetBundle(AssetBundle&, core::basic_string<char, core::StringStorageDefault<char> > const&, ScriptingSystemTypeObjectPtr)

AssetBundle.bindings.gen.cpp - Line 265
AssetBundle_CUSTOM_LoadAsset_Internal(Il2CppObject*, Il2CppString*, Il2CppObject*)

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 3860516

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 7811508

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 21918336

Il2CppInvokerTable.cpp - Line 4336374932
RuntimeInvoker_Void_t1185182177_RuntimeObject_IntPtr_t(void (*)(), MethodInfo const*, void*, void**)

Runtime.cpp - Line 521
il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**)

ScriptingApi_Il2Cpp.cpp - Line 245
scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool)

ScriptingInvocation.cpp - Line 341
ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool)

ScriptingInvocation.h - Line 71
Coroutine::InvokeMoveNext(ScriptingExceptionPtr*)

Coroutine.cpp - Line 257
Coroutine::Run(bool*)

MonoBehaviour.cpp - Line 742
MonoBehaviour::TryCreateAndRunCoroutine(ScriptingObjectPtr, ScriptingMethodPtr, Coroutine**)

MonoBehaviour.cpp - Line 760
MonoBehaviour::StartCoroutineManaged2(ScriptingObjectPtr)

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 3882300

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 26739188

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 1768320

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 1768580

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 2599928

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 2602580

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 2037840

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 21782776

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 21782776

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 26101624

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 22342724

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 22341516

unsigned long long flatbuffers::HashFnv1a<unsigned long long>(char const*) + 22339732

Il2CppInvokerTable.cpp - Line 99434
RuntimeInvoker_Void_t1185182177(void (*)(), MethodInfo const*, void*, void**)

Runtime.cpp - Line 521
il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**)

ScriptingApi_Il2Cpp.cpp - Line 245
scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool)

ScriptingInvocation.cpp - Line 341
ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool)

MonoBehaviour.cpp - Line 571
MonoBehaviour::CallUpdateMethod(int)

Behaviour.cpp - Line 176
void BaseBehaviourManager::CommonUpdate<BehaviourManager>()

PlayerLoop.cpp - Line 340
ExecutePlayerLoop(NativePlayerLoopSystem*)

PlayerLoop.cpp - Line 357
ExecutePlayerLoop(NativePlayerLoopSystem*)

RecursionLimit.h - Line 45
PlayerLoop()

LibEntryPoint.mm - Line 299
UnityPlayerLoopImpl(bool)

UnityAppController+Rendering.mm - Line 280
UnityRepaint

UnityAppController+Rendering.mm - Line 77
-[UnityAppController(Rendering) repaintDisplayLink]
QuartzCore
CA::Display::DisplayLink::dispatch_items(unsigned long long, unsigned long long, unsigned long long) + 632

And again, a dozen similar ones.

Answer 1

The first thing to do is try to narrow down why EXC_BAD_ACCESS errors are occurring. A common cause is attempting to access a deallocated object, also known as a dangling pointer error.

You can check for this in XCode by enabling zombie objects. Doing so allows deallocated objects to be kept around as 'zombies' for debugging purposes. Accessing a zombie will still result in a EXC_BAD_ACCESS. However, it will give you more information about what object you were trying to access, which makes debugging easier.

If that doesn't help, the next step is to have Xcode analyze your project. This will generate a list of issues. These are warnings about things that may need attention. One approach is to filter through the issues & their linked code looking for things that seem most relevant to the problem at hand. If time / resources permit it might be better to go through them and address/correct as many as possible.

Note: sometimes the warnings will not be applicable to your particular situation. However, keep in mind that your code is crashing; don't be quick to convince yourself that you are right & the analysis tool is wrong.

Answer 2

I would recommend running with Address Sanitizer. In Xcode's scheme settings for your app, select the "Run" tab on the left, then the "Diagnostics" tab on the top. The first checkbox is "Address Sanitizer." (If it's grayed out, turn off "Malloc Guard Edges" or other settings in that group.) Check the box for Address Sanitizer and check the box for "Detect use Of stack after return". Rebuild, and run. Do whatever it is that triggers serialization. Address Sanitizer will likely stop your application and point out any address misuse you have. It's possible that even though it's not crashing on your system, you may still trigger Address Sanitizer when you read or write past the end of an array, or something similar.

Answer 3

Turned out this was a Unity bug that they have fixed in the latest Unity update.