As you may know I write a small OS X called Sail. Over the past few months that it has been available I've received a few crash reports about an issue I wasn't able to reproduce. Today I decided I wanted to dive into them and see if I could at least figure out the root of the issue and fix it with my next release.
This lead me down the rabbit hole of symbolication, something I personally hadn't dealt with myself before (since Crashlytics does it for you). I was hoping I would be able to find something around the internet about this, unfortunately what I mostly came up with was a lot of iOS related answers that didn't seem to work the same way and two links to Apple documentation that have been removed. Other than the process for symbolicating reports for OS X apps seems to be different than iOS apps which there is plenty of documentation for (I'm not bitter). Daniel Jalkut has a post about these but his exact method didn't seem to work for me.
Here is what did work for me. For my first abridged crash report I had this
Process: Sail  Path: /Applications/Sail.app/Contents/MacOS/Sail Load Address: 0x106823000 Identifier: com.keithsmiley.SailOSX Version: 4 (1.2.0) Code Type: x86_64 (Native) Parent Process: launchd  Date/Time: 2013-07-19 16:09:24.097 +0200 OS Version: Mac OS X 10.8.4 (12E55) Report Version: 8 Thread 0: 13 Accounts 0x00007fff839fd1b1 -[ACAccountStore accountTypeWithAccountTypeIdentifier:] + 230 14 Sail 0x00000001068308f7 15 Sail 0x0000000106830798 16 Sail 0x0000000106825249 17 CoreFoundation 0x00007fff82465eda _CFXNotificationPost + 2554 18 Foundation 0x00007fff8611b7b6 -[NSNotificationCenter postNotificationName:object:userInfo:] + 64 31 AppKit 0x00007fff812cc1a3 -[NSApplication run] + 517 32 AppKit 0x00007fff81270bd6 NSApplicationMain + 869 33 libdyld.dylib 0x00007fff869d07e1 start + 0 Binary Images: 0x106823000 - 0x106896fff com.keithsmiley.SailOSX (1.2.0 - 4) <D1F313B6-21F6-341B-8627-5480C5D1DB20> /Applications/Sail.app/Contents/MacOS/Sail
Just glancing at this crash report it's not too difficult to understand a bit
about what was going on. A notification was sent, some methods were called in my
application and then
accountTypeWithAccountTypeIdentifier was called. Based on
the small number of times I call that method I was quickly able to assume where
the issue was but I still wanted to see exactly what methods of mine were being
This brings me to
atos the command line tool Apple provides to symbolicate
these reports. This is where my experience differs with most of what I found
online. My usage looked like this:
atos -arch x86_64 -o Sail.app.dSYM/Contents/Resources/DWARF/Sail -l 0x106823000
This uses my dSYM file that was generated with the archive build I submitted to the app store along with the knowledge that it was running on an x86_64 architecture and the most important part, for me, the load address.
To find my dSYM file that was generated when I did my archive build I simply
noted the version and build number from the crash report, went to the Archives
tab in the Xcode organizer, found the build with the same number, right clicked
and clicked "Show in Finder." This takes you directly to the
on disk which you can right click and click "Show Package Contents." From there
I copied my dSYM to the desktop so I didn't overwrite anything unintentionally.
The load address is the starting memory address of your application. The tool
uses this address as an offset to find the correct methods in your symbols. In
the above crash report
Load Address is a provided field. This was the only
report I saw that had that, typically I needed to look under the
section for the address range of my application. In this example it was
0x106823000 - 0x106896fff.
atos command then provides an interactive prompt where you can paste
addresses into the stdin and it will tell you the corresponding methods. Mine
looked like this:
0x00000001068308f7 -[KSAccountsPreferences dealloc] (in Sail) (KSAccountsPreferences.m:77) 0x0000000106830798 -[KSAccountsPreferences viewDidLoad] (in Sail) (KSAccountsPreferences.m:73) 0x0000000106825249 -[KSAppDelegate openAboutWindow:] (in Sail) (KSAppDelegate.m:59)
Here I can see that the notification that was being posted was probably an
NSApplicationDidFinishLaunchingNotification starting off some methods in my
app delegate. I then load the accounts preferences, which would make sense to
ACAccountStore method, but then
dealloc is called. Seeing this was
an immediate red flag since
KSAccountsPreferences should be retained since it
provides information about available accounts to the rest of the application.
I had another crash report from a different issue that was a little bit harder to parse without symbolicating the methods.
Process: Sail  Identifier: com.keithsmiley.SailOSX Version: 1.2.0 (4) Code Type: X86-64 (Native) Parent Process: launchd  User ID: 502 Application Specific Information: *** Terminating app due to uncaught exception 'NSRangeException', reason: '*** -[__NSArrayM objectAtIndex:]: index 0 beyond bounds for empty array' abort() called terminate called throwing an exception Application Specific Backtrace 1: 2 CoreFoundation 0x000000010c5008ec -[__NSArrayM objectAtIndex:] + 252 3 Sail 0x000000010b838eed Sail + 61165 4 Sail 0x000000010b838f36 Sail + 61238 5 Sail 0x000000010b838b19 Sail + 60185 6 Sail 0x000000010b83785d Sail + 55389 7 libdispatch.dylib 0x000000010fa07f01 _dispatch_call_block_and_release + 15 Binary Images: 0x10b82a000 - 0x10b89dff7 +com.keithsmiley.SailOSX (1.2.0 - 4) <47EC2733-B543-31EA-A6AA-9D998FB65803>
Obviously this was caused by an invalid access to an array but that's a little harder to track down. So I again used
atos with the dSYM and new memory location.
atos -arch x86_64 -o Sail.app.dSYM/Contents/Resources/DWARF/Sail -l 0x10b82a000
I got this output for my memory addresses
0x000000010b838eed -[KSAccountsPreferences selectedADNUser] (in Sail) (KSAccountsPreferences.m:238) 0x000000010b838f36 -[KSAccountsPreferences selectTwitterUsername] (in Sail) (KSAccountsPreferences.m:243) 0x000000010b838b19 -[KSAccountsPreferences populateTwitterAccounts] (in Sail) (KSAccountsPreferences.m:211) 0x000000010b83785d __36-[KSAccountsPreferences viewDidLoad]_block_invoke (in Sail) (KSAccountsPreferences.m:70)
This ended up giving me the exact line where out of bounds issue was happening
depending on a certain number of accounts. I noticed that this issue had been
fixed since my last release so I did a
diff on the tag I created for that
specific release with
git difftool HEAD..1.2.0\(3\)
Then in Kaleidoscope I was able to figure out what changed had fixed the issue. Just because it's so pretty here is what it looked like.
Symbolicating crash reports is definitely vital to tracking down bugs your users are experiencing when you can't reproduce them yourself. Once you figure out how it's obviously worth it.