Fixing Mail import crashes on Leopard

After restoring from a Time Machine backup, Apple Mail would crash every time I tried to re-import my email archive. The problem seems to be affecting a lot of people who have mail that pre-dates OS X. The same crash also happens after migrating to a new machine.

Quick fix

Copy the following command and paste it into your Terminal. Press return and wait a few minutes. Once the command finishes, Mail should be able to import your messages.

grep -lZr 'Content-disposition: attachment' ~/Library/Mail/Mailboxes/ | xargs -0 ruby -i -pe 'gsub(/(Content-type:[^;]*;\s*name=)"(.*)"/){$1+(if !$2.nil? then $2.dump.gsub(/\\\\/, "\\") end)}'

The problem

The problem is specific to older, MacRoman encoded email messages with attachments whose filename includes non-ascii characters. For whatever reason, that freakshow edge-case combination will crash Apple Mail every time it tries to import those files. If by some stroke of luck you already have these messages in your email archive (I did), they can still crash Mail when trying to rebuild the containing mailbox.

The solution is sort of simple, just rename the attachment in the .emlx file. The harder part is finding which file to edit.

We’ll use grep to recursively search for a common token in these older files, then pass the filenames to a Ruby snippet which will filter the line containing the bad characters. Here is a section of a suspect message’s header:

Content-type: application/octet-stream; name="SPWH 4.01ü.sit";
Content-disposition: attachment
Content-transfer-encoding: base64

The first bolded line contains the attachment name which is causing the problem, who knows what that umlaut was originally. The second bolded line is what we’re telling grep to locate, I couldn’t get anything to dependably match the oddball characters. “Content-disposition” appears to be an older attachment syntax and didn’t appear in any of my messages from after 2001.

While the Ruby script could be run from find’s exec command, it wouldn’t be particularly efficient. Calling the script from find would pass every .emlx file in the Mailboxes directory through Ruby’s gsub, which is almost wholly unnecessary (and much slower). Only 10 of my 57,007 messages needed fixing, 99.98% of them were fine.

Is this safe?

Since the most common way to encounter this bug involves Time Machine restores or migration to a new machine, most users should already be backed up. If something should go wrong, just restore the backup’s Library/Mail folder over the messed up one. You can also copy or zip the Library/Mail folder to another drive to be even safer.

That said, I ran dozens of iterations of this solution against copies of my personal Mail archive without issue. And besides, if you’re reading this, you might not have any of your old mail, so how much worse could it really get?

If you’re still worried, copy your ~/Library/Mail/Mailboxes folder to another drive, run the script, then compare directories with something like Apple’s FileMerge. That will show you exactly which files have changed and what was changed inside them.

Renaming the attachment should be perfectly safe since the full encoded file contents are stored in the message. All the attachment name does is specify the filename, in a sense, it’s totally arbitrary.

I first submitted this bug with Apple back in May, if you have a developer account with Apple, please file a dupe for radar: 5912997

Fixing CS3: All apps crash when saving

Post updated, jump directly to the improved solutions.

I spent the better part of today remotely trying to figure out why our latest Creative Suite 3 installation was crashing. Illustrator, Photoshop and InDesign all crashed when saving. CS3 (Design Premium) was being installed onto a new 24" iMac, everything else with the machine is fine.

For whatever reason, CS3’s shared VersionCueUI.framework component was not installed. Here’s what was showing multiple times in the logs of the problem machine:

2007-08-10 19:30:33.926 Adobe InDesign CS3[919] CFLog (21): Cannot find executable for CFBundle 0x2e1b5a50 </Library/Application Support/Adobe/Adobe Version Cue CS3/Client/3.1.0/VersionCueUI.framework> (not loaded)

Even though we aren’t using Version Cue (I’ve yet to meet anyone who does), that component is necessary for all save functions. If it can’t be called, as I found to be the case here, all CS3 apps will crash out immediately upon invoking Save or Save As. We aren’t installing the Version Cue server on any stations, but I did try installing it once to see if that would fix this. It didn’t.

I re-installed. I repaired. I wiped everything with maccs3clean, restarted and reinstalled — three times. From two different accounts. If you’ve ever installed CS3, you know how much time that eats.

Then I gave up on the installer.

Checking /Library/Application Support/Adobe/Adobe Version Cue CS3/Client/3.1.0/ against a functioning install showed that VersionCueUI.framework had the wrong byte-count. I decided to try replacing it with a copy from a working installation.

Starting from a working installation, I tgz’d the framework with this command:

tar -cvzf ~/VersionCueUI.framework.tgz VersionCueUI.framework

Next I copied the archive to the machine with the problem install, untarred it, moved the bundle to the right directory and changed ownership to root:admin:

tar xvfzp VersionCueUI.framework.tgz


sudo chown -R root:admin VersionCueUI.framework 

I’m sure there’s an easy way to integrate user, groups and permissions into the tar commands to save the chown, but after a day of dealing with this I wasn’t in the mood to look it up.

After that, saving from various CS3 appears to be working perfectly and updates installed without errors.

This experience was very similar to the solution I found to my Illustrator 13.0.1 upgrade problems. Manually doing the installer’s job solved the problem there too.

Dear CS3 Installer,
Thanks so much for ejecting the DVD after a failed install. Everyone I called to shove the DVD back in for me were really glad they could help.

Update: Several great self-contained solutions in comments, I’ll be trying these first if I run into this again. Thanks to everyone who posted.

  • Dave Pijuan-Nomura’s solution looks the simplest and builds on Dusty’s earlier method:
    1. Delete /Library/Application Support/Adobe/Adobe Version Cue CS3/Client/3.1.0
    2. Run Adobe Updater
  • Dusty was first to report success after deleting /Library/Application Support/Adobe/Adobe Version Cue CS3/Client/3.1.0/. He also paused Adobe Updater to make a copy of the Version Cue installer, Updater will otherwise delete the file after the install fails. That standalone updater now appears to be here: Adobe Version Cue CS3 client 3.1.0 update.
  • Dave Henderleiter got it working by renaming an older version of Version Cue:

    In [/Library/Application Support/Adobe/Adobe Version Cue CS3/Client/3.1.0/]… I had a 3.1.0 version and a 3.0.0 version. I deleted the 3.1.0 version and renamed 3.0.0 to 3.1.0 and all the apps worked right away without even a restart.

    Faking CS3 into using older component versions makes me nervous, but several people reported success with this method.

iPhone and the non-deleting email

I’ve seen this happen a few times, an email message just will not delete. Clicking the trash can shows the message delete animation, but the message just reappears behind the animation. The same thing happens in list view. The solution is to reboot the phone, everything should be fine after a restart.

To reboot the phone, hold the wake button for several seconds. A slider will come up asking you to power off. Shut it down, wait a second or two and then press wake to turn it back on. Your email should work normally.

I’ve found rebooting every few days is a good way to clear any slightly odd behavior or minor slowdowns. iPhone is a 1.0 release and these are software bugs. I’m confident they’ll be fixed in a future update.

Update: Force-quitting Mail can also help. To force-quit any iPhone application, just hold the home button for about 10 seconds, until the home screen displays.

Illustrator CS3, 13.0.1 and application.sif

While updating several machines to CS3, one (otherwise identical) machine would not successfully complete the Illustrator 13.0.1 update. I discovered a solution which I posted to Adobe’s forums, where plenty of other people seemed to be running into the same problem. Here’s the text since their support pages are not especially Google-friendly.

After spending way too many hours beating my head against CS3’s lethargic installer, I finally found a fix for the impossible-to-update Illustrator.

I should note that I’ve installed CS3 on a half-dozen machines, five of those installations were flawless (though deathly slow). The remaining one however, on identical hardware, was a disaster. (all were iMac G5s running OSX 10.4.10)

On this one machine, Illustrator would not update to 13.0.1. I’d already given up on Adobe Updater, so I was running the downloaded standalone Illustrator update. It failed every time at “application.sif”. In reading this thread, one line jumped out at me in the Console dump above: “File to add already exists. Need not add.”

Renaming the “application.sif” file inside the Illustrator application allowed the 13.0.1 update to finish. I didn’t delete the file until after the update finished, but I suspect that would work too.

Here’s the path to the file to rename/delete:

/Applications/Adobe Illustrator CS3/Adobe

To find that file, open the Illustrator CS3 application bundle (control-click on Illustrator CS3 and choose Show Package Contents). Open Contents. Open Resources. Open AMT and there it is. Rename or delete it and try running the update again.

Hopefully this works for more than just me.

CS3 is really an exceptionally solid release, but Adobe seems to have completely dropped the ball on the installer. It’s hard to describe exactly how slow it is when things work right, should something fail and need to be re-installed you can kiss your day goodbye. In this case, I ended up losing the night as well.

The care and testing which went into the applications is badly served by a lousy installer, something should be done about this in the next point release. CS3 should not be stuck behind this abomination for the suite’s entire 12-18 month product cycle.

Mbox files and in 10.4

One of the big under-the-hood changes to in 10.4 is that messages are no longer in mbox files, this allows Spotlight to index individual messages without having to first parse out the contents of the entire mailbox. Despite being unused, the old mbox files are often still on the drive, which means that most everyone’s mail is now taking up almost twice as much space as it did with 10.3. (my mail folder went from 1.4 to 2.8 gigs). If installing Tiger devoured a lot of hard drive space, that might account for a significant portion of where it went.

After an Archive & Install upgrade, my ~/Library/Mail directory still has folders labeled *.mbox, but those folders each now contain a “Messages” directories which holds thousands of numbered *.emix files. Those mostly appear to be plain text files each containing one message. There is a small glob of XML plist data attached to the end of each file, as well an integer at the top of the file. The first integer is the message’s character/byte count from the end of the integer to the beginning of the XML data.

In theory, a fairly simple shell script could glom everything together into a standard mbox. Not sure how processor intensive that would be, but the steps to reassemble the data would be trivial. At very least Apple’s decision to move away from the mbox format can be easily reversed with no data loss.

Not much has been written about this, but I found this MacOS X Hints mbox thread which confirms what I’m seeing:

I used to be able to use mutt or pine to view the mbox mailboxes in ~/Library/Mail/<account>/<box>/mbox . In 10.4 these are still present, but appear not to be updated any more. The up to date emails are in ~/Library/Mail/<account>/<box>/Messages/*.emlx which I believe is required for spotlight to be able to index messages – it only indexes file-based entities, not subportions of files.

Because Carbon Copy Cloner doesn’t work with 10.4 yet, I can’t comfortably back up my drive and experiment with deleting the old mboxes. It seems like it should be safe to remove all mbox files and associated files, nothing outside the Messages directories has been modified since I upgraded to 10.4. If anyone has more information, please leave a comment.

(While reading a little background on the mbox format, I found the original RFC for email as a text file. The W3c also has an HTML version of RFC822, partially converted by (sir) Tim Berners-Lee. It’s fun to encounter raw history like that.)

Update I posted a simple command to delete unused Mbox files.

getSelection() Workaround for Safari 1.3, 2.0 and Firefox 1.0.3

update 2: These workarounds also work with Safari 2.0 in Mac OS X v10.4 .
update: There’s a simpler fix, jump to the bottom.

Yesterday morning I noticed a change to the JavaScript/DOM getSelection() behavior in the new Safari 1.3 (in 10.3.9) and the most recent version of Firefox 1.0.3.

I’ve been using this method for years to pull selected text from web pages for several of my bookmarklets. The one I use most frequently generates a link from whatever text is selected. If nothing is selected, it grabs the document’s title. The change in getSelection() broke that bookmarklet, no selected text was recognized.

After a bit of research, I found Mozilla’s Safely accessing content DOM from chrome page which describes the security fixes behind the modification and detailing other problems the changes had caused. Based on Mozilla bug 290777 and this post by Buzz Anderson, both browsers seem to have problems with the change. Despite those bugs, I managed to find the simple workaround as described below.

What Safari and Firefox now seem to be doing is creating a DOM selection object from getSelection() instead of treating it as a simple string. The result is that getSelection() appear to be a string, but few of the string manipulation functions work without additional considerations.

The following examples are all intended to be tested as bookmarklets, drag them to your bookmarks bar for testing:

  • getSelection() test 1

    Works as expected in Safari 1.2.4, Safari 1.3 and Firefox 1.03, popping an alert containing the selected text. Trying to measure that returned string fails in Safari 1.3 and Firefox 1.0.3 but works in Safari 1.2.4:

  • getSelection() test 2

    The older version of Safari returns a character count for the string of selected text. Firefox and Safari 1.3 return “undefined”. There are quite a few other problems:

  • getSelection() test 3

    Works in Firefox and Safari 1.2.4 but not in Safari 1.3.

  • getSelection() test 4

    Getting the length after toString works in Safari 1.2.4 and Firefox.

Further inconsistencies between Safari 1.3 and Firefox 1.0.3:

  • getSelection() test 5

    Returns “Range” in Safari with a selection, returns “Caret” or “None” with nothing selected. Fails with “undefined” in Firefox. (I think the Firefox 1.0.3 DHTML regression bug might be preventing it from working in Firefox but I didn’t try any of the recent nightly builds.)

  • getSelection() test 6

    Fails silently in Safari, returns selected text in Firefox. Safari dumps this into the Console log:
    [5956] :TypeError - Value undefined (result of expression d.getRangeAt) is not object.

  • getSelection() test 7

    Fails with “undefined” in both.

There is some good news:

  • getSelection() test 8

    Works in both Firefox and Safari 1.3; fails in Safari 1.2.4 as “undefined”. This means (finally!) there is a workaround for my problem.

I was using the length property to determine whether a selection was empty or not, then fetching the title of the window if that value was 0. Knowing that length no longer works in Firefox and Safari, isCollapsed can be used as a conditional switch.

  • getSelection() Workaround

    That will return any selected text or the document title if there is no selection. Tested successfully in Safari 1.2.4, Safari 1.3, Firefox 1.0.3 and presumably Safari 2.0 as well.

Line breaks were added to visible code examples because my style-sheet choked on long lines and I can’t redo the CSS right now…

Update: After working through all of the above, I realized there is a far simpler solution: +''. The Safari problem seems to be that string methods do not work on the returned object from getSelection(). Forcing the result into a string by concatenating with an empty string fixes all of my bookmarklets. Concat() fails because it’s a method of string, use the "+" joining operator and an empty string '' instead.