I got an email this morning asking if I knew anything about the iPhoto color shifting issue with edited images. I hadn’t been paying much attention to this while working on other stuff, but once I looked for the problem it it was very apparent and troubling.

Based on a lot of good information in the MacInTouch iPhoto Report, a massive thread on Apple Discussions and my own experience with Apple’s image manipulation tools, I think I’ve found the basis of this bug. James Bailey’s observation on MacInTouch specifically identifies the problem:

Using Photoshop 7.0 to help diagnose this, here is what seems to be happening. My original photo seems to have an embedded ColorSync profile of sRGB IEC61966-2.1. […] Opening the duplicated and cropped photo I do not get the same warning which means that Photoshop thinks the embedded color profile is now ColorSync RGB – Generic RGB Profile for the new file. [emphasis added]

That describes exactly what I’m seeing.

SIPS, Apple’s Scriptable Image Processing System offers tools for working with images and manipulating things like ICC color profiles. I have two photos, DSC04737.JPG and DSC04737_1.JPG, the second is a duplicate of the first with a few lines cropped off the bottom to trigger the color change.

When opened in Photoshop, I see the following Color Management warnings:
DSC04737.JPG
Embedded: sRGB IEC1966-2.1

DSC04737_1.JPG
Embedded: Generic RGB Profile

However, when checking with SIPS, I see this:

$ sips -g profile DSC04737.JPG
profile: Generic RGB Profile

$ sips -g profile DSC04737_1.JPG
profile: Generic RGB Profile

SIPS appears to wrongly identify the embedded profile in the original image, believing it is Generic RGB. It gets slightly more interesting, and confirms another suspicion of mine: With 10.3.9, SIPS can’t find a profile in the unmodified image and returns the following:

$ sips -g profile DSC04737.JPG
profile: <nil>

$ sips --extractProfile profiledump.icc DSC04737.JPG
Error: No profile in image

At this point I’m comfortable pointing to the underlying ColorSync foundations updated in 10.4 as the culprit. (These are used by SIPS and presumably by iPhoto too, despite iPhoto’s unique EXIF parsing engine, the foundations are meant to be used throughout the system.) My only hesitation is whether or not this problem appears on 10.3.x after installing QT7.

Comparing the two images without color management in Photoshop shows that the image is permanently changed. Unfortunately, this means that whatever caused the conversion applied a lossy correction to all pixel values. While there is the potential to somewhat reverse this problem with some profile voodoo and dumb luck, the result would be twice shifted and especially prone to visible artifacting and posterization.

Based on the above, I’m guessing that when iPhoto reads the originals, it wrongly interprets the files before applying changes to it. This would mean the modified files are ruined before iPhoto’s adjustments are even applied.

If I were feeling especially adventurous and had lots more time, I might try replacing all or some of the ColorSync frameworks with versions from 10.3.9. I’d find the files to replace here:

/System/Library/Frameworks/
ApplicationServices.framework/Versions/
A/Frameworks/ColorSync.framework
/Versions/A/

Of course I’d be doing that at my own risk, with the knowledge that I could render my computer unbootable and only with a full backup ready to go. But it is an interesting theory and I wonder if anyone’s tried it yet.

Rumor has it that this is all fixed in 10.4.2. (update: not fixed

Apple’s WWDC starts next week and the rumor sites are uncharacteristically quiet even though Steve Jobs’ keynote is only four days away. It’s very unlikely we’ll see any talk about 10.5, since 10.4 just shipped and there are a ton of wrinkles to iron out. I’m expecting to see a lot of new hardware development, or else a two hour buzzkill Tiger demo. There are 18 TBA sessions scheduled including at least one in the big room, so something new is likely to be announced.

The past year seems like it’s been a good for hardware innovation. Now that AMD and Intel both have multicore and 64bit chips in the market the G5 doesn’t seem as competitive as it was two years ago (not to mention being two years overdue for 3ghz). IBM’s Cell Processor is another unknown, but arguments could be made about cramming one into a PowerBook. A friend with reasonably good contacts thinks we’re going to finally see a G5 PowerBook. I hope so, and my credit card is ready to go. Hopefully we’ll at least get that Powerbook HD referenced by the typo in the current PowerBook manuals. And maybe PowerBooks will do a little better to close the speed gap with desktops (or at least iMacs). According to MacRumors Buyer’s Guide, Powerbooks will be 126 days into their product cycle and iBooks will be overdue for a refresh at 230 days. Seems like a good time to announce new portables, especially since nearly everyone at the conference is carrying one.

The tracks seem a lot more technical this year, which is good. The previous two years I ended up in several sessions where I was bored or didn’t learn anything.

My primary goals for this year are a bit more modest than last year. First, I’m focusing on related technologies to FXScript, FCP and graphics stuff. This includes all the pro-video apps, especially Motion and the Shake SDK, plus the Quartz Composer labs. I’m still very interested in CoreImage and CoreVideo, but there aren’t very many of those sessions scheduled.

I have a feeling I’d be bored by too many Dashboard sessions since I’m pretty good with JavaScript and CSS and don’t really have any ideas for widgets that haven’t been done already. Unix scripting and shell commands will be a focus, I’d like to see how other people work with them since I taught myself and feel like I’m often stumbling around.

The best thing I got last year was better programming practices, so I’ll be making a more deliberate effort to be in sessions related to source-code control, development tools and better working methods.

Aside from that, I’m hoping to pick up a bit of Cocoa, get over the AppleScript Studio hump, and find more fascinating stuff that doesn’t fly too far over my head.

Wednesday’s Brown Bag Lunches are a problem because I want to attend three of them; Python Today with the language’s creator Guido van Rossum, PHP on a PowerBook with that language’s creator Rasmus Lerdorf and the MySQL and SQLite lunch with author Brian Jepson. I’ll probably be at either the PHP or Python lunch, PHP because I’ve done a lot of work with it, Python because I’ve been meaning to learn it. Thursday I’m going to the Advanced Scripting with brian_d_foy, who I’m looking forward to meeting.

I’ll also be attending Buzz Andersen’s WWDC 2005 Weblogger Dinner on Monday night. Looking forward to meeting a bunch of people there.

One of the big under-the-hood changes to Mail.app in 10.4 is that messages are no longer in mbox files, this allows Spotlight to index individual messages without having to first parse out the contents of the entire mailbox. Despite being unused, the old mbox files are often still on the drive, which means that most everyone’s mail is now taking up almost twice as much space as it did with 10.3. (my mail folder went from 1.4 to 2.8 gigs). If installing Tiger devoured a lot of hard drive space, that might account for a significant portion of where it went.

After an Archive & Install upgrade, my ~/Library/Mail directory still has folders labeled *.mbox, but those folders each now contain a “Messages” directories which holds thousands of numbered *.emix files. Those mostly appear to be plain text files each containing one message. There is a small glob of XML plist data attached to the end of each file, as well an integer at the top of the file. The first integer is the message’s character/byte count from the end of the integer to the beginning of the XML data.

In theory, a fairly simple shell script could glom everything together into a standard mbox. Not sure how processor intensive that would be, but the steps to reassemble the data would be trivial. At very least Apple’s decision to move away from the mbox format can be easily reversed with no data loss.

Not much has been written about this, but I found this MacOS X Hints mbox thread which confirms what I’m seeing:

I used to be able to use mutt or pine to view the mbox mailboxes in ~/Library/Mail/<account>/<box>/mbox . In 10.4 these are still present, but appear not to be updated any more. The up to date emails are in ~/Library/Mail/<account>/<box>/Messages/*.emlx which I believe is required for spotlight to be able to index messages – it only indexes file-based entities, not subportions of files.

Because Carbon Copy Cloner doesn’t work with 10.4 yet, I can’t comfortably back up my drive and experiment with deleting the old mboxes. It seems like it should be safe to remove all mbox files and associated files, nothing outside the Messages directories has been modified since I upgraded to 10.4. If anyone has more information, please leave a comment.

(While reading a little background on the mbox format, I found the original RFC for email as a text file. The W3c also has an HTML version of RFC822, partially converted by (sir) Tim Berners-Lee. It’s fun to encounter raw history like that.)

Update I posted a simple command to delete unused Mbox files.

While everyone’s rightly worrying about Dashboard and widget security issues, I’ve found a more immediately annoying problem — widgets that run processes while hidden. If a widget appears in Activity Viewer repeatedly sucking up CPU while hidden, it’s gone. I noticed this because one (extra geeky) binary clock was constantly using 8-20% of my cpu cycles while invisible. Hula girl has been known to run off on her own as well.

Sidenote: Clocks in Dashboard are a waste of screen space. Yes I have Hula Girl and the Butterfly, but no clocks.

While responding to an email asking about IPTC data in iPhoto, I started thinking how great it would be to store iPhoto’s information as Spotlight extended attributes, but I found that Apple did it already! This is a great idea and I was glad to see it.

The mdls [file] terminal command will list all metadata associated with the specified file. (The easiest way to see this is to type “mdls”, space, then drag a file into the terminal window)

Anything in iPhoto with keywords or a title should have these two additional bits of metadata attached to them as HFS extended attributes:

kMDItemKeywords = ("3rd Avenue", Mirror, Sidewalk, NYC)
kMDItemTitle = "Man Walking with Mirror 1"

It’s great to see this but it’s not especially portable yet. Anyone know how to write IPTC data from the command line?

There were a ton of things I forgot to do before updating to Tiger. Since I chose Archive & Install, a lot of UNIXy things didn’t transfer over. Thankfully I created a full, bootable backup of my 10.3 disk before installing, so recovering the missing parts hasn’t been too difficult.

A long, long time ago, when computers had to be restarted every 10 minutes, I’d learn how to do things by constantly doing them again and again. Here in the future, I set up CVS once after WWDC last year, and it’s worked ever since.

Steps to rebuild my CVS repository:

1. Initialize the CVSROOT variable using the instructions on Apple’s Version Control with CVS on Mac OS X page.
2. Copy over all the files from the old disk into the newly initalized CVS repository (using cp in the terminal).
3. Run sudo chmod -R g+w on the CVS repository to set all the files to the correct permissions.

Everything seems to be working, file revision histories are intact and I can forget about all of this for another 18-24 months until 10.5 comes out.

Update: WordPress or my server was barfing on this post, so I’m editing it to refresh the database

In 10.3, I had hacked Mail.app to display a widescreen view. It really helped me get close to having a handle on my mail, and was a far better use of my PowerBook’s widescreen than the standard stacked three-view. No luck so far in 10.4. Something about Tiger Mail’s custom ‘ExpandingSplitView’ keeps expanding the splitter to fill most of the window when it’s switched to a vertical split.

update: Still not working, but I’ve found that the width of the splitter bar is tied to the height of the window.

Small Splitter (short window):

Huge Splitter (regular sized window):

update: I’m putting this away for a while. No luck and I’m suspecting the splitter is somehow glued into the application code. If so, it’s probably never going to work from an Interface Builder hack, short of re-building the whole nib (no, I will not try to rebuild the whole nib anytime soon, no no no…)

Final update There’s now a widescreen addon for mail.