A Reason for Using Externally Referenced Masters

by James Duncan Davidson

For a long time, I was perfectly happy with the managed library structure introduced with Aperture 1.0. The simplicity of having the application manage all those pesky RAW files outweighed the fact that they were packaged in an opaque structure. And the ease of use of Aperture Vaults to back data up was the frosting on the cake. Of course, this wasn't everyone's cup of tea and I was happy for those people to see the introduction of support for externally referenced masters in Aperture 1.5, released last September. I thought, however, that I would ignore that feature and stick with the managed library. I really did.

Eastern Sierra


But then, in October and November, I had a change of heart. For the first time in a bit over a year, I started printing large format prints again. While the RAW converter in Aperture gives good results in many cases, I much preferred the prints I could make of many of my photographs with different tools, such as Photoshop with ACR or Lightroom. With the RAW files tucked away in the managed library, however, it was a bit of a hassle to use external converters. I was exporting the RAW files out, opening them up in those other tools, and wading through the resulting duplication of files—something that I was glad to get away from when I started using Aperture.

22 Comments

Markus Nagl
2007-01-23 00:16:36
Hi! I´m just wondering if your external drive is FAT32 or Mac file format. I heard that Aperture has some problems with FAT32 drives.
Thanks for your informative blog!
James Duncan Davidson
2007-01-23 00:30:15
Markus: You're welcome! :) All my drives are HFS+ at this point. I wouldn't have a problem using FAT32 on a remote filesystem for backup purposes, but for day to day operations, I'd certainly try to keep everything HFS+ as much as possible.
Jay
2007-01-23 01:09:33
Thanks, I have been frustrated trying to export to Capture NX, and this Finder idea works perfectly (just tried it). If Aperture can keep track of referenced pics you'd think they could develop a Vault routine for this - that would be awesome.
Rob Brown
2007-01-23 01:22:58
If you're after a good alternative to the vault backup (you still have to back up the vault for your settings though), then look no further than the free iBackup tool that allows you to schedule backups of any directory. In the preferences you can set the app up to remove files on your backup drive that have been deleted on your source.


I use the referenced library and backup using the vault for my settings and iBackup for my photos - works a treat!


http://www.grapefruit.ch/iBackup/

Kim
2007-01-23 02:13:10
Why rsync? Why not a standard backup tool like Retrospect?
Michael Early
2007-01-23 04:22:51
James, I like your approach. But rather than the rync I have been using Deja Vu as that lets me make a file copy of my data base and that way if there is a problem I can just plug my external esata drive into my laptop and continue my work. (With something like Retrospect - great product - I would have to do a restore.)


I have not gone to the point of backing up every few hours yet however, just every night seems to be working fine for me. Keep up with the good info......

Don Landwehrle
2007-01-23 06:08:29
"This last little bit isn't as straight forward as it should be--it should be just a drag and a drop into the Aperture library window--but it's workable for now."


If you command-option drag the new image back onto the Aperture Project you want the image in, Aperture will reference the new file instead of managing it.

Eric
2007-01-23 07:21:21
Kim -


rsync is *the* standard unix backup tool. One of the great things about it is that it works over the net as well as locally, and only moves files that are different. Also, if you drop the '--delete' flag, it will keep all files in the backup, even if you delete them from the original.

Roger Nolan
2007-01-23 07:45:11
for more geekery, you could use launchd to perform the sync when your FW drive is plugged in and Aperture is not running :-)
Daniel Mendez
2007-01-23 08:01:27
I use a managed library during the week and export to my referenced version when I sort through things and do some adjustments. I only connect to the external lib when I need to do backups, add images or retrieve images. Sort of like a hybrid approach.


James, one question:
When you need another editor to work on a file, why not create a new version from master (option-g), then open that version using an external editor like photoshop? This is the same for referenced files or not.
This way, the new file never leaves Aperture and is already stacked for you. When you save the file, aperture updates the thumbnail (and preview if you have one). You can choose your editor and the format of the external file in the preferences.
Is there a reason why you don't do it this way?
thanks.

David Crellen
2007-01-23 09:51:14
I feel that important to store your masters independent of any application as you stated in your blog, James. I do just that and avoid having to eftw geeky as I use a nice backup app called NTI Shadow. It's under $30 (Mac and Win) from http://www.ntius.com/shadow.asp.


I setup three backup strategies. The first backs up all of my User directory data on a daily basis onto a 750gb external drive I call "Master Backup" which I save and replace with a new drive at end of each year. (I allow multiple versions of the data - this is similar, I think, to what OS 10.5 will do with their Time Machine).


The second strategy backs up my entire hard drive once a week onto the same "Master Backup" external drive. Thus, I have the equivalent of my entire iMac 24 drive up to date on a weekly basis.


The last, the most important, backs up all of my "Pictures" directory onto a separate drive I call "Photography". I set this to do on a manual basis as I only need to do this periodically.


In the above scenarios, NTI Shadow only backs up changed files so you aren't unnecessarily chunking away the entire drive.

James Duncan Davidson
2007-01-23 09:57:26
Kim: As Eric said, rsync is a standard Unix tool. It's on every Mac OS X install out of the box. And, its ability to push files over the network to any box with a login is priceless. And, like Micheal's solution with Deja Vu, the files are usable without a restore.


Roger: Hehehe. That's a nice thought. I may have to play around with that sometime.


Don: Holy moly! That's not intuitive, but it works! Thanks for the tip!


Daniel: If I created a new version to work on in an exteranl editor, that version would still be decoded from RAW and passed off to the external application as a TIFF or PSD. I'm doing this precisely so that I can use a different RAW decoder on the image for those cases where I'm not happy with the Aperture rendering. If I were happy with the rendering and just wanted to perform some edits with a different tool, say Nikon Capture NX, then I'd do exactly what you indicate.


James Duncan Davidson
2007-01-23 14:37:46
Jay: I agree that Aperture, since it's keeping track of referenced files, should be able to Vault them. But I expect that the true reason it's not there at this point is a UI issue. Doing this well would require the application handling whether or not referenced files were "online" and asking lots of questions about what to do in those cases. Hopefully, the next major release of Aperture will address this.


Or maybe we'll all be happy with Time Machine at that point. :)


Micah
2007-01-23 17:41:33
On Backing up stuff.... just thought I would ad my 2 cents. I have been really hapy with Synk Professional - Decimus Software


The app is really flexible and can do things like backup as root (doesnt matter who is logged in--good for computers with multiple acocunts) and it can backup to several locations at once.


At hearts it is a syncing app, so it can;t do things like make incremental backups to DVD. For this I turn to .Mac's Backup utility. not as flexible as Synk, but gets the job done.


I beleive deja vu works as a wraparound gui to rsync, as do many backup apps out there. The big question right now with Synk Pro, is if the new Apple Time Machine will make it obsolete. Now that I hear the new OS will have support for ZFS, things could get pretty interesting!

Henry Trentman
2007-01-23 19:47:26
James- I am puzzled at your reason for using referenced files. I can see many reasons for them but am unsure of yours. If you are printing to large format and assuming you are using a rip like Colorbyte software, why would Adobe PS be any better than Aperture at converting the raw to a tiff or jpeg to be printed on a large format printer? Adobe is more flexible and better I'm told at resizing than Aperture but a rip will do a better job than either of them. I use a 7600 and the output is as sharp at 22" wide as the picture is on the screen. Is there something else that PS does that improves the output besides enlargement? Also, if you were going to drop Aperture for some other RAW processor or file manager you are still going to have to export your files from Aperture whether you use referenced or managed files if you wish to export all the side cars with the metadata in them. You still loose the adjustment data but that would be the case with any raw processor. At least by exporting the side cars you would probably not have to reenter all the keywords, captions etc.
I really enjoy your posts. I have found them very helpful, but feel a little out in left field on this one. Best, Henry
James Duncan Davidson
2007-01-23 20:27:46
Henry: I'm not using a RIP. My printer of choice right now is the HP B9180. I'm currently lusting over a Z3100, but that's probably not going to happen just yet. I use the HP drivers and print straight out of the applciations to the printer.


The first reason I'm using different converters for my larger prints is that the RAW converters aren't equal. You get different results out of each converter--especially in the realm of sharpness and absolute detail. In my photographs, the conversions from Aperture are softer and have less detail than my conversions using Lightroom. It's not typically the kind of thing you'll see in a web-sized image. Or even photographs destined for offset printing in normal sizes. But in a 13x19 print, it's noticeable in direct comparisons--at least with my images. YMMV since Nikon RAW handling may differ from Canon RAW handling in any particular application.


The second reason I'm using Lightroom to print my big images is that, beyond the detail it can pull from my RAW files, it has a really kick ass print sharpening routine. The print sharpening in Lightroom is far better than almost any other pre-print sharpening I've tried. Prints come out tack sharp where they need to and without any evidence of halos or oversharpening. I haven't directly compared it to something like Nik Sharpener Pro (which I've used before), but it feels like it's in the same league. Its the kind of thing that, when you see it, you just immediately think "Now, that's what I've been looking for!"


In direct A/B printing of the same image with Aperture vs. Lightroom at 12x18, I like what comes out of Lightroom more. It's as simple as that.


Now, could I perform some set of adjustments in Aperture which would result in better prints? Maybe. I've tried most everything though, including radically tweaking with the RAW converter settings in Aperture. I haven't found anything that is satisfying in A/B comparisons.


As for dropping Aperture, let me be really clear on this point: I have no current plans to do so. Aperture is just the kind of tool that I've been wanting ever since I started shooting RAW in 2000 with the Canon D30. There's nothing else out there in released form that comes close to it in terms of metadata handling, organization, and the like. It would have to be a heck of a tool to make it worth the data migration. And, from what I've seen of Lightroom as of B4, it's not going to be in the same league as Aperture for management and organization when it's released.


This strategy is all about using Aperture where it excels and taking advantage of other tools when the need arises. Since I only print a very small number of my images at large size (less than 1%), going to an external tool to make these prints is acceptable at this point.

Andy Delcambre
2007-01-23 23:45:09
rsync with --delete is dangerous. I think everyone who uses rsync on any sort of regular basis has a --delete horror story, I know I do. The problem with using --delete for a backup like that is if you accidentally delete something but don't realize immediately, and "backup" your photos, rsync will happily delete the other copy of the file you accidentally deleted.


This problem is compounded if you are running the rsync backup in a cron job.


Were it me, I would operate without the --delete for a daily basis, then if the backup got full, run a --delete after ensuring you aren't doing anything you don't want to.

Andrew Turner
2007-01-24 05:59:35
I was just thinking about changing my strategy to use external files. Especially since I want to use some Automator tools and scripts to geotag my images from GPS tracks.


However, I have just imported, tagged, and organized several weeks of photos from a trip to New Zealand. I now want to reference the external images as the Master. Is there a way to easily 'replace' the referenced master to an external file without having to re-import and modify all of the photos?

Karen Buckland
2007-01-24 08:56:42
Duncan, like you I have found Aperture lacking in RAW conversion quality for large fine art prints, and I've adopted a referenced file strategy for the same reason. I am also unwilling to give up on Aperture's superior organizational tools. Fortunately, I don't need to choose between Aperture, Lightroom, Photoshop, or CaptureOne, and use the best tool for the job at hand.


Since so many people have indicated their favorite referenced master backup progam, I thought I would mention ChronoSync. It will do an incremental synchronize backup (or a mirror backup) and it will also place deleted files in an archive so they may be restored if you accidentally deleted the originals before you synchronized.

balijsamina
2007-01-30 13:07:41
Hello!
since many years I go for my vacation in Indonesia, in the wonderful island of Bali.
I just love that place. Maybe because of the sun and the beaches, maybe because of the cristalline water. I don`t know. What I know for sure is that I want to find a way to express this love.
My project is to get information and foto`s about Bali.
With such material i would like to make a website with travel information for those of you that wanna go to Bali, inclusive bali travel and more.
Please replay or pm if interested.
Bye.
Andrew Turner
2007-02-10 13:06:17
I figured out the answer to my own question.


To change from imported images to externally referenced images, and keep all your edits, Select your library, then in "File"->"Relocate Masters for Library..." This will essentially move all your images out of the Aperture Library back to a file structure while maintaining all of your references and metadata.

EXL
2007-04-26 10:55:42
I'm considering using externally referenced files rather than the opaque library. One thing that I'm trying to understand are crash/corruption scenarios. If all the versions, edits, and enhancements are kept in the vault, and let's say I keep the masters on an external firewire drive, if the external or my computer crashes (and I have individually backed up both), will I be able to "sync" up the versions and metadata with the masters? I don't know too much about the mechanics here - just looking to see if there are any hidden vulnerabilities using reference file approach. Would appreciate any advice on this one. thanks!