XMP Sidecar Files and Lightroom

by Michael Clark

I've recently fielded some questions about XMP sidecar files in Adobe Lightroom and felt that would be a good topic for this weeks blog post. Before we get into XMP sidecar files and Lightroom, let's first explain why they are needed. Lightroom is a non-destructive image editor, meaning that it does not alter the original RAW image in any way save for renaming the file. Hence Lightroom, as well as Photoshop CS2 and CS3, both use what are called XMP sidecar files that describe the changes and additions applied to a RAW image. These include the addition of metadata, all of the adjustments made in Lightroom or Adobe Camera RAW and any other adjustments or alterations added such as keywords and the like.

Hence, in Lightroom, the XMP sidecar files contain the metadata, keywords and adjustments made to an image. In its default mode, Lightroom does not automatically write these sidecar files into the folder containing your images. You have to go into the preferences and check the "Automatically write changes into XMP" box to turn this feature on as in the image below.

xmp_image_1_15.jpg

Once you have this box checked, Lightroom will create XMP sidecar files that show up in the same folder along with your images and with the same name as the RAW image file but with an .xmp file extension.

Now, I know many of you are asking why would I want to do this? When you import images into Lightroom all of the changes, metadata and keywords are recorded in the Lightroom cache while working on your images so additionally saving this information into the folder with your images might seem redundant. My reasoning for doing this is partly my archival strategy and partly for convenience as well as how my workflow works.

I want to have the XMP sidecar file next to my RAW image file in the same folder because I want that information to travel with the RAW file wherever it goes. After I have backed up my images to two hard drives I also burn them to DVD's - my third back up. I have had two occasions where I've had to reload images from those DVD's (my third backup) and I was very glad to have the XMP sidecar files right there so I did not have to redo all of the work I had done before. This is a huge advantage to having the XMP sidecar files in with my RAW images.

Another convenient reason for using the XMP sidecar files is transferring images I have partially worked up on my laptop to my office computer for final prep. I can just copy a group of images along with the Lightroom XMP sidecar files onto a hard drive, dump them onto my main imaging computer, then import them into Lightroom and voila, all of my settings and everything are read by Lightroom and I can continue working without missing a beat because Lightroom reads those settings from the XMP sidecar files.

And finally, my workflow is such that I don't import all of my folders into Lightroom and leave them there indefinitely. I bring folders in that I am working on, then after a month or two, I delete those folders out of Lightroom so that it stays snappy and fast. Hopefully soon, Lightroom will also become the end-all-be-all image cataloging and archiving software of choice but as of yet it is not. When it is, then I'll leave all of my folders and images in Lightroom.

That's it for this week.

Adios, Michael Clark

24 Comments

Phil
2007-04-30 04:25:12
Michael,


I have this option checked, yet no sidecar files are ever generated. Is this because the XMP info is written to the actual files? I use only JPEGs and DNGs, and I seem to recall that the metadata is written to these files. So does this mean that the only files to have XMP sidecars will be TIFFs, BMPs, PNGs, etc., or will DNGs and JPEGs sometimes have sidecars? Thanks!

Steve
2007-04-30 04:58:19
I was annoyed to find that I could not get Lightroom to save sidecar files for JPG files at all. I usually have my original CR2 and JPG files marked as read only, and assumed that this would make Lightroom use sidecars for both - it doesn't. It just generates error messages for the JPGs.
Jim N.
2007-04-30 07:05:11
Phil,


I'm pretty sure the JPGs don't use xmp sidecars (I learned that after trying to export many jpegs from Aperture for use in LR). Also, I think that the xmp data lives inside the DNG file, along with a small jpeg preview. Michael, is that correct?

Michael Clark
2007-04-30 07:42:54
You are all right, XMP sidecar files are not written for JPEG's. It wouldn't really work even if it did because you have to resave the jpeg to actually get that information into them. XMP sidecar files have and pretty much are for a RAW workflow only. And since Lightroom is a RAW processing software application - which also happens to work with tiff and jpegs - I would highly recommend shooting the RAW file format for the best image quality.


In terms of DNG, the metadata and keywords are stored within the DNG file without altering the original RAW format so that is why with DNG's there are no XMP sidecar files. Imagine your original RAW file is the yolk of an egg and the DNG file format is the egg shell - then all of the runny stuff in side is the metadata, keywords, and thumbnail previews for that image.


Hope this helps...

Andrés Vattuone
2007-04-30 10:32:00
Snapshot and history info is not stored in sidecar files. Neither are virtual copies, as far as I know. As a partial solution to the problem of recovering this info from backup files I decided to jointly store a backup of lightroom´s library (at least the relevant part of it that corresponds to the images to be saved). But then there is no way of merging libraries in the future when I want to recover the files with all info.
Any suggestions are welcome.
Bill Wood
2007-04-30 20:15:43
The XMP information for JPGs and TIFFs is stored inside the actual file itself. There is no need for a separate .XMP file.
Scott
2007-05-01 11:53:24
Are XMPs in Lightroom read both ways - i.e. if I were to edit a photo in Lightroom with "Automatically write changes into XMP" turned on and get an XMP for my RAW file, then change the XMP via Camera Raw, and then go back to Lightroom, would it automatically and instantaneously read back in from the XMP or does it rely on the database?


Also, if I write out to XMP, does my Lightroom database still contain all the photo edit information as well?


Thanks.

Michael Clark
2007-05-01 12:45:32
Scott -


Yes and yes are the right answers to your questions I think. I have not upgraded yet to Photoshop CS3 so I have not tested it out myself but I have been told that XMP files would go both ways. Whether or not that is actually the case - hopefully someone else can chime in and tell us.


And yes, the Lightroom database still contains all the info even if XMP's are created.

Carol Parker
2007-05-02 12:06:25
Your second to last paragraph is EXACTLY what I have been trying to do. I partially edit new photos on my laptop. When I get back to my desktop I copy the entire folder of NEFs and XMPs to the desktop hard drive, then use Lightroom Import. The xmp files are greyed out in my hard drive folder, and Lightroom does not pick up any of the editing changes applied in the laptop. I do have the box checked for XMP files to be generated. Any idea what's going wrong? I would sure like to get this working...it is essential for my workflow.
Thanks!
Michael Clark
2007-05-02 15:08:42
Carol -


I don't know why the xmp files would be grayed out in your hard drive folder - that is very strange. But I just remembered there is a little trick to all of this. Once you have made all of your settings and before copying the files, first, select all of your images, then Export the XMP metadata by going to (in the menu bar) Metadata > XMP > Export XMP Metadata to File. This should save all of your settings to the XMP file and update it.


I just noticed that Lightroom doesn't automatically update the XMP everytime you adjust a file. So at the end of every session you will need to do this if you want the image settings to be in the folder with the file - very strange. Seems like this should be automatically done. Hope this helps...

bbbrinson
2007-05-04 17:12:53
Michael -
The info that LR does not AUTOMATICALLY save to xmp files is very helpful. I had (incorrectly) assumed this worked similarly to in Bridge - metadata changes need not be 'pushed' out to xmp's after the default was set in preferences.


Did confirmation ever come in re: bi-lateral xmp sharing between LR and Photoshop/Bridge?

John G. Walter
2007-05-10 06:38:03
Michael,


I was also confused about this, as after importing some images from a card, AND making changes to the files by both adding to the Meta Data and making adjustments in 'Develop', there were still no XMP files present.


Doing what you suggested, "Metadata > XMP > Export XMP Metadata to File" seemed to correct the problem, however, it also appears to have created XMP files (or at least update them) for every other file in my Library.


I haven't really investigated that issue, but I have studied the Import procedure, discovering how it works, related to XMP files.


First, if the box "Automatically write changes into XMP" in Edit > Preferences > File Management > Metadata is not checked, future changes to your files will not update the XMP files automatically after every change. With the box checked, updates will be made immediately after a change.


With the box unchecked, XMP files won't even be created even if you have chosen to add metadata via the Import dialog. Interestingly, checking the box IMMEDIATELY creates the XMP files, even before clicking OK.


I'm not sure why anyone would want to leave this box unchecked, but I suppose if the option were not there, somebody would complain about their inability to do just that.

Jeremy Rowland
2007-05-17 14:14:37
I did an experiment while trying to build a workflow with Lightroom. I shoot with a number of cameras (digital SLR to point-and-shoot) so I get a mix of RAW/DNG and JPG files, and I want to make sure I can work with both equivalently.


I made a number of adjustments to JPG files that I imported straight off the card, including both keyword/copyright tagging and develop adjusts, then wrote the XMP back out to the files. Being JPGs, no XMP sidecars were created. I moved them to my archive on another drive then re-imported them. In addition to the keywords, Lightroom remembered the develop settings for each image, so clearly the Lightroom settings are being stored inside the JPG files themselves. Other viewing apps (color-managed, of course ;) ) show the unmodified JPG, as expected.


The reason RAW files get the sidecar XMPs is due to the proprietary, undocumented nature of the files, the developers feel (quite justifiably) that storing XMP data in the original files is risky at best. JPGs, on the other hand, and DNGs both include a well-documented place within the file to store XMP data, so Lightroom uses it.


What this means in practical terms is that you can easily migrate JPEGs between installations of Lightroom (eg., laptop to desktop) and keep all the adjustments you've made, while not having to worry about degrading the original JPEG file...just remember to export the metadata!

Michael Clark
2007-07-11 16:33:49
See my latest blog post on Lightroom Catlogs in Lightroom 1.1 - that solves all of the problems....
chris dunkley
2007-08-15 03:02:52
hi michael a quick question i have just started to use lightroom for my wedding business and exported my first images to tiff files but when i open them again on another computer for the client see, the changes have been lost?! any ideas
Michael Clark
2007-08-15 08:10:11
Chris -


I don't exactly know what kind of circumstances you are talking about - give me a call and maybe I can help out.

sunny
2007-08-19 23:54:10
hi I used lightroom for my son birthday party images for showing the other family members all are jpegs files were lost. I have 80gb hard drive but not a singal file in their. After than i found any recovery centre on the website, suddenly i got it hdrconline.com recovery service centre. When i contact with him they suggest me you took out with your hdd only. They wants hard drive for testing after testing they gave you some time for recovery. Whenever you check your data you feel happy because they recoverd the whole data of your hard drive. You need some more details on this website :www.hdrconline.com go there and take out of tension.


best regards

alphazo
2007-09-03 00:38:36
when I import a JPG + XMP sidecar that contains hierchical keywords into Lightroom... keywords don't get imported. However if I do the same with DNG + XMP I get the keywords right. Is this a limitation of Lightroom?


Dany


PS: Apparently I cannot post the XMP content here but basically it contains
lr:hierarchicalSubject
rdf:Bag
rdf:li ... keyword |keyword

Hans
2007-09-28 19:01:37
Phil,


The trouble with sidecar files is actually the standard specification! The official XMP specs from Adobe describe that the sidecar file needs to have the same file name, with the extension ".XMP". In my opinion, this is a major flaw in the XMP specs. Example: a file named birthday.dng will get a sidecar file called birtday.xmp - instead of birthday.dng.xmp.
This may sound trivial, but you can already see how LR (and currently any other XMP app with sidecar support) stumbles upon this issue. By definition and design, sidecar files can theoretically be used with ANY file format (even XMP supporting formats).
I posted numerous request about this on the Adobe XMP developer forum and never got an answer. I suppose the XMP spec writers dug themselves a deep hole here...and fell in.

Korayem
2007-10-16 04:58:25
I follow the exact same archiving pattern in my workflow.


I have a laptop with a small hard drive. I work in lightroom to develop the photos, but after that, I dump them onto a bigger archival harddrive and backup to dvds to leave room for more photos.


I have torn my hair in regards to this issue, I said there must be a way, until I read about the nifty checkbox ;)


I just stumbled upon this post which discusses the same issue. Thank you.

Vida
2007-11-26 21:40:09
I edit files for other photographers. Does this mean I have to re-burn all these JPG's for those photographers who shoot JPG instead of RAW? If so, what incredible waste of resource and a huge oversight and mistake on Adobe's part.
Eric
2007-12-02 20:27:15
Michael,


I convert all my RAWs to DNG upon import. Do I need to have the "Automatically Write Changes to XMP" checked to have it write the changes I make in LR to the DNG files or does it write it to the DNG regardless?

Michael Clark
2007-12-02 20:30:28
Eric -


The XMP is stored within the DNG file itself - so if you are working with DNGs you don't have to worry about it. I may be wrong about this but I think this is right....

gary Chapman
2007-12-04 12:23:49
I import as DNG files and have the "Auto write" button checked, but the keywords added after import don't seem to be showing up in Bridge or iView.