Lightroom 1.1 as a Digital Asset Management Tool

by Michael Clark

In my last blog post, talking about the Tom Hogarty interview, I decided to investigate just how well Lightroom would work as a digital asset management tool. I must say that I am reporting the facts as they are here for my hardware and images, these results may not be the same for your hardware and your images, especially if you only shoot jpegs. Below is part of my last blog post:

At the moment, I have about 5,200 images imported into Lightroom. This week I plan to import about 20,000 images just to see how much the program slows down and if it slows down at all. As a test to see how fast or slow Lightroom is working, I will render 1:1 previews for one folder that contains 287 images as it is now (with 5,200 images in the catalog), then repeat this test once I have imported the new images. With 5,200 images total in my catalog it took 38 minutes 54 seconds to render the 1:1 previews for a folder containing 287 images. I will also of course work on images and see if there are other ways in which the program slows down or if it seems to run just the same as it does currently.

Before we get too far along here I would like to share with you some hardware specs because these matter. I conducted this test using an Apple G5 dual 2.0 Ghz tower with 4.5 GB of RAM. All of my images and the Lightroom catalog were on a SATA hard drive. And all of the raw images, save for one folder, were Nikon D2x raw image files which are approximately 12 MB in size.

Earlier this week I imported another 4,929 images into Lightroom bringing my grand total to 10,129 raw images in my Lightroom catalog. I ran the same test as before, rendering 1:1 previews for that same folder of 287 images and with 10,129 images in the catalog it took over twice as long to render the 1:1 previews, clocking in at 1 hour 28 minutes 15 seconds. I wasn't really surprised by this number too much but it wasn't too exciting either. On a separate note, most of Lightroom's functionality seemed to run at about the same speed - changing to the Develop module and working with images was just as it was at 5,200 images in the catalog. It was only rendering previews or selecting large groups of images and trying to add metadata that took much longer than it did previously.

I continued to import images to find out what the numbers would be at 15,000. But by the time I got to 11,365 and tried to render 1:1 previews the progress bar didn't move much at all and Lightroom became much slower than I would have wanted. Again using the Develop module didn't seem that much slower but going to 1:1 on any images took quite a while for it to snap sharp. So, I never made it to 15,000 images and finished my testing at 11, 365. While I found that Lightroom was quite slow at this number of images this is not a defacto standard for everyone. Read on and I'll go more into how different computing power and image size will affect just how many images you can import into Lightroom before you see a significant slow down. It is all relative.

While importing images from an assignment mid last week I decided that I needed Lightroom to run faster - so I deleted about 8,000 images out of Lightroom (bringing my total number of images in the catalog down to around 4,200). The catalog did not fair so well after that and it took a few hours for Lightroom to get back to it's normal self. It seems like it took a while for Lightroom to remove all of those previews from the catalog after I deleted approximately 15 folders. In the end, I had to make a new catalog to work on another set of images because the old catalog did not seem to recover from the massive importation of images and their subsequent deletion.

Hence, the bottom line for my computer, my raw files and my workflow is that while Lightroom may be able to manage 10,000 images, the speed of operation takes a big hit. I won't go so far as to say that Lightroom will not work as a digital asset management (DAM) tool but for right now with my workflow and hardware the application was a little slower that usual. If I was shooting with a lower resolution camera and had a wicked fast brand-spanking new Mac Pro tower with some serious RAM then I'm sure I could go up to 20,000 images in a catalog without any issues.

As it is now, my workflow with Lightroom works fantastic as long as I keep the number of images below 7,000 images. Not a big deal since I tend to use Lightroom as a raw processor with perks. I import images into Lightroom, render the previews, edit and process the raw images, create slideshows and web galleries then when I am finished with an assignment I delete the folder out of Lightroom. If I need to access those images again I can view them in Bridge CS3 and work with the exported tiff files in Photoshop CS3 or even reprocess them using Adobe Camera Raw 4.1 (or re-import them into Lightroom). It works just fine.

Please note that this is not a slight on Lightroom, I wanted to see for myself where the limits were for my hardware and my workflow. I had been warned by others that going over 10,0000 raw images in Lightroom would severely slow down the software and well, I found that out for myself. And I understand perfectly now why Adobe does not put out any concrete numbers or limits on how many images one should import into Lightroom. The application speed depends on so many factors that no number would really be accurate. My aging Apple G5 tower seems to work extremely well with Photoshop and all of my other applications but I think this test has exposed some of weaknesses of a 2 year old computer. If any of you have different experiences I would love to hear about them. If any of you have 100,000 high res raw files in a catalog and Lightroom continues to run like a champ I would really like to hear about that. Perhaps it is time to upgrade to a Mac Pro.

Either way, Lightroom is still the application of choice for my workflow. I don't really need it to be a digital asset management application and I am sure this is high on the list of things Adobe is working on for version 2.0 (I don't know for sure - just guessing).

That's it for this session. See you next week....

Adios, Michael Clark

16 Comments

Markus
2007-08-27 02:41:33
Running on Windows Xp Pro with a 3.4GHz P4, 2GB RAM PC with SATA-Drives: No problems so far with 20.000 images, 80% Nikon D70 and D80 RAW-Files (converted to DNG), 20% JPEG
Sean McCormack
2007-08-27 06:59:59
Hi Michael,
I'm currently running 58,895 images in my Library. I'm on a 2.16Ghz Core 2 Duo iMace with 2Gb Ram. Images are on the internal and 3 external drives. I'll probably get one 500Gb drive and try and consolidate onto that. I don't find it particularly slow, but in a forthcoming magazine article for the SWPP, I do recommend using a new catalog to do your selections for speed. Once you've deleted the crud, you can then import this into your main catalog.
One thing that does happen with the large catalog is that occasionally Lightroom will stop and take it's time loading an image in Develop. Other than that moving between images etc, is fine speedwise.


David Medina
2007-08-27 07:12:05
Hi Michael:


Thanks for your honest assesment. I am not sure what you mean by slow down but I have 26,000 raw images and PSDs in my LR library. Some are RAW files from the D70 which are around 5mb each, and RAW files from the D70 which are around 15mb, plus close to 1,000 PSDs.


I am using a MACPRO with 4gb memory with a dedicated 750gb HD just for images and LR 1.1.


I cannot say that LR has slow down over time... not sure ... it probably has... but it has not affected my workflow at all. I experience "slow sown" when LR first open, but I can't say I have experience slow down while adjusting images, etc.


But I agree with you... I am looking to some Library improvement, specially in features. I would like to see some sort of "smart albums" as well as some sort of Light Table "a la Aperture". I think it would be nice if the folder and Collection can be integrated under one folder so we don't have to be jumping in the panel.


In the develo module I would like to see real localized adjustments sort of like the one you see in Capture NX.


I think those changes would make LR unbeatable.

Michael Clark
2007-08-27 08:03:18
Thanks for the input guys. Maybe I just have a slow computer right now. Do you ever run "render 1:1 previews" on your images with those large catalogs? This is where I experienced the biggest time syncs with a bunch of images....


Also, A great idea Sean about working with a new catalog for speed then inporting that catalog into the larger one - I hadn't thought of that.


Also Markus, D70 files are much smaller than my D2x files so I am not surprised that you can have 26,000 of them in a catalog (with a fast computer) and not have issues. I'll have to keep playing with this and see what happens. Thanks for the input I appreciate it for my own understanding and I hope to hear a lot more.

td
2007-08-27 09:42:42
WinXP, 1GB memory, 3.2GHz single core hyperthreading. Lightroom1.1. Single Catalog.


I am running with about 25,000 images. About 20,000 of those would be JPG, the rest are mostly RAW with some PSD. Most of the things I do don't seem to suffer from performance problems. The one thing that seems to take 'too' long is working with meta data. Selecting several dozen photos to change the meta data - tagging - therein.

Tim
2007-08-27 10:42:38
Michael
After reducing the number of images in your catalog, did you optimize it?
Michael Clark
2007-08-27 10:50:24
Tim - Yes, I optimized it several times after I deleted the folders out of Lightroom. Optimizing helped but didn't get it back to the way it was. I should try that old catalog again and see how it goes. From what others are saying it looks like I need to upgrade my tower here at some point.


One other interesting point is that after deleting the folders out of the catalog the catalog size did not immeadeately decrease as I thought it would. It took it several hours to reduce the catalogs actual size. Don't know what that means.

Colin Wright
2007-08-27 11:11:32
I've seen similar slowing in my sole LR catalog. While it may not be the right tool for DAM, that's the primary thing I've been using it for, whilst playing with web exports, etc. When it comes to playing with my images, I still haven't quite crossed the chasm between using PS for everything and using LR for many things.


My rig is a PowerMac Quad G5 with 5GB memory. Library's on the internal SATA disk, and referenced images are on a Firewire 800 connected RAID drive. The library is comprised of images from five different cameras used over the last five years, with JPGs and RAWs primarily out there -- probably 60%-40% split. And the library? Well, it's got a little over 60,000 images in it right now, across seven top level folders with hundreds of folders beneath each of those.


It seems that after I've done something big in the catalog, the resulting effect is that LR will sit for hours chewing up at least one full processor. I don't have any tasks running inside LR, that's just the idle point after a bunch of purges, addition of previews, etc. Leave it overnight, and it's back to a respectable idle level (20% or less of a full processor). Even an optimize causes this effect for a while. Once it settles down, things are fine.


Now, I'm sure I can go to a smaller catalog size -- carve up my library in some way -- and that's probably the right answer, but I really had designs on using the search capabilities inside LR to be able to carry a small version of my library via an LR catalog on my MacBook.


I guess my sense is that LR 1.1 isn't quite a DAM environment when it comes to real big catalogs. (Or I've got something configured wrong on my side! Never discount the operator....) Maybe that's coming in LR2, or maybe that's not in the sweet spot of what LR is supposed to be doing. Maybe if LR2 could search across multiple catalogs to bring back a result set, that would hit the mark for using it as a manager across a huge archive, split into multiple LR catalogs.


Anyway, that's my experience. YMMV.

Chris Kresser
2007-08-27 12:09:50
I only have about 2,000 images in LR, but about half of them are very large PSDs (100 MB +). I'm a fine art photographer and tend to do heavy manipulation on images. My system is almost identical to yours, Michael (G5 dual-core 2.0 Ghz with 4.5 GB RAM). I've got a 74 GB Raptor as a boot drive running OS X and LR, and my images and user directory are on a separate 400 GB drive.


I'm not really satisfied with LR's performance... it doesn't seem as snappy as it should be considering my hardware. I've been considering a RAID 0 set-up with two 750 GB drives in the G5, and/or maxing out the RAM... but I'm not sure that would help much. I'm guessing that $$ would be better applied to a new MacPro. But with the expensive RAM it would be significantly more expensive than just upgrading my G5.


I'm hoping that a future LR update will improve performance. The LR team has mentioned this as a priority, so I'll keep my fingers crossed.

Tim
2007-08-27 18:43:40
I guess I must be doing something wrong. I've got 5 year old PC I put together myself. P 4 2.75 GHZ with 1.75 GB RAM with A GEForce 6200 Video card with 256MB WIN XP Pro. I've got 15,000 images of mostly RAW images and I've never had reason to complain about LR's performance. After reading some of the other posts it makes me wonder if the RAW/JPG/PSD mix in the catalog of images makes a difference in the performance of LR.

2007-08-28 15:51:36
A NEf file for a D70 is around 4 to 5mg, but when converted to PSD 16 bits it gets upward of 30mb. A D200 NEf (RAW) file is about 15mb and you an imagine the PSd that comes out of it. I have close to 30k images(among RAW, PSD and JPG) which equal to 300GB. I cannot say I have experience any significant slow down. I don't usually make 1:1 previews but I understand it may run faster.
Bradley Kents
2007-08-29 07:03:42
Interesting. I'm not so sure it's hardware (I'm also running a G5 dual with 4.5 megs of RAM), or catalog location (I've tested my catalog and photos running off both an internal hard drive and a big 1.5T FireWire 800 drive), or volume of photos (I have just 6000). Everything was just fine and snappy UNTIL I rendered 1:1 previews for the last month of photos (a process that took hours and ran overnight). I get why I want to do that and am actually taking the tip from your e-book, Mike, but -- to repeat -- once I did it, I have very slow performance. Interestingly, even the refresh on the standard previews used on the thumbnails is now quite slow ...


Any comments?


2007-08-29 07:07:54
Ooops! Correction needed for post immediately below. The location of the catalog DOES matter. I didn't realize I was still running off the external drive. When I select the correct catalog on the internal drive, things are snappy again. Even with the month's worth of 1:1 previews. Sorry for the error.
Michael Clark
2007-08-29 08:08:05
Bradley -


Yup, the hard drive makes a huge difference in sppeed with Lightroom, hence that is why all of my images are saved on SATA drives and why my catalogs is on a SATA drive as well because they have the fastest read and write speeds of any drives out there.


My Lightroom is back to normal. It was very interesting that even at 12,000 images Lightroom ran just fine save for selecting large groups of images to change metadata or trying to render previews.

Sean McCormack
2007-08-29 23:21:07
A quickie again on this Michael. Lightroom keeps deleted stuff referenced in the Library for a little bit before dumping it, in case you need an undo. Previews remain for about 5 minutes to avoid having to rerender if you suddenly decide you shouldn't have deleted the images.
Lightroom is trying to be smart for when we're dumb.
Bradley
2007-08-30 04:58:39
Sorry to prolong this, Michael, but one other wrinkle here that may be an idle thought: when LR is running a little slower than you'd like, it seems like it actually takes longer for LR to get, say, 80 thumbnails refreshed (in the Grid View) than it takes to get one 1:1 preview up at full size (in Loupe View). I guess that makes sense. 80 small things to do vs one task (though a larger size). No comment necessary unless someone has something specific to come back with.