iPhoto, iTunes Falling Down on Library Size

by Scot Hacker


Just so you don't think all I do is kvetch about Apple, let me say up front that I thought today's keynote was a total gas, and I've already fallen head-over-heels for Safari. I think Final Cut Express perfectly fills the gaping maw between iMovie and FCP, I think Keynote is going to be a PowerPoint killer on the Mac, and that the new PowerBooks are right on target. All that said, I'm royally steamed about a glaring omission in the iLife suite.





I've got 1GB of memory in my 867MHz PowerMac. But at 15,500 tracks, iTunes is starting to become difficult (not quite impossible) to use. It hums along fine if I don't touch it. But simply selecting a track can result in 20 seconds of spinning beachball. Editing an ID3 tag can take more than 30 seconds. Dragging tracks to a playlist, same. Trying to use iTunes as the music database it's designed to be has started to become more chore than fun. I'm not even done encoding all my CDs yet, and already I can tell iTunes is not going to make it to 25,000 tracks, let alone what it might grow to in the future. What kind of digital hub is that? I can run a FileMaker or MySQL database with a million records and they won't bog my machine down like this - iTunes needs to work on efficiency, period.




iPhoto is even worse. At around 800 images, it started to bog down on me. At 1,000, it started getting confused, and refused to allow me to add more images (or rather, it would add them to the library, but would not create thumbnails). 800? I've only had the digital camera for 18 months, and we edit our images down pretty scrupulously, deleting all the deadwood. We were totally hamstrung until we found iPhoto Librarian, but it's a major pain to have to switch back and forth between libraries, and there's no easy way to get images from one library into another's albums.




I'm not steamed because there are bugs. I'm steamed because Apple announced a whole raft of iLife features today -- great features, no doubt -- but made no mention of addressing the one thing that thwarts the very people who take the digital hub sales pitch seriously. I spent two hours of my Christmas holiday helping Dad clear out his iPhoto library because I knew he'd be hitting the same glass ceiling soon.




So I spoke to an Apple employee about this today and she agreed 100% - it makes no sense to add new features before basic scalability is taken care of. She was steamed too (but asked to remain anonymous), and implored me to tell my friends to use the iPhoto feedback page to register my complaints. Apparently, the feedback really does get read, and cumulative mail amounts to real pressure on developers.




If you're struggling with quote-unquote "large" iPhoto or iTunes libraries, let Apple know they have to take care of the basics before they can sell us on the fun bits.


37 Comments

anonymous2
2003-01-08 00:31:27
The random bugs are also funny
The deletes that happens if you let your hard disk fill up, and the apps run out of disk space on which to save your precious meta-data.
Or the fact that iPhoto crashes on startup if you move your photo collection on the disk.
anonymous2
2003-01-08 01:32:12
25,000 Tracks
So you have 25,0000 tracks on your Mac. How many ears do you have? Do you REALLY need to archive them all? I did the same until I realised I was just wasting space archiving material that I listened to once every 5 or 6 years. I'm being flippant, I know but I do agree that the bugs have to be sorted. The problem is that at some point surely a ceiling has to be set and I'm not sure how Apple will define that. I would assume that you are in a minority having that many tracks (you must have disk space to burn! - No Pun intended). Most casual/non-geek users (which is who the iApps are designed to attract) will never have that many tracks in their entire Vinyl/CD collection let alone on their Mac. If you assume an average of about 8 tracks per CD/Album then that's a LOT of them in one place. I would prefer to see other more serious issues tackled prior to this one in iTunes. The iPhoto one is a pain in the old back passage without a doubt - That should have been fixed ages ago, so I'll send some feedback and hopefully we can see a decent response to your problem!
anonymous2
2003-01-08 03:04:32
25,000 Tracks
As a former dj, I own over 1,000 cds which works out to approximately 18,000 tracks, painstakingly ripped (some twice after a hard drive failure). I'm not sure that I understand the argument about "most casual/non-geek users" given that Macs are the platform of choice for media professionals, people who want to USE their computer in an integrated, intuitive fashion, decidedly non-geeky in my opinion (although possibly still obsessive about media). What kind of sales pitch can be made by saying: "iTunes is a great, mind-numbingly easy to use tool to keep track of your music, but don't load it up too much, or you might have problems." Due to the responsiveness challenges that I was also experiencing with my collection, I find myself using the web interface of the netjuke(netjuke.sourceforge.net) mp3 player with its MySQL backend more often than iTunes. Since it's a web app, it doesn't seem to have the same auto-updating everytime you make any change of state time lags as iTunes (changing track information in iTunes will give me the spinning beach ball for 20-30 seconds in library view, but isn't quite as unmanageable in a playlist view). And installing netjuke once you have your web sharing turned on and php/mysql activated was pretty much cake.
dicklacara
2003-01-08 04:34:23
Database File System
At some point in time OS X will likely have a database as part of the file system.


This would be the natural way to resolve the data/file limitations of the all the iApps.


You could likely store all the reference info (Title, Artist, etc) in the db and have very rapid access/drill-down.


The actual sounds/images would, likely, continue be stored as simple files.


Dick

anonymous2
2003-01-08 05:33:18
25,000
When I made the original post about the 25,000 songs and referred to geeks (I count myself as one) I was trying to illustrate that Apple is aiming this kind of app at the consumer guys more than anyone. I mean, it's FREE for a start! From what I can gather MegaSeg appears to be a more targeted solution to DJ'ing using MP3's but not having used it I can't really say either way. What percentage of iTunes users do you honestly think want to Rip that many CD's? Would you say that because of this particular issue iTunes "Falls down"? I certainly wouldn't. I thinnk it is a shortcoming to a degree but I've heard very little complaint from other Mac users about the problem (doesn't mean it doesn't exist - I just haven't heard about it or seen it). I won't argue that it can cause irritation but frankly there are other things, IMVHO, that Apple need to sort out with their iApps before they get to this one. As far as the "Geeky" thing goes, the platform will never expand if all we do is keep attributing this "creative types" label to Mac users. I'm so fed up with that - It really damages Apple sometimes. The amount of times I have told people I use a Mac and they then say "Oh. I heard they're good for Graphics but not much else" is ridiculous.
The irony is that making very powerful but incredibly simple to use applications is Apple's forté and as such it's very well suited to the sort of person who would never have even heard of UNIX or O'Reilly.com and who just wants to surf the web, send some mail, mess around with some spreadsheets and rip some of their CD's simply. I'm not saying that Apple shouldn't target professionals such as yourself, or that the feature you ask for shouldn't work but you have to at least realise that what you are asking for (and what the original article relates to) is not by any means a "Standard" scenario. Not everyone is a DJ!


I wonder if iTunes 4 may offer the fix you're looking for since it is supposed to be Rendezvous enabled. If they start pulling in playlists from the rest of the network then I think more people will definitely run into this problem.


Anyway - Apologies for dragging this off topic a little. I just think it's important to keep hese things in perspective. What I should have done was merely just say that I disagreed that iTunes "falls down" over this issue.

anonymous2
2003-01-08 06:24:36
Netjuke
iTunes really isn't meant to store that many tracks. If you want to share all your tracks with non-Mac clients, it also doesn't work.


I also picked Netjuke to store the ~800 CDs in my collection, running on a Linux box in the corner.


"Favourite" music remains on my Mac, and everything else is only a URL or fileshare away.


Off to iPhoto feedback form...

anonymous2
2003-01-08 06:31:59
25,000
My iTunes library is over 20,000 songs and continues to grow. I really want to hear them all (49 days of non-stop listening would do the trick) and I want to use iTunes for its organizational features and ease of use. Being able to digitize my lp collection, rip my cds, get it all in one place and play anything I want whenever I want is part of what having a mac is all about. Thank goodness for this article - I thought I was the only one with these problems! By the way... I'm no DJ, just a joe-consumer type with a lot of music to listen to.
anonymous2
2003-01-08 07:24:11
2266 photos in my library--so far so good
I haven't noticed too many problems or gross slow downs with my 2200+ photo library. It takes while on my GhzTibook (1 gig ram) to get everything from disk into memory, but once there, it scrolls through the thumbs fine. I have dozens of albums and can create and delete them easily.


That said...
I have had problems with iPhoto tanking on a Ti 400 with 384 MB RAM and about 2000 photos.

trekkie
2003-01-08 07:36:38
Several Thousand photos
I've got a Canon D60 and a baby girl, so I fire away with no abandon on my camera whever she is looking cute, which as most parents know is about ever 2.3 seconds.


I pull all the photos into iPhoto so my wife can use them as a screen saver. I mainly use iPhoto as a 'what is the filename of that photo' type of program.


Right now, I don't put all my photos in because on my iMac FP 800MHz the beach ball thing gets a work out, and it's hard to find pictures.


But just about any program I used on Windows would do the same thing with that many files in directories. Are we seeing library speed problems, or are we seeing too many files in single directory file system problem? Or is it a combination of both?

anonymous2
2003-01-08 07:57:18
iPhoto 2 adds multi-albums
Hey Scott, lighten up!


Looks like help is on the way. Scroll down for more info on archiving images in offline albums.


Check it out:
http://www.apple.com/iphoto/organize.html

anonymous2
2003-01-08 08:12:00
Upper Limits...
Looks like you've reached the upper limits of iTunes. 25,000 tracks is well over a 1,000 CD's! That's one hell of an audio collection that exceeds what most DJ's haul around in boxes.


iTunes appears to use a binary Library file along with an XML file. If the binary file is using ObjC objects that are archived then parsing that much data would cause the slowdown.


To deal with that amount of data you really need something faster that's designed to dish out data rapidly. Something like an RDBMS for example.


The aforementioned NetJuke would probably be perfect. Don't know if it will run on OS X but it's worth a shot.


I seriously doubt many have been impacted by the 20k+ record limits you are experiencing with iTunes.


I'll agree they will need to look at this issue and improve the scalability but I can almost guarantee that the design specs didn't take that many MP3's/Photo's into account.


At that stage of the game, you will need a professional commercial solution. iPhoto is really coded in AppleScript, so it's no wonder that tool is bogged down. But again the Library storage and retrieval is the real problem.


The spec's you need are definitely a server type of solution that would require a RDBMS to manage the library storage and retrieval. I can see some business opportunities for media savvy developers to build server based systems for photo, mp3 storage. Even QuickTime Movie archives.


How about an RDBMS application that can organize, store and retrieve data rapidly. Stream said data, import/export iLife Library Files. It could be web based or have a services interface. Hmmm a Sherlock plugin would be nice. Rendevzous, etc. Oooh I got it! How about a home server appliance that's a true digital hub! Less than an XServe and more than a TiVO! Something with SCSI expansion so you can RAID a bunch of drives, a DVD burner/player, Dolby Digital/DSP input/outputs, etc. But then again, that would cost a ridiculous amount of money with most of it being devoted to the drive storage. It would never work unless some form of high capacity high speed storage was suddenly invented that cost 1/10th of what we have now. Something like Holographic storage on crystal... Sigh, maybe in 20-50 years....

anonymous2
2003-01-08 08:30:36
iPhoto sluggishness
You can reduce sluggishness in iPhoto *considerably* by making sure to keep all of the "film rolls" in the Library closed.


This is not a solution but rather a workaround while Apple works on cranking up the app speed.

jcteo
2003-01-08 08:34:26
Yes!

I totally agree with this article. At 2744 tracks, I haven't had any problems with iTunes. But iPhoto becomes unuseable very quickly. iLibrarian isn't really a solution because keeping my photos spread out across 6 (and growing) libraries is getting out of hand. It defeats the organize and search features if I first have to know which library a photo I'm looking for is in.


iPhoto needs to scale!!! Scot mentioned MySQL, I think iPhoto should make use of a relational DBMS like MySQL instead of handling its own database.


anonymous2
2003-01-08 08:56:57
Exactly
I love Mac OS X and all it's iApps...But performance continues to be the main issue. I'm having the same issues. I've got 4000+ CD's and have ripped most to mp3 (not all are in iTunes because of disk space issues) and iTunes is slow with 5000 songs. iPhoto is a dog slow with 2000 photos which isn't that many at all the way we all shoot digital photos (esp. having kids).
I think hearing complaints from folks with g4 1Ghz with 1 GB of RAM should be heard from Apple...with Apple really behind in the CPU speeds this only compounds the issue.


my 2 cents

anonymous2
2003-01-08 09:10:37
iTunes Feedback
You can send iTunes feedback here: http://www.apple.com/feedback/itunes.html
anonymous2
2003-01-08 09:16:01
iPhoto =/= free
A couple of people have mentioned that iPhoto is free -- and while it is true that there is no *marginal* cost to buying iPhoto, it is not "free" -- Apple touts the iApps as "features" of MacOS (and thus mac hardware), and neither the OS or the hardware comes for free.
anonymous2
2003-01-08 09:20:59
iPhoto and 4,000+ photos
I have close to 4,000 photos in iPhoto and although it's not very fast, it's still doing its job.
anonymous2
2003-01-08 11:19:12
3500+ Photos in iphoto
i too have over 3500 images in iphoto running on a 640MB ibook 500 MHZ G3. most are 2.1 megaPIX compressed to just under 800K JPEGS.


i've found that if I collapse the film rolls (i have over 100) and limit the number of previews I see at any time to just one or two rolls, iphoto is still zippy enough on my ibook to import and preview new images w/o a hitch.


cavaet: sloppy film and preview management slows everything down to a crawl. iphoto is not suitable for applications which require large preview libraries.


.02 U.S. cents, .015 canadian cents, .02 euros ...
ur change may vary


-born 2 design guy

dougadams
2003-01-08 12:52:05
Upper Limits...
I believe the limit for iTunes is around 32K tracks.
anonymous2
2003-01-08 12:56:40
Astonishingly incompetent design ...
... is the only explanation for iPhoto. My 8086 640k 8MHz IBM PC clone with a 20MB hard drive could easily handle a database of several thousand records (and iPhotos problem seems to be a metadata problem, it leaves images in the file system).


iPhoto chokes once one exceeds about 600 images with comments, titles, and multi-album membership. That's incredible.


I don't know WHAT they did to screw up so badly. My uninformed guess is that they're using the file system to keep track of small bits of metadata that should have been kept in a simple database (a 100,000 image library probably has about 1-2 MB of metadata in maybe 130,000 records -- we're not talking anything that would have strained an 80386).


I'd love an explanation of what went wrong in iPhoto, and how the heck it wasn't fixed.


jfaughnan@spamcop.net
john faughnan

anonymous2
2003-01-08 13:18:46
Digital Hub?


As much as I love Apple this sounds like Digital Traffic jam rather than Digital Hub...

anonymous2
2003-01-08 16:14:43
Jaguar
iPhoto runs much faster on my machine (400 MHz TiBook) after upgrading to 10.2.3 from 10.1.x (whatever the last revision of 10.1 was). I have 1119 photos.
anonymous2
2003-01-09 08:19:44
Decision Breaker
Thanks for this insight. I was just about to buy a Mac to manage the growing library of photos from my digital camera. I am using windows xp, which is feature starved, but does scale. I add about a 1000 pictures a month, currently at about 20,000. The whole Mac solution looked attractive, but if it doesn't scale, well it won't meet my needs. I kind of wondered if these Mac apps are just honeypot apps.
anonymous2
2003-01-09 08:28:53
Lazy programmers
Thanks for the article Scott. I just had a similar run in with scalability, on a Kodak DC265 digital camera. It originally came with a 16MB CF card. I recently bought a 256MB CF card for an overseas trip, but found that by the time the card was half full, the camera was taking over a minute to boot and recycle between shots, making it all but unusable. Worse, this didn't seem to be a storage-time issue (the blinking light took a few seconds as always), but most likely a design problem with the routine which merel determines how much space is left on the card (it apparently traverses the stored pictures to get their average sizes, which doesn't scale well). Kodak admitted to a problem, but is not about to "fix" a four year old camera. They claimed that their newer cameras don't suffer from this, but since it's admittedly hard to test, I wonder what will happen when we stick a 10GB card in a camera a few years from now. The point is, programmers (and managers) need to seriously consider scalability in all of their applications, all the time. That or put usability limits in writing.
anonymous2
2003-01-09 08:30:06
Several Thousand photos
I have a collection of 7,323 Music files and I use Windows XP w/SP1 installed and I have never had a problem browsing my files. Everything runs rather well on my 2200+ w/512MB of RAM and this is while I am using my machine to capture video with PVCR in MPEG2 @ the highest available settings.


Winamp3 and MusicMatch Jukebox(which both have some form of an internal library feature) have no problems with my collection as well, and in the case of winamp3, that also includes my video collection. I have a digital camera but I am not much of a picture person but I've not heard a single complaint from my roommate who uses the camera to take photos of his cats at least 10 times a day. They are his babies so go figure.

anonymous2
2003-01-09 09:30:07
iView MediaPro
If you have an ever-growing collection of digital photos, I'd heartily recommend buying iView MediaPro. Not as pretty as iPhoto, but far, far more powerful and flexible.
anonymous2
2003-01-09 11:36:04
seconding iView...
second vote: iView MediaPro is the way to go for large photo library managment (and for any other media management, too -- does a nice job with MP3s or video clips). I have over 5000 images in one iView file and no performance issues on any machine I've ever tried it on. I use iPhoto for the Web printing/book assembly features but I've never tried to make it into a real archiving and search tool.


http://www.iview-multimedia.com/


to "Decision Breaker" -- it's not really fair to assume that just because the free iApps may not scale all that well for professional-level volume requirements, therefore the entire Mac platform doesn't scale. It's apples & oranges (excuse the pun). There are scads of media management tools on the Mac, including iView but plenty of others, that have the stones to handle whatever volume of data you care to throw at them. Before you settle on XP, check in with a local Apple Store or versiontracker.com and see what apps are out there to handle your requirements, other than iPhoto.


-Mike Rose

anonymous2
2003-01-09 14:05:04
6733 photos
I have 6,733 photos in iPhoto and while it's not fast (it takes a minute or more to launch on my iBook/800), it does work.


I am worried that it will stop working. And it was certainly a pain to use my 110 free .Mac prints, and I'm a little annoyed I spent all the time trying to get them done before Christmas, only to have the deadline extended.


I wonder how other apps like iView MediaPro handle this kind of load. (I have a license for the older version, both through Toast and separately.)

anonymous2
2003-01-09 14:58:18
archive in iPhoto 2
hi


your article failed to mention that apple have come out with an archive to cd or dvd feature in iPhoto 2. This means that you don't have to keep all your photos in there at once, hense you can enjoy faster speeds again. displaying all those thumbnails at once was a really impressive feat. I can understand when it slows with 2000+ photos in there at once but now they have given us a solution (or will have when it evenually gets released). Stick the ones you don't use on a CD and free up your library once more.


Mark

anonymous2
2003-01-09 19:58:43
The keynote was right on
There may be fixes already in the works for these problems. Not everyone has run into these problems yet. Steve Job's keynote was not a good place for Apple to admit to glaring problems with the packages they are trying to get everyone excited about. It's a place to get everybody excited about the inovations that Apple's trying to acomplish. All of these applications are still just babies. They still have a long life ahead of them. Lot's of changes. And, hey, why are we complaining??? They're free!!!!
anonymous2
2003-01-13 19:22:24
iphoto is too fat
thanks for the info
anonymous2
2003-10-23 12:16:51
Several Thousand photos
Adobe Album 2.0 on a PC can handle thousands of photos w/o a problem. It uses a microsoft access database to handle the metadata and never touches the underlying images (unless you edit the image from within adobe album)


infinity005
2005-10-02 13:13:12
slowest win32 application yet.
I thought Outlook was the most poorly written application used on a daily basis until iTunes. As much as I hate Microsoft, they seem to code better than Apple.


When is someone going to write an MP3 manager with a mysql backend? This flat file stuff is for losers. (Wait, i'm using it, oops)

infinity005
2005-10-02 13:15:37
The keynote was right on
"they're free"


...yes, but I'd rather pay for a good one. The best mp3 application I found was a perl script I wrote. Everything else is too inflexible/and featureless or slow because of the flat file system.

infinity005
2005-10-02 13:17:52
Upper Limits...
"At that stage of the game, you will need a professional commercial solution"


What is this commercial solution you speak of called?

peppepop
2006-01-12 04:09:43
Still an issue on a G5
I though that iTunes 6 would fix the big library problem, but no... It still won't scale, and that is bad. Over 20k songs and it's still the same as in the article, almost three years ago! If I were to test software like this, I would work with libraries of 100k pictures or songs and see if it works. Why not start at 1 million? If it works then, it should scale pretty well. It seems as if they are doing the tests with 10 ripped CD's, thats it. Yes, iTunes rocks with a library that size.



Yawn.


/Peter

Wolf
2006-06-15 08:16:24
35K Tracks, painstakingly ripped, categorized, labeled, lost, ripped again, transferred from one hard disk to the next as space got smaller and smaller (and hard disks got bigger and cheaper).


and all along iTunes is dragging its feet. slower .. and ... slo-wer ... aaanndd .... *yawn* slooooooowww... ;-/


I have looked for alternatives, but found none. (on os x or windows, that is; i'm still not into unix/linux)
iTunes capability of categorizing things is just great and something that I got used to. wouldn't want to miss is.
funny thing though, my 1 ghz mac cube is starting and quitting itunes much quicker than my 3,6 ghz pc, raid-array hd's and all. can't exactly explain it, but it's fun to notice...