"read the input into memory" considered harmful

Article:		Top Ten Data Crunching Tips and Tricks
Subject:		"read the input into memory" considered harmful
Date:		2005-06-22 05:49:40
From:		GregWilson
Response to: "read the input into memory" considered harmful
I agree, out-of-core algorithms that don't pull everything into memory at once are absolutely necessary in a lot of cases. However, the focus of the book was on automating odds-and-ends tasks, like pulling sales ranking data off Amazon and finding peaks and valleys. (Gosh, why would I be doing that...? ;-) If you can process your data record by record, that's great; if you can't, and your data won't fit in core, then what you have is a real programming task, rather than a one-off throwaway script.