We've expanded our news coverage and improved our search! Visit
oreilly.com for the latest or search for all things across O'Reilly!
Article:
 |
|
Top Ten Data Crunching Tips and Tricks
|
Subject: |
|
"read the input into memory" considered harmful |
Date: |
|
2005-06-22 05:49:40 |
From: |
|
GregWilson
|
Response to: "read the input into memory" considered harmful
|
I agree, out-of-core algorithms that don't pull everything into memory at once are absolutely necessary in a lot of cases. However, the focus of the book was on automating odds-and-ends tasks, like pulling sales ranking data off Amazon and finding peaks and valleys. (Gosh, why would I be doing that...? ;-) If you can process your data record by record, that's great; if you can't, and your data won't fit in core, then what you have is a real programming task, rather than a one-off throwaway script. |