Building a Simple Search Engine with PHP
Subject:   Optimizing multiple occurences of same word on same page
Date:   2002-10-30 08:07:18
From:   anonymous2
Instead of inserting several records for the same word on the same page, you could add another field to the occurence table which indicates the number of occurences of the word on the referenced page. This reduces the number of records per page to the number of distinct words. Also stop words (excluding some words from the index is always good ["a","the","and"]).