Is Search Going Downhill?

by William Grosso

Related link: www.google.com



A friend and I had an interesting conversation the other day. It turns out that both of us think that, in general, searching the internet for high quality partially specified information is harder than it was a couple of years ago. Putting that into English: We use search engines less, and are more frustrated when we use them. In particular, I find Google frustrating these days-- with the exception of the cases noted below, I always wind up making three or four searches to find anything (not "looking at three or four results" -- "making three or four searches").


So we both use search engines less. And we're both building lists of "reliable sites" (frequently weblogs) that we read regularly, and decreasing our reliance on search engines these days.


Of course, there are lots of useful cases where the internet has gotten easier to search. Here's, for example, are two searches I make all the time:


  • Want to find something that you know is on MSDN? Use Google with MSDN as a keyword string. It's wonderful. Google indexes MSDN much better than Microsoft does (which might well explain Microsoft's recent efforts. In general, if you know enough details about the thing you're looking for, and there's a single canonical result, most search engines do fine.


  • If you know the Java class name, the easiest way to pull up the javadocs for the class is often to use google with three keywords: "java", "class" and the actual classname.


But, in general, I've felt that things have slid backwards in the past year (in fact, Seruku grew out of personal frustration). And I've been noticing that other people are starting to voice the same opinion.


And what I'm wondering is:



Is this really true? Is it getting harder to find things on the web? Are search engines really getting more frustrating?


The reason I'm asking is, of course, that it's entirely possible that I might simply be spoiled. Since the conversation I mentioned above, I've asked 13 more people. 9 agreed with me; 3 think the quality of search engines has held constant, and 1 thinks that things have improved in the last year.


But that's my circle of acquaintances. I'm still not sure what's really going on in the wider world.


What do you think? How has your "search engine experience" changed in the past year?


19 Comments

gulch
2003-09-20 13:15:13
Search is certainly harder, though now there's more to search
I agree, searching does seem to be a lot harder. For me a part of this is because 5 or 6 years ago, I had Alta Vista searching down to a fine art. When Google came along, with a reduced set of advanced search terms, although the basic search was better, fine-tuning things became a lot harder.


Of course, the web has also got much bigger than it used to be, so it could be that we're just trying to find needles in a much bigger haystack. And marketers have got better at getting their site to the top of the search rankings, so for example if I search for a band or musician then I'm likely to have to trawl through pages of search results from CD stores or encyclopedic music sites, many of which don't actually have any information on the artist but want to pull in potential customers regardless. The same applies to most consumer products and services, where I'll be pulled in by a Dooyoo or a Kelkoo regardless of whether or not they have any information on the product I'm after.

anonymous2
2003-09-20 16:48:32
search ain't easy
Although there's lots of noise out there about it, doing a search *well* is NOT easy. Google may be somewhat frustrating, but it's doing a pretty good job overall.


Singingfish (www.singingfish.com) does search for just a limit domain: Streaming Media - and it does it well enough that it's driving the search in Real player AND Windows Media.


The question is how do you determine if something is relevant... Google has the pagerank(tm) gig that based it's relevancy factors basically off human nature of building web pages. Altavista, Inktomi, etc have based there searches off Metadata (with some interesting comments there from Joi Ito's blog - http://joi.ito.com/archives/2003/09/20/if_i_were_microsoft.html


I might suggest using something like Copernic if you want a client to help you out - it's been around for quite some time and it really quite good at culling out info from vast sources, and not constrained by the "need for a quick response" which limits so many other search mechanisms.

anonymous2
2003-09-20 21:40:24
There's just more content
As content keeps building on the web, it becomes harder to find what you want because there's no 'context' in a search engine. When you want to search for info on a Java class, you want to only search "programmer documentation" for a Java class. But there's no way to tell a search engine that, so a sentence like "this Java is in a class of its own" from an online coffee shop would also pop into the results. While Google and other engines have tweaked their algorithms and implemented little tricks, there really isn't any way that they can "determine" context without the user's help.


It's not so much that search engines are going downhill, it's that they've reached a limit to their usefulness. What is needed now, I think, is a contextual search engine that still has powerful search capabilities, but is focused on a particular topic. In other words, Google should start hosting "foundries" that are a mix between Slashdot and Yahoo, being both a directory, news site and search engine. A sort of "focal point" for a particular community to find information relevant to them.

alexvaldez
2003-09-20 23:55:33
It's not just more content...
There are also those who go to great lengths to artificially boost their Google ranking, which lowers the signal-to-noise ratio of the search results.
anonymous2
2003-09-21 03:27:11
Google searching


Agree absolutely...
Now that Google has partnered with AOL,
the searching quality has diminished.


Several other sites I used 4-5 years ago
are now defunct.


Another factor anyone can list an address
as long as they use simple html coding.
Several IPS and email providers provide
templates, so many teens no have their own
websites, fan websites and many other personal
or personalized sites that are of little interest
or use to the general public..


You can probably locate a number relating
to the incremental increases-then query Google
for a response.


Also dilike their present news formatting
Should supplement is with actual media sites, as
before; but maybe they had to cease at the request
of media outlets.
(Have found a better, more complete source
recently.


MNO

wegrosso
2003-09-21 08:45:01
More details, please
I'm not sure what the AOL deal had to do with Google search quality. Could you explain that a little more?


Also: "Have found a better, more complete source ..." please give a url?

dkurman
2003-09-21 09:08:30
How much do search engines affect the content searched?
Do search engines develop better queries over time?


Is that offset by advertisers/promoters tweaking the content to score better on searches?


It seems to me that resource-type information searches have improved. While searches involving content seeking impressions has stayed the same or lost quality.

anonymous2
2003-09-21 20:54:44
You have to pay now to be listed
One of the reasons why Google is not as good as it used to is that now when you advertise a site, you are not likely to be admitted unles you pay.


This places commercial sites ahead of all others.


Even though individuals may have the better sites.

wegrosso
2003-09-21 21:57:12
You have to pay now to be listed
I'm not sure what you're saying. Are you saying the main google search results are biased towards paying advertisers? Or that the "adwords" on the side are? In any case, this seems to be directly counter to Google's policy (at http://www.google.com/ads/faq.html#increase). Do you have any references for this claim?
jwenting
2003-09-22 02:02:30
You have to pay now to be listed
Yes, most major searchengines will now list paying sites before non-paying sites (and usually list paying sites in decreasing order of their fees).
Altavista conscientiously lists them separate from others in a special block on the beginning of the page, Google I believe does not.
jwenting
2003-09-22 02:08:17
How much do search engines affect the content searched?
Many (if not most) searchengines are in a constant battle with people trying to get their pages on the top of the listing by tricking the searchbots.
They seem to be largely succeeding too, given the reduction in the number of pornsites coming up on unrelated searches 0:-) (those sites used to be infamous for using enormous numbers of hidden text made up of random words that were frequently used by searchengines just to end up on as many search results as possible).


Overall the results produced by searchengines have improved, they are now usually better related to the queries entered.
Problem is that as the amount of information on the internet keeps growing (and a lot of it is unreliable or out of date) searchengines have an ever harder time indexing it all and your search results will be ever longer lists unless you are extremely specific in your search (which may well miss out pages now that were for some reason not indexed for those specific criteria).

anonymous2
2003-09-22 06:45:42
This is false
Check your facts before posting, Google does not do this. While they do display paid advertisement links, they are clearly marked as they always have been.
anonymous2
2003-09-22 08:08:50
funny you should mention...
Results Search
[Advanced Search]


We're sorry, search is temporarily unavailable. Please try again soon.


We're sorry, search is temporarily unavailable. Please try again soon.


No Results Found, Please Try Your Search Again


Search the contents of over 1000 online books on Safari:
O'Reilly Home | Privacy Policy


2003, O'Reilly & Associates, Inc.
webmaster@oreilly.com

wegrosso
2003-09-22 10:03:38
References?
Again, http://www.google.com/ads/faq.html#increase would seem to indicate that Google does not let ads affect search rankings.


Do you have a reference for why you think otherwise?


Also: do you have a link to altavista's policy? I don't see one on their site.

anonymous2
2003-09-22 11:46:32
Signal/Noise is decreasing
I'd say search is getting harder-- & I feel pretty sure it's mainly the noise that's increasing-- 'Noise' meaning 'stuff that's not what I'm looking for'. Part of the problem is that the information is clumped the noise is correlated-- consequently a 'smart' search engine will decide that there's a signal in there, & raise its relevance.
tom_davies
2003-09-22 20:25:04
Google searching
What do you mean by "Should supplement is [sic] with actual media sites, as before"? Each story on the news.google.com has a link to the actual story.
anonymous2
2003-09-23 07:16:55
This is false
Running a couple of searches in Google backed you up -- Google does not appear to list paid links first.
anonymous2
2003-09-23 20:44:17
You do not have to pay to be listed
> Altavista conscientiously lists them separate
> from others in a special block on the beginning
> of the page, Google I believe does not.


Have you actually used Google?


Sponsored links are clearly set aside in a different-colored box that says "Sponsored Link". The formatting is different, the phrasing is different, and did I mention that they're labeled as sponsored (paid) links?


the FUD is unreal. thankfully, all anyone has to do to refute you is to type search into google.


Google DOES NOT accept paid placements within resultsets. If you run a query via the SOAP API for Google (free, by the way), you get the matches from PageRank. No ads, no BS. Just the matches.


Even if competing search engines had anything comparable to the Google web API, I strongly doubt that the resultsets would be free of commercial influence.


Google doesn't sell results. Never has. Never will. (Disclaimer: I work for Google.)

anonymous2
2003-09-28 19:20:17
Yes, it has going downhill
Quality of search engine such as Google has been compromised by the need for revenue from sponsored links.


I used to like Google until I started to search for "lawsuit Bernard Haldane" and I've got more than 30 pages full of wonderful things about this company and links to their site. When I searched for "lawsuit bernard haldane lawsuit", I found links to lawsuits against this company for frauds.


lle