Search Engines That Work

Last updated Thursday, November 20, 2008.

Are you finding all the sites you search for? Some search engines actively prevent you from finding all the web pages you want to see.

Mind you, this is not about pornography, nor even about censorship. Clean-cut, wholesome sites are being blocked. Certain defective search engines are randomly ignoring all kinds of web pages.

These search engines work well:

These search engines could work better:

These search engines fail:

UPDATE: In 2008, the search engines listed in red (all combined) accounted for less than one percent of all traffic to this website.
So, if you're a search engine programmer, you'd be well advised to read the following.

Criteria:

A grade no higher than "could work better" will be given to search engines with any of these problems:

• a recent history (anytime in the last 1-2 years) of unreliable performance.
• Mission creep and frequent redesigns affect scoring, since these create a "permanent learning curve," i.e., they cause customers to have to re-learn the interface frequently.
• Over-reliance on graphics or HTML tables or JavaScript (a language which unfortunately will never be stable), causing long browser rendering times. This promotes the "Digital Divide" by encouraging (or forcing) users of older computers to go elsewhere. (Excite)
• Whole-phrase search (quoted string search) not supported or just doesn't work.

To receive a failing grade, a search engine company must have any of these professionalism problems:

• False Promises. When you submit your page, the engine responds with a promise that it will index (or "process") your page within a stated time period. But it does not. This non-responsiveness is the most common reason why search engines receive a failing grade.
• Excessive Limits. The company openly and proudly admits it has a policy of discriminating against large domains like AOL and GeoCities. For example, Infoseek and HotBot once accepted only 50 new URL's (per day!) from the major ISP domains -- simply not appropriate for domains which have millions of members. HotBot seems to have improved.
• You Can't Get Indexed. Lycos falls into this category. I gave up trying in April 2K. As of November 2001, Lycos accepts new URLs with the promise "Once your site has been spidered, it will be entered into our catalog within 2 to 3 weeks," but it has yet to enter any of my pages.
• Weird Professionalism. Certain web directories have a paid submission process, whereby if you pay a huge fee up front, you get a guarantee that an editor will review (although not necessarily accept) your listing. The offer blatantly denies you any right to a refund if your listing is refused, and allows you no say in how your listing will appear. The Yellow Pages couldn't have survived doing business like that. Yet Yahoo has now embraced this business model. So has AltaVista.
• Instability; Spontaneous De-Indexing. Certain engines drop valid URLs from their indexes. No one seems to know why, but maybe this happens because the URL's host machine is down or overloaded when the search engine makes its periodic visits to confirm the URL. Your web site's listing can be deleted in a single night -- and it may take weeks to get indexed again. Random de-indexing, combined with the difficulty of getting re-indexed, makes some search engines next to useless. Google once had this deficiency, years ago. And as of March 2004, AltaVista apparently no longer exists as its own search technology; in yet another example of never-ending mission creep, AltaVista is now powered by Yahoo, and when the change-over happened, AltaVista spontaneously de-indexed nearly all of my web pages.
• Software Malfunction. Suppose you type your search keywords and press "Submit." Suppose the search engine tells you there are 600 matches, and shows you the first ten. Suppose the search engine provides a button labeled "click here to see the next 10 matches." Suppose you click it. Suppose the search engine then says "no more matches." Recently I saw AOL.com Search doing this. I reported the problem to them on Oct. 24, 2001; I never heard back, so I never went back.
• Politics. One particular search engine (the "Open Directory") claims "Humans Do It Better!" -- but by hiring thousands of unsupervised volunteer editors, the company has unwittingly created a nasty side effect: Each category is ruled by a feudal warlord, who acts arbitrarily and without mercy. If you operate a web site, expect your listing to be frequently moved from one category to another, without notice. Expect some editors to carelessly rewrite your listing, in ways that show they did nothing more than quickly scan only your top-level home page. Expect some editors to embellish your listing with inaccuracies, misspellings, and poor grammar. Some will even presume to edit the very name of your site or business! When you complain, be prepared for a retaliatory deletion. "Humans do it better"? Humans are difficult for the sake of being difficult.

Search engines are supposed to bring web pages and users together. But if your favorite search engine is ignoring large parts of the Web, then chances are it won't help you find the information you're seeking. Tired of not finding what you're looking for? Try the "good" search engines listed above.

Concepts: search engine, search engines, rating, ratings, rated, ranking, rank, de-index, de-indexed, how to index a web page, spider, 
altavista, aol search, aol search problems, aol search complaints, dmoz, dmoz problems, dmoz complaints, excite, go.com, google, hotbot problems, hotbot complaints, reviews of hotbot, infoseek, lycos, netfind, northern light, open directory problems, open directory complaints, reviews of the open directory, webcrawler, yahoo 
	

Further Reading at Other Sites

• Search Engine Watch - detailed ratings (at www.searchenginewatch.com)

• Home