We get pretty fed up with the quality of results produced at times by web search engines. It was one of the motivations behind us writing the Foboz - Meta Search Engine.

We have therefore decided to take an occasional benchmark of the results produced by the major web search engines. Looking at two criteria.

• Relevence - Do the results reflect what you are looking for.
• Legitimacy  - were the search engines themselves spammed into producing this result by aggressive placement strategies or by downright illegititmate strategies such as hidden keyword placement.

We will look at the top 10 results produced by each engine on the fairly specific search text. This text will change for each report we do, and we plan to produce about one report a month. The text will be chosen on the basis that a user might have some legitimate expectation of good results. For example:

"download meta search engine program"

Search engines will be ranked on a scale of 0 - 10. Based on the sum of each of its top 10 results graded on a scale of 0-1.

Each results grade will be (R x L)/4.

These measures are not integral. e.g. a result may be 0.75. Placement value for a site will generally be 2.0, we cannot penalise sites that make good use of meta tags to help their ranking. The "L" rating begins to slip when this gets beyond the pale, with for example page titles designed for the search engine rather than the human reader.