In any website, search is an important user interface element. Research shows that around 10% of users will elect to use search as their primary means of locating content within a website. However, most users only turn to search when they have been unable to locate content via a site’s navigation structure. In addition, when it comes to search, Google sets users' expectations to a high standard.

So, how do we evaluate the performance of a site search from a usability perspective? Well, this question can be addressed from two separate angles. From a classic usability evaluation perspective we can compare how the site search performs against search best practice and observe users searching the site. Alternatively, we can look at Key Performance Indicators (or KPIs) to monitor how the site search is performing over time.

We’ll start by briefly covering how to evaluate the usability of a site search, before looking in more detail into KPIs for search, as these are rather similar across websites.

To evaluate the usability of a site search, UsabilityOne developed a 66-point checklist of known best search practice. This enables us to present a set of recommendations based on an expert review of the site. To provide further insight we perform a set of one-on-one user tests - shedding light on how real users experience the search.

KPIs assist in benchmarking the effectiveness of search and provide a way to measure improvements resulting from future changes. While most KPIs rely on quantitative data extracted from analytical tools (such as Google Analytics), some qualitative measures also need to be developed in order to understand the relevance of search results.

The first obvious KPI is to look at the search terms used on the site. A clear understanding of these terms is important to update the site content and support the user. As well as closely monitoring search terms used, below are six KPIs that rest on quantitative data and can be measured using most analytical tools:

1. Average searches per visit: this KPI helps examine visitor behaviour in respect to search and determining the frequency visitors use search.
2. Percent of visitors using search: tracking the percentage of visitors who use the search tool helps monitor any change in visitors’ understanding and expectations of the site.
3. Percent of zero “results searches”: nothing is less satisfying for users than entering a search term and getting a “sorry, no results” error back. Monitoring closely this KPI helps better support users’ expectations.
4. Percent of zero yield search & 5-Results Page views per search: serving up search results is good but only if visitors are actually clicking on links and finding useful information. Tracking search yield and results page views can help understand how likely visitors see a result they believe to be compelling enough to click.
5. Search result to site exits ratio: similar to percent zero yield search results, this KPI helps understand whether visitors find the internal search engine useful.
6. Percent of search refinements: this metric presents the percentage of searches that resulted in another search (i.e. a keyword refinement).

The above KPIs may be measured automatically and inspected regularly to note trends and improvements in the site search, but they do not measure how relevant the search results are. To do this, here is a manual way – that although time consuming, is well worth the effort. It provides a benchmark that the search engine's results can be measured against over time. The process is outlined below:

Step 1: Set a predefined sample of queries. Research has shown that 50 is a good set to work with.

Step 2: Run the search on the sample set.

Step 3: Inspect the results for their relevance. Relevance levels can be binary - simply asking if a result is relevant or not, or graded - indicating results with a varying degree of match between the topic of the result and the information need.

When changing and improving the search engine parameters or the site content, running this check again on the same initial sample will provide a clear indication on improvement achieved.

UsabilityOne has extensive experience, not only in evaluating search functionalities across a wide range of websites, but also in helping its clients gather user requirements. This all helps design a user experience which possesses optimum search functionality.

References

The UsabilityOne 66-point checklist of known best search practice and the KPIs presented above are extracted from UsabilityOne's expertise and also from world experts in that domain. Key references are:

• Avinash Kaushik's books and blog (http://www.kaushik.net/) - accessed 27 Oct 2010
• The Big Book of Key Performance Indicators, by Eric Peterson (http://www.webanalyticsdemystified.com/downloads/The_Big_Book_of_Key_Performance_Indicators_by_Eric_Peterson.pdf ) - accessed 27 Oct 2010
• The KPI library (http://kpilibrary.com/) - accessed 27 Oct 2010