An evolutionary approach for combining different sources of evidence in search engines

Abstract

Modern Web search engines use different strategies to improve the overall quality of their document rankings. Usually the strategy adopted involves the combination of multiple sources of relevance into a single ranking. This work proposes the use of evolutionary techniques to derive good evidence combination functions using three different sources of evidence of relevance: the textual content of documents, the reputation of documents extracted from the connectivity information available in the processed collection and the anchor text concatenation. The combination functions discovered by our evolutionary strategies were tested using a collection containing 368 queries extracted from a real nation-wide search engine query log with over 12 million documents. The experiments performed indicate that our proposal is an effective and practical alternative for combining sources of evidence into a single ranking. We also show that different types of queries submitted to a search engine can require different combination functions and that our proposal is useful for coping with such differences.

Introduction

Modern search engines usually use several sources of evidence to compute the ranking of documents that satisfy a user query. Examples of such sources of evidence are anchor text concatenation, the URL of each page, page title, the link structure available on the Web, geographic information about the user or about the Web pages, user profiles, etc. The final result produced by each search engine can be seen as a combination of partial results where each source of relevance evidence produces an individual ranking for each query. This requires a combination of all sources of evidence to provide a single final ranking.

Previous work in the literature present simple generic solutions to this problem, such as computing a linear combination of pieces of evidence [1], or modeling each source of evidence as an independent relevance probability [2], [3]. Such solutions have the disadvantage of missing important detailed information about each evidence that might affect the overall system quality. Examples of properties which are difficult to generalize are dependencies among different sources of evidence (e.g., dependencies among textual and link evidence based on query terms), or discrepancy in the values (or in their distribution) provided by each individual evidence.

Another alternative approach is to use learning methods for combining sources of relevance evidence [4]. Following this strategy, we propose the use of genetic programming (GP) as a tuning approach to solve the problem of combining different sources of evidence into a single ranking function. GP has been proved to be a successful framework for discovering ranking functions in information retrieval systems [5]. By using GP to find evidence combination functions, we aim not only at providing a good solution to combine different sources of evidence, but also at generating valuable information that allows a detailed analysis of the usefulness of each individual piece of evidence to different types of query submitted to a search engine.

In order to evaluate the proposed approach we have performed experiments with a Web collection and a real search engine query log using three different sources of evidence explored in the literature. The first is the result of the well-known vector space model applied to the full text of each document. The second also uses the vector space model, however, it is applied to the concatenation of all anchor text referring to a document in the collection. The third is the well-known Pagerank [6] of the Web pages, which is computed by the analysis of the Web link structure.

Our experiments were conducted taking into consideration two distinct types of queries as described in [7]: navigational and informational queries. These two query types were further divided into sets of popular and non-popular queries, thus producing four distinct query sets. The experiments confirmed that each of these sets is affected in a different way by the three pieces of evidence studied. Therefore, distinct ranking functions are required to combine the studied pieces of evidence according to each query type. The performed experiments indicate that our approach achieves a good final ranking based on the combination of pieces of evidence. In addition it is also useful for studying the importance of each piece of evidence to the final ranking.

This paper is organized as follows. In Section 2 we discuss related work. Section 3 presents a brief overview of GP. Section 4 describes how we use GP to generate ranking functions that combine distinct sources of evidence. Section 5 discusses the methodology we use to conduct our experimentation. Section 6 presents the results obtained by our evolutionary approach confronted with the best results obtained with each individual evidence used in isolation and with other methods. Finally, Section 7 presents final remarks, with directions to future work.