Fixed-cost Pooling Strategies based on IR Evaluation Measures

Type: 
Proceedings contribution
Proceedings: 
ECIR 2017: Advances in Information Retrieval
Publisher: 
Springer, Cham
Pages: 
357 - 368
ISBN: 
ISBN: 978-3-319-56608-5
Year: 
2017
Abstract: 
Recent studies have reconsidered the way we operationalise the pooling method, by considering the practical limitations often encountered by test collection builders. The biggest constraint is often the budget available for relevance assessments and the question is how best - in terms of the lowest pool bias - to select the documents to be assessed given a fixed budget. Here, we explore a series of 3 new pooling strategies introduced in this paper against 3 existing ones and a baseline. We show that there are significant differences depending on the evaluation measure ultimately used to assess the runs. We conclude that adaptive strategies are always best, but in their absence, for top-heavy evaluation measures we can continue to use the baseline, while for P@100 we should use any of the other non-adaptive strategies.
TU Focus: 
Computational Science and Engineering
Reference: 

A. Lipani, J. Palotti, M. Lupu, F. Piroi, G. Zuccon, A. Hanbury:
"Fixed-cost Pooling Strategies based on IR Evaluation Measures";
in: "ECIR 2017: Advances in Information Retrieval", herausgegeben von: Springer; Springer, Cham, Cham, 2017, ISBN: 978-3-319-56608-5, S. 357 - 368.

Zusätzliche Informationen

Last changed: 
15.12.2017 16:50:35
TU Id: 
264549
Accepted: 
Accepted
Invited: 
Department Focus: 
Author List: 
A. Lipani, J. Palotti, M. Lupu, F. Piroi, G. Zuccon, A. Hanbury
Abstract German: