Fine-Grained Relevance Annotations for Multi-Task Document Ranking and Question Answering

Authors: 
Sebastian Hofstätter
Markus Zlabinger
Mete Sertkan
Michael Schröder
Type: 
Speech with proceedings
Proceedings: 
Proceedings of the 29th ACM International Conference on Information & Knowledge Management
Publisher: 
Association for Computing Machinery
Pages: 
3031 - 3038
ISBN: 
ISBN: 9781450368599
Year: 
2020
Abstract: 
There are many existing retrieval and question answering datasets. However, most of them either focus on ranked list evaluation or single-candidate question answering. This divide makes it challenging to properly evaluate approaches concerned with ranking documents and providing snippets or answers for a given query. In this work, we present FiRA: a novel dataset of Fine-Grained Relevance Annotations. We extend the ranked retrieval annotations of the Deep Learning track of TREC 2019 with passage and word level graded relevance annotations for all relevant documents. We use our newly created data to study the distribution of relevance in long documents, as well as the attention of annotators to specific positions of the text. As an example, we evaluate the recently introduced TKL document ranking model. We find that although TKL exhibits state-of-the-art retrieval results for long documents, it misses many relevant passages.
TU Focus: 
Information and Communication Technology
Reference: 

S. Hofstätter, M. Zlabinger, M. Sertkan, M. Schröder, A. Hanbury:
"Fine-Grained Relevance Annotations for Multi-Task Document Ranking and Question Answering";
Vortrag: CIKM 2020: International Conference on Information & Knowledge Management 2020, Virtual Event, Ireland; 03.12.2020 - 05.12.2020; in: "Proceedings of the 29th ACM International Conference on Information & Knowledge Management", Association for Computing Machinery, (2020), ISBN: 9781450368599; S. 3031 - 3038.

Zusätzliche Informationen

Last changed: 
27.11.2020 15:30:05
TU Id: 
291499
Accepted: 
Accepted
Invited: 
Department Focus: 
Business Informatics
Abstract German: 
Author List: 
S. Hofstätter, M. Zlabinger, M. Sertkan, M. Schröder, A. Hanbury