"Computer: Analyse the distribution of the pieces that we have, correcting for changes in star configurations over four billion years, then extrapolate for the missing piece" (Star Trek, The Chase)
News
1/7/2018: We have discovered an issue with the partitioning of the dataset and are releasing ComplexWebQuestions version 1.1 with a new partitioning of the dataset. The Question file format remains the same, except that we added an additional field with extra supervision describing the answer of a decomposed question. Please read our report
A dataset for answering complex questions that require reasoning over multiple web snippets.
ComplexWebQuestions is a new dataset that contains a large set of complex questions in natural language, and can be used in multiple ways:
-
By interacting with a search engine, which is the focus of our paper (Talmor and Berant, 2018);
-
As a reading comprehension task: we release 12,725,989 web snippets that are relevant for the questions, and were collected during the development of our model;
-
As a semantic parsing task: each question is paired with a SPARQL query that can be executed against Freebase to retrieve the answer.
Question: The actress that had the role of Martha Alston, plays what role in Finding Nemo?
Answer: "Dory"
Title: "Ellen DeGeneres - Wikipedia"
Web Snippet: "..She also played Martha Alston in the 1996 Touchstone Pictures film Mr. Wrong and.. "
Title: "Ellen DeGeneres | Disney Wiki | FANDOM powered by Wikia"
Web Snippet: "... provided the voice of Dory in Disney/Pixar's 2003 animated film, Finding Nemo. ..."
Leaderboard
The dataset contains 34,689 examples, each containing:
-
A complex question
-
Answers (including aliases)
-
An average of 366.8 snippets per question
-
A SPARQL query (against Freebase)
Freebase version used is freebase-rdf-2015-08-02-00-00
Sample Questions
-
“Which school that Sir Ernest Rutherford attended has the latest founding date?”
-
“what movies does Leo Howard play in and that is 113.0 minutes long?"
-
“Where is the end of the river that originates in Shannon Pot?”
-
Test results may be submitted every two weeks, and scores will be computed using the official Eval Script
-
Please send predictions in JSON format (according to the Eval Script) to alontalmor@mail.tau.ac.il
-
If you find this data helpful in your work, please cite this paper:
@inproceedings{talmor18compwebq,
author = {A. Talmor and J. Berant},
booktitle = {North American Association for Computational Linguistics (NAACL)},
title = {The Web as a Knowledge-base for Answering Complex Questions},
year = {2018},
}