Overview

Claims are short phrases that an argument aims to prove. The goal of the Claim Sentence Search task is to detect sentences containing claims in a large corpus, given a debatable topic or motion. The dataset contains results of the q_mc query – sentences containing a certain topic, as described in the paper – containing 1.49M sentences. In addition, the dataset contains a claim sentence test set containing 2.5k top predicted sentences of our model, along with their labels. The sentences were retrieved from Wikipedia 2017.

The dataset includes:
– readme_mc_queries.txt – Readme of the claim sentence search results
– readme_test_set.txt – Readme of the test set
– q_mc_train.csv – Sentences retrieved by the q_mc query on 70 train topics
– q_mc_heldout.csv – Sentences retrieved by the q_mc query on 30 heldout topics
– q_mc_test.csv – Sentences retrieved by the q_mc query on 50 test topics
– test_set.csv – Top predictions of our system along with their labels

The three CSV files, q_mc_train.csv, q_mc_heldout.csv and q_mc_test.csv, contain the following columns for each sentence:
1. id – the topic id, as specified in the appendix of the paper from note (1)
2. topic – the motion topic
3. mc – the main Wikipedia concept of the topic
4. sentence
5. query_pattern – the query pattern that matches the sentence
6. score – the DNN score on the sentence, between 0 and 1
7. label – the gold label of the sentence, 1 for positive and 0 for negative
8. url – link to source Wikipedia article

The CSV file test_set.csv includes the following columns for each sentence:
1. id – the topic id, as specified in the appendix of the paper from note (1)
2. topic – the motion topic
3. mc – the main Wikipedia concept of the topic
4. sentence
5. query_pattern – the query pattern that matches the sentence
6. score – the DNN score on the sentence, between 0 and 1
7. label – the gold label of the sentence, 1 for positive and 0 for negative
8. url – link to source Wikipedia article

Dataset Metadata

Format License Domain Number of Records Size Originally Published
CSV
CC-BY-SA 3.0 Natural Language Processing 1.49M records
571MB August 20, 2018

Example Records

# From the q_mc_heldout.csv file (the q_mc_train and q_mc_test have s similar format):
# id,topic,mc,sentence,suffix,prefix,url

86,Randomized controlled trials bring more harm than good,Randomized controlled trial,"(Smith & Iadarola, 2015) Several recent studies on Floortime were cited in the article including the recent randomized clinical trial studies.", studies.,"(Smith & Iadarola, 2015) Several recent studies on Floortime were cited in the article including the recent ",https://en.wikipedia.org/wiki/Floortime

# From the test_set.csv file:
# id,topic,mc,sentence,query_pattern,score,label,url

136,The American Bar Association brings more harm than good,American Bar Association,"In 1989 the ABA's House of Delegates adopted a resolution stating that "the American Bar Association and each of its entities should use gender-neutral language in all documents establishing policy and procedure."",q_strict,0.951,0,https://en.wikipedia.org/wiki/American_Bar_Association

Citation

@inproceedings{levy-etal-2018-towards,
title = "Towards an argumentative content search engine using weak supervision",
author = "Levy, Ran and
Bogin, Ben and
Gretz, Shai and
Aharonov, Ranit and
Slonim, Noam",
booktitle = "Proceedings of the 27th International Conference on Computational Linguistics",
month = aug,
year = "2018",
address = "Santa Fe, New Mexico, USA",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/C18-1176",
pages = "2066--2081",
abstract = "Searching for sentences containing claims in a large text corpus is a key component in developing an argumentative content search engine. Previous works focused on detecting claims in a small set of documents or within documents enriched with argumentative content. However, pinpointing relevant claims in massive unstructured corpora, received little attention. A step in this direction was taken in (Levy et al. 2017), where the authors suggested using a weak signal to develop a relatively strict query for claim{--}sentence detection. Here, we leverage this work to define weak signals for training DNNs to obtain significantly greater performance. This approach allows to relax the query and increase the potential coverage. Our results clearly indicate that the system is able to successfully generalize from the weak signal, outperforming previously reported results in terms of both precision and coverage. Finally, we adapt our system to solve a recent argument mining task of identifying argumentative sentences in Web texts retrieved from heterogeneous sources, and obtain F1 scores comparable to the supervised baseline.",
}
  • Project Debater Project Debater is the first AI system that can debate humans on complex topics. The goal is to help people build persuasive arguments and make well-informed decisions. This dataset contributed to training the models in Project Debater.