2021 Call for Code Awards: Live from New York, with SNL’s Colin Jost! Learn more

IBM Debater® Multi Word Term Relatedness Benchmark


Measuring word relatedness is an important ingredient of many NLP applications. Several datasets have been developed in order to evaluate such measures. The main drawback of existing datasets is the focus on single words, although natural language contains a large proportion of multiword terms. We propose the new TR9856 dataset which focuses on multi-word terms and is significantly larger than existing datasets. The new dataset includes many real world terms such as acronyms and named entities, and further handles term ambiguity by providing topical context for all term pairs. We report baseline results for common relatedness methods over the new data, and exploit its magnitude to demonstrate that a combination of these methods outperforms each individual method.

Dataset Metadata

Format License Domain Number of Records Size Originally Published
CC-BY-SA 3.0 Natural Language Processing 9,856 labeled pairs of terms
2.6 MB 2014-01-01

Example Records

blasphemy should be criminalized,south park,religious,null,Unrelated,null,null,null,Unrelated,null,null,Unrelated,Unrelated,null,null,Unrelated,Unrelated,null,Unrelated,Unrelated,Unrelated,null,null,null,Unrelated,0
the sale of violent video games to minors should be banned,game,violent video games,null,null,null,null,Related,Related,Related,null,null,Related,null,null,Related,null,Related,null,null,null,Related,Related,Unrelated,Related,0.9
parents should be allowed to genetically screen fetuses for heritable diseases,quad test,presymptomatic testing,null,Related,null,null,null,Unrelated,null,null,Unrelated,Related,null,null,Related,Unrelated,null,Unrelated,Related,Related,null,null,null,Unrelated,0.5


    author    = {Levy, Ran  and  Ein-Dor, Liat  and  Hummel, Shay  and  Rinott, Ruty  and  Slonim, Noam},
    title     = {TR9856: A Multi-word Term Relatedness Benchmark},
    booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
    month     = {July},
    year      = {2015},
    address   = {Beijing, China},
    publisher = {Association for Computational Linguistics},
    pages     = {419--424},
    url       = {http://www.aclweb.org/anthology/P15-2069}
  • IBM Project Debater Project Debater is the first AI system that can debate humans on complex topics. The goal is to help people build persuasive arguments and make well-informed decisions. This dataset contributed to training the models in Project Debater.