Common Crawl

Tutorials

• Analysing Petabytes of Websites by Mark Litwintschik
  Amazon EMR
• Common Crawl Index Athena by Edward Ross
  Amazon Athena
• Index to WARC Files and URLs in Columnar Format by Sebastian Nagel
  Amazon Athena
• Large-scale graph mining with Spark by Win Suen
  
• One click to download all the web pages you may want by Jader Dias
  Amazon AthenaAWS Lambda
• Search the Common Crawl Using Lambda Functions by Andres Riancho
  AWS Lambda

Tools & Applications

• All Around The World: The Common Crawl Dataset - Attack Surface Research by Aliz Hammond
  
• CCNet: Extracting high quality monolingual datasets from web crawl data by Facebook AI Research
  
• Dresden Web Table Corpus (DWTC) by Database Systems Group Dresden
  
• Glove: Global vectors for word representation by Jeffrey Pennington, Richard Socher, Christopher D. Manning
  
• Learning word vectors for 157 languages by Facebook AI Research
  
• Ransacking your password reset tokens by Lukas Euler
  
• Search the html across 25 billion websites for passive reconnaissance using common crawl by Ryan Elkins
  

Publications

• Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures by Pedro Javier Ortiz Suárez, Benoît Sagot, Laurent Romary
  
• Building a Web-Scale Dependency-Parsed Corpus from CommonCrawl by Alexander Panchenko, Eugen Ruppert, Stefano Faralli, Simone Paolo Ponzetto, Chris Biemann
  
• C4Corpus: Multilingual Web-Size Corpus with Free License by Ivan Habernal, Omnia Zayed, Iryna Gurevych
  
• CC-News-En: A large English news corpus by Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R. Trippas, J. Shane Culpepper, Alistair Moffat
  
• CCAligned: A Massive collection of cross-lingual web-document pairs by Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, Philipp Koehn
  
• Coyo-700m: Image-text pair dataset by Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, Saehoon Kim
  
• Defending against neural fake news by Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, et al
  
• Index fun by Philippe Suter
  
• LAION-5B: An open large-scale dataset for training next generation image-text models by Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, et al
  
• Language is not all you need: aligning perception with language models by Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, et al
  
• Language models are few-shot learners by Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, et al
  
• Large-scale analysis of style injection by relative path overwrite by Sajjad Arshad, Seyed Ali Mirheidari, Tobias Lauinger, Bruno Crispo, Engin Kirda, William Robertson
  
• LLaMA: open and efficient foundation language models by Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, et al
  
• Mapping languages: The Corpus of Global Language Use by Jonathan Dunn
  
• mT5: A massively multilingual pre-trained text-to-text transformer by Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, et al
  
• Multimodal C4: an open, billion-scale corpus of images interleaved with text by Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, et al
  
• N-gram counts and language models from the Common Crawl by Christian Buck, Kenneth Heafield, Bas van Ooyen
  
• No Language Left Behind: scaling human-centered machine translation by Costa-jussà, Marta R., James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, et al
  
• Of using Common Crawl to play Family Feud by Paul Masurel
  
• On the impact of publicly available news and information transfer to financial markets by Metod Jazbec, Barna Pásztor, Felix Faltings, Nino Antulov-Fantulin, Petter N. Kolm
  
• Using open data to predict market movements by DELL EMC
  
• Web Data Commons - RDFa, microdata, and microformat data sets by Christian Bizer, Robert Meusel, Anna Primpeli
  

Sudachi Language Resources

Tutorials

• analysis-sudachi Tutorial by Works Applications
  
• chiTra Tutorial by Works Applications
  
• chiVe Tutorial by Works Applications
  
• Sudachi Tutorial by Works Applications
  
• SudachiPy Tutorial by Works Applications
  

Tools & Applications

• analysis-sudachi: Sudachi pluglin for Elasticsearch by Works Applications
  
• chiTra: SudachiPy for hugging face Transformers by Works Applications
  
• jdartsclone: TRIE Data Structure using Double-Array by Works Applications
  
• Kintoki: Dependency Parser by Works Applications
  
• Sudachi: Japanese Tokenizer for Business by Works Applications
  
• sudachidict_core on pypi.python.org - a Python module to download and install SudachiDict for the python tokenizer by Works Applications
  
• sudachidict_full on pypi.python.org - a Python module to download and install SudachiDict for the python tokenizer by Works Applications
  
• sudachidict_small on pypi.python.org - a Python module to download and install SudachiDict for the python tokenizer by Works Applications
  
• SudachiPy: Python version of Sudachi by Works Applications
  

Publications

• chiVe 2.0: SudachiとNWJCを用いた実用的な日本語単語ベクトルの実現に向けて by 河村宗一郎, 久本空海, 真鍋陽俊, 髙岡一馬, 内田佳孝, 岡照晃, 浅原正幸
  
• chiVe: 製品利用可能な日本語単語ベクトル資源の実現へ向けて ～形態素解析器Sudachiと超大規模ウェブコーパスNWJCによる分散表現の獲得と改良～ by 久本空海, 山村崇, 勝田哲弘, 竹林佑斗, 髙岡一馬, 内田佳孝, 岡照晃, 浅原正幸
  
• Sudachi: a Japanese Tokenizer for Business by Kazuma Takaoka, Sorami Hisamoto, Noriko Kawahara, Miho Sakamoto, Yoshitaka Uchida, Yuji Matsumoto
  
• 形態素解析器『Sudachi』のための大規模辞書開発 by 坂本美保, 川原典子, 久本空海, 髙岡一馬, 内田佳孝
  
• 複数粒度の分割結果に基づく日本語単語分散表現 by 真鍋陽俊, 岡照晃, 海川祥毅, 髙岡一馬, 内田佳孝, 浅原正幸
  
• 詳細化した同義関係をもつ同義語辞書の作成 by 高岡一馬, 岡部裕子, 川原典子, 坂本美保, 内田佳孝
  

Synthea synthetic patient generator data in OMOP Common Data Model

Tutorials

• Create data science environments on AWS for health analysis using OHDSI by James Wiggins
  AWS CloudFormationAmazon EC2Amazon RDS
• Map clinical notes to the OMOP Common Data Model and healthcare ontologies using Amazon Comprehend Medical by James Wiggins
  Amazon ComprehendAWS CloudFormationAmazon EC2Amazon RDS
• Predict patient health outcomes using OHDSI and machine learning on AWS by James Wiggins
  AWS CloudFormationAmazon EC2Amazon RDS

Tools & Applications

• OHDSIonAWS by James Wiggins
  

Japanese Tokenizer Dictionaries

Tutorials

• Fugashi Word Count Tutorial by Paul O'Leary McCann
  Amazon SageMaker

Tools & Applications

• unidic-py by Paul O'Leary McCann
  

Publications

• How to Tokenize Japanese in Python by Paul O'Leary McCann
  

MIMIC-III (‘Medical Information Mart for Intensive Care’)

Tutorials

• Building predictive disease models using Amazon SageMaker with Amazon HealthLake normalized data by Ujjwal Ratan, Nihir Chadderwala, and Parminder Bhatia
  AWS GlueAWS HealthLakeAmazon SageMaker
• Perform biomedical informatics without a database using MIMIC-III data and Amazon Athena by James Wiggins, Alistair Johnson
  Amazon AthenaAWS Glue

Tools & Applications

• MIMIC-code GitHub repository by Alistair Johnson
  

REDASA COVID-19 Open Data

Tools & Applications

• Curadr - Curation Platform by REDASA Consortium, Imperial College London
  

Publications

• Using a Secure, Continually Updating, Web Source Processing Pipeline to Support the Real-Time Data Synthesis and Analysis of Scientific Literature: Development and Validation Study by Uddhav Vaghela, Simon Rabinowicz, Paris Bratsos, Guy Martin, Epameinondas Fritzilas, et al.
  

CMS 2008-2010 Data Entrepreneurs’ Synthetic Public Use File (DE-SynPUF) in OMOP Common Data Model

Tutorials

• Create data science environments on AWS for health analysis using OHDSI by James Wiggins
  AWS CloudFormationAmazon EC2Amazon RDS
• Map clinical notes to the OMOP Common Data Model and healthcare ontologies using Amazon Comprehend Medical by James Wiggins
  Amazon ComprehendAWS CloudFormationAmazon EC2Amazon RDS
• Predict patient health outcomes using OHDSI and machine learning on AWS by James Wiggins
  AWS CloudFormationAmazon EC2Amazon RDS

Tools & Applications

• OHDSIonAWS by James Wiggins
  

Common Screens

Tutorials

• IAB Text Classification by Common Screens
  

Discrete Reasoning Over the content of Paragraphs (DROP)

Publications

• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs by Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner
  

End of Term Web Archive Dataset

Publications

• Moving the End of Term Web Archive to the Cloud to Encourage Research Use and Reuse by Mark Phillips and Sawood Alam
  

MultiCoNER Datasets

Publications

• Dynamic Gazetteer Integration in Multilingual Models for Cross-Lingual and Cross-Domain Named Entity Recognition by Besnik Fetahu, Anjie Fang, Oleg Rokhlenko and Shervin Malmasi
  
• Gazetteer Enhanced Named Entity Recognition for Code-Mixed Web Queries by Besnik Fetahu, Anjie Fang, Oleg Rokhlenko and Shervin Malmasi
  
• GEMNET: Effective Gated Gazetteer Representations for Recognizing Complex Entities in Low-context Input by Tao Meng, Anjie Fang, Oleg Rokhlenko and Shervin Malmasi
  
• MultiCoNER: A Large-scale Multilingual Dataset for Complex Named Entity Recognition by Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, Oleg Rokhlenko
  

Quoref

Publications

• Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning by Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, Matt Gardner
  

Reasoning Over Paragraph Effects in Situations (ROPES)

Publications

• Reasoning Over Paragraph Effects in Situations by Kevin Lin, Oyvind Tafjord, Peter Clark, Matt Gardner
  

Gretel Synthetic Safety Alignment Dataset

Tutorials

• Building Datasets to Enable Safer AI Responses by Gretel.ai
  
• Getting Started with the Gretel Safety Alignment Dataset by Gretel.ai
  

Tools & Applications

• Gretel Navigator Data Designer by Gretel.ai
  

ABEJA CC JA

Tutorials

• Tutorial of ABEJA CC JA dataset by Kyo Hattori
  

Publications

• Building a Large-Scale Japanese Corpus from Common Crawl and Its Preprocessing by Kyo Hattori
  

Amazon-PQA

Publications

• Answering Product-Questions by Utilizing Questions from Other Contextually Similar Products by Ohad Rozen, David Carmel, Avihai Mejer, Vitaly Mirkis, and Yftah Ziser
  

Answer Reformulation

Publications

• Voice-based Reformulation of Community Answers by Simone Filice, Nachshon Cohen & David Carmel
  

Automatic Speech Recognition (ASR) Error Robustness

Publications

• Using Phoneme Representations to Build Predictive Models Robust to ASR Errors by Anjie Fang, Simone Filice, Nut Limsopatham and Oleg Rokhlenko
  

DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue

Publications

• DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue by Shikib Mehri, Mihail Eric, Dilek Hakkani-Tur
  

Enriched Topical-Chat Dataset for Knowledge-Grounded Dialogue Systems

Publications

• Policy-Driven Neural Response Generation for Knowledge-Grounded Dialogue Systems by Behnam Hedayatnia, Karthik Gopalakrishnan, Seokhwan Kim, Yang Liu, Mihail Eric & Dilek Hakkani-Tur
  

Helpful Sentences from Reviews

Publications

• Identifying Helpful Sentences in Product Reviews by Iftah Gamzu et al (2021)
  

Humor Detection from Product Question Answering Systems

Publications

• Humor Detection in Product Question Answering Systems. by Yftah Ziser, Elad Kravi & David Carmel
  

Humor patterns used for querying Alexa traffic

Publications

• “Alexa, Do You Want to Build a Snowman?” Characterizing Playful Requests to Conversational Agents by Shani C., Libov A., Tolmach S., Lewin-Eytan L., Maarek Y., and Shahaf D.
  

Learning to Rank and Filter - community question answering

Publications

• IR Evaluation and Learning in the Presence of Forbidden Documents by David Carmel, Nachshon Cohen, Amir Ingber & Elad Kravi
  

Low Context Name Entity Recognition (NER) Datasets with Gazetteer

Publications

• GEMNET: Effective Gated Gazetteer Representations for Recognizing Complex Entities in Low-context Input by Tao Meng, Anjie Fang, Oleg Rokhlenko and Shervin Malmasi
  

Multi Token Completion

Publications

• Simple and Effective Multi-Token Completion from Masked Language Models by Oren Kalinsky, Guy Kushilevitz, Alex Libov & Yoav Goldberg
  

Multilingual Name Entity Recognition (NER) Datasets with Gazetteer

Publications

• Gazetteer Enhanced Named Entity Recognition for Code-Mixed Web Queries by Besnik Fetahu, Anjie Fang, Oleg Rokhlenko and Shervin Malmasi
  

PASS: Perturb-and-Select Summarizer for Product Reviews

Publications

• PASS: Perturb-and-Select Summarizer for Product Reviews by Nadav Oved and Ran Levy (2021)
  

Phrase Clustering Dataset (PCD)

Publications

• McPhraSy: Multi context phrase similarity and clustering by Amir DN Cohen, Hila Gonen, Ori Shapira, Ran Levy, and Yoav Goldberg
  

Pre- and post-purchase product questions

Publications

• "Did you buy it already?", Detecting Users Purchase-State From Their Product-Related Questions by Lital Kuchy, David Carmel, Thomas Huet & Elad Kravi
  

Product Comparison Dataset for Online Shopping

Publications

• Generating Explainable Product Comparisons for Online Shopping by Nikhita Vedula, Marcus Collins, Eugene Agichtein and Oleg Rokhlenko
  

Shopping Humor Generation

Publications

• Evaluating Humorous Response Generation to Playful Shopping Requests by Natalie Shapira, Oren Kalinsky, Alex Libov, Chen Shani, Sofia Tolmach
  

VoiSeR

Publications

• VoiSeR: A New Benchmark for Voice-Based Search Refinement by Simone Filice, Giuseppe Castellucci, Marcus Collins, Eugene Agichtein & Oleg Rokhlenko
  

WikiSum: Coherent Summarization Dataset for Efficient Human-Evaluation

Publications

• WikiSum: Coherent Summarization Dataset for Efficient Human-Evaluation by Nachshon Cohen, Oren Kalinsky, Yftah Ziser & Alessandro Moschitti
  

Wizard of Tasks

Publications

• Wizard of Tasks: A Novel Conversational Dataset for Solving Real-World Tasks in Conversational Settings by Jason Ingyu Choi, Saar Kuzi, Nikhita Vedula, Jie Zhao, Giuseppe Castellucci, Marcus Collins, Shervin Malmasi, Oleg Rokhlenko and Eugene Agichtein