Datasets for English Named Entity Recognition

Datasets for English Named Entity Recognition

Annotated Corpus for Named Entity Recognition: Corpus for entity classification with enhanced and popular features by Natural Language Processing applied to the data set.

i2b2 Challenges: By the Informatics for Integrating Biology & the Bedside (i2b2) center, these clinical datasets were created for named entity recognition.

CoNLL 2003: Dataset that contains 1,393 English news articles with annotated entities LOC (location), ORG (organization), PER (person) and MISC (miscellaneous).

NLPBA 2004: Medical data tagged with protein/DNA/RNA/cell line/cell type (2,404 MEDLINE abstracts).

Resume Entities for NER: Document annotation dataset to be used to perform NER on resumes from indeed.com.

Enron Emails: Over 500,000 email messages tagged with names, dates and times.

MIT Movie Corpus: A semantically tagged training and test corpus in BIO format. The eng corpus are simple queries, and the trivia10k13 corpus are more complex queries.

Annotated GMB Corpus: An annotated corpus using GMB (Groningen Meaning Bank) corpus for entity classification with enhanced and popular features by Natural Language Processing applied to the data set.

Best Buy E-Commerce NER Dataset: A dataset containing Best Buy search queries labeled with entities such as Brand, Model name, Category Name, and etc.

WNUT 17 Emerging Entities Dataset: Text from YouTube, Stack Overflow, Twitter and Reddit comments filtered to prefer text that is likely to contain named entities.

Finnish NER dataset

https://github.com/mpsilfve/finer-data

The directory data contains a corpus of Finnish technology related news articles with a manually prepared named entity annotation (digitoday.2014.csv). The text material was extracted from the archives of Digitoday, a Finnish online technology news source (www.digitoday.fi). The corpus consists of 953 articles (193,742 word tokens) with six named entity classes (organization, location, person, product, event, and date). The corpus is available for research purposes and can be readily used for development of NER systems for Finnish. The corpus is described in the article

"A Finnish News Corpus for Named Entity Recognition" (in review)

English NER dataset

Dataset	Domain	License	Reference	Availablility
CONLL 2003	News	DUA	Sang and Meulder, 2003	Easy to find
NIST-IEER	News	None	NIST 1999 IE-ER	NLTK data
MUC-6	News	LDC	Grishman and Sundheim, 1996	LDC 2003T13
OntoNotes 5	Various	LDC	Weischedel et al., 2013	LDC 2013T19
BBN	Various	LDC	Weischedel and Brunstein, 2005	LDC 2005T33
GMB-1.0.0	Various	None	Bos et al., 2017	http://gmb.let.rug.nl/data.php
GUM-3.1.0	Wiki	Several (*2)	Zeldes, 2016	✔ Included here
wikigold	Wikipedia	CC-BY 4.0	Balasuriya et al., 2009	✔ Included here
Ritter	Twitter	None	Ritter et al., 2011	No split , Train/test/dev split
BTC	Twitter	CC-BY 4.0	Derczynski et al., 2016	✔ Included here
WNUT17	Social media	CC-BY 4.0	Derczynski et al., 2017	✔ Included here
i2b2-2006	Medical	DUA	Uzuner et al., 2007	http://www.i2b2.org
i2b2-2014	Medical	DUA	Stubbs et al., 2015	http://www.i2b2.org
CADEC	Medical	CSIRO	Karimi et al., 2015	http://data.csiro.au/
AnEM	Anatomical	CC-BY-SA 3.0	Ohta et al., 2012	✔ Included here
MITRestaurant	Queries	None	Liu et al., 2013a	http://groups.csail.mit.edu/sls/
MITMovie	Queries	None	Liu et al., 2013b	http://groups.csail.mit.edu/sls/
MalwareTextDB	Malware	None	Lim et al., 2017	http://www.statnlp.org/
re3d	Defense	Several (*1)	DSTL, 2017	✔ Included here
SEC-filings	Finance	CC-BY 3.0	Alvarado et al., 2015	✔ Included here
Assembly	Robotics	X	Costa et al., 2017	X

한국어 NLP dataset 모음

코퍼스 명	용도	설명	링크
Naver sentiment movie corpus v1.0	분류	네이버 영화 리뷰 (긍정, 부정) 분류 라벨링 됨	https://github.com/e9t/nsmc
Chatbot_data	분류	채팅 대화 (일상,긍정,부정) 분류 라벨링 됨	https://github.com/songys/Chatbot_data
청와대 국민청원 사이트의 만료된 청원 데이터 모음	RAW	일자,카테고리,제목,내용 등 만료된 청원 Raw 데이터	https://github.com/akngs/petitions
Korean NER Corpus	NER	한국어 NER 용 데이터 (NER, 형태소)	https://github.com/machinereading/KoreanNERCorpus
Korean Parallel corpora	번역	번역용 한국어/영어, 한국어/불어 병렬 데이터	https://github.com/j-min/korean-parallel-corpora
KorQuAD 1.0	MRC	MRC 용 Wikipedia에 대한 질문 답변 데이터	https://korquad.github.io/category/1.0_KOR.html
KorQuAD 2.1	MRC	MRC 용 Wikipedia에 대한 질문 답변 데이터 (1.0 보다 데이터가 큼)	https://korquad.github.io/
AI허브 AI데이터	다양	법률,특허,상식,대화 등 다양한 분야의 학습용 데이터 제공 (데이터 신청 별도 해야함)	http://www.aihub.or.kr/ai_data
국립국어원 언어정보나눔터	다양	말뭉치, 대화 자료등등 방대한 한국어 데이터 제공 (학습을 위해서는 전처리가 많이 필요함)	https://ithub.korean.go.kr/user/total/database/corpusManager.do