List of Finnish Datasets for NLP Projects

 https://metatext.io/datasets-list/finnish-language

FI News Corpus

Dataset is a collection of news headlines and short summaries of text, organized by date. The news articles were published between 2012-2020.

CC100-Finnish

This dataset is one of the 100 corpora of monolingual data that was processed from the January-December 2018 Commoncrawl snapshots from the CC-Net repository. The size of this corpus is 15G.

FinChat

Dataset contains conversations with message timestamps, sender’s id, and metadata information. It contains 86 conversations with 3,630 messages, 22,210 words with the average word length of 5.6, and on the average 14 turns per each conversation.

Finnish News Corpus for Named Entity Recognition

Dataset contains 953 articles (193,742 word tokens) with 6 named entity classes: organization, location, person, product, event, and date.

Finlex

Dataset is a collection of legislative and other judicial information of Finland, which is available in Finnish and Swedish.

Fiskmö

Dataset is a parallel corpus of Finnish and Swedish Languages.

FinChat

Dataset contains conversations with message timestamps, sender’s id, and metadata information. It contains 86 conversations with 3,630 messages, 22,210 words with the average word length of 5.6, and on the average 14 turns per each conversation.

FI News Corpus

Dataset is a collection of news headlines and short summaries of text, organized by date. The news articles were published between 2012-2020.

CC100-Finnish

This dataset is one of the 100 corpora of monolingual data that was processed from the January-December 2018 Commoncrawl snapshots from the CC-Net repository. The size of this corpus is 15G.

FinChat

Dataset contains conversations with message timestamps, sender’s id, and metadata information. It contains 86 conversations with 3,630 messages, 22,210 words with the average word length of 5.6, and on the average 14 turns per each conversation.

Finnish News Corpus for Named Entity Recognition

Dataset contains 953 articles (193,742 word tokens) with 6 named entity classes: organization, location, person, product, event, and date.

Finlex

Dataset is a collection of legislative and other judicial information of Finland, which is available in Finnish and Swedish.

Fiskmö

Dataset is a parallel corpus of Finnish and Swedish Languages.

FinChat

Dataset contains conversations with message timestamps, sender’s id, and metadata information. It contains 86 conversations with 3,630 messages, 22,210 words with the average word length of 5.6, and on the average 14 turns per each conversation.

text summarise dataset

 **Paper:**

https://arxiv.org/abs/1908.08345

**Dataset:**

1) the CNN/DailyMail news highlights dataset: somewhat Extractive

- News Articles & Related Highlights: Provides a brief overview of articles

- Input document: limited to 512 tokens

- https://www.kaggle.com/datasets/gowrishankarp/newspaper-text-summarization-cnn-dailymail

2) the New York Times Annotated Corpus (NYT): somewhat Extractive

- Contains 110,540 articles with abstract summaries

- Input document : limited to 800 tokens

- https://research.google/resources/datasets/ny-times-annotated-corpus/

3) XSum: Abstractive

- 226,711 news articles answering the question of ‘What is this articles about?’ + one-sentence summaries

- Input document: limited to 512 tokens

- https://github.com/google-research-datasets/xsum_hallucination_annotations

Naver sentiment movie corpus v1.0 (korean)

https://github.com/e9t/nsmc

Recommendation and ranking systems

Recommendation and ranking systems

• Movielens: Movie ratings dataset from the Movielens website, in various sizes ranging from demo to mid-size.
• Million Song Dataset: Large, metadata-rich, open source dataset on Kaggle that can be good for people experimenting with hybrid recommendation systems.
• Last.fm: Music recommendation dataset with access to underlying social network and other metadata that can be useful for hybrid systems.
• Book-Crossing dataset:: From the Book-Crossing community. Contains 278,858 users providing 1,149,780 ratings about 271,379 books.
• Jester: 4.1 million continuous ratings (-10.00 to +10.00) of 100 jokes from 73,421 users.
• Netflix Prize:: Netflix released an anonymized version of their movie rating dataset; it consists of 100 million ratings, done by 480,000 users who have rated between 1 and all of the 17,770 movies. First major Kaggle style data challenge. Only available unofficially, as privacy issues arose.

Sentiment

Sentiment

• Multidomain sentiment analysis dataset An older, academic dataset.
• IMDB: An older, relatively small dataset for binary sentiment classification. Fallen out of favor for benchmarks in the literature in lieu of larger datasets.
• Stanford Sentiment Treebank: Standard sentiment dataset with fine-grained sentiment annotations at every node of each sentence’s parse tree.

Question answering

Question answering

• Maluuba News QA Dataset: 120K Q&A pairs on CNN news articles.
• Quora Question Pairs: first dataset release from Quora containing duplicate / semantic similarity labels.
• CMU Q/A Dataset: Manually-generated factoid question/answer pairs with difficulty ratings from Wikipedia articles.
• Maluuba goal-oriented dialogue: Procedural conversational dataset where the dialogue aims at accomplishing a task or taking a decision. Often used to work on chat bots.
• bAbi: Synthetic reading comprehension and question answering datasets from Facebook AI Research (FAIR).
• The Children’s Book Test: Baseline of (Question + context, Answer) pairs extracted from Children’s books available through Project Gutenberg. Useful for question-answering (reading comprehension) and factoid look-up.

Text Datasets

Text Datasets

• 20 newsgroups: Classification task, mapping word occurences to newsgroup ID. One of the classic datasets for text classification) usually useful as a benchmark for either pure classification or as a validation of any IR / indexing algorithm.
• Reuters News dataset: (Older) purely classification-based dataset with text from the newswire. Commonly used in tutorial.
• Penn Treebank: Used for next word prediction or next character prediction.
• UCI’s Spambase: (Older) classic spam email dataset from the famous UCI Machine Learning Repository. Due to details of how the dataset was curated, this can be an interesting baseline for learning personalized spam filtering.
• Broadcast News: Large text dataset, classically used for next word prediction.
• Text Classification Datasets: From; Zhang et al., 2015; An extensive set of eight datasets for text classification. These are the benchmark for new text classification baselines. Sample size of 120K to 3.6M, ranging from binary to 14 class problems. Datasets from DBPedia, Amazon, Yelp, Yahoo! and AG.
• WikiText: A large language modeling corpus from quality Wikipedia articles, curated by Salesforce MetaMind.
• SQuAD: The Stanford Question Answering Dataset — broadly useful question answering and reading comprehension dataset, where every answer to a question is posed as a segment of text.
• Billion Words dataset: A large general-purpose language modeling dataset. Often used to train distributed word representations such as word2vec.
• Common Crawl: Petabyte-scale crawl of the web — most frequently used for learning word embeddings. Available for free from Amazon S3. Can also be useful as a network dataset for it’s a crawl of the WWW.
• Google Books Ngrams: Successive words from Google books. Offers a simple method to explore when a word first entered wide usage.
• Yelp Open Dataset: The Yelp dataset is a subset of Yelp businesses, reviews, and user data for use in NLP.

Datasets for English Named Entity Recognition

Datasets for English Named Entity Recognition

Annotated Corpus for Named Entity Recognition: Corpus for entity classification with enhanced and popular features by Natural Language Processing applied to the data set.

i2b2 Challenges: By the Informatics for Integrating Biology & the Bedside (i2b2) center, these clinical datasets were created for named entity recognition.

CoNLL 2003: Dataset that contains 1,393 English news articles with annotated entities LOC (location), ORG (organization), PER (person) and MISC (miscellaneous).

NLPBA 2004: Medical data tagged with protein/DNA/RNA/cell line/cell type (2,404 MEDLINE abstracts).

Resume Entities for NER: Document annotation dataset to be used to perform NER on resumes from indeed.com.

Enron Emails: Over 500,000 email messages tagged with names, dates and times.

MIT Movie Corpus: A semantically tagged training and test corpus in BIO format. The eng corpus are simple queries, and the trivia10k13 corpus are more complex queries.

Annotated GMB Corpus: An annotated corpus using GMB (Groningen Meaning Bank) corpus for entity classification with enhanced and popular features by Natural Language Processing applied to the data set.

Best Buy E-Commerce NER Dataset: A dataset containing Best Buy search queries labeled with entities such as Brand, Model name, Category Name, and etc.

WNUT 17 Emerging Entities Dataset: Text from YouTube, Stack Overflow, Twitter and Reddit comments filtered to prefer text that is likely to contain named entities.

Finnish NER dataset

https://github.com/mpsilfve/finer-data

The directory data contains a corpus of Finnish technology related news articles with a manually prepared named entity annotation (digitoday.2014.csv). The text material was extracted from the archives of Digitoday, a Finnish online technology news source (www.digitoday.fi). The corpus consists of 953 articles (193,742 word tokens) with six named entity classes (organization, location, person, product, event, and date). The corpus is available for research purposes and can be readily used for development of NER systems for Finnish. The corpus is described in the article

"A Finnish News Corpus for Named Entity Recognition" (in review)

English NER dataset

Dataset	Domain	License	Reference	Availablility
CONLL 2003	News	DUA	Sang and Meulder, 2003	Easy to find
NIST-IEER	News	None	NIST 1999 IE-ER	NLTK data
MUC-6	News	LDC	Grishman and Sundheim, 1996	LDC 2003T13
OntoNotes 5	Various	LDC	Weischedel et al., 2013	LDC 2013T19
BBN	Various	LDC	Weischedel and Brunstein, 2005	LDC 2005T33
GMB-1.0.0	Various	None	Bos et al., 2017	http://gmb.let.rug.nl/data.php
GUM-3.1.0	Wiki	Several (*2)	Zeldes, 2016	✔ Included here
wikigold	Wikipedia	CC-BY 4.0	Balasuriya et al., 2009	✔ Included here
Ritter	Twitter	None	Ritter et al., 2011	No split , Train/test/dev split
BTC	Twitter	CC-BY 4.0	Derczynski et al., 2016	✔ Included here
WNUT17	Social media	CC-BY 4.0	Derczynski et al., 2017	✔ Included here
i2b2-2006	Medical	DUA	Uzuner et al., 2007	http://www.i2b2.org
i2b2-2014	Medical	DUA	Stubbs et al., 2015	http://www.i2b2.org
CADEC	Medical	CSIRO	Karimi et al., 2015	http://data.csiro.au/
AnEM	Anatomical	CC-BY-SA 3.0	Ohta et al., 2012	✔ Included here
MITRestaurant	Queries	None	Liu et al., 2013a	http://groups.csail.mit.edu/sls/
MITMovie	Queries	None	Liu et al., 2013b	http://groups.csail.mit.edu/sls/
MalwareTextDB	Malware	None	Lim et al., 2017	http://www.statnlp.org/
re3d	Defense	Several (*1)	DSTL, 2017	✔ Included here
SEC-filings	Finance	CC-BY 3.0	Alvarado et al., 2015	✔ Included here
Assembly	Robotics	X	Costa et al., 2017	X