Datasets | Mozilla Data Collective

Sewá Morales

Ralámuli Ra’icháala TTS

Aproximadamente 2 horas con 20 minutos de habla leída de la lengua ralámuli, de una sola hablante.

License: CC-BY-NC-4.0

Locale: tar

Task: TTS

Format: WAV, TSV

Size: 422.68 MB

MDC Curators

CoVoST 2 English - Swedish

The dataset includes audio files in English and their translations in Swedish.

License: CC-BY-NC-4.0

Locale: en, sv-SE

Task: MT

Format: MP3, TSV

Size: 12.56 GB

PROXIMA

Urdu Alpaca

Filtered Urdu translation of the Stanford Alpaca instruction tuning dataset for Urdu LLM instruction fine-tuning.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: NLG

Format: Parquet

Size: 20.26 MB

PROXIMA

Sindhi OCR by Proxima AI

Synthetically generated Sindhi OCR dataset of 278,000 text image pairs using dictionary words and multi word phrases.

License: CC-BY-NC-SA-4.0

Locale: snd

Task: CV

Format: PARQUET

Size: 1.34 GB

MDC Curators

Common Voice Scripted Speech 26.0 - Southern American English

A closed-form Southern-American-English-accent subset of Mozilla Common Voice Scripted Speech 26.0 (English, validated). MP3 audio + TSV metadata.

License: CC0-1.0

Locale: en-US

Task: ASR

Format: TSV, MP3

Size: 32.34 MB

Community

Priangan Dialect-Sundanese Parallel Corpus

This dataset consists of a parallel corpus of Sundanese translations of the Priangan dialect especially from Cianjur City in West Java, Indonesia.

License: CC-BY-NC-SA-4.0

Locale: sun

Task: MT

Format: tsv

Size: 366.64 KB

Institute of African Digital Humanities

A Bakoko Sociocultural Dataset

Sociocultural dataset for the Bakoko community of Cameroon: identity, kinship, religion, economy, onomastics and a linguistic component.

License: NOODL-1.0

Locale: bkh

Task: OTH

Format: TSV, PDF

Size: 78.82 KB

MDC DatasetsCompensated

🚧 African Actions Annotated Videos for AI (Health Domain) 🚧

A large multimodal dataset of actions performed in Nigeria: 10 egocentric videos and 1,666 time-stamped action-object-tool segments in the "health" domain.

License: MDC DLA 1.0

Locale: en, ig, ha, yo

Task: OTH

Format: MP4, JSON

Size: 4.81 GB

PROXIMA

MMLU ProX Sndhi

Sindhi translation of the MMLU-ProX benchmark for evaluating LLM reasoning and knowledge in a low resource language setting.

License: MIT

Locale: snd

Task: NLP

Format: CSV

Size: 2.04 MB

Institute of African Digital Humanities

A Kanuri Sociocultural Dataset

Sociocultural dataset for the Kanuri community of Cameroon: identity, kinship, religion, economy, onomastics and a linguistic component.

License: NOODL-1.0

Locale: knc

Task: OTH

Format: TSV, PDF

Size: 77.79 KB

PROXIMA

Sindhi Alpaca

Filtered Sindhi translation of the Stanford Alpaca instruction tuning dataset for Sindhi LLM instruction fine tuning.

License: CC-BY-NC-SA-4.0

Locale: snd

Task: NLG

Format: Parquet

Size: 18.94 MB

PROXIMA

Sindhi Names by Proxima AI

Curated collection of 10,300 Sindhi personal names in Arabic script for NLP and language modeling tasks.

License: CC-BY-NC-SA-4.0

Locale: snd

Task: NLG

Format: CSV

Size: 57.15 KB

PROXIMA

English Medical Speech Dataset

English medical speech dataset of 11,563 audio-text pairs covering clinical terminology and medical education content.

License: CC-BY-NC-SA-4.0

Locale: en

Task: ASR

Format: Parquet

Size: 5.69 GB

PROXIMA

Synthetic Urdu 2

Synthetic Urdu sentences audio dataset of 85,327 verse style audio-text pairs.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: TTS

Format: Parquet

Size: 14.80 GB

PROXIMA

Urdu Legal Instruct

Curated Urdu language legal question answer corpus covering Pakistani family law, religious ordinances, and criminal statutes.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: NLG

Format: Parquet

Size: 132.33 KB

PROXIMA

Helpful and Harmless RLHF Urdu

Urdu translation of Anthropic's HH-RLHF preference dataset for RLHF/reward-model research in Urdu.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: NLP

Format: Parquet

Size: 10.37 MB

PROXIMA

Urdu Financial Phrasebank

Urdu translation of the Financial PhraseBank sentiment dataset, covering financial news sentences labeled by polarity.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: NLP

Format: Parquet

Size: 194.21 KB

Bahasa Bahana

BAHANA-Speech Corpus of English Learners from Indonesia

A corpus of read speech by learners of English from Indonesia.

License: CC-BY-SA-4.0

Locale: en

Task: CALL

Format: WEBM, WAV, TSV

Size: 1.14 GB

Taruen

Kumyk Proverbs and Sayings Text Corpus

A corpus of traditional Kumyk proverbs and sayings extracted from a 2015 dictionary, designed for NLP tasks and LLM cultural competence evaluations.

License: CC0-1.0

Locale: kum

Task: NLP

Format: TXT

Size: 86.48 KB

PROXIMA

Syntheic Urdu Audios

Synthetic Urdu conversational speech dataset of 7,963 audio text pairs featuring casual, humorous dialogue exchanges.

License: CC-BY-NC-SA-4.0

Locale: urd

Task: TTS

Format: PARQUET

Size: 4.58 GB

Institute of African Digital Humanities

A Bum Sociocultural Dataset

Sociocultural dataset for the Bum community of Cameroon: identity, kinship, religion, economy, onomastics and a linguistic component.

License: NOODL-1.0

Locale: bmv

Task: OTH

Format: TSV, PDF

Size: 84.33 KB

Institute of African Digital Humanities

A Foulbe / Pular Sociocultural Dataset

Sociocultural dataset for the Foulbe / Pular community of Cameroon: identity, kinship, religion, economy, onomastics and a linguistic component.

License: NOODL-1.0

Locale: fub

Task: OTH

Format: TSV, PDF

Size: 76.62 KB

Institute of African Digital Humanities

A Gbaya Sociocultural Dataset

Sociocultural dataset for the Gbaya community of Cameroon: identity, kinship, religion, economy, onomastics and a linguistic component.

License: NOODL-1.0

Locale: gya

Task: OTH

Format: TSV, PDF

Size: 83.68 KB

PROXIMA

Urdu Medical SFT

Safety aligned Urdu medical question answer dataset of 3,072 records for supervised fine-tuning of medical-domain LLMs.

License: CC-BY-SA-4.0

Locale: urd

Task: NLG

Format: PARQUET

Size: 357.97 KB

All Datasets

Compensated datasets

Conversational Speech in Gujarati in the Finance Domain

Conversational Speech in Manipuri in the Finance Domain

Conversational Speech in Manipuri in the Medical Domain

Ralámuli Ra’icháala TTS

CoVoST 2 English - Swedish

Urdu Alpaca

Sindhi OCR by Proxima AI

Common Voice Scripted Speech 26.0 - Southern American English

Priangan Dialect-Sundanese Parallel Corpus

A Bakoko Sociocultural Dataset

🚧 African Actions Annotated Videos for AI (Health Domain) 🚧

MMLU ProX Sndhi

A Kanuri Sociocultural Dataset

Sindhi Alpaca

Sindhi Names by Proxima AI

English Medical Speech Dataset

Synthetic Urdu 2

Urdu Legal Instruct

Helpful and Harmless RLHF Urdu

Urdu Financial Phrasebank

BAHANA-Speech Corpus of English Learners from Indonesia

Kumyk Proverbs and Sayings Text Corpus

Syntheic Urdu Audios

A Bum Sociocultural Dataset

A Foulbe / Pular Sociocultural Dataset

A Gbaya Sociocultural Dataset

Urdu Medical SFT