1. Executive Summary & Problem Context
Nepali is spoken by tens of millions of people worldwide but remains severely low-resource in the modern LLM era (reporting only ~4.18B public tokens, over 80× fewer than English). Existing Devanagari text in large multilingual web scrapes (such as FineWeb-2, Sangraha, and IndicCorp) suffers from silent Hindi leakage, broken legacy fonts (Preeti/Kantipur), high duplication rates, and severed provenance.
Once scraped text is merged into a flat file, its original publisher, license, crawl date, and annotator identity disappear. Nepali Data Studio (NDS) is an all-in-one data operating system built on Datasette and SQLite that tracks the full life of Nepali data with 100% auditable lineage.
2. The 6-Database Architecture
NDS avoids fragile infrastructure by running on SQLite with Write-Ahead Logging (WAL) and Litestream replication. Data is partitioned into six purpose-built databases:
registry.db: Sources, licenses, crawl policies, and external dataset aliases (linking past KU research papers and Hugging Face benchmarks).raw.db: Untouchable vault of immutable, SHA-256 hashed HTML, PDF, image, and audio files.corpus.db: Normalized (NFC, Preeti-converted) text, Devanagari sentences, tokens (MPP 43 & UPOS), and gazetteers.annotation.db: Standoff annotation workbench, multi-rater votes, and gold adjudications.instruct.db: Prompt templates, instruction-response pairs, and DPO/RLHF preference pairs.release.db&admin.db: Release snapshots, Croissant-RAI metadata, contributor credit ledgers, and audit logs.
3. Annotation Hub & Plugin Manifests
Instead of building custom software for every task, NDS tasks are specified as declarative YAML manifests + JSON Schemas.
Core P0 Tasks: 10-class Named Entity Recognition (NER), Part-of-Speech tagging (MPP 43 tagset), Spelling & Grammar Error Correction (GEC), Question Answering (QA), and Gemma/T5 chat instruction pairs.
origin = machine and never auto-promoted to Gold.