DoAI × ILPRLKathmandu University
Home / Projects / BalEval
In ProgressEvaluation & Benchmarks

BalEval: Multimodal Nepali Foundation Benchmark

A Unified Empirical Evaluation Suite Pairing Natively Multimodal Retrieval with Generative Foundation Reasoning for Nepali.

1. Charter & Research Scope

The primary objective of BalEval is to execute a rigorous, publication-grade empirical milestone for the Nepali language: consolidating 22 years of fragmented legacy NLP tools into a unified foundation benchmark and fine-tuned baseline suite.

BalEval pairs EmbeddingGemma 2 (740M) for natively multimodal retrieval with Gemma 4 E4B (4.2B) for generative reasoning across Devanagari text, scanned historical archives, and speech transcripts.

Core Objective: Replace disjoint pipelines with a single open foundation benchmark (AST tool evaluation, IFEval verifier, and lexical audit).

2. 12-Week Sprint Schedule

  • Month 1 (Archival Mining & Auditing): Harvest ILPRL legacy datasets and public sector archives into standardized JSON Lines schema; audit zero-shot baselines.
  • Month 2 (Multimodal Engineering & Training): Index scanned citizen charters and audio transcripts into EmbeddingGemma 2 vector store; fine-tune Gemma 4 E4B via 4-bit QLoRA.
  • Month 3 (Rigorous Evaluation & Capstone Packaging): Automated AST and IFEval execution, qualitative linguistic review with supervisors, ablation studies, and open-source release (ILPRL/BalEval).