Whole verses, from texts models haven't memorised.
v0's questions turned out too easy for today's models: one word or one form at a time, from texts they can recite. v1 asks for whole verses from texts they can't recite (we tested), plus verb forms that rarely occur in any text. Every answer key still comes from the corpus and from Pāṇini's rules, never from a model.
How today's models do
Preview results. The test questions are private, so these scores can't be trained on; the answer keys are being reviewed question by question, and scores are updated as keys are corrected.
What models can recite
We gave each model the first half of 30 verses from each of 23 texts and asked for the second half. A model that returns it almost word for word has memorised the text, and questions from that text would test memory more than Sanskrit. v1 uses only texts the strongest model recited 10% of the time or less.
Share of verses whose second half was reproduced near-verbatim (character similarity of 0.9 or more). Grey rows are well-known texts tested for comparison; v1 marks the texts v1 uses.
What the model is asked
The examples below use well-known verses so the private questions stay private. A verse counts only if every part of the answer is right.
Padapāṭha
Undo all the sandhi of a verse: every word in its stand-alone form, in order. Compounds may be written whole or with hyphens.
dharmakṣetre kurukṣetre samavetā yuyutsavaḥ
Parse
The lemma and grammar, in context, of every inflected word of a verse. One wrong word fails the verse.
māmakāḥ pāṇḍavāś caiva kim akurvata sañjaya: akurvata
Scan
Mark every syllable of a verse heavy or light, and name the meter: anuṣṭubh, mandākrāntā, śikhariṇī, vasantatilakā and others.
kaścit kāntāvirahaguruṇā svādhikārāt pramattaḥ
GGGG LLLLLG GLGGLGG · mandākrāntāGenerate
Produce a verb form that rarely occurs in texts: desideratives, causatives and intensives, the precative, conditional and periphrastic future, the passive.
The desiderative of √gam, laṭ, 3rd singular, active.
Where the questions and answers come from
- Texts from the Digital Corpus of Sanskrit, chosen by the recitation test above: .
- Padapāṭha keys are the corpus's word split, kept only where Vidyut's sandhi rules rebuild exactly the printed verse from it.
- Parse keys are the corpus's annotation, kept only where Vidyut confirms every word and the split rebuilds the verse. Forms that look the same in two readings (masculine and neuter outside the nominative, for example) are accepted either way.
- Scan keys come from Vidyut's prosody module, only for verses whose meter it identifies by rule. The last syllable of each pāda isn't graded.
- Generate keys are every form Vidyut derives from Pāṇini's rules; where several are allowed, each counts.
- Private test set. The questions are drawn with a secret seed and not published, so they can't be trained on. Models are run by Samskritam.ai; to have a model added, get in touch.
- Review. Reviewers check the answer keys question by question on an invitation-only page. Corrections are applied and the scores recomputed; what changed is listed here.
What this does not measure yet
- Meaning, translation, commentary and śāstric reasoning.
- Keys are only as good as the corpus and Vidyut; that is why they are reviewed, and why this is a preview.
- One run per model. Differences of a few points are within the margin shown on each score.
A Sanskrit grammar test graded by rule, not by another model.
Models write Sanskrit that looks fluent. This measures whether the grammar underneath is right: building verb and noun forms, reading them back, parsing words in real verses, and scanning meter. Every answer is an exact form or analysis. The answer key comes from an implementation of Pāṇini's rules and from a scholarly corpus, and was reviewed by hand.
How today's models do
| Model | Overall487 qs | Tiṅanta150 | Subanta94 | Analysis100 | Gītā93 | Chandas50 |
|---|
How to read it: an overall score is good to about ±2 points, and a task score to about ±3 points for the larger tasks and ±12 for Chandas (50 questions); hover a score for its margin. Gaps smaller than that are not meaningful. Each model was run once.
Every model's answers, graded question by question: .
The first results will appear here once the runs are complete and the wrong answers have been checked by hand for grading mistakes.
Any model on OpenRouter, in about 30 minutes. The runner and grader need only Python.
git clone https://github.com/ssriramhere/samskritam
cd samskritam && pip install -r pipeline/requirements.txt
export OPENROUTER_API_KEY=...
python bench/run.py --provider openrouter --model vendor/model
python bench/grade.py bench/results/openrouter__vendor_model.jsonlSend us the results file and we'll review the wrong answers and add the model. Other providers: --provider anthropic|openai|google|sarvam.
Items and answers are CC BY 4.0; the code is Apache 2.0.
@misc{samskritam-bench-v0,
title = {Samskritam Bench v0: a Sanskrit grammar
benchmark graded by rule},
author = {Subramanian, Sriram},
year = {2026},
url = {https://samskritam.ai/bench},
note = {487 items; answer keys from Vidyut and the
Digital Corpus of Sanskrit}
}Corrections to the answer keys
The answer keys are being reviewed question by question, including the 13 questions removed before publishing. v0 stays as published: the scores above don't change. Corrections are listed here, with each model's score re-graded from its published answers using the corrected keys.
Loading…
What the model is asked
Real questions from the item set. The model must answer in JSON; the grader accepts IAST or Devanagari.
Tiṅanta
Build a finite verb from a root, lakāra, person and number.
Give the verb form of √vah (bhvādi, "to carry") in laṅ (imperfect), uttama-puruṣa, ekavacana, kartari.
Subanta
Decline a noun or adjective stem in a given case, number and gender.
Decline the stem deva (puṃliṅga) in ṣaṣṭhī ekavacana.
Analysis
Read a verb form backwards: name its lakāra, puruṣa and vacana.
bubudhivahe is a form of √budh (divādi, "to understand"), active voice. Which lakāra, puruṣa and vacana?
Gītā in context
Give a word's dictionary form and grammar inside a real verse.
In ṛṣibhirbahudhā gītaṃ chandobhirvividhaiḥ pṛthak (13.5), analyse vividhaiḥ.
Chandas
Scan a half-verse into heavy (guru) and light (laghu) syllables.
Scan sādhureva sa mantavyaḥ samyagvyavasito hi saḥ (9.30).
GLGLLGGG GGLLLGLGWhere the questions and answers come from
- Questions are generated from fixed templates by
bench/generate.pywith a fixed seed, so anyone can rebuild the identical set. - Verb questions use 54 common roots, each pinned to its dhātupāṭha entry so the gaṇa and meaning are unambiguous, across seven lakāras: laṭ, laṅ, loṭ, vidhiliṅ, lṛṭ, liṭ and luṅ.
- Noun questions use the most frequent noun and adjective stems of the Bhagavad Gītā.
- Answers for verbs and nouns are derived by Vidyut, an implementation of more than 2,000 sūtras of the Aṣṭādhyāyī. Where Pāṇini allows more than one form, every form is accepted. Each noun answer is also checked against Vidyut's dictionary of inflected forms; two questions failed that check (the dual of mahat and the feminine of bhagavat) and were removed.
- Hand review. Before publishing, every question that any model got wrong was reviewed by hand. Eleven answer keys turned out to be wrong or genuinely disputable, for example the feminine of rājasa (it is rājasī) and the gender of dhṛti in 18.34. Those eleven were removed, leaving 487. Each removal and its reason is listed in
bench/generate.py. Reviewing only the questions models missed can favour the models, so the full set is open for anyone to check. The keys, and these removals, are now being reviewed question by question; see Errata. - Answers for the Gītā questions are the hand annotations of the Digital Corpus of Sanskrit, used only where Vidyut's rules allow the same analysis. The rules confirm the analysis is possible for the form; the choice in context is the annotators'.
- Answers for meter are computed from the rules of prosody. The last syllable of each pāda is not graded, since either weight is allowed there.
What counts as right
- The exact form is required. A near miss is wrong.
- IAST and Devanagari are both accepted, and so are both spellings of a nasal before a consonant (saṃjaya and sañjaya).
- For Gītā questions, the lemma and every grammatical field given by the annotation must match. Common citation forms are accepted for pronouns (yuṣmad for tvad, for example). The boundary between noun and adjective is not graded.
- An answer that is not valid JSON in the requested shape is wrong.
What this does not measure yet
- Meaning, translation, commentary and śāstric reasoning. Those need human judgement.
- Sandhi splitting, compounds and kāraka roles. These are planned for the next version.
- The answer key is only as complete as Vidyut. A valid form it does not generate would be marked wrong, which is why wrong answers are reviewed by hand before results are published.
- The Gītā is famous and well represented in training data, so the Gītā and Chandas tasks may partly measure memory. The verb and noun tasks mostly ask for forms that rarely occur in any text.
Run it on your own model
The items, the runner and the grader are in the GitHub repository. The runner needs only Python; the grader also needs the vidyut package.
pip install -r pipeline/requirements.txt
python -c "import vidyut; vidyut.download_data('vidyut-data')"
export OPENROUTER_API_KEY=...
python bench/run.py --provider openrouter --model <vendor/model> --rpm 18
python bench/grade.py bench/results/<file>.jsonl
Each model's raw answers are published too, one line per question with the grade, in bench/results-v0, so the scores can be re-graded and the hand review checked.
Items and results are licensed CC BY 4.0. Please credit Samskritam.ai, the Digital Corpus of Sanskrit (Oliver Hellwig) and Vidyut (Ambuda).