Skip to content

Dataset · themohal

Saraiki LLM evaluation benchmark

themohal/saraiki-llm-bench

A set of seven test tasks to check the quality and safety of language models in the low-resource Saraiki language.

What's inside

Inside are text tasks in Saraiki (Shahmukhi script) for testing knowledge (including an MMLU equivalent), mathematics, instruction following, and safety. The total volume ranges from 1 to 10 thousand entries split into a test set and a small dev split for few-shot examples. The dataset is intended strictly for model evaluation; training on it is prohibited.

What you can do with it

  • Evaluating multilingual LLMs for understanding the low-resource Saraiki language
  • Testing models on instruction following and safety in few-shot and zero-shot modes
Tasks
Question answering, Multiple choice, Text generation
Languages
skr
Size
1K–10K rows
Downloads this month
48.3K
Likes
0
Access
on request on Hugging Face

How to load it

from datasets import load_dataset

ds = load_dataset("themohal/saraiki-llm-bench")