Dataset · themohal
Saraiki LLM evaluation benchmark
themohal/saraiki-llm-bench
A set of seven test tasks to check the quality and safety of language models in the low-resource Saraiki language.
What's inside
Inside are text tasks in Saraiki (Shahmukhi script) for testing knowledge (including an MMLU equivalent), mathematics, instruction following, and safety. The total volume ranges from 1 to 10 thousand entries split into a test set and a small dev split for few-shot examples. The dataset is intended strictly for model evaluation; training on it is prohibited.
What you can do with it
- Evaluating multilingual LLMs for understanding the low-resource Saraiki language
- Testing models on instruction following and safety in few-shot and zero-shot modes
- Tasks
- Question answering, Multiple choice, Text generation
- Languages
- skr
- Size
- 1K–10K rows
- Downloads this month
- 48.3K
- Likes
- 0
- Access
- on request on Hugging Face
How to load it
from datasets import load_dataset
ds = load_dataset("themohal/saraiki-llm-bench")