Dataset · google
Google engineering task benchmark
google/physical-engineerings-last-exam
A set of complex multi-step tasks from licensed engineers for evaluating AI agents.
What's inside
It contains English engineering tasks across 4 disciplines created by licensed professionals. In each task, the agent must work in a workspace featuring blueprints, specifications, tables, and CAD/BIM tools to produce calculations, code, models, and reports. The repository features a dev set comprising fewer than 1,000 examples.
What you can do with it
- Evaluating autonomous AI agents in complex engineering and CAD/BIM tasks
- Testing model capabilities with blueprints, code, and calculations
- Benchmarking multimodal LLMs for technical design
- Tasks
- Text generation, Question answering
- Languages
- English
- Size
- up to 1K rows
- Downloads this month
- 378.3K
- Likes
- 191
- Access
- on request on Hugging Face
How to load it
from datasets import load_dataset
ds = load_dataset("google/physical-engineerings-last-exam")