Dataset · docling-project
1.21m labeled desktop screenshots with clicks
docling-project/DeskForge-1M
A corpus of desktop screenshots annotated with UI elements and click transitions for training GUI agents.
What's inside
Contains 1.21 million English desktop screenshots with 159.7 million annotated UI elements and 917,000 click transitions. The data was generated in a controlled DeskForge environment using real applications. Created by researchers from ETH Zurich, IBM Research Zurich, and Microsoft.
What you can do with it
- Training VLM agents for autonomous computer interface operation
- Detecting and parsing GUI elements in screenshots
- Evaluating desktop control models on click scenarios
- Tasks
- Image → text, Object detection
- Languages
- English
- Size
- 1M–10M rows
- Downloads this month
- 25.3K
- Likes
- 52
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("docling-project/DeskForge-1M")