Skip to content

Dataset · docling-project

1.21m labeled desktop screenshots with clicks

docling-project/DeskForge-1M

A corpus of desktop screenshots annotated with UI elements and click transitions for training GUI agents.

What's inside

Contains 1.21 million English desktop screenshots with 159.7 million annotated UI elements and 917,000 click transitions. The data was generated in a controlled DeskForge environment using real applications. Created by researchers from ETH Zurich, IBM Research Zurich, and Microsoft.

What you can do with it

  • Training VLM agents for autonomous computer interface operation
  • Detecting and parsing GUI elements in screenshots
  • Evaluating desktop control models on click scenarios
Tasks
Image → text, Object detection
Languages
English
Size
1M–10M rows
Downloads this month
25.3K
Likes
52

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("docling-project/DeskForge-1M")