UI controls detector
UI element detector
2 min readupdated
A model based on RF-DETR-Medium, trained to find clickable elements on webpage screenshots and classify them into icons, text controls, and input fields. The checkpoint was trained on 599 screenshots with annotations derived from page structure. The model stands out for its high processing speed (0.3 seconds per image on Apple M-series) and a smaller weight size compared to alternatives. It will come in handy for developers and researchers working on browser automation and web interface analysis.
Key features
- Finds icons, text buttons, and input fields on 1440×900 screenshots
- Fast performance: 0.3 seconds per image on Apple M-series processors
- Compact weight size (142 MB)
- Shows an element detection accuracy of 91.6% at a 0.2 threshold
What you can do
- Use it to analyze webpage screenshots in automation pipelines
- Automatically extract coordinates of clickable interface zones
- Integrate the detector into Python scripts using the rfdetr library
Example
from PIL import Image
from rfdetr import RFDETRMedium
model = RFDETRMedium(pretrain_weights="ui-controls-rfdetr-medium-1280.pth", resolution=1280)
det = model.predict(Image.open("screenshot.png").convert("RGB"), threshold=0.2)
for box, cls, score in zip(det.xyxy, det.class_id, det.confidence):
print(["icon", "text_control", "input"][cls], [round(v) for v in box], round(float(score), 2))Read more
DetailsMetrics: finds 91.6% of elements at a 0.2 threshold and up to 96.3% at lower thresholds
Limitations: tested only on webpages with a 1440×900 resolution
Annotations: generated automatically from page structure and OCR
License: Apache 2.0










Comments