CAT
Terrain traversability estimation for different robots from a single RGB-D frame
2 min readupdated
Developed by researchers at the University of São Paulo and presented at IROS 2026, CAT generates a dense traversability map from a single RGB-D image (a color frame paired with a depth map). Its standout feature is robot-aware prediction: the output adapts to the capabilities of the specific platform traversing the terrain, meaning an obstacle passable for a tracked vehicle might be flagged as impassable for a small wheeled rover. Under the hood, it relies on frozen backbones (feature-extraction networks) facebook/dinov3-vitb16-pretrain-lvd1689m and CIDAS/clipseg-rd64-refined. It is geared toward developers working on autonomous navigation and mobile robotics in unstructured, off-road environments.
Key features
- Robot-specific adaptation: traversability predictions account for the capabilities and scores of the selected platform.
- Single-frame inference: generates a map from a single RGB and depth image pair without needing a sequence of frames.
- DINOv3 and CLIPSeg backbone: builds on frozen visual and multimodal features from pretrained models.
- Bundled sensor statistics: ships with
depth_stats.jsoncontaining per-sensor depth statistics alongside thebest_model.pthweights.
What you can do
- Off-road path planning: integrate the model into an autonomous robot's navigation stack to select safe trajectories across uneven terrain.
- Cross-platform traversability comparison: run the same RGB-D scenes using different robot capability profiles (via
scores.yaml) to see where each vehicle can pass. - Test on the NaviTrace dataset: run the ready-to-use
cat_demo.pyscript on sample data from the project repository to evaluate segmentation quality.
Example
uv run --locked --extra cu128 hf download CAT-IROS-2026/cat best_model.pth depth_stats.json --local-dir checkpoints/cat
uv run --locked --extra cu128 python cat_demo.py \
--checkpoint checkpoints/cat/best_model.pth \
--rgb examples/navitrace/*/rgb.png \
--depth examples/navitrace/*/depth.png \
--scores examples/navitrace/scores.yaml \
--sensor ts
Read more
DetailsBase models: facebook/dinov3-vitb16-pretrain-lvd1689m and CIDAS/clipseg-rd64-refined (downloaded automatically on first run).
Release contents: best_model.pth checkpoint and depth_stats.json depth statistics file.
Licenses: the checkpoint is distributed under the DINOv3 License (as it was trained on DINOv3 features), and the code is under the MIT License.
VRAM requirements and benchmark figures are not listed in the model card (see arXiv:2607.20679 for details).










Comments