Skip to content

Dataset · jat-project

Multimodal data for a general agent

jat-project/jat-dataset

A mix of expert RL demonstrations, image-caption pairs, and text for training a multimodal agent.

What's inside

The dataset combines multiple separate sets: expert RL agent demonstrations, image-caption pairs, text data, and more. This is part of the JAT project to create a general-purpose multimodal agent. Time series are also specified in the metadata. The total size ranges from 100 million to 1 billion examples.

What you can do with it

  • Train a multimodal agent with reinforcement learning
  • Fine-tune VLMs on image-text pairs
  • Evaluate QA and text generation
Tasks
Reinforcement learning, Text generation, Question answering
Size
100M–1B rows
Downloads this month
128.3K
Likes
89

How to load it

from datasets import load_dataset

ds = load_dataset("jat-project/jat-dataset")