Skip to content

Dataset · AnalystAIlab

AI assistant failure examples in everyday tasks

AnalystAIlab/ai-assistant-failure-cases

A small collection of everyday queries where popular AI assistants made notable errors.

What's inside

The repository contains fewer than 1,000 entries in text and tabular formats in English, collected by the analyst.ai project. Each case contains a real-world everyday prompt where a consumer AI assistant failed (plus one case where no one failed). The authors warn that the dataset is created to demonstrate types of failures rather than to calculate overall model rankings.

What you can do with it

  • Manual testing and regression checks for custom AI assistants on complex cases
  • Analysis of typical hallucinations and LLM errors in everyday scenarios
  • Preparing edge-case prompts to evaluate model reliability
Tasks
Text generation, Question answering
Languages
English
Size
up to 1K rows
Downloads this month
212
Likes
11

How to load it

from datasets import load_dataset

ds = load_dataset("AnalystAIlab/ai-assistant-failure-cases")