Brain MRI multimodal benchmark
2 min readupdated
A comprehensive set of 155 checkpoints and code for multimodal disease classification using clinical descriptions and head MRI scans. The model combines a BiomedBERT text encoder and a BiomedCLIP visual encoder, evaluating the contribution of each component to the final diagnosis. The research demonstrates how text context affects the accuracy of medical data recognition. The package is intended for researchers in the field of medical artificial intelligence.
Key features
- 155 checkpoints: covers various seeded runs and modality combinations.
- Multimodal approach: combines text reports, MRI scans, and tabular data.
- Advanced architecture: uses fine-tuned encoders with late fusion.
- Verified baseline: trained on the MultiCaRe dataset with publication-year splitting.
What you can do
- Run ready-to-use medical case classification pipelines using the mm_release.py script.
- Explore the impact of text prompts and MRI images on model accuracy.
- Test individual encoders for feature analysis from open clinical reports.
Example
from huggingface_hub import snapshot_download
import sys
d = snapshot_download('Kashif-Mahmood007/brainmri-casereport-multimodal-benchmark')
sys.path.insert(0, d)
import mm_release as R
m = R.load_best_model(seed=42)
p = R.predict_best_model(m, text=text, images=images, age=54, sex='F')
print(dict(zip(R.CLASS_NAMES, p.round(3))))Read more
DetailsBenchmarks: best macro-F1 on internal test is 0.559.
Requirements: about 4.2 GB of disk space for weights.
Limitations: for research purposes only, weak labels derived from headers.
Data: trained on 1,542 patients from the open MultiCaRe corpus.










Comments