MOSAIC
Multimodal model for survival prediction in histopathology
2 min read
The MOSAIC model combines histopathological data, structured clinical information, and genomic alterations to predict overall survival (OS) and progression-free survival (PFS). It accepts WSI (whole slide images) patch features, clinical data, or SNV/CNV gene lists in various combinations as input, returning corresponding risks and 768-dimensional embeddings. The repository includes a language encoder, WSI aggregator, visual adapters, and multimodal fusion modules. The tool targets researchers in bioinformatics and medical analytics.
Key features
- Flexible data input: support for clinical features, WSI visual patches, and SNV/CNV genomic panels individually or in combination.
- Automated processing of WSI spatial representations using built-in patch grid analysis.
- Joint use of pretrained language encoder and SSL encoder components for feature extraction.
- Calculation of independent risk scores for overall survival (OS) and progression-free survival (PFS).
What you can do
- Evaluate patient risks using only available clinical data.
- Perform multimodal analysis by combining clinical information with histological slide visual features and genomic data.
- Extract 768-dimensional WSI embeddings for further tissue image analysis.
- Compare various combinations of input modalities for oncology research tasks.
Example
import torch
from transformers import AutoModel
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModel.from_pretrained(
"Astaxanthin/MOSAIC",
trust_remote_code=True,
).to(device).eval()
clinical = {
"age": 65,
"sex": "Male",
"stage": "II",
"grade": "G2",
"subtype": "Pancreatic Adenocarcinoma",
}
output = model.predict(
cancer_type="Pancreatic Cancer",
clinical=clinical,
)
print("OS risk:", output["os_risk"].item())Read more
DetailsBenchmarks on the HCMI sample (patient-level OS C-index): Clinical — 0.675; Clinical + vision — 0.701; Clinical + TP53 SNV — 0.689; Clinical + vision + TP53 SNV — 0.724.
Requirements: PyTorch >= 2.6, transformers == 4.57.1, nystrom-attention.
Limitations: the model provides relative risk estimates for research purposes and is not suitable for making standalone clinical decisions.










Comments