JECT_320_aligned
A chest CT encoder for analysis and report retrieval
2 min readupdated
The JECT_320_aligned model is a 3D encoder for chest computed tomography, trained using joint embedding predictive coding (JECT) and aligned with radiological reports. The complete system includes a pre-trained image encoder, a latent-attention pooler, and a text adapter based on chest2vec. The tool enables generating vector embeddings of CT scans, performing report retrieval using images, and calculating zero-shot pathology scores. This development is useful for researchers and medical software developers working on radiological image analysis.
Key features
- Joint training: The image encoder, latent pooler, and text adapter are aligned end-to-end for precise matching between scans and text descriptions.
- Flexible data preparation: Supports NIfTI (.nii.gz) formats and preprocessed cubes (.npz) with multiple lung retrieval modes.
- High accuracy: Tested on the CT-RATE validation dataset using macro-AUROC and Recall@K metrics.
What you can do
- Report retrieval: Search for text radiological descriptions using an uploaded chest CT scan.
- CT embeddings: Extract volume vector representations for classification and pathology analysis.
- Pipeline integration: Use the preprocessing scripts from the repository to automate work with scans.
Example
import torch
from transformers import AutoModel
repo = "lukeingawesome/JECT_320_aligned"
model = AutoModel.from_pretrained(repo, trust_remote_code=True).cuda()
from huggingface_hub import snapshot_download
import sys; sys.path.insert(0, snapshot_download(repo))
from prepare import load_volume
from modeling_ject import load_text_model
volume = load_volume("scan.nii.gz", model.config).cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
out = model(volume, ct_type=["non-contrast"])
text = load_text_model(repo, device="cuda")
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
texts = text(["[Lungs] - A 9 mm solid nodule in the right upper lobe."])
print(model.similarity(out["embedding"].float(), texts.float()))
Read more
DetailsBenchmarks: image-to-report R@1 is 0.330, report-to-image R@1 is 0.359 on the CT-RATE dataset.
Requirements: A graphics card with PyTorch and CUDA support for fast volume processing.
Limitations: The model is trained exclusively on adult patients and English-language reports. It is not a medical device.










Comments