Introduction
Medical imaging plays an important role in clinical diagnosis and decision-making. Chest X-rays, for example, contain visual evidence that can be interpreted alongside radiology reports describing anatomical findings and potential abnormalities.
Developing machine learning models that understand both medical images and clinical text can support tasks such as image classification, medical image retrieval, visual question answering, and automated report generation.
However, collecting large-scale medical datasets with expert annotations is expensive and time-consuming. Vision-language pretraining offers a way to learn useful representations from existing image-report pairs without requiring every image to have extensive additional annotations.
Many vision-language models follow a dual-encoder architecture. An image encoder and a text encoder independently process their respective inputs, and contrastive learning encourages the representations of matching image-text pairs to become similar.
Other approaches use fusion encoders that allow visual and textual information to interact directly through attention mechanisms. These architectures can capture richer relationships between modalities and support more complex reasoning tasks.
Despite these advances, many existing approaches primarily focus on understanding and aligning information rather than generating visual content from text. Some methods incorporate masked image modeling, but this does not necessarily provide a complete mechanism for generating coherent visual representations from textual prompts.
Medical images also contain subtle visual features that may be important for interpretation, including small nodules and slight opacities. Learning representations that preserve these details is an important challenge.
MedUnifier addresses these limitations by integrating text-grounded image generation into a unified vision-language pretraining framework. Instead of treating image generation as a separate downstream task, it incorporates visual generation directly into the pretraining process.
Methodology
1. MedUnifier Architecture
MedUnifier consists of three principal components:
An image-text encoder for learning multimodal representations.
A text generator for generating clinical reports from visual information.
An image generator for reconstructing or generating images using discrete visual representations.
The framework uses learnable embeddings to facilitate interaction between visual and textual information. Its image-text encoder is based on a 12-layer Transformer architecture, while a pretrained Vision Transformer (ViT) extracts preliminary visual features.
The visual encoder is kept frozen during training to reduce computational costs and preserve the pretrained visual representations. Learnable embeddings and clinical report tokens are processed through Transformer layers that use different attention-masking strategies depending on the learning objective.
The architecture is trained jointly using four complementary objectives. These objectives encourage the model to align image and text representations, identify matching pairs, generate reports from images, and generate images from multimodal information.
2. Image-Text Contrastive Learning (ITC)
Image-text contrastive learning encourages the model to place matching medical images and clinical reports close to one another in the learned representation space.
The model extracts visual and textual representations and projects them into a shared embedding space. It then compares matching pairs against other image-text combinations within the training batch.
The contrastive objective encourages the similarity of matching pairs to increase relative to mismatched pairs.
This allows the model to learn broad associations between medical visual patterns and the language used to describe them.
For example, a chest X-ray and a report describing its major findings should have compatible representations, while unrelated images and reports should be distinguishable.
ITC provides a foundation for cross-modal retrieval and other tasks that require meaningful alignment between images and text.
3. Image-Text Matching (ITM)
While contrastive learning focuses on representation similarity, image-text matching directly predicts whether an image and a report correspond to one another.
The model uses bidirectional attention to allow visual-related learnable embeddings and text tokens to interact. The resulting multimodal representation is passed to a classifier that predicts whether the pair is matched.
This objective encourages the model to learn finer-grained relationships between image content and textual descriptions.
Together, ITC and ITM support both broad cross-modal alignment and more detailed matching between medical images and their associated reports.
4. Image-Grounded Text Generation (ITG)
Image-grounded text generation enables MedUnifier to generate clinical descriptions conditioned on medical images.
The framework uses a causal language modeling objective. During generation, each word is predicted based on the image-conditioned representation and the preceding text tokens.
A dedicated decoding mechanism generates the report one token at a time.
This objective encourages the model to learn relationships between visual findings and the language used to describe them in clinical reports.
It also provides a generative capability within the same framework used for multimodal representation learning, rather than requiring a completely separate language model to be introduced afterward.
5. Text-Grounded Image Generation (TIG)
The principal innovation of MedUnifier is its Text-grounded Image Generation module.
TIG enables the model to reconstruct and generate visual content using information learned from images and corresponding reports. Its design addresses the challenge of preserving detailed visual information while incorporating the semantic information contained in text.
The module uses a vector-quantized variational autoencoder (VQ-VAE) to learn discrete visual representations.
Unlike approaches that rely entirely on continuous visual embeddings, vector quantization maps visual features to entries in a learned discrete representation space. This provides a structured representation that can be used for image reconstruction and generation.
The researchers introduce hierarchical latent representations and adapters to connect the multimodal encoder with the image generation module.
The framework combines two levels of representation:
Top-level representations capture more abstract multimodal information, including textual implications learned through image-text interaction.
Bottom-level representations retain local visual information that supports the reconstruction of fine-grained image details.
Latent adapters transform these representations into spatial feature maps, which are then quantized into discrete representations. A hierarchical decoder reconstructs the image from the resulting representations.
This design allows the generation process to use both high-level semantic information and lower-level visual details.
6. Joint Optimization
MedUnifier combines the four objectives into a unified pretraining loss:
Image-text contrastive learning (ITC).
Image-text matching (ITM).
Image-grounded text generation (ITG).
Text-grounded image generation (TIG).
Each objective contributes different information to the learned representations.
ITC improves image-text alignment, ITM strengthens matching accuracy, ITG learns to generate clinical descriptions, and TIG encourages the model to retain and reconstruct detailed visual information.
The researchers train the objectives jointly so that the model can benefit from the complementary information learned through each task.
This approach aims to create a more versatile medical vision-language model without requiring a separate image tokenizer pretraining stage or an iterative diffusion-based image generation process.
Experimental Evaluation
The researchers evaluate MedUnifier on medical vision-language benchmarks, focusing on chest X-ray datasets and three categories of downstream tasks: unimodal, cross-modal, and multimodal.
The experiments compare MedUnifier with existing methods for medical image representation learning, image-text retrieval, report generation, and medical image generation.
1. Unimodal Tasks
Unimodal evaluation examines the quality of the visual representations learned by the model.
The researchers evaluate image classification performance on datasets including RSNA Pneumonia, SIIM-ACR, and COVIDx. They also investigate how performance changes when only a limited proportion of training data is available.
The reported experiments show improvements over the compared methods across multiple classification settings. These results suggest that the combination of multimodal objectives and visual generation can improve the usefulness of learned representations for downstream medical image tasks.
The study also examines data efficiency by fine-tuning models with different proportions of training data, including 1%, 10%, and 100% of the available training samples.
2. Cross-Modal Tasks
Cross-modal evaluation investigates how effectively the model connects medical images with their corresponding textual descriptions.
The researchers evaluate image-text retrieval and zero-shot image classification using medical datasets, including MIMIC-CXR and RSNA.
For retrieval, the model must identify relevant image-text pairs from a collection of candidates. For zero-shot classification, it must use learned image-text relationships to distinguish categories without relying exclusively on task-specific training.
The reported results demonstrate improvements over the compared approaches on the evaluated tasks.
These findings support the use of a unified architecture that learns image-text alignment alongside generative objectives.
3. Multimodal Tasks
The multimodal experiments examine the model's ability to generate clinical reports from medical images.
MedUnifier is evaluated on natural language generation metrics, including BLEU, METEOR, and ROUGE-L.
The paper reports that MedUnifier achieves the best performance across the evaluated metrics in its comparison table.
These results indicate that the learned visual and textual representations can support report generation, in addition to classification and cross-modal retrieval.
4. Medical Image Generation
The researchers also evaluate the quality of reconstructed and synthetic medical images.
Image generation is assessed using the Fréchet Inception Distance (FID). The study examines both image reconstruction and the generation of synthetic samples from learned latent representations.
The authors report that reconstructed visual samples are nearly indistinguishable from authentic radiographs under direct visual inspection. They also report that synthetic samples generated from multimodal priors exhibit diversity, suggesting potential for augmenting datasets that differ from the original training distribution.
These findings demonstrate the framework's ability to combine medical vision-language representation learning with visual generation.
However, the quality of synthetic medical images and their suitability for clinical use require careful validation beyond visual similarity or generative benchmark metrics.
Ablation Study
The researchers conduct ablation experiments to investigate the contribution of each learning objective.
The experiments compare configurations that progressively introduce ITC, ITM, ITG, and TIG.
The model trained with image-text contrastive learning alone achieves the lowest performance among the evaluated configurations. Adding image-text matching and image-grounded text generation improves the results, demonstrating the value of richer cross-modal interactions and language generation.
The experiments also show that incorporating TIG can outperform a configuration using ITG without TIG on some evaluated tasks.
The complete model, which combines all four learning objectives, achieves the strongest overall results in the reported ablation study.
These findings support the central idea of MedUnifier: visual generation can contribute useful learning signals to a medical vision-language model, complementing the objectives traditionally used for multimodal understanding.
Discussion
MedUnifier expands medical vision-language pretraining beyond feature extraction and cross-modal alignment by incorporating visual generation into the same learning framework.
Its central contribution is the integration of discrete visual representations with multimodal learning. The TIG module allows the model to use textual information while reconstructing fine-grained visual details, while hierarchical latent representations provide a mechanism for combining abstract semantic information with local image features.
The four learning objectives serve complementary purposes. Contrastive learning aligns images and text, matching improves pair-level discrimination, report generation connects visual findings with clinical language, and image generation encourages the preservation of visual information.
This unified design can be useful for research involving medical image classification, image-text retrieval, report generation, and synthetic medical image generation.
Nevertheless, strong benchmark performance does not establish clinical reliability. Generated reports may omit findings or introduce inaccurate details, and synthetic images may not faithfully represent the full range of real-world pathology. Further evaluation is necessary before such systems can be relied upon in clinical workflows.
Conclusion
MedUnifier introduces a unified medical vision-language pretraining framework that combines multimodal understanding with text-grounded image generation.
The architecture integrates image-text contrastive learning, image-text matching, image-grounded text generation, and text-grounded image generation. Through vector quantization and hierarchical latent adapters, the model learns discrete visual representations that support image reconstruction while incorporating multimodal information.
The reported experiments demonstrate improvements across evaluated unimodal, cross-modal, and multimodal tasks. The ablation study further supports the contribution of incorporating visual generation into the pretraining framework.
Overall, this research presents an approach to building more versatile medical vision-language models by combining visual understanding, clinical language generation, and image generation within a single training framework.
Research Publication
Conference: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2025).