ENHANCING MEDICAL DIAGNOSTICS: A CONTRASTIVE LEARNING APPROACH TO MULTI-MODAL MEDICAL IMAGE SEGMENTATION

Abstract

Modern diagnosis rarely depends on a single image. Clinicians combine magnetic resonance imaging, positron emission tomography, radiographs, and laboratory results, yet most segmentation models are trained on one modality at a time and fail when the modality, scanner, or institution changes. Pooling data across institutions would help, but patient privacy rules make sharing difficult. This paper presents a contrastive learning approach to multi-modal medical image segmentation that addresses these three obstacles together. A fusion stage built on a variational transformer autoencoder combines modalities into local latent representations. A contour-aware generative adversarial network produces synthetic examples whose boundaries follow anatomical contours, and a contrastive loss aligns synthetic and real examples with the local latents so that the representation is consistent across modalities and domains. A differentially private synthetic data generation component provides a formal privacy guarantee when data leave an institution. We evaluate the approach on cross-modality brain tumor segmentation, multi-modal image fusion, and interpretability on two diagnostic classification tasks, Parkinson's disease imaging and melanoma. In cross-modality segmentation, the method reached a Dice score of 0.789, compared with 0.727 for the strongest domain adaptation baseline and 0.836 for fully supervised training, and reduced the 95th percentile Hausdorff distance to 9.1 millimeters. Removing the contrastive loss lowered Dice by 4.5 points, the largest single loss among the ablations, and a privacy budget of 8 cost about 1.2 points. Attention maps from the contrastive model concentrated on the striatum in Parkinson's imaging and on the lesion in melanoma images, in contrast to diffuse attention from single-modal baselines.