Search

Rare dermatological conditions (or orphan diseases) present major diagnostic challenges due to their low prevalence, limited clinician exposure, and the scarcity of well-labeled datasets, which together hinder the development of conventional AI systems. As a result, most deep learning models trained on supervised approaches perform well only on common skin diseases while failing to generalize to rare conditions, leaving a significant gap in clinical support and contributing to delayed diagnoses and worse patient outcomes, especially in regions with limited specialist access. To address this limitation, contrastive language-image pre-training offers a promising alternative by leveraging paired dermatological images and unstructured clinical notes from electronic health records in a self-supervised manner. This allows models to learn meaningful visual–textual relationships without requiring large-scale manual annotation. The framework typically includes an image encoder, a clinical text encoder, a contrastive alignment objective, and a zero-shot classification mechanism based on prompt similarity. By learning from existing multimodal clinical data, such systems can generalize to previously unseen rare conditions and enable zero-shot diagnosis, reducing dependence on labeled datasets. This approach transforms routine physician documentation into a rich supervisory signal, helping overcome annotation bottlenecks and improving AI applicability in real-world dermatology settings. Ultimately, foundation models trained in this way offer a scalable path toward more inclusive and effective AI-assisted diagnosis of rare skin diseases.

Rare diseases collectively affect over 300 million people globally, yet individual conditions are often missed due to low clinician familiarity and non-specific presenting symptoms that mimic common disorders. Supervised machine learning requires large numbers of labeled examples for training, but rare diseases have too few diagnosed cases to develop condition-specific predictive models using traditional approaches. We propose a multimodal foundation model pretrained on 10 million de-identified electronic health records (EHRs) combining clinical notes and laboratory values for zero-shot rare disease diagnosis without requiring labeled training examples. The framework comprises four components: a clinical note encoder based on a large language model, a laboratory value encoder using a time-series transformer, a multimodal fusion module with cross-attention, and a zero-shot classifier that compares patient embeddings to disease descriptions. Pretraining on large-scale EHR data enables the model to learn general medical knowledge and disease patterns, allowing diagnosis of rare conditions by recognizing manifestations even when no labeled examples of that specific disease were used for training.