Search

Transformer-based architectures have significantly advanced clinical natural language processing by improving the capture of contextual relationships in unstructured electronic health records compared to earlier recurrent and convolutional models, with domain-specific variants such as ClinicalBERT and BioBERT designed to better handle clinical terminology, abbreviations, and specialized language, thereby improving information extraction performance, although the relative impact of different pre-training strategies remains insufficiently synthesized and requires systematic evaluation of corpus selection and fine-tuning approaches; this systematic review mapped studies focusing on pre-training corpora, fine-tuning methods, and named entity recognition performance across entity types such as medications, diseases, procedures, laboratory tests, and social determinants of health, using PRISMA-guided methods and searches across PubMed, ACL Anthology, arXiv, and IEEE Xplore, identifying 32 eligible studies from 1,247 records; findings showed that ClinicalBERT, BioBERT, and PubMedBERT were the most frequently evaluated models, pre-trained on datasets such as MIMIC-III, PubMed abstracts, and mixed biomedical corpora, with consistent evidence that domain-specific pre-training outperforms general-domain BERT models on benchmarks like i2b2 and n2c2 despite variation across entity types and fine-tuning strategies, while clinical pre-training on large EHR corpora improves named entity recognition and optimized fine-tuning approaches such as lower learning rates and data augmentation further enhance performance, particularly for medications and diseases, underscoring the importance of domain adaptation and the need for more standardized evaluation protocols in clinical NLP research.

Rare diseases collectively affect over 300 million people globally, yet individual conditions are often missed due to low clinician familiarity and non-specific presenting symptoms that mimic common disorders. Supervised machine learning requires large numbers of labeled examples for training, but rare diseases have too few diagnosed cases to develop condition-specific predictive models using traditional approaches. We propose a multimodal foundation model pretrained on 10 million de-identified electronic health records (EHRs) combining clinical notes and laboratory values for zero-shot rare disease diagnosis without requiring labeled training examples. The framework comprises four components: a clinical note encoder based on a large language model, a laboratory value encoder using a time-series transformer, a multimodal fusion module with cross-attention, and a zero-shot classifier that compares patient embeddings to disease descriptions. Pretraining on large-scale EHR data enables the model to learn general medical knowledge and disease patterns, allowing diagnosis of rare conditions by recognizing manifestations even when no labeled examples of that specific disease were used for training.

Clinical notes, problem lists, medication orders, and billing codes are core components of the electronic health record. When these components conflict, the record may become less reliable for care delivery, quality measurement, and reimbursement. Current inconsistency detection is largely manual, episodic, and dependent on documentation audits. This approach is difficult to scale across encounters, specialties, and longitudinal records. This article proposes a deep learning NLP model for detecting contradictions among clinical notes, problem lists, medication orders, and billing codes. The goal is to support continuous documentation integrity surveillance. The proposed model uses transformer-based encoders for clinical text and embedding layers for structured coded fields. Cross-attention mechanisms align concepts across EHR modules before classifying consistency relationships. Conceptually, the model could surface discrepancies such as a diagnosis documented in a note but absent from the problem list, or a billing code unsupported by physician documentation. Its output would include an inconsistency category and an interpretable explanation for clinician review. A unified NLP model for cross-module inconsistency detection could improve EHR trustworthiness, documentation quality, and clinical audit workflows. Such a system should be evaluated prospectively before operational deployment.