Transformer-based architectures have significantly advanced clinical natural language processing by improving the capture of contextual relationships in unstructured electronic health records compared to earlier recurrent and convolutional models, with domain-specific variants such as ClinicalBERT and BioBERT designed to better handle clinical terminology, abbreviations, and specialized language, thereby improving information extraction performance, although the relative impact of different pre-training strategies remains insufficiently synthesized and requires systematic evaluation of corpus selection and fine-tuning approaches; this systematic review mapped studies focusing on pre-training corpora, fine-tuning methods, and named entity recognition performance across entity types such as medications, diseases, procedures, laboratory tests, and social determinants of health, using PRISMA-guided methods and searches across PubMed, ACL Anthology, arXiv, and IEEE Xplore, identifying 32 eligible studies from 1,247 records; findings showed that ClinicalBERT, BioBERT, and PubMedBERT were the most frequently evaluated models, pre-trained on datasets such as MIMIC-III, PubMed abstracts, and mixed biomedical corpora, with consistent evidence that domain-specific pre-training outperforms general-domain BERT models on benchmarks like i2b2 and n2c2 despite variation across entity types and fine-tuning strategies, while clinical pre-training on large EHR corpora improves named entity recognition and optimized fine-tuning approaches such as lower learning rates and data augmentation further enhance performance, particularly for medications and diseases, underscoring the importance of domain adaptation and the need for more standardized evaluation protocols in clinical NLP research.
Hospitals generate dense streams of timestamped operational events, including orders, transfers, staff actions, queue changes, and system interactions. These events describe how care actually unfolds, yet much of their value remains unused because they are rarely labeled for prediction tasks. Existing operational predictive models often depend on task-specific labels, handcrafted features, and local workflow assumptions. This limits their ability to scale across hospitals, departments, and evolving operational conditions. This manuscript designs a self-supervised representation learning model that pre-trains on diverse healthcare operations event streams. The goal is to learn a generalizable embedding of hospital operational state that can be adapted to multiple downstream prediction tasks. The proposed model uses a transformer-based architecture trained with masked event modeling and temporal contrastive learning. Timestamped orders, transfers, staff actions, queue transitions, system interaction logs, and unit-level workflow signals are represented as time-aware event sequences, and the pre-trained backbone is later fine-tuned for specific operational tasks. Conceptually, the model could learn semantic and temporal regularities of hospital workflow, such as common discharge sequences, clustered STAT order activity, and operational precursors to bottlenecks. These representations would be expected to support downstream tasks such as delay forecasting, anomaly detection, and resource demand estimation when labeled data are limited. Self-supervised learning could unlock the latent value of healthcare operations logs by creating reusable representations of hospital workflow. Such a model could become a foundation for operational analytics, enabling faster and more adaptable development of predictive tools.