Search

The integration of artificial intelligence (AI) into healthcare systems marked a pivotal evolution in clinical analytics architectures and governance structures, transforming data-driven decision-making from siloed, retrospective analyses to dynamic, predictive, and integrated frameworks. This period witnessed rapid advancements in machine learning (ML) applications for healthcare infrastructure, encompassing electronic health records (EHRs), imaging diagnostics, population health management, and real-time monitoring systems. Key developments included the shift toward federated learning to address data privacy concerns, the emergence of explainable AI (XAI) to enhance clinical trustworthiness, and the standardization of regulatory pathways for AI as medical devices. Architecturally, healthcare systems evolved from static analytics pipelines—where data ingestion, model training, and inference occurred in isolated phases—to adaptive, closed-loop configurations that incorporate feedback mechanisms for continuous model refinement and human-AI collaboration. Governance structures are adapted accordingly, emphasizing ethical frameworks to mitigate bias, ensure data equity, and promote algorithmic accountability, particularly for underserved populations. This review synthesizes literature from this timeframe, highlighting how AI-enabled analytics architectures facilitated precision medicine by integrating multimodal data sources, such as genomics, wearables, and social determinants of health, into cohesive systems. Challenges in interoperability and scalability were addressed through consensus guidelines like CONSORT-AI and SPIRIT-AI, which promoted transparent reporting of AI interventions in clinical trials. Moreover, the COVID-19 pandemic accelerated AI deployment in pandemic response systems, underscoring the need for resilient architectures capable of handling real-time data surges and uncertainty communication. Governance evolved to include multi-stakeholder perspectives, from regulatory bodies such as the FDA to clinical practitioners, ensuring that AI tools align with evidence-based medicine. This narrative review provides an original systems-level framing, organizing the literature around data-to-decision cycles, infrastructural integration, and governance maturation. By examining cross-study insights, it reveals how AI has fostered intelligent healthcare ecosystems, reducing diagnostic bias across diverse cohorts and enhancing decision support without over-relying on black-box models. Ultimately, this synthesis underscores the transition from AI as a supplementary tool to a foundational element of healthcare systems, paving the way for equitable, efficient clinical analytics.

The integration of artificial intelligence into clinical decision support systems offers improved diagnostic accuracy and efficiency, but the opacity of many machine learning models raises concerns about trust, accountability, and regulatory compliance. Explainable artificial intelligence (XAI) has been proposed to address this by making model predictions interpretable to clinicians; however, its true clinical value remains uncertain, and evaluation has not kept pace with methodological development. This systematic review aimed to identify XAI methods used in clinical decision support systems, assess how they are evaluated with clinicians, and determine whether explanations improve diagnostic accuracy, trust, mental models, and efficiency. Following PRISMA guidelines, we searched PubMed, Web of Science, IEEE Xplore, ACM Digital Library, and Scopus for studies published between 2017 and 2024. Eligible studies included original research evaluating XAI in clinical decision support systems with clinician participants and reporting quantitative or qualitative outcomes. Risk of bias was assessed using adapted QUADAS-2 and ROBIS tools, and findings were synthesized narratively with subgroup analyses. From 2,847 records, 68 studies were included. The most common XAI methods were SHAP-based feature attribution (38%), saliency or heatmap methods (29%), concept-based approaches such as TCAV (15%), and counterfactual or example-based explanations (12%). Radiology was the dominant field (54%), followed by dermatology (18%) and pathology (12%). Evaluation approaches were highly inconsistent, with few validated instruments and most studies relying on Likert-scale trust measures or qualitative feedback. Only 16% of studies showed improved diagnostic accuracy with explanations, 67% showed no significant effect, and 17% reported reduced accuracy due to over-reliance or misinterpretation. Although 82% of studies reported increased clinician trust, trust rarely correlated with actual diagnostic performance. Overall, while XAI methods are widely studied in clinical decision support, their evaluation is inconsistent and their benefits are limited. Explanations tend to increase clinician trust without reliably improving diagnostic accuracy, and may sometimes worsen performance, highlighting a trust–accuracy gap that poses important safety concerns for clinical deployment.

Large language models (LLMs) have rapidly advanced since the transformer architecture was introduced in 2017, with systems such as GPT-3, GPT-4, Med-PaLM, and Claude increasingly explored for applications in medical education, clinical documentation, decision support, and patient communication, raising both optimism and concerns regarding safety and reliability. This systematic review synthesizes evidence across studies retrieved from PubMed, arXiv, ACL Anthology, IEEE Xplore, and Google Scholar that empirically evaluated LLMs in clinical settings using quantitative performance metrics, with risk of bias assessed using an adapted PROBAST framework for machine learning research. Findings show that LLMs achieve 60–90% accuracy on USMLE-style examinations, with leading models such as GPT-4 and Med-PaLM 2 reaching or surpassing passing thresholds, while in clinical documentation tasks they can reduce physician workload by approximately 30–50% in generating outputs such as discharge summaries, though human review remains consistently required. Performance in clinical decision support is more variable and specialty-dependent, and hallucination rates ranging from 5–30% have been reported, alongside persistent issues of bias and overconfidence in incorrect outputs. Overall, while LLMs demonstrate strong capabilities in structured medical knowledge tasks and documentation support, current limitations including hallucinations, bias, and lack of prospective clinical validation prevent safe autonomous deployment, making clinician oversight and robust safety safeguards essential for any clinical use.