Search

The integration of artificial intelligence into clinical decision support systems offers improved diagnostic accuracy and efficiency, but the opacity of many machine learning models raises concerns about trust, accountability, and regulatory compliance. Explainable artificial intelligence (XAI) has been proposed to address this by making model predictions interpretable to clinicians; however, its true clinical value remains uncertain, and evaluation has not kept pace with methodological development. This systematic review aimed to identify XAI methods used in clinical decision support systems, assess how they are evaluated with clinicians, and determine whether explanations improve diagnostic accuracy, trust, mental models, and efficiency. Following PRISMA guidelines, we searched PubMed, Web of Science, IEEE Xplore, ACM Digital Library, and Scopus for studies published between 2017 and 2024. Eligible studies included original research evaluating XAI in clinical decision support systems with clinician participants and reporting quantitative or qualitative outcomes. Risk of bias was assessed using adapted QUADAS-2 and ROBIS tools, and findings were synthesized narratively with subgroup analyses. From 2,847 records, 68 studies were included. The most common XAI methods were SHAP-based feature attribution (38%), saliency or heatmap methods (29%), concept-based approaches such as TCAV (15%), and counterfactual or example-based explanations (12%). Radiology was the dominant field (54%), followed by dermatology (18%) and pathology (12%). Evaluation approaches were highly inconsistent, with few validated instruments and most studies relying on Likert-scale trust measures or qualitative feedback. Only 16% of studies showed improved diagnostic accuracy with explanations, 67% showed no significant effect, and 17% reported reduced accuracy due to over-reliance or misinterpretation. Although 82% of studies reported increased clinician trust, trust rarely correlated with actual diagnostic performance. Overall, while XAI methods are widely studied in clinical decision support, their evaluation is inconsistent and their benefits are limited. Explanations tend to increase clinician trust without reliably improving diagnostic accuracy, and may sometimes worsen performance, highlighting a trust–accuracy gap that poses important safety concerns for clinical deployment.

Case management programs are intended to reduce avoidable utilization and improve coordination for patients with complex medical, social, and engagement needs. Because case management capacity is limited, health systems need prioritization tools that are both clinically sensible and transparent. Existing referral methods often rely on clinician judgment, simple utilization thresholds, or proprietary risk scores that provide limited explanation. These approaches may overlook social needs, missed appointments, and care gaps that shape patient complexity and influence whether an intervention is feasible. This article proposes an explainable machine learning model that stratifies patients by risk of future high utilization and provides patient-specific reasoning. The model is designed around prior utilization, chronic disease burden, social needs documentation, missed appointments, and care gap indicators. The conceptual architecture uses a gradient-boosted classification model with a SHAP-based post-hoc explanation layer. The model would output both a risk score and a ranked list of contributing factors for each patient considered for case management referral. Conceptually, the model would identify patients who may benefit from case management and explain why each patient was prioritized. These explanations could help case managers tailor outreach, match patients to intervention pathways, and distinguish medical complexity from social instability or disengagement. An explainable risk stratification model could turn a blind referral process into a transparent, clinically sensible prioritization workflow. Its value would depend on careful implementation, fairness monitoring, and alignment with real case manager decision-making.

Prior authorization delays impede timely patient care and contribute to administrative pressure across clinical and revenue cycle workflows. These delays can affect scheduling, medication access, procedural planning, and patient confidence in the care process. Current authorization management tools are largely reactive and often focus on tracking request status after submission. They rarely predict which requests are likely to experience approval delays or explain the operational, clinical, or payer-specific reasons behind those delays. This article proposes an interpretable machine learning model for predicting the likelihood of prior authorization approval delays. The model is designed to provide transparent, request-level explanations that can guide pre-submission correction and authorization preparation. The proposed framework uses a gradient-boosted tree model trained conceptually on historical authorization requests. Inputs include payer-specific rules, clinical documentation features, procedure type, medical necessity indicators, and historical approval timelines, with SHAP used to attribute predicted delay risk to individual features. Conceptually, the model would output a delay probability and an explanation of the dominant drivers of that prediction. These drivers could include incomplete documentation, mismatch with payer medical necessity criteria, procedure categories associated with additional review, or payer-procedure combinations with historically slow turnaround. An interpretable prior authorization delay model could support earlier correction of incomplete requests, reduce administrative waste, and improve patient access. By aligning predictive analytics with transparent explanations, the framework could make authorization preparation more proactive and accountable.