Search

The integration of artificial intelligence into clinical decision support systems offers improved diagnostic accuracy and efficiency, but the opacity of many machine learning models raises concerns about trust, accountability, and regulatory compliance. Explainable artificial intelligence (XAI) has been proposed to address this by making model predictions interpretable to clinicians; however, its true clinical value remains uncertain, and evaluation has not kept pace with methodological development. This systematic review aimed to identify XAI methods used in clinical decision support systems, assess how they are evaluated with clinicians, and determine whether explanations improve diagnostic accuracy, trust, mental models, and efficiency. Following PRISMA guidelines, we searched PubMed, Web of Science, IEEE Xplore, ACM Digital Library, and Scopus for studies published between 2017 and 2024. Eligible studies included original research evaluating XAI in clinical decision support systems with clinician participants and reporting quantitative or qualitative outcomes. Risk of bias was assessed using adapted QUADAS-2 and ROBIS tools, and findings were synthesized narratively with subgroup analyses. From 2,847 records, 68 studies were included. The most common XAI methods were SHAP-based feature attribution (38%), saliency or heatmap methods (29%), concept-based approaches such as TCAV (15%), and counterfactual or example-based explanations (12%). Radiology was the dominant field (54%), followed by dermatology (18%) and pathology (12%). Evaluation approaches were highly inconsistent, with few validated instruments and most studies relying on Likert-scale trust measures or qualitative feedback. Only 16% of studies showed improved diagnostic accuracy with explanations, 67% showed no significant effect, and 17% reported reduced accuracy due to over-reliance or misinterpretation. Although 82% of studies reported increased clinician trust, trust rarely correlated with actual diagnostic performance. Overall, while XAI methods are widely studied in clinical decision support, their evaluation is inconsistent and their benefits are limited. Explanations tend to increase clinician trust without reliably improving diagnostic accuracy, and may sometimes worsen performance, highlighting a trust–accuracy gap that poses important safety concerns for clinical deployment.

Prostate cancer metastasis to bone and lymph nodes marks a critical transition to incurable disease, with five-year survival dropping dramatically compared to localized disease. Early identification of patients at high risk of metastasis enables timely intensification of treatment, including androgen deprivation therapy, salvage radiation, or systemic therapies. Current deep survival models that integrate serial PSA measurements and genomic risk scores achieve high predictive accuracy for time-to-metastasis but operate as black boxes, providing no explanation for why a particular patient is predicted to have early or late metastasis. Clinicians cannot trust or act upon predictions without understanding which PSA features or genomic markers drive the risk assessment. We present an explainable deep survival framework that combines a deep survival model for time-to-metastasis prediction with Integrated Gradients attribution, a method that distributes the model's hazard prediction among input features. The framework produces patient-specific explanations showing how each serial PSA value and each genomic score component contributes to the predicted metastasis hazard. The framework consists of three core components: (1) a deep survival model (DeepSurv architecture) with a PSA time-series encoder and genomic risk encoder, (2) Integrated Gradients attribution computed over the hazard function, and (3) visualization tools for individual and population-level interpretations. Integrated Gradients attributes the predicted hazard to individual PSA measurements across time and specific genomic markers, enabling clinicians to distinguish between risk driven by rapid PSA kinetics versus high genomic risk scores. This interpretability transforms a black-box survival prediction into an actionable clinical decision support tool.