Rule of Thumb: Explaining Artificial Intelligence Systems using Partial Information

1. Introduction

Artificial intelligence (AI) in the form of large language models has become ubiquitous. One of the key components that has made it disruptive and widespread is its ability to perform zero-shot tasks. Given an applicant’s resume, it can say if they are a good fit for a job, or identify an animal in a photo. But does it work? As it stands, the only way to find out if a large language model (LLM) can perform a task, is to label a significant subset of data, and measure how consistent the LLM is with these labels. This data labeling process substantially decreases the disruptive nature of AI. While the AI itself may be zero-shot, the process of deciding if we should deploy it is not. Explainable AI (XAI) has long been promised as a tool to help with this kind of deployment decision. Better understanding of how, and in some cases, why AIs make decisions can increase confidence in deployment and help users identify and correct errors [1, 2].

However, precisely when you would expect XAI to be making a resurgence, it is nowhere to be seen. In no small part, this is because of fundamental limitations of the methods. Existing Explainable AI methods try to either peer inside the algorithmic black-box and use access to the internals of models (grad-cam [4], integrated gradients [5], counterfactuals [6], or mechanistic interpretability [7]) which is inapplicable to many widely used closed source LLMs that only allow access via an application programming interface (API), or are combinatorially enumerative, and rely on passing a large amount of synthetic data to the model (SHAP [8], LIME [1], Counterfactual explanations [6] or Recourse [9]), which is either prohibitively expensive or simply infeasible.

---

Feature importance explanations, such as those from LIME or SHAP, are post-hoc explanations of individual predictions in the form of a signed vector of scores, denoting the importance of each input feature to the prediction1. Prior techniques have been variants of sensitivity analysis, operationalising the intuition that important input features are those which, upon synthetic perturbation, cause the greatest changes in model outputs. We focus on post-hoc, local (per-datapoint) feature importance explanations because they are a popular and easy to use form of XAI, do not require retraining AI systems, and can be implemented separately from the predictions they seek to explain.

(a) Rule of Thumb (RoT) explanation for a datapoint. Glucose and age (in blue) contribute most to reducing the predicted probability

(b) SHAP explanation for the same datapoint. Glucose (in blue) is still most important in reducing output probability, but here age (in red) increases it

2. Rule-of-Thumb: Explanations for Emerging XAI Applications

We propose “Rule-of-Thumb” (RoT), a novel form of XAI that considers important inputs to be those that are most predictive of the outputs of an AI system. Rather than formulating XAI in terms of how altering an input value would alter the system output, we ask how knowledge of particular feature values should alter our predictions of the AI system’s behaviour.

The outputs from RoT are standard feature importance explanations, which can be analysed using existing visualisations such as pixel or token level saliency maps that highlight positive or negative contributions towards an output (Figures 1 and 2). These figures show the unique ability to use RoT even when model access is available only via API, where other explainers are unviable2. For tabular data where model access is common, we are able to compare with explainers like SHAP and LIME using standard force-plots for a single datapoint (Figure 3) or swarmplots for a collection of datapoints (Figure 7).

In these latter comparisons, we note that applying sensitivity analysis implicitly requires several nontrivial assumptions: (i) that it is possible to query the AI system with new datapoints; (ii) the system responds similarly to synthetically perturbed data in the same way it does to real-world data, and (iii) The outputs of a system are continuous, and typically vary when a limited number of features are changed3. RoT makes none of these assumptions, conferring specific advantages in the following three emerging XAI application areas:

1. Explaining Zero-shot Classification with LLMs: Classification with LLMs presents two practical challenges. First, weights and gradients are inaccessible for closed systems accessed via API, and second, even when the AI system is open-source, producing each explanation requires computing many additional LLM inferences, making explanations too cumbersome and slow to compute. We experiment with a movie sentiment prediction task and a judicial case outcome prediction task to reflect each of these settings. In both cases, we find RoT explanations are aligned with expected explanations produced by human annotators. We also find RoT feasible to compute quickly on consumer hardware, and as RoT explanations do not require the training a model for each datapoint explained, our computational benefits increase with the number of data points explained. When explaining an open-weight LLM, asymptotically, 13 million new RoT explanations can be computed in the time taken for a single additional SHAP explanation. We also conduct an additional experiment filtering resume texts, where we demonstrate the unique RoT ability to explain predictions in terms of non-input features.

2. Auditing Proprietary Models: Auditing AI systems such as e-commerce recommendation engines can be challenging because access to the underlying AI is often restricted. A standard workaround is to first construct an additional mimic model to replicate the AI system, and then explain the mimic model in turn as a proxy for the restricted AI system itself. We go on to show that explanations from this commonly employed workaround can be contested by varying the mimic model used; whereas RoT directly provides explanations without depending on mimic models.

3. Scientific Discovery using XAI: We identify an emerging strategy to generate scientific hypotheses, which first collects data about a phenomenon of interest, then trains an AI system using the data, and finally explains its predictions in order to provide insights into the phenomenon itself. Using sensitivity analysis based explainers here is challenging because sometimes they generate incorrect hypotheses. We analyse datasets about diabetes diagnoses, criminal justice, and lending; comparing hypotheses from RoT, SHAP, and LIME explanations. Through a carefully constructed scenario where a specific input feature is unpredictive of model behaviour, and another where a specific input feature is predictive, we demonstrate that only RoT explanations correctly identify these features as unimportant and important respectively.

2.1 Formulation

We wish to determine which features are most informative in estimating a given prediction from an AI system. Using additive functions that model the AI system outputs from individual features, we can both estimate the prediction and determine the relative importance of each feature value.

We make some simplifying assumptions to answer this efficiently. Given a model C(·) defined over features J, and dataset X, we want to find the optimal set of additive functions such that for any subset of features J ⊆ J and datapoint x ∈ X we estimate C(x) using their sum.

where F is a sigmoid function (for classification) or an identity function (for regression), f_θj(xj) is a learnt function representing the information provided by knowing that the j^th feature takes value xj, and G a global bias term that indicates what prediction should be made without knowing anything about the datapoint x. We refer to f_θj(xj) as the importance of a feature j taking value xj. This corresponds to how much a prediction of C(x) should update, on average, after discovering this information.

Given a random subset of features selected using uniform probability p, we can approximate C(x) as a the sum of importances. This idea of always having a rough estimate given any amount of limited information gives rise to the name Rule of Thumb. To find the optimal weights θj ∀j ∈ J for our model, we minimize the following objective L.

Here the term ℓ[C(x), F(∑_j∈Jf_θj(xj) + G)] is a standard training loss such as log loss or squared error that penalizes mispredictions of C(x). This loss is not only marginalized over all datapoints, as is common in standard machine learning, but also all possible masks of the data. Although computing L exactly is expensive and requires computing the standard loss over the entire dataset 2^|J| times, it can be efficiently optimized using dropout[14] applied to the feature importances f_θj(xj). Using dropout, the training time for fitting is almost the same as fitting the simple additive form of equation (1) that does not consider subsets of features. After fitting the model, the feature j with the largest absolute value |f_θj(xj)| is the one that should most change our confidence in the prediction, and can be considered the most important input feature.

Comparison with existing methods

Gradient-based methods[4, 11, 12] are computationally efficient but require direct access to model weights, which are often restricted. With LLMs ,it is common to provide access only via API access. Gradient-based approaches cannot be applied to any model that only offers API access.

Sensitivity-based approaches LIME and SHAP optimise a per-datapoint objective. While RoT masks information and fits one model for the entire dataset, these approaches peturb datapoints and fit one linear model per datapoint. This creates substantial differences in applicability – not only is sensitivity analysis badly suited for analyzing opaque AI systems that only return yes/no decisions (such as those provided by an LLM API), but it also requires many times (typically 500 for SHAP to 5000 for LIME) more inferences or API calls to explain each prediction. Finally, by definition, sensitivity analysis can explain AI systems only in terms of their inputs. RoT can utilise non-input features by fitting on extended datasets if required. This enables explaining model predictions using external models such as MobileNet[15] or BERT[16] to generate more informative explanations. See Appendix A for additional details situating RoT with respect to prior XAI approaches.

3. Experiments

Explanations obtained through sensitivity-based XAI techniques often serve as an aid for users and developers in understanding and debugging model behaviour. However, they fall short in key emerging application areas: zero shot classification with LLMs, auditing black-box AI, and scientific discovery using AI. RoT confers key advantages here: it can be used even if models are slow, expensive, accessible only via API, or inaccessible for additional inferences entirely. It can generate model explanations using input and non-input features, allowing explanations to supplement inputs with other data to explain model behaviour. Finally, RoT makes no causal assumptions and is invulnerable to adversarial attacks that make other explainers ill-suited to proposing scientific hypotheses or auditing black-box systems.

Compared to SHAP, RoT reveals how informative it is to know on average, that a feature takes a certain value, while SHAP reveals the average change that occurs upon altering a feature to a certain value from its mean. This is valuable because it allows us to visualise RoT explanations using the same familiar plots as SHAP explanations (eg. the force-plots in Figure 3). This relationship makes SHAP a natural basis for experimental comparisons with RoT.

3.1 Model Evaluation and Zero-Shot Classification with LLMs

LLMs are often used for zero-shot classification when there is no labelled dataset to train a classifier model. For each datapoint, an LLM is prompted to output a classification. Since there are no target labels to compare predictions with, typical model performance metrics such as accuracy, precision or recall cannot be readily computed. XAI offers a complementary path to performance evaluation, allowing us to inspect and verify that the “correct” features are predictive of the model’s behaviour.

A key challenge is that new predictions can be difficult to obtain for arbitrarily modified model inputs. When using commercial models, this manifests as increased costs (by default, SHAP requires 500 additional API calls to explain a single API prediction, while LIME requires 5000). Even with an open source model, this causes explanations to be prohibitively slow to compute. By contrast RoT does not require additional API calls for explanation. To improve RoT’s predictive power on text, we generate per token embeddings and treat these as features, performing dropout at the token level.

We replicate three uses of LLMs for zero-shot classification: the use of commercial LLM APIs to filter resumes [10], the use of API based LLMs to classify movie reviews [18, 19], and the use of a fine-tuned language model to predict the outcomes of judicial appeals from a court summary[17]. Examples of explanations from the resume filtering task can be seen in Figure 2. A word-cloud constructed from token-level importances across the corpus is visualised in Figure 5. Since this experiment was conducted entirely via APIs, no other explainers can be used to provide baseline comparisons. We run into the same problem with our movie review sentiment classification experiment, but we are able to compare RoT generated explanations against “ground-truth” token importances from human annotators. This provides us with a quantitative measure of the agreement between humans explaining movie review sentiments and RoT explaining an APIs classification of the same sentiments. For Judicial case outcome prediction, we take advantage of access to the model weights to compare RoT explanations with other explainers, and visualise the agreement competing explanations have with human annotations in Figure 4. Overall, we find RoT and SHAP explanations to be better aligned with human annotators, and RoT explanations fastest to compute.

(a) A datapoint with text segments highlighted as per RoT importances

(b) The same datapoint with segments highlighted as per SHAP importances

3.1.1 Explaining Judicial Case Predictions from a Fine-Tuned RoBERTa Model

To test RoT explanations on LLMs, we study an AI system[17] to predict appeal outcomes in Indian courts. Figure 4 presents an abridged example of explanations from one prediction. The data includes annotation labels for individual segments in each case. These indicate which portions of the text were found important to the judgement by human annotators[17], allowing us to measure explanation quality4 by computing the area under the reciever operating characteristic (AUROC) between RoT explanation importances and known ground-truth importances per text segment. As a fine-tuned RoBERTa classifier is used here, we are able to use other explainers, and Table 1 presents AUROC scores from different explainers. A detailed analysis of program runtimes is provided in Figure 9, but it is worth noting that RoTis the only explainer able to produce results in reasonable time on consumer hardware. Details about data preparation, explanation generation, and weighted average AUROC evaluation can be found in Appendix D.2, and individual case explanations can be perused at: https://kairawal.github.io/Rule-of-Thumb-Explaining-Artificial-Intelligence-Systems-using-Partial-Information/JudicialCaseOutcomePrediction/Code/results.html.

3.1.2 Explaining Movie Review Classifications from an LLM API

We conduct a similar experiment with zero-shot movie review sentiment classification, using an AI system built upon OpenAI’s GPT-4.1-nano LLM, accessible only via API. When compared with ground truth importance annotations[18, 19] for review sentiments, RoT achieves a weighted AUROC of 0.72, while a random baseline only achieves 0.50. Gradient based explainers (Integrated Gradients) cannot be compared, as access to GPT-4.1-nano weights is restricted, and perturbation based explainers (LIME, SHAP) are infeasible to apply even at this modest scale. This showcases a unique RoT ability, where we can explain discrete decisions without either access to its weights, ors without making additional API calls, simply by looking at predictions already made by the AI. Additional details are in Appendix D.3.

3.1.3 Explaining Resume Filtering from Proprietary LLMs

In addition to classifying movie reviews, we replicated a zero-shot resume classification system to filter candidates for information-technology (IT) jobs using the GPT-4.1-nano API [10]. RoT was then used to generate explanations identifying which parts of the resume text were important to selecting candidates as suitable for an IT position. We augment our text-based features with three attributes – race, gender, and political orientation, utilising RoT’s unique ability to produce explanations using non-input features. We verify findings from the original study indicating that, on this dataset, GPT ascribes negligible importance to race, gender, and political orientation5 when making hiring decisions. We evaluate the system with a corpus of resumes containing both IT workers and other occupations. The word-cloud in Figure 5 presents the text identified as predictive of selection and rejection across the resume corpus, and Figure 2 shows examples of an IT worker’s and a teacher’s resume, with tokens highlighted according to the RoT importances. Additional data and implementation details can be found in Appendix D.4.

3.2 Auditing Proprietary Black-Box AI Systems

We replicate an audit of Amazon’s e-commerce recommendation system conducted by The Markup [20]. Lacking access to Amazon’s AI system, we adopt the standard practice[21–24] of using a dataset of scraped listings from Amazon to train a mimic AI that predicts product rankings. We then explain mimic predictions using SHAP and LIME, and find these to vary significantly depending on the choice of mimic. This ambiguity is missing from competing RoT explanations, which explain the AI recommendation system directly without using any mimic AI. Our analysis using RoT is also able to surface insights missed by mimic based explanations.

When explaining a proprietary AI system, restrictions can prevent access to model weights as well as the ability to query outputs for synthetically perturbed inputs. This usually disqualifies sensitivity based explainers, but a common workaround is to collect input-output pairs from real-world deployments of the AI system, and use this dataset to build a “mimic” AI system. With unrestricted access to the mimic, predictions can be obtained on new synthetically-perturbed inputs, allowing typical post-hoc explanations such as those from SHAP or LIME to be computed[21–24]. This is distinct from XAI that consists of an interpretable “glass-box” surrogate model, which approximates and AI system holistically as a global explanation (different from local feature importance explanations) itself. Building a mimic AI system and then using it to compute post-hoc feature importance explanations is a distinct explanation category that involves two potential sources of error: the overall approximation of the AI system by the mimic AI system, and the local approximation of the mimic using sensitivity analysis, like SHAP.

Mimics can however suffer from the “Rashomon effect” where they can make predictions identical to the original AI system, but through different internal mechanisms [25–27], yielding different local sensitivity and consequently different explanations [28, 29]. In our audit experiment, the SHAP or LIME explanations from TheMarkup’s mimic AI (Figure 6 panel a) reveal the most sensitive input feature to be whether the product brand is Amazon (“brand is amazon”), followed by “product reviews”. However, switching to a different, equally performant mimic AI (Figure 6 panel b) alters these feature importances. We find many equally performant (seeTable 2) possible mimics in the Rashomon set of roughly equally performant AI systems, which seem to have different internal decision mechanisms (see Table S6).

(a) Explanations from The Markup’s Random Forest mimic model. Using SHAP, they report “brand is amazon” and “product reviews” to be important

(b) Explanations from the scikit-learn default Random Forest mimic model differ, but continue to indicate the importance of “brand is amazon”

(c) RoT importances obtained without mimic models: “brand is amazon” and “sold by amazon” are important, “product reviews” is unimportant

This ‘different local behaviour but equivalent performance’ casts doubt on the mimic approach. An audit using a particular mimic can be undermined by adversarially selecting a different mimic. Further, AI audit results can be contested because insights from any explanation can be attributed to the choice of mimic rather than the underlying AI system. By contrast, RoT explanations (Figure 6 panel c) can be computed without access to the original AI system or the ability to obtain outputs for new inputs. This makes RoT valuable for enforcing regulations which do not grant access to third-party AI systems or source code. For example under Article 74(12) of the EU AI Act[30], regulators are granted access to observe AI outputs on fixed input data, but not the ability to obtain new predictions on arbitrary inputs.

Like other explainers, RoT also uncovers “brand is amazon” to be the most important feature. However, critically, it also shows “sold by amazon” to be an important feature, something that remained hidden in preceding mimic based analysis. This is a new insight signalling potentially self-preferencing behaviour where the Amazon platform does not just promote its own brands, but also third party product sold directly by Amazon over the same product sold by third-party merchants. This finding is harder to undermine because the explanation is independent of idiosyncrasies arising from the choice of mimic. As such, RoT explanations can be useful in identifying potentially self-preferencing practices in regulatory contexts such as the EU Digital Markets Act, where establishing causal links is not strictly necessary[31].

3.3 Scientific Discovery using XAI

We consider a diabetes prediction scenario where XAI needs to award low feature importance scores to unimportant features, and recidivism and loan approval scenarios where XAI needs to award high importance scores to important features in the presence of misleading alternatives. In both situations, RoT is good at hypothesis generation because it correctly identifies the unimportant and important input features. These explanations are being sought not to explain the behaviour of a specific AI system, but to understand the underlying data it was trained on. XAI has been advocated as a means of scientific discovery [32–36] in diverse fields like medicine[37], engineering[38], chemistry[39], and many more [40–47]

Upon analyzing 1151 research articles at the intersection of XAI and science, we found 4.9% of them used explanations of AI systems as a means of proposing novel scientific hypotheses. Of these, 70% used SHAP. We detail the methodology behind these statistics in Appendix D.6. XAI use in scientific discovery suffers the same challenges as AI auditing – like mimics, AI systems trained to replicate scientific phenomena are not guaranteed to faithfully duplicate underlying causal mechanisms. This can be problematic when using SHAP, because sensitivity analysis generates explanations by making predictions on synthetic, out-of-distribution datapoints, which may include physically implausible values. The behaviour of the AI system on non-realistic datapoints could be anything, and consequently the feature importances could be arbitrary [48–50]. By contrast, RoT importances are inspired by feature “predictiveness” rather than sensitivity, overcoming these limitations without implicitly introducing harder problems such as learning causal models [51] or density estimation [52].

RoT also satisfies leading AI regulations better, and its advantage over explainability methods such as SHAP and LIME can even inform (future) best practices in drug discovery. Specifically, the European Medicines Agency (EMA) and US Food and Drug Administration (FDA) recently published joint guidance noting a set of principles whereby interpretability and explainability are enumerated as best model design and development practices[53]. The principles do not mandate specific XAI methods, but the guidance makes clear that transparency, reliability and accountability are key to acceptable deployment.

3.3.1 Correctly Identifying Uninformative Features

If an explainer indicates that a specific feature impacts a phenomenon of interest, but that feature is extraneous to the phenomenon, then the explainer is poorly suited for scientific discovery. But sensitivity based explainers like SHAP and LIME consider any input feature important if applying any perturbations to it leads to changes in the AI-modeled output phenomenon[48], even if the changes are not meaningful. Uninformative features can inadvertently seem important this way.

We consider the same diabetes prediction task as in Figure 3, using a dataset specially collected to study the disease [13]. Using a predictive AI system in such a manner is one way to study diabetes [54]. However, for our experiment, we manually transform the dataset using FairPCA[55] to ensure that the “age” input feature is statistically uncorrelated with the “diabetes” target feature being predicted. Since age has been deliberately rendered uninformative, an XAI technique useful for scientific discovery should correctly identify that it does not affect diabetes.

(a) RoT finds age to be unimportant

(b) SHAP finds age to be somewhat important

(c) LIME finds age to be somewhat important

Figure 7 displays feature importances from RoT, SHAP, and LIME on transformed diabetes data. RoT is the only explainer to consider age unimportant. SHAP and LIME both lead to spurious hypotheses that indicate the importance of age in diabetes prediction. However, if our interest is in learning about the laws and relations that govern the data and phenomenon of diabetes, RoT is the better method because it correctly identifies age as unpredictive. Model details and our use of FairPCA[55] to ensure statistical independence between age and diabetes are detailed in Appendix D.7.

3.3.2 Identifying Important Features in the presence of Misleading Features

We have seen that explainers can be vulnerable to arbitrary out-of-distribution behaviour from an AI system. Adversarial attacks have been proposed to exploit this vulnerability by modifying AI systems to deliberately poison out-of-distribution outputs [56]. This effectively presents different behaviour to SHAP or LIME than to real-world inputs, masking the actual importances of “sensitive” features and manipulating explainers into selecting misleading “foil” features as highly important instead. In black-box AI audits, adversial attacks cause “fairwashing” by allowing malicious actors to deliberately hide AI systems with discriminatory behaviour behind benign-looking explanations [56]. For scientific discovery, the same thing can cause candidate hypotheses generated using XAI to be misleading.

(a) Distribution of feature importances from SHAP and RoT explanations across “sensitive”, “foil” or other features for an adversarial model

(b) Distribution of feature importances from LIME and RoT explanations across “sensitive”, “foil” or other features for an adversarial model

To verify that RoT does not generate such spurious explanations, we replicate adversarial AI systems for credit-scoring and criminal justice, which deliberately hide the importance of an important “sensitive” feature by introducing one or more misleading “foil” features. We find that this is ineffective at misleading RoT, which is able to correctly identify the sensitive features (race for recidivism prediction, and gender for loan approvals) as important and the foil features (simulated, extraneous features) as unimportant. In Figure 8 we show the distribution of SHAP, LIME, and RoT importances over three feature categories: sensitive, adversarial, and other, from AI systems subjected to an adversarial attack [56]. Details about the datasets used, model behaviours introduced, “sensitive” input features masked and adversarially misleading “foil” features introduced are available in Appendix D.8.

We report detailed statistics about the proportion of explanations that identified the sensitive feature as most important (success), or the foil feature (adversarial failure) or other features (non-adversarial failure) inTable S7. Across 10 experiments, for LIME or SHAP the rate of recovering the sensitive feature as most important does not exceed 5% except once, where adversarial failures (foil feature is most important) still outnumber success (sensitive feature is most important). RoT has a success rate always greater than 89%, and in 6 of the 10 experiments RoT shows a perfect 100% recovery rate, always correctly identifying the sensitive feature to be the most important.

3.4 Computational Efficiency and Environmental Impacts

Of all the experiments listed so far, generating RoT, SHAP, LIME, and Integrated Gradients explanations for Judicial Case Outcome Prediction with a finetuned LLM was the largest in terms of dataset size and computations involved for prediction. For this experiment, generating the first RoT explanation took 1160 seconds, and each additional explanation took less than 0.1 milliseconds. By contrast, the average time taken for each SHAP explanation was 13 million times longer at 1320 seconds. This was without GPU acceleration, to simulate consumer hardware conditions. In practice, it is impractical to compute SHAP or LIME explanations for LLM predictions, except if only a handful of datapoints need explaining. Gradient methods are quicker, but require access to model weights when most commercial LLMs can only be accessed via API. RoT is the first explainer that can actually be deployed in practice, bringing feature importance XAI to LLMs.

(a) Explanation Runtimes: Judicial Case Outcome Prediction with LLMs

(b) Cumulative Explanation Runtimes (Quickest First): Case Prediction

RoT was also found to be orders of magnitude faster than SHAP and LIME in our benchmark XAI comparison, detailed in Appendix D.9, which we document in Tables S4 and S5. This increase in computational efficency, evidenced by faster program running times, is a direct result of RoT’s unique formulation that does not require obtaining additional predictions for each explanation. Local feature importance explainers often function by collecting data about an AI system’s behaviour and then fitting an interpretable model. Both SHAP and LIME need to fit a new interpretable model for each prediction they explain. This requires data collection by making arbitrarily time-consuming calls to the AI system for each additional explanation. By contrast, RoT uses dropout to fit a single interpretable ensemble on the entire unperturbed dataset, and each explanation is generated using per-feature importance scores from this model. Since this inference is quicker than continuous training for each explanation, this makes RoT extremely fast once fit. When explaining predictions from large AI systems such as computationally intensive LLMs, the efficiency gains from RoT would translate directly to reduced environmental impacts.

4. Regulations Concerning Emerging XAI Applications

Compared to other XAI methods, RoT has a range of strengths that make it an ideal tool for making existing AI systems auditable and accountable. Its narrow focus on directly predicting the behaviour of an existing system on real data, gives it robustness against adversarial attacks, and removes the need for mimic models. This gives it a clear advantage in meeting the EU AI Act’s requirements for accuracy, robustness, and adversarial testing of high-risk AI systems under Article 15 and for robustness and resilience under Article 15 (4) – (5) [30]. This in turn, supports transparency and instructions of use requirements under Article 13 (1) and (3) (b) (ii). RoT’s computational efficiency compared to approaches requiring construction of a mimic model give it clear advantages in meeting environmental sustainability goals [57]. Most importantly, it allows auditing proprietary models, overcoming inherent transparency limitations found in many AI regulatory frameworks [58].

RoT enables participatory governance of both narrow and general-purpose AI systems, facilitating “independent external testing” and adversarial testing of general-purpose systems with systemic risk, as required under Recital 114; Article 55 (1) (a) of the AI Act [59]. In this regard, RoT may prove particularly valuable for enforcement of the AI Act. While Article 74(13) allows regulators to access testing datasets and model outputs, it does not always allow full model access or the ability to study model responses on new data, as would be required for typical sensitivity based explainers.

RoT can also be a powerful tool to identify and challenge anti-competitive practices of major technology companies that are increasingly embedding AI systems across their platforms. RoT can help identify self-preferencing practices, such as search result manipulation, that may be illegal under horizontal competition laws and the EU Digital Markets Act (DMA) that require establishing a link between model behaviour and anti-competitive practices [60, 61]. We elaborate further on the specific methods and limitations of RoT in Appendix C.

In addition to auditing, explainability and interpretability are also important safeguards for AI use in drug development[62] and related scientific discovery[63, 64], with each fulfilling distinct expectations depending on context. For instance, the EMA notes that for the review and monitoring of black box models, explainability metrics, such as SHAP and LIME should be used[64]. The EMA–FDA joint guidance provides a broad view on the use of AI in evidence generation and monitoring across all phases of a medicine including early research. While this recent document does not recommend a specific explainability metric, it aims to promote a baseline on the use of transparent models that are reliable, robust, generalisable and utilises data that is ‘fit-for-purpose’ [53]. Here we see the added value of RoT, clarifying why SHAP and LIME would not be sufficient for ensuring accountability, transparency and reliability, and providing the more appropriate baseline for future policy guidance and reform.

5. Conclusion

This paper presents a new approach to explainable AI based on the predictiveness of individual feature values, rather than their sensitivity to alteration. This Rule of Thumb approach allows us to approximate any decision making process by summing the importance response over any subset of feature values. The most useful feature values for a prediction are naturally those with largest importance, and their predictive power overrides the contribution of others. As such, RoT can be understood as a post-hoc and per-datapoint explainer with more in common with ablation studies than sensitivity analysis.

RoT addresses several long-standing concerns in the community about the validity and practical utility of XAI, and importantly does so without making additional assumptions. It is flexible and useful in a range of emerging XAI applications areas where other forms of XAI simply can not be used – evaluating LLMs and explaining their behaviour in zero-shot classification, auditing proprietary AI systems where we can only observe their predictions and not alter input features, and using XAI to understand real-world phenomena and propose novel scientific hypotheses. As a local, post-hoc explanation method, RoT can be used as a drop in replacement for other explainers in existing XAI codebases, and visualised the same way, providing a fast and computationally efficient (and therefore environmentally friendly) method of explaining AI.

Acknowledgments

The work of all the authors has been supported through research funding provided by the Wellcome Trust (grant nr 223765/Z/21/Z), Sloan Foundation (grant nr G-2021-16779), Department of Health and Social Care, EPSRC (grant nr EP/Y019393/1), and Luminate Group. Their funding supports the Trustworthiness Auditing for AI project and Governance of Emerging Technologies research programme at the Oxford Internet Institute, University of Oxford. Kai Rawal is funded by an EPSRC doctoral scholarship. The work of Sandra Wachter, Daria Onitiu, Chris Russell, and Kai Rawal has also been supported by the Alexander von Humboldt Foundation in the framework of the Alexander von Humboldt Professorship (Humboldt Professor of Technology and Regulation) endowed by the Federal Ministry of Education and Research via the Hasso Plattner Institute.

Data and materials availability: All code used in our experiments can be found at https://github.com/KaiRawal/Rule-of-Thumb-Explaining-Artificial-Intelligence-Systems-using-Partial-Information. The scripts rely on permissively licensed publicly available datasets, and instructions for downloading these, where applicable, are included with the code.

References

1. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “why should i trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342322. doi: 10.1145/2939672.2939778. URL https://doi.org/10.1145/2939672.2939778.
2. Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 618–626, October 2017.
3. Bhavik Jikadara. Cats and Dogs Classification Dataset. https://www.kaggle.com/datasets/bhavikjikadara/dog-and-cat-classification-dataset, 2024. Kaggle dataset.
4. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014. URL http://arxiv.org/abs/1312.6034.
5. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3319–3328. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/sundararajan17a.html.
6. Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech., 31:841, 2018.
7. Leonard Bereska and Stratis Gavves. Mechanistic Interpretability for AI Safety - A Review. Transactions on Machine Learning Research, 2024. URL https://mlanthology.org/tmlr/2024/bereska2024tmlr-mechanistic/.
8. Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 4768–4777, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
9. Kaivalya Rawal and Himabindu Lakkaraju. Beyond individualized recourse: Interpretable and interactive summaries of actionable recourses. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12187–12198. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/8ee7730e97c67473a424ccfeff49ab20-Paper.pdf.
10. Akshaj Kumar Veldanda, Fabian Grob, Shailja Thakur, Hammond Pearce, Benjamin Tan, Ramesh Karri, and Siddharth Garg. Investigating hiring bias in large language models. In R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023. URL https://openreview.net/forum?id=erl90pLIH0.
11. Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. CoRR, abs/1706.03825, 2017. URL http://arxiv.org/abs/1706.03825.
12. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 3145–3153. JMLR.org, 2017.
13. Jack W Smith, James E Everhart, William C Dickson, William C Knowler, and Robert Scott Johannes. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the annual symposium on computer application in medical care, page 261, 1988.
14. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html.
15. Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019.
16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423.
17. Shubham Kumar Nigam, Anurag Sharma, Danush Khanna, Noel Shallum, Kripabandhu Ghosh, and Arnab Bhattacharya. Legal judgment reimagined: PredEx and the rise of intelligent AI interpretation in Indian courts. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 4296–4315, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.255. URL https://aclanthology.org/2024.findings-acl.255/.
18. Omar Zaidan, Jason Eisner, and Christine Piatko. Using ”annotator rationales” to improve machine learning for text categorization. In Proceedings of the conference of the North American chapter of the Association for Computational Linguistics (NAACL), pages 260–267, 2007.
19. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. Eraser: A benchmark to evaluate rationalized nlp models, 2019.
20. Adrianne Jeffries and Leon Yin. Amazon puts its own “brands” first above better-rated products. The Markup, October 2021. URL https://themarkup.org/amazons-advantage/2021/10/14/amazon-puts-its-own-brands-first-above-better-rated-products.
21. Krishna Gade, Sahin Geyik, Krishnaram Kenthapadi, Varun Mithal, and Ankur Taly. Explainable ai in industry: Practical challenges and lessons learned. In Companion Proceedings of the Web Conference 2020, WWW ’20, page 303–304, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450370240. doi: 10.1145/3366424.3383110. URL https://doi.org/10.1145/3366424.3383110.
22. Arize AI. Surrogate Model. URL https://docs.arize.com/arize/machine-learning/how-to-ml/explainability/surrogate-model. Accessed: 2024-08-02.
23. Fiddler AI. Surrogate Models. URL https://docs.fiddler.ai/ui-guide/explainability-ui-giude/surrogate-models. Accessed 2024-08-02.
24. Microsoft. Model interpretability - Azure Machine Learning, 2023. URL https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-interpretability. Accessed 2024-08-02.
25. Lucas Monteiro Paes, Rodrigo Cruz, Flavio P. Calmon, and Mario Diaz. On the inevitability of the rashomon effect. In 2023 IEEE International Symposium on Information Theory (ISIT), pages 549–554, 2023. doi: 10.1109/ISIT54713.2023.10206657.
26. Cynthia Rudin, Chudi Zhong, Lesia Semenova, Margo Seltzer, Ronald Parr, Jiachang Liu, Srikar Katta, Jon Donnelly, Harry Chen, and Zachery Boner. Amazing things come from having many good models, 2024. URL https://arxiv.org/abs/2407.04846.
27. Xavier Renard, Thibault Laugel, and Marcin Detyniecki. Understanding prediction discrepancies in classification. Machine Learning, Aug 2024. ISSN 1573-0565. doi: 10.1007/s10994-024-06557-4. URL https://doi.org/10.1007/s10994-024-06557-4.
28. Charles Marx, Flavio Calmon, and Berk Ustun. Predictive multiplicity in classification. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6765–6774. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/marx20a.html.
29. Sebastian Müller, Vanessa Toborek, Katharina Beckh, Matthias Jakobs, Christian Bauckhage, and Pascal Welke. An empirical evaluation of the rashomon effect in explainable machine learning. In Danai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Elena Baralis, and Francesco Bonchi, editors, Machine Learning and Knowledge Discovery in Databases: Research Track, pages 462–478, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-43418-1.
30. European Union. Regulation (eu) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence and amending certain union legislative acts (artificial intelligence act). Official Journal of the European Union, L 202, 12 July 2024, pp. 1–241, 2024. URL https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32024R1689. OJ L 202, 12.7.2024, p. 1–241.
31. European Union. Regulation (eu) 2022/1925 of the european parliament and of the council of 14 september 2022 on contestable and fair markets in the digital sector and amending directives (eu) 2019/1937 and (eu) 2020/1828 (digital markets act). https://eur-lex.europa.eu/eli/reg/2022/1925/oj, 2022. OJ L 265, 12.10.2022, p. 1–66.
32. Ribana Roscher, Bastian Bohn, Marco F. Duarte, and Jochen Garcke. Explainable machine learning for scientific insights and discoveries. IEEE Access, 8:42200–42216, 2020. doi: 10.1109/ACCESS.2020.2976199.
33. Jacques A. Esterhuizen, Bryan R. Goldsmith, and Suljo Linic. Interpretable machine learning for knowledge generation in heterogeneous catalysis. Nature Catalysis, 5(3):175–184, Mar 2022. ISSN 2520-1158. doi: 10.1038/s41929-022-00744-z. URL https://doi.org/10.1038/s41929-022-00744-z.
34. Richard Dybowski. Interpretable machine learning as a tool for scientific discovery in chemistry. New J. Chem., 44:20914–20920, 2020. doi: 10.1039/D0NJ02592E. URL http://dx.doi.org/10.1039/D0NJ02592E.
35. Quinn Dickinson and Joel G. Meyer. Positional shap (poshap) for interpretation of machine learning models trained from biological sequences. PLOS Computational Biology, 18(1):e1009736, 2022. doi: 10.1371/journal.pcbi.1009736. URL https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009736.
36. Mateusz Garbulowski, Klev Diamanti, Karolina Smolińska, Nicholas Baltzer, Patricia Stoll, Susanne Bornelöv, Aleksander Øhrn, Lars Feuk, and Jan Komorowski. R.rosetta: an interpretable machine learning framework. BMC Bioinformatics, 22(1):110, Mar 2021. ISSN 1471-2105. doi: 10.1186/s12859-021-04049-z. URL https://doi.org/10.1186/s12859-021-04049-z.
37. Chukwuebuka Joseph Ejiyi, Zhen Qin, Joan Amos, Makuachukwu Bennedith Ejiyi, Ann Nnani, Thomas Ugochukwu Ejiyi, Victor Kwaku Agbesi, Chidimma Diokpo, and Chidinma Okpara. A robust predictive diagnosis model for diabetes mellitus using shapley-incorporated machine learning algorithms. Healthcare Analytics, 3:100166, 2023. ISSN 2772-4425. doi: https://doi.org/10.1016/j.health.2023.100166. URL https://www.sciencedirect.com/science/article/pii/S2772442523000333.
38. Liangliang Hu, Xiaolin Meng, Yilin Xie, Craig Hancock, George Ye, and Yan Bao. Examination of load-deformation characteristics of long-span bridges in harsh natural environments based on real-time updating artificial neural network. Engineering Structures, 308:118022, 2024. ISSN 0141-0296. doi: https://doi.org/10.1016/j.engstruct.2024.118022. URL https://www.sciencedirect.com/science/article/pii/S0141029624005844.
39. Qiming Fu, Tao Xu, Chenggong He, Daomiao Wang, Meiling Liu, and Chao Liu. Machine learning-assisted study of ren(x)c(6-x)-doped graphene as potential electrocatalysts for oxygen electrode reactions. Langmuir, 40(20):10726–10736, 2024. doi: 10.1021/acs.langmuir.4c00803. Epub 2024 May 8.
40. Gian Maria Campedelli. Explainable machine learning for predicting homicide clearance in the united states. Journal of Criminal Justice, 79:101898, 2022. ISSN 0047-2352. doi: https://doi.org/10.1016/j.jcrimjus.2022.101898. URL https://www.sciencedirect.com/science/article/pii/S0047235222000186.
41. Xinyi Niu, Chengpeng Lu, Ying Zhang, Yong Zhang, Chengcheng Wu, Ebrima Saidy, Bo Liu, and Longcang Shu. Hysteresis response of groundwater depth on the influencing factors using an explainable learning model framework with shapley values. Science of The Total Environment, 904:166662, 2023. ISSN 0048-9697. doi: https://doi.org/10.1016/j.scitotenv.2023.166662. URL https://www.sciencedirect.com/science/article/pii/S0048969723052877.
42. Andrés Cremades, Sergio Hoyas, Rahul Deshpande, Pedro Quintero, Martin Lellep, Will Junghoon Lee, Jason P. Monty, Nicholas Hutchins, Moritz Linkmann, Ivan Marusic, and Ricardo Vinuesa. Identifying regions of importance in wall-bounded turbulence through explainable deep learning. Nature Communications, 15(1):3864, May 2024. ISSN 2041-1723. doi: 10.1038/s41467-024-47954-6. URL https://doi.org/10.1038/s41467-024-47954-6.
43. Zhen Sun, João Santos, Elsa Caetano, and Catarina Oliveira. Interpreting cumulative displacement in a suspension bridge with a physics-based characterisation of environment and roadway/railway loads. Journal of Civil Structural Health Monitoring, 13(2):387–397, Mar 2023. ISSN 2190-5479. doi: 10.1007/s13349-022-00647-4. URL https://doi.org/10.1007/s13349-022-00647-4.
44. Shuo Wang, Tianzhuo Zhang, Ziheng Li, and Jinglan Hong. Exploring pollutant joint effects in disease through interpretable machine learning. Journal of Hazardous Materials, 467:133707, 2024. ISSN 0304-3894. doi: https://doi.org/10.1016/j.jhazmat.2024.133707. URL https://www.sciencedirect.com/science/article/pii/S0304389424002863.
45. Zicheng Cao, Zekai Qiu, Feng Tang, Shiwen Liang, Yinghan Wang, Haoyu Long, Cai Chen, Bing Zhang, Chi Zhang, Yaqi Wang, Kang Tang, Jing Tang, Junhong Chen, Chunhui Yang, Yuzhe Xu, Yulin Yang, Shenglan Xiao, Dechao Tian, Guozhi Jiang, and Xiangjun Du. Drivers and forecasts of multiple waves of the coronavirus disease 2019 pandemic: A systematic analysis based on an interpretable machine learning framework. Transboundary and Emerging Diseases, 69(5):e1584–e1594, Sep 2022. doi: 10.1111/tbed.14492. URL https://doi.org/10.1111/tbed.14492. Epub 2022 Mar 13.
46. Xinyuan Cao, Jisi Huang, Kexin Du, Yawen Tian, Zhixin Hu, Zhu Luo, Jinlong Wang, and Yanbing Guo. Machine-learning-assisted descriptors identification for indoor formaldehyde oxidation catalysts. Environmental Science & Technology, 58(19):8372–8379, 2024. doi: 10.1021/acs.est.4c01691. Epub 2024 May 1.
47. Ruikun Gou, Jinshu Chi, Jiangong Liu, Yunpeng Luo, Ankit Shekhar, Lidong Mo, and Guanghui Lin. Atmospheric water demand constrains net ecosystem production in subtropical mangrove forests. Journal of Hydrology, 630:130651, 2024. ISSN 0022-1694. doi: https://doi.org/10.1016/j.jhydrol.2024.130651. URL https://www.sciencedirect.com/science/article/pii/S0022169424000453.
48. I. Elizabeth Kumar, Suresh Venkatasubramanian, Carlos Scheidegger, and Sorelle Friedler. Problems with shapley-value-based explanations as feature importance measures, 2020. URL https://arxiv.org/abs/2002.11097.
49. Dominik Janzing, Lenon Minorics, and Patrick Blöbaum. Feature relevance quantification in explainable ai: A causal problem, 2019. URL https://arxiv.org/abs/1910.13413.
50. Muhammad Faaiz Taufiq, Patrick Blöbaum, and Lenon Minorics. Manifold restricted interventional shapley values, 2023. URL https://arxiv.org/abs/2301.04041.
51. Leland Gerson Neuberg. Causality: Models, reasoning, and inference, by judea pearl, cambridge university press, 2000. Econometric Theory, 19(4):675–685, 2003. doi: 10.1017/S0266466603004109.
52. Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998. ISBN 978-0-471-03003-4.
53. European Medicines Agency and U.S. Food and Drug Administration. Guiding principles of good AI practice in drug development. Technical report, European Medicines Agency and U.S. Food and Drug Administration, January 2026. URL https://www.ema.europa.eu/en/documents/other/guiding-principles-good-ai-practice-drug-development_en.pdf. Principle 7: Model design and development practices.
54. Victor Chang, Jozeene Bailey, Qianwen Ariel Xu, and Zhili Sun. Pima indians diabetes mellitus classification based on machine learning (ML) algorithms. Neural Comput Appl, pages 1–17, March 2022.
55. Matthäus Kleindessner, Michele Donini, Chris Russell, and Muhammad Bilal Zafar. Efficient fair pca for fair representation learning. In International Conference on Artificial Intelligence and Statistics, pages 5250–5270. PMLR, 2023.
56. Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES ’20, page 180–186, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450371100. doi: 10.1145/3375627.3375830. URL https://doi.org/10.1145/3375627.3375830.
57. Philipp Hacker. Sustainable ai regulation. Common Market Law Review, 61(2), 2024.
58. Sandra Wachter. Limitations and Loopholes in the EU AI Act and AI Liability Directives: What This Means for the European Union, the United States, and Beyond. Yale Journal of Law & Technology, 26(3), 2024. doi: 10.2139/ssrn.4924553. URL https://ssrn.com/abstract=4924553. Available at SSRN: https://ssrn.com/abstract=4924553 or http://dx.doi.org/10.2139/ssrn.4924553.
59. European Commission. Contents of the code of practice on generative ai. https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai, 2024. Accessed: 2025-09-10.
60. European Court of Justice. Google LLC and Alphabet Inc v European Commission, 2024. URL https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:62022CJ0048. Case C-48/22.
61. General Court of the European Union (Ninth Chamber, Extended Composition). Google and Alphabet v. Commission (Google Shopping), 2021. URL https://curia.europa.eu/juris/document/document.jsf?text=&docid=250881&pageIndex=0&doclang=en&mode=lst&dir=&occ=first&part=1&cid=10134666. Case T-612/17.
62. U.S. Food and Drug Administration. Considerations for the use of artificial intelligence to support regulatory decision-making for drug and biological products. Draft guidance for industry and other interested parties, U.S. Food and Drug Administration, January 2025. URL https://www.fda.gov/media/184830/download. Draft guidance; distributed for comment purposes only.
63. U.S. Food and Drug Administration. Using artificial intelligence & machine learning in the development of drug & biological products. Discussion paper and request for feedback, U.S. Food and Drug Administration, February 2025. URL https://www.fda.gov/media/167973/download. Originally published May 2023; revised February 2025.
64. European Medicines Agency. Reflection paper on the use of artificial intelligence (AI) in the medicinal product lifecycle. Reflection Paper EMA/CHMP/CVMP/83833/2023, European Medicines Agency, September 2024. URL https://www.ema.europa.eu/en/documents/scientific-guideline/reflection-paper-use-artificial-intelligence-ai-medicinal-product-lifecycle_en.pdf.
65. Ankur Taly and Kaz Sato. Monitoring feature attributions: How google saved one of the largest ml services in trouble, September 29 2021. URL https://cloud.google.com/blog/topics/developers-practitioners/monitoring-feature-attributions-how-google-saved-one-largest-ml-services-troubl Google Cloud Blog.
66. Jianlong Zhou and Fang Chen. Human and Machine Learning: Visible, Explainable, Trustworthy and Transparent. Springer Publishing Company, Incorporated, 1st edition, 2018. ISBN 3319904027.
67. Luca Deck, Jakob Schoeffer, Maria De-Arteaga, and Niklas Kühl. A critical survey on fairness benefits of explainable ai. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’24. ACM, June 2024. doi: 10.1145/3630106.3658990. URL http://dx.doi.org/10.1145/3630106.3658990.
68. Andrea Saltelli, Marco Ratto, Stefano Tarantola, and Francesca Campolongo. Sensitivity analysis for chemical models. Chemical Reviews, 105(7):2811–2828, 2005. doi: 10.1021/cr040659d. URL https://doi.org/10.1021/cr040659d. PMID: 16011325.
69. H Rabitz, M Kramer, and D Dacol. Sensitivity analysis in chemical kinetics. Annual Review of Physical Chemistry, 34(Volume 34, 1983):419–461, 1983. ISSN 1545-1593. doi: https://doi.org/10.1146/annurev.pc.34.100183.002223. URL https://www.annualreviews.org/content/journals/10.1146/annurev.pc.34.100183.002223.
70. Richard H. McCuen. The role of sensitivity analysis in hydrologic modeling. Journal of Hydrology, 18(1):37–53, 1973. ISSN 0022-1694. doi: https://doi.org/10.1016/0022-1694(73)90024-3. URL https://www.sciencedirect.com/science/article/pii/0022169473900243.
71. Linda Lilburne and Stefano Tarantola. Sensitivity analysis of spatial models. International Journal of Geographical Information Science, 23(2):151–168, 2009. doi: 10.1080/13658810802094995. URL https://doi.org/10.1080/13658810802094995.
72. Hans B. Nielsen. Sensitivity analysis in linear systems (assem deif). SIAM Review, 31:706–707, 1989. doi: 10.1137/1031159. URL https://doi.org/10.1137/1031159.
73. Richard Meyes, Melanie Lu, Constantin Waubert de Puiseau, and Tobias Meisen. Ablation studies in artificial neural networks. CoRR, abs/1901.08644, 2019. URL http://arxiv.org/abs/1901.08644.
74. B. Pegourie, J.-M. Picchiottino, H.-W. Drawin, A. Geraud, and M. Chatelier. Pellet ablation studies on tore supra. Nuclear Fusion, 33(4):591, apr 1993. doi: 10.1088/0029-5515/33/4/I06. URL https://dx.doi.org/10.1088/0029-5515/33/4/I06.
75. Michael Keidar, Iain D. Boyd, Anthony Williams, and Richard Beyer. Ablation study in a capillary sustained discharge. IEEE Transactions on Magnetics, 43(1):308–312, 2007. doi: 10.1109/TMAG.2006.887672.
76. Arnon Rosenthal. The gdnf protein familygene ablation studies reveal what they really do and how. Neuron, 22(2):201–203, 1999. ISSN 0896-6273. doi: https://doi.org/10.1016/S0896-6273(00)81077-6. URL https://www.sciencedirect.com/science/article/pii/S0896627300810776.
77. Elisabeth A. Murray. What have ablation studies told us about the neural substrates of stimulus memory? Seminars in Neuroscience, 8(1):13–22, 1996. ISSN 1044-5765. doi: https://doi.org/10.1006/smns.1996.0003. URL https://www.sciencedirect.com/science/article/pii/S1044576596900032.
78. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2668–2677. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/kim18d.html.
79. Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in neural information processing systems, 34:4699–4711, 2021.
80. Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. Openxai: towards a transparent evaluation of post hoc model explanations. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9781713871088.
81. Sandra Wachter, Brent Mittelstadt, and Luciano Floridi. Why a right to explanation of automated decision-making does not exist in the general data protection regulation. International data privacy law, 7(2):76–99, 2017.
82. Andrew D Mitchell, Dominic Let, and Lingxi Tang. Ai regulation and the protection of source code. International Journal of Law and Information Technology, 31(4):283–301, 2023.
83. David Hartmann, José Renato Laranjeira de Pereira, Chiara Streitbörger, and Bettina Berendt. Addressing the regulatory gap: moving towards an eu ai audit ecosystem beyond the ai act by including civil society. AI and Ethics, pages 1–22, 2024.
84. Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, et al. Black-box access is insufficient for rigorous ai audits. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 2254–2272, 2024.
85. Sandra Wachter, Brent Mittelstadt, and Chris Russell. Why fairness cannot be automated: Bridging the gap between eu non-discrimination law and ai. Computer Law & Security Review, 41:105567, 2021.
86. Sandra Wachter. The theory of artificial immutability: Protecting algorithmic groups under anti-discrimination law. Tul. L. Rev., 97:149, 2022.
87. European Union. Regulation (eu) 2022/2065 of the european parliament and of the council of 19 october 2022 on a single market for digital services and amending directive 2000/31/ec (digital services act). Official Journal of the European Union, 10 2022. URL https://eur-lex.europa.eu/eli/reg/2022/2065/oj. OJ L 277, 27.10.2022, p. 1–102.
88. United States District Court for the District of Columbia. United States of America et al. v. Google LLC, 2024. Case pending.
89. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024. URL https://arxiv.org/abs/2412.13663.
90. Takaya Saito and Marc Rehmsmeier. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One, 10(3):e0118432, March 2015.
91. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423/.
92. Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. How we analyzed the compas recidivism algorithm, May 2016. URL https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm.
93. Michael Redmond. Communities and Crime. UCI Machine Learning Repository, 2002. DOI: https://doi.org/10.24432/C53W3X.
94. Hans Hofmann. Statlog (German Credit Data). UCI Machine Learning Repository, 1994. DOI: https://doi.org/10.24432/C5NC77.
95. Yuji Ikeda. yuzie007/mpltern: 1.0.4. Zenodo, 2024. URL https://doi.org/10.5281/zenodo.11068993. Version 1.0.4.
96. J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9(3):90–95, 2007. doi: 10.1109/MCSE.2007.55.

A Motivation and Related Work

A.1 Explainability in Machine Learning, Feature Attribution Methods, and SHAP

Machine learning models range from inherently interpretable (eg. decision stumps, linear regressions, rule-sets, or single nearest neighbour classifiers) to uninterpretable black boxes (eg. random forest ensembles, deep neural networks, or markov random fields). In order to explain black-box models, methods have been developed to produce post-hoc model explanations (eg. saliency maps, counterfactual explanations, anchors, prototypes, etc). These can be either global, explaining model behaviour in general (eg. PDPs, Functional Decomposition, Mechanistic Interpretability etc), or local, explaining individual predictions made by a model (eg. ICE, LIME, anchors, counterfactuals). Some model explanation methods are specific to certain kinds of models (eg. TCAV, Saliency Maps, gini-importance), whereas others are considered to be model agnostic (counterfactual explanations, LIME, anchors, etc). SHAP is a widely used model-agnostic, post-hoc, local explanation method, that produces scores indicating the relative importance of various input features that led to a model prediction [8]. SHAP is often the first technique used by practitioners interested in explaining the behaviour of black box models.

A.2 Explainability in Experimentation and Production

One use of explainability in machine learning is to continuously explain the predictions being made by artificial intelligence systems that have been deployed in the real world. This can help debug errors [65], comply with regulations, and engender trust [66]. Often, explanations are used to gauge if protected attributes are significantly impacting model outputs and bring transparency to otherwise opaque decision making processes [67]. When models are being deployed in real world settings and affecting large numbers of users, it can be important to monitor their behaviour and have explainability in production, which requires explanations to be computed in a scalable manner.

A different use of explainability is in the lab, where researchers may want to understand model behaviour in order to study underlying phenomena. When using explainability during model development or exploratory data analysis, explanations can be used to understand the data generating process itself and building machine learning models and then using explainability techniques such as SHAP has been proposed as a means of scientific discovery [32–36]. This strategy has been employed in diverse fields such as policy-making [40], physics [42], engineering [38, 43], epidemiology [44, 45], chemistry [39, 46], healthcare [37], ecology [47], and geology [41], research where model explanations are used to derive insights about underlying natural phenomena using a machine learning model, rather than explain the behaviour of a model used in decision making. This can be termed explainability in experimentation, and requires explanations to be robust and consistent.

These two use cases for model explanations are different, and present themselves with different desiderata and technical challenges. RoTprovides a scalable method that is important in production settings, as well as a robust method that does not suffer from causal vulnerabilities or the Rashomon effect important in experimentation settings.

A.3 Implicit Assumptions about Causality

Machine learning classification is concerned with learning the conditional probability of obtaining a response y from a given set of inputs x. Borrowing from Judea Pearl’s causal hierarchy [51], this involves “level 1” reasoning - the ability to observe associations. There is broad consensus today that machine learning alone is unable to produce systems with higher “level 2” or “level 3” reasoning abilities involving the ability to answer “interventional” and “counterfactual” questions, respectively. [51]

The AI explainability literature so far has been surprisingly forgiving of this deficit typical in all modern machine learning systems. Sensitivity based model explanation methods like SHAP implicitly assume that models produce predictions using level 3 reasoning. If a model only possesses “level 1” observational reasoning, but an explanation method probes it with inputs that do not exist in naturally observed training data, then model predictions in these regions cannot be considered indicative of model behaviours.

SHAP’s reliance on perturbing model inputs causes explanations to be overly sensitive to out of distribution model behaviour, which can be arbitrary [48]. The solutions proposed to this problem such as manifold-SHAP [50] or causal SHAP [49] necessarily involve solving the harder problems of density estimation [52] or causal modeling [51], as opposed to learning a simpler classifier decision boundary. Adversarial attacks have been proposed to exploit this vulnerability, and it has been shown that explanations can be manipulated this way, leading to fair-washing. [56]

While these two approaches are possible to use in theory, the associated problems with each often make them impossible to implement in practical scenarios. These problems are especially pronounced for uses of model explainability in experimentation, where many models are built and explained iteratively, and the focus is more on the explanations themselves than on deploying the model to real world production settings. In such scenarios, both requiring the user to first build a causal model or estimate the entire density of the data is a non-starter.

By contrast, RoT makes no causal assumptions about the model, and does not need to solve the “harder” problems of density estimation either. RoTgenerates explanations entirely through “level 1” observation.

A.4 Ablation Studies and Sensitivity Analysis

Feature importances can be motivated in at least two distinct ways, involving the effect of:

1. removing or ablating input features on model outputs to determine feature importances through their predictiveness (ablation studies); or
2. perturbing input features and measuring the corresponding perturbations on model outputs to determine feature importances through their impact on the model output (sensitivity analysis).

Both these notions have been used widely. Examples of senstivity analysis include applications in chemistry [68] [69], geology [70] [71] and engineering [72]. Ablation studies have roots in medicine and are widely used in machine learning [73] today, and sometimes in other fields like physics [74] [75] and biology [76] [77].

In existing machine learning explainability literature, there exist global methods motivated by sensitivity analysis, local methods motivated by sensitivty analysis, global methods motivated by ablation studies, but no local methods motivated by ablation studies. RoT is the first model agnostic post-hoc local explanation method that defines feature importance using the predictiveness of features rather than the sensitivity of the output to perturbations, akin to ablation studies rather than sensitivity analysis. Table S1 provides a non-exhaustive list of model explanation methods, classified along different axes: whether the explanations are global or local, and whether feature importance is determined by sensitivity or predictiveness. RoT is a unique explanation method, as can be seen from the table.

A.5 Problems with Model Explanations in Practice

Returning to our previous examples of trying to debug LLM behaviour by explaining it’s classification outputs, and of trying to monitor a large-scale proprietary e-commerce recommendation system, we can illustrate the common practical problems explaining machine learning predictions.

When trying to explain the behaviour of an LLM, it is not possible to arbitrarily modify the input features to the model. For an explanation method like SHAP, it then becomes unclear how to perturb the inputs features. Additionally, explanations can only be used to ascribe importances to input features, and nuanced analyses studying the relationship between the output and external concepts or latent features is difficult. RoT is useful because not only does it not rely on input perturbations, but also because it allows the inclusion of external concepts when generating model explanations. TCAV is a global explanation method that explicitly uses latent concepts to provide model explanations [78], but RoT is able to do this on a local, per-prediction level.

When trying to audit the behaviour of proprietary systems, often model artifacts cannot be accessed to obtain output predictions for new input points . This makes explanation methods like SHAP impossible to use without first creating a model to mimic the original. In real world settings, it is often easier to explain such a mimic model, rather than explaining the actual model [23] [21], and it is hard to assert that the mimic explanations generated also apply to the original model due the predictive multiplicity of classifiers [28]. This “Rashomon effect”, where two models have similar predictions but different internal mechanisms [25], makes model explanations obtained using a mimic model inherently untrustworthy when explaining the original model behaviour [29]. RoT sidesteps this issue entirely and does not need access to a model at all, it generates model explanations instead by looking at its predictions.

B RoT Implementation Details

Recall our formulation from Equation 2. In all experiments, we treat ℓ as the log loss, and F as the sigmoid function, arriving at an objective similar to logistic regression. We consider three variants of the importance function f_θj(x_j).

The linear form

A more general additive form consisting of a linear form plus a mixture of Gaussians

While we explored the use of other additive models including polynomial kernels and the double exponentiated form proposed in [79], the mixture of Gaussians combined with a linear term, showed better stability, and robustness to extreme values.

And finally, for explaining the behaviour of text-based classifiers, such as ChatGPT for zero-shot classification, we use a linear form shared over all tokens

Here x_j is now a vector representing the text embedding taken from a model such as BERT, and a and b are common parameters shared over all token locations j ∈ J. Based on this, we provide two canonical implementations of RoT. One uses dropout on the features and a simple linear model to learn classifier logits or regression outputs using either cross entropy or mean squared error loss, and the other uses text embeddings, performing dropout on both embedding features and tokens, to explain zero shot classification tasks.

Our experiments were performed on a MacBook with 24 GB memory and an Apple M4 Pro Processor. All the code used for our experiments will be made public. We also plan to make RoT available for the community as an open-source software package. We make use of existing implementations of explainers such as SHAP [8] and LIME [1], other open sourced data and models [10, 13, 17, 20], and evaluation benchmarks and code [56, 80] previously released by the community.

B.1 Derivation

We wish to explain the behavior of a model C(·), over an empiric data distribution X. Given a data-point, x ∈ X , we use x_j to refer to the j^th component of the feature vector of x. We define our explanations as a set of functions, similarly indexed by j, and refer to f_j(x_j) as the importance of x_j. Given an arbitrary subset of features J ⊆ J over a data-point x, we wish to be able make predictions about the model response C(x) using only the sum of importance weights Σ_j∈J f_j(x_j)

In this way, we wish to be able to make a prediction about the model response C(x) using incomplete information about the feature vector of x, where only j ∈ J of the feature values are known. The f_j(x_j) individually serve as feature importances, and when summed are used to predict the model output.

B.1.1 Formulating Explanations from Incomplete Information

We give a general formulation suitable both for modelling discrete decisions using a logistic regression approximation, and for modelling continuous regression tasks using a linear regression approximation. In practice, these can be replaced with any additive model.

The functions f_j are parameterized by weights θ_j and the model is approximated by a function F of their sum and a global bias term G. Recall that we want to find the optimal set of additive functions f such that for any data point x ∈ X and arbitrary subset of features J we can approximate C(x).

As mentioned above, we use the linear and logistic approximations of C(·), in which f_θj(x_j) = a_jx_j + b_j. In the case of linear regression F is the identity function, whereas for logistic regression F is the logistic function σ.

For regression models, this yields:

and for classification this yields:

For regression models, we minimise the mean squared error, further simplifying equation S5. The per feature bias term b_j can disregarded if the mean of every feature j is 0, and the global bias term becomes equal to the mean model response C(x).

For classification models, we maximize the likelihood by marginalising over a distribution of randomly sampled features. We seek the values of θ_j that minimize the expected loss L[·, ·] (typically cross-entropy error) over all subsets of features J ⊆ J and data-points x ∈ X.

Like the formulation of SHAP, this objective is exponentially large with respect to the feature space. However, a key difference that allows us solve it effectively is that the number of function calls to C is equal to the number of data points |X|, and completely independent of the feature space J. By commuting this is equivalent to

Upon careful inspection, this is equivalent to performing dropout with probability p [14] on the initial approximation model σ(Σ_j∈Jf_θj(x_j) + G). This allows us to directly find the weights θ_j (recall we defined f_θj = a_jx_j+ b_j) and G using stochastic optimization by treating this as a regression problem regulated by dropout. Implementation of the method is straightforward, with the only subtlety to be aware of is that the weights and biases should be zero-initialized to avoid problems with vanishing gradients in the logistic regression loss.

B.1.2 Uniqueness and Stability

Each component of the objective defined in equation (S8)is convex. As such, if for every estimated variable there exists a strictly convex component containing that variable, the problem as a whole is strictly convex. Therefore, the overall objective has a unique minimum provided the individual functions f_θj(·) are well-posed and strictly convex with each function having a single minimum associated with it.

B.1.3 From Approximation Model Weights to Interpretable Feature Attributions

The optimization procedure returns a set of functions f_θj that then need to be applied to each feature. The response f_θj(x_j) can be directly interpreted as a signed measure of the importance of feature j on data-point x. For logistic regression, the sum Σ_j∈Jf_θj(x_j) can be directly interpreted as probability in logit space of the classifier response C(x); while for linear regression the same sum is an estimate of the model prediction C(x).

C XAI for Regulation and Auditing

Enforcement of existing AI regulation often falters due to a lack of evidence, stemming from weak or missing requirements for transparency and model access for regulators, third-party auditors, and researchers [81–83]. Gathering sufficient evidence of potentially illegal model behaviour is highly difficult without full API access that would allow regulators to feed the model data to evaluate its responses or oversee model outputs holistically [84–86]. By working without model call access and requiring only observations of prior model outputs or predictions,RoT can bypass significant barriers to regulatory enforceability created by trade secrecy and confidentiality. RoT thus democratizes access to explaining and auditing AI systems. This is crucial because strong provisions enabling access or auditing by third parties (e.g., civil society organizations, vetted researchers) are uncommon in AI regulation, with Article 40 of the EU Digital Services Act being a rare exception [58, 87].

While Article 74(13) grants market surveillance authorities access to the source code of a high-risk AI system under a limited set of circumstances6, Article 74(12) grants market surveillance authorities access to the training, validation, and testing datasets to carry out investigations into these systems. Importantly, this is limited to dataset access, and normally does not include full model access or enable regulators to feed new data to a model to check its responses 7. RoT offers a clear advantage over other methods in this regard because it can compute accurate explanations using only these datasets, while others require full API access or training a mimic model.

RoT can also be used to identify anti-competitive practices and prevent self-preferencing. By explaining model predictions and recommendations,RoT can be used to infer “how” a digital service makes certain recommendations or “why” it behaves in a certain way. Competition law, however, does not always require establishing causality. A recent case against Google in the United States has shown, for example, that proving the mere existence of anti-competitive conditions is sufficient to show the law has been breached [88]. In this context, even if specific instances of this behaviour in a production environment (e.g., actual transactions with customers) cannot be shown due to a lack of access to model outputs, RoTcan demonstrate that a model displays anti-competitive behaviour. RoT can help demonstrate that model behaviour is likely to contravene the competition law prohibitions, and thus assist in the enforcement of competition law. Unlike measuring the sensitivity of mimic models, RoTmakes no causal assumptions and thus its ability to predict behaviour of models directly makes it a stronger tool to demonstrate potentially discriminatory behaviour.

D Experimental Details

D.1 Explaining Zero-Shot Image Classification from the GPT-4-mini API

We use the GPT-4o-mini API from OpenAI to label a balanced subset of 5000 images from a Kaggle dataset of cats and dogs [3]. To do this we pass each image to the language model via the API, along with the prompt ‘Is this a dog or a cat?’. We then extract features from the images using mobilenetv3[15] as a backbone. The image embeddings extracted this way are used as input features for a RoT classifier, trained using 75% of the labelled images. Explanations, comprising of positive or negative importances for the mobilenetv3 features, are computed for the 25% test split, and visualised after per-image-normalisation as overlaid saliency maps. The GPT-4o-mini API has perfect classification accuracy, excluding ambiguous or incorrectly labelled images from the dataset, and the RoTexplainer has a training accuracy of 98.5% and test accuracy of 97.6%.

D.2 Explaining Judicial Case Predictions from a Fine-Tuned RoBERTa Model

From the published PredEx [17] data, we start by removing all annotations that contain hallucinations or text that we could not reliably match in the raw case text. We drop datapoints with missing case text or missing annotations, and datapoints where no text segments were found to be important or where the entire case text was found to be important. The data was annotated by law students who tried to identify text responsible for the judgement. However, the finetuned model making predictions does not always match the ground-truth predictions. Since the explanation annotations are produced from the ground-truth, but our explainers try to explain the model, there is a discrepancy. To resolve this, we drop all cases where the model predictions were incorrect, since we are interested in explaining the model but comparing our outputs with annotations which were produced independent of the model. From the remaining subset, we further drop those cases where highlighted segments were too “short” to be reliable: we enforce the condition that the minimum character length of a highlighted or non-highlighted segment in the text should be at least 75 characters. This threshold is large enough to ensure that common phrases do not get highlighted unnecessarily, and small enough to ensure we still have enough datapoints pass the filter and be included in our experiment.

This finally leaves us with a set of 502 cases, from which we compute SHAP and RoT explanations on a random 20% of cases (101 cases total). TheRoT explainer we build uses the remaining 401 cases as background, but results using all 502 cases or only the 101 explanation cases as background are qualitatively similar. For the 401 case corpus, RoT is only fit on the final RoBERTa prediction per case, and not on the per-segment annotations within the case text.

We report the AUROC of the explanations with respect to the ground truth human annotations (Table 1). Computing this metric involves comparing explanation importances (which can be negative) with human annotations (which are a binary 0 or 1). Since the annotations indicate whether text is important without direction (appeal rejection or acceptance), we post-process the feature importance by aligning them with the case prediction manually. That is, if the case appeal was rejected, we consider negative values to denote higher importance by flipping the sign of the importance for all segments, and if the case appeal was accepted, we consider the importance values as is. This way, positive values always indicate higher importance as defined by the annotators. This post-processing is not specific to RoT and is common to all explainers because it is a peculiarity arising from the PredEx data.

For RoT explanations, we use a straightforward implementation where dropout is used both on input features and on input tokens. For SHAP, we use the canonical partition explainer implementation (which defaults to 500 background samples) available for use with language models8. For LIME we use the text explainer with its default bag-of-words tokenizer, with its default 5000 background samples. In addition to abridged example in Figure 4, program runtimes are documented in Figure 9. TheRoT and SHAP explanations for the unabridged case can be seen in Figures S2 and S3. Additionally, LIME and Integrated Gradients explanations for this particular case can be seen in Figures S5, S6, and S4 respectively. Results from other cases can also be analysed via https://kairawal.github.io/Rule-of-Thumb-Explaining-Artificial-Intelligence-Systems-using-Partial-Information/JudicialCaseOutcomePrediction/Code/results.html.

D.3 Explaining Movie Review Classifications from an LLM API

The ERASER[19] benchmark provides multiple text datasets with annotations indicating importance of constituent tokens. Most of the tasks however consist of retrieval (from an LLM context window), which can be evaluated using these annotations. The movie review dataset[18], however, presents an exception. The task here is simple zero-shot classification and does not involve any explicit retrieval.

We use the GPT-4.1-nano API to obtain predictions for movie reviews (positive or negative), which we then seek to explain using RoT. Our dataset consists of of 1600 movie reviews from the ERASER benchmark dataset [19]. To explain these LLM review sentiment classifications we repeat our process from explaining fine-tuned RoBERTa model predictions for judicial cases exactly, producing RoT explanations for each text segment from the movie reviews and measuring how well they align with human annotations. The notable difference between the two setups is that when explaining judicial cases, we reused the embeddings from the finetuned RoBERTa model itself, whereas now we use ModernBERT [89].

Choice of Evaluation Metric For both LLM experiments discussed so far, we evaluate explanation quality using the area under the receiver-operating-characteristic (a plot of the true positive rate against the false positive rate). We choose this metric over the area under a precision-recall curve because the precision-recall curve is sensitive to annotation density [90]. That is, the precision recall metric is highly sensitive to what proportion of tokens are deemed important for any given text. Since the annotations are booleans, this is dependent entirely on annotator idiosyncracies – we simply do not have ground-truth importance scores per segment). To illustrate the impact of this, consider the areas under the precision-recall curve reported in the top row of Table S3. If we simply flip all the labels in the ground truth annotation and simultaneously flip the signs of theRoT importances (the equivalent of finding / deleting unimportant segments instead of finding / selecting important segments from a given review), the metrics change as is visible in the bottom row of the table.

D.4 Explaining Zero Shot Resume Filtering using Proprietary LLM APIs

We replicate a study that found GPT-3 to be largely unbiased in resume filtering tasks [10]. Starting with a synthetic corpus of resumes consisting of information-technology workers, accountants, aviation workers, construction workers, chefs, advocates, teachers, and salespeople; we use GPT-4.1-nano to select the IT workers from this corpus. In such a scenario where zero shot classifications are obtained from an LLM API which cannot be queried at will, RoT is still able to explain predictions.

Just like explaining case outcome prediction and movies, we are able to produce token level importance scores using RoT. This time, we use BERT[91] embeddings instead of ModernBERT[89], showing how our method is embedding agnostic and can be used with various text embeddings. We additionally augment explanations with non-input features so predictions are explained not just in terms of the resume, but also in terms of the demographic attributes of the candidates. The original study used race, gender, and political orientation, and we explicitly add this information to our explanation text (even though the predictions were made using only the resume summary). This is an additional ability unique to RoT, and is particularly useful for black-box algorithmic auditing. It allows us to generate feature importance explanations using a superset of the model inputs – we include race, gender, and political orientation. Our results match the original study, and find that these features have negligible impact on model predictions.

In addition to the wordcloud in Figure 5, we visualise the importance ascribed to a particular token across resume summaries. A cursory examination of these shows that IT related words are positively correlated with IT workers, whereas words belonging to other professions are negatively correlated. Some of these can be seen visualised in Figure S7.

D.5 Auditing Proprietary Black-Box AI Systems without Mimic Models

Journalists at The Markup audited Amazon’s product recommendation system. Since the model used is proprietary, it cannot be queried at will, which renders all previous explainers inapplicable. To circumvent this, product recommendation data was scraped from the Amazon website, consisting of product pairs – the first and second product recommended for any particular search. A model would be built as a “proxy” to mimic Amazon product rankings, formulated as a binary classifier selecting one of the products from the scraped pair. Feature inputs to this model were also scraped, consisting of attributes like product stars (the difference in number of stars between the pair of models), and whether the picked model brand was Amazon, was sold by Amazon, shipped by Amazon, etc.

The Markup analysed this data by building a Random Forest model and then computing SHAP explanations of this model9. Following this methodology, they discovered Amazon to systematically promote products from its own brand over others, as SHAP considered “brand is Amazon” to be the most important feature, and “product reviews” to be second. The model trained and explained was not arbitrary, but chosen after a typical hyperparameter search. When we replicated this audit, we found several models that were (marginally) more accurate, which we list in Table 2. We found the scikit-learn default Random Forest had better performance than the Random Forest with hyperparameters chosen from The Markup’s hyperparameter search. We also found the default Logistic Regression in scikit learn to have better performance, as did both regularised Logistic Regressions (C=0.01).

All our models had similar accuracies and formed a Rashomon set. Their internal mechanisms were distinct, as can be verified from the global explanations summarised in Table S6. This variation is also evident from the SHAP explanations for these models, which led to different conclusions about input feature importances depending on the mimic used. All models found “brand is amazon” to be the most important input feature, but following this, subsequent features had different importances depending on the model selected. For brevity, we present SHAP explanations from just Random Forest models in Figure 6, but include results from all models in Figure S8 and Figure S10.

RoT explanations, by contrast, do not require mimic models. Results obtained from RoT are therefore stronger in an auditing context, because they cannot be attributed to peculiarities in any mimic model, which can otherwise be used to absolve the original inaccessible, proprietary model.

D.6 Literature Review to Determine the Prevalence of XAI for Scientific Discovery

We measured how often explainability is used as a means of scientific discovery to understand the natural phenomena that underlie models by performing a literature review.

To do this, we analysed papers fetched from the Scopus database related to various scientific fields that used some form of interpretable machine learning. We queried the Scopus API using the TITLE-ABS-KEY field with a search string constructed as follows:

This yielded 1151 articles, which we filtered using LLMs to analyse the abstracts for mentions of XAI for forming new scientific hypotheses. Using the LLM filter, we reduced our corpus to 111 articles, which we checked manually to verify the use of XAI in each paper. We found that 56 of these papers used XAI, and 39 of the 56 did so using the SHAP explainer. These article lists can be found at:

• https://github.com/KaiRawal/Rule-of-Thumb-Explaining-Artificial-Intelligence-Systems-using-Partial-Information/blob/main/LitReview/a.Corpus_1151.csv
• https://github.com/KaiRawal/Rule-of-Thumb-Explaining-Artificial-Intelligence-Systems-using-Partial-Information/blob/main/LitReview/b.LLM_filtered_111.csv
• https://github.com/KaiRawal/Rule-of-Thumb-Explaining-Artificial-Intelligence-Systems-using-Partial-Information/blob/main/LitReview/c.Manual_XAI_56.csv
• https://github.com/KaiRawal/Rule-of-Thumb-Explaining-Artificial-Intelligence-Systems-using-Partial-Information/blob/main/LitReview/d.Manual_SHAP_39.csv

respectively. Our reported statistic of 4.9% papers using XAI for scientific discovery is thus a lower bound – there might be papers using XAI that were missed by our LLM filter and never included in set of 111 papers we manually inspected. If the LLM false negative rate is assumed to be the same as the false positive (55/111) rate, our final result could be as high as 49%. Our objective however is not to perform a thorough literature review or identify what proportion of papers use SHAP to generate new scientific discoveries. Instead, we merely wish to demonstrate that a significant amount of modern scientific discovery relies on sensitivity based explainers to generate hypotheses that seek to causally explain real-world phenomena.

D.7 Identifying Uninformative Features as Unimportant for Scientific Discovery

When using XAI for scientific discovery, we want explanations not to consider uninformative input features as important. We deliberately transform[55] the “age” feature of the “Pima Indians”[13] diabetes prediction dataset to be uncorrelated with diabetes, the target feature, and then compute explanations with various explainers to test which ones still find “age” to be important.

To make a feature statistically uncorrelated with the target, we adopt the FairPCA[55] method to transform our dataset. For our dataset with 8 input features, this transformation works by learning a low dimensional space of 7 features, such that age information is deliberately lost. By transforming our data into this space, any model trained cannot utilise age information. Finally, we learn a Logistic Regression classifier on the data, and use standard SHAP, LIME, and RoT explainers to obtain feature importances. As seen in Figure 7, we find sensitivity based explainers unable to identify the unimportance of age, but RoT does this perfectly.

D.8 Invulnerability to Adversarial Attacks that Mask Important Features

We consider a scenario where an adversary tries to mislead explainers into selecting spurious “foil” features as important, instead of the “sensitive” features the model actually uses to make decisions.

We use the same datasets as those used in the paper proposing the adversarial attack [56], and follow its setup exactly: for the COMPAS [92] and Communities & Crime [93] datasets, the “sensitive” feature is “race” whereas for the German Credit [94] dataset it is “gender”. All models are binary classifiers formulated as simple if statements depending on the sensitive feature, predicting recidivism risk for the former two datasets, and loan repayment probabilities for the latter.

The “foil” attributes used are synthetically added features for COMPAS and Communities and Crime, whereas for the German Credit [94] dataset, the foil used is not synthetic but a new feature that is a function of other input features. Further, instead of 3 experiments (one for each dataset), for COMPAS and Communities and Crime we conduct experiments twice – once with a single synthetic foil, and once with 2 synthetic foils. The adversary modifies models to mislead explainers. These modifications work can be tailored to the particular perturbations used by a sensitivity based explainer, and we consider models designed to mislead both SHAP (m_S) and LIME (m_L), comparing both with RoT.

Like in the original paper, this setup leads to 5 experiments with each explainer: 1 German Credit, 2 COMPAS, and 2 Communities and Crime. For each datapoint in each experiment, we consider the absolute importance of the sensitive feature, average absolute importance of the foil features, and average absolute importance of all other features, forming a sample from a ternary distribution. We visualise this distribution of importances in Figure 8 using publicly available software [95, 96], combined from across each of the 5 experiments so each experiment has the same weight. We also present exact numerical results for detailed perusal in Table S7.

D.9 Benchmark Evaluations of RoT Explanations with OpenXAI

We analysed RoT in traditional tabular modalities using an existing XAI benchmark, since it can be used as a drop-in replacement for other feature-importance explainers. Evaluating explanation quality is a nuanced problem, and we find standard benchmarks unsuitable to measure quality, but are able to observe that RoT produces similar explanations as SHAP in a faster and more computationally efficient manner. Sensitivity analysis underlies XAI evaluation benchmarks just like it underlies explainers. The OpenXAI benchmark [80] consists of explanation evaluation metrics that either perform sensitivity analysis, or comparisons with “ground-truth” explanations.

OpenXAI considers Logistic Regression coefficients as the “ground-truth” explanation for all datapoints. Regardless of the specific feature values for a given datapoint, such ground-truth comparison metrics consider a single explanation as ideal across all input datapoints, promoting a single unchanging explanation regardless of specific input values. Using Logistic Regression coefficients OpenXAI deliberately promotes gradient methods or LIME, which are Taylor-series like approximations of AI system behaviour and can recover the Logistic Regression coefficients exactly for any input datapoint. While these explainers score better than SHAP in the published OpenXAI benchmark scores, they are undesirable in practice because they cannot explain individual predictions and always produce identical feature importance outputs. Due to these aforementioned issues, instead of reporting the official metrics from OpenXAI, we adopt the benchmark to verify that RoT behaviour mirrors SHAP, and every input datapoint does not yield the same explanation output.

We compute explanations of the pretrained OpenXAI models by running publicly available code from the benchmark[80] as provided10. We present results from the COMPAS dataset because it has the fewest features from among OpenXAI datasets, allowing us to present results succinctly. To select an initial point and a final point for plot (a), we find pairs of datapoints with different model predictions such that l0 distance between them is 2 or less, and l1 distance between them is as large as possible. For plot (b), we consider all pairs where l0 distance is exactly 1 while maximising l1 distances as before, and ensuring the feature changed is always “length of stay”. While we present select results, we are able to repeat this process for all provided pretrained models and datasets: COMPAS, Home Equity Line of Credit (HELOC), Adult Income, and German Credit. These provide the runtime statistics seen in Tables S4 and S5.

Figure S1 shows how changing input feature values do not always lead to changes in the corresponding feature importances. Only the SHAP and RoT explainers show desirable behaviour, where the local explanation is truly a function of the datapoint being explained. LIME and gradients show the same feature importance regardless of the feature values, undermining the usefulness of OpenXAI metrics which consider these explanations to be the best. All gradient based methods in OpenXAI generated similar explanations, and here we only show Integrated Gradients for clarity.

E Extended Results

E.1 Benchmark Evaluations and Computational Efficiency

We found fundamental problems with the OpenXAI benchmark that prevented its adoption as-is. In Figure S1 we show how RoT and SHAP explanations for example datapoints in the benchmark are the only ones that actually vary when the prediction being explained varies. The “ground-truth” metrics in the OpenXAI leaderboard are known not to score SHAP explanations highly. However, we were able to use the benchmark to measure runtimes across explainers. Tables S4 and S5 document RoT runtimes against SHAP and LIME across the OpenXAI benchmark and all our experiments with tabular data.

(a) Feature importances for an initial datapoint and for a final datapoint with a changed prediction. We expect input feature changes to change the corresponding importance, but this happens only for RoT and SHAP. LIME and Gradients are constant despite changes in “age” and “length of stay”.

(b) Changes in “length of stay” feature importance for changes in input feature values. Like in (a), RoT and SHAP change but LIME and Gradients are constant

E.2 Evaluating Zero-Shot LLM Classifier Models with XAI

Unabridged examples of the case prediction explanation from Figure 4 using RoT (Figure S2), SHAP (Figure S3), Integrated Gradients (Figure S4), LIME reduced to 500 background samples to reduce program runtime (Figure S5), and LIME with 5000 default background samples (Figure S6).

E.3 Auditing Proprietary Black-Box AI Systems without Mimic Models

Replicating The Markup’s audit of Amazon’s Product Recommendation Model Using SHAP on a mimic model is not guaranteed to recover explanations faithful to the original model being audited. We constructed different “mimic” models that emulated the Amazon product recommendation system, and showed that SHAP explanations contradicted each other for the Markup created Random Forest, and the default hyper-parameter Random Forest. We now extend this analysis to the other logistic regression mimic models we created, producing extended versions of Figure 6 showing SHAP explanation distributions and SHAP explanation feature rankings respectively in Figures S10 and S8 respectively. We also show global importances for all models, verifying that mimic models do indeed have different internal mechanisms despite producing predictions with similar accuracy rates.

(a) RoT importances for “infrastructure” across our resume corpus

(b) RoT importances for “network” across our resume corpus

(c) RoT importances for “teaching” across our resume corpus

(d) RoT importances for “financial” across our resume corpus

(e) RoT importances for “construction” across our resume corpus

(f) RoT importances for “professional” across our resume corpus

(a) The Markup model finds the most important feature to usually be either “brand is amazon” or “product reviews”

(b) The default random forest model picks a more diverse set of features as most important

(c) The logistic regression model picks a different set of features as most important

(d) The L1 regularized logistic regression picks only “brand is amazon” as important

(e) The L2 regularized logistic regression model picks a different set of features as important

(f) RoT usually finds “brand is amazon” to be the most important, and often finds “sold by amazon” to be the second most important

(a) The Markup model finds the most important feature to be “brand is amazon” with “sold by amazon” second

(b) The default random forest model picks a more diverse set of features as most important

(c) The logistic regression model picks a different set of features as most important

(d) The L1 regularized logistic regression picks only “brand is amazon” as important

(e) The L2 regularized logistic regression model picks a different set of features as important

(f) RoT usually finds “brand is amazon” to be the most important, and often finds “sold by amazon” to be the second most important

(a) The Markup model finds “brand is amazon” and “product reviews” to have importances of high magnitude

(b) The default random forest model generates a different profile of importance distributions

(c) The logistic regression model finds a different set of importances than both random forest models

(d) The L1 regularized regression model produces considers only “brand is amazon” to be important

(e) The L2 regularized regression model produces a different importance distribution

(f) RoT importances can be computed without any mimic models

(a) The Markup model finds “brand is amazon” and “product reviews” to have importances of high magnitude

(b) The default random forest model generates a different profile of importance distributions

(c) The logistic regression model finds a different set of importances than both random forest models

(d) The L1 regularized regression model produces considers only “brand is amazon” to be important

(e) The L2 regularized regression model produces a different importance distribution

(f) RoT importances can be computed without a mimic model

E.4 Scientific Discovery using XAI

Identifying Important Features in the presence of Misleading Features Table S7 effectively summarises the most important feature across datapoints, as picked by various explainers. This is the underlying explanation data that was plotted in the ternary distribution plots in Figure 8. To provide a more granular analysis, we also provide additional confusion matrices comparing the first, second, and third most important features as selected by various explainers across various datasets. in Tables S8, S9, S10, S11, and S12.