Home Explainable deep learning improves human mental models of self-driving cars

Explainable deep learning improves human mental models of self-driving cars

Health & Medicine

September 2, 2026

Explainable deep learning improves human mental models of self-driving cars

Architecture

The black-box ML planner uses a modified version of the DriveIRL architecture^1,2 (Fig. 1b). For the trajectory generator G, we use a heuristic generator that produces 143 jerk-optimal trajectories to anchor waypoints along the route. For the scene encoder H, we use the hierarchical vector transformer (HiVT)⁵¹ pretrained for multi-agent motion prediction. Apart from the scene embedding h, this produces an additional 3 trajectories for the AV, for a total of k = 146 candidate trajectories. In the scene-trajectory encoder E, trajectories are encoded using a recurrent neural network and then fed jointly with the scene embedding into a transformer layer, which produces the scene-trajectory embeddings z_i. The reward model R is a multilayer perceptron (MLP). In CW-Net (Fig. 1c), the classifier C and the new reward model ${R}^{{\prime} }$ are MLPs.

We avoided testing other methods apart from CW-Net because, at the time of writing, we are unaware of other works that are capable of modelling interpretable-by-design IRL systems. Moreover, we broadly seek a general comparison of concept-based explanations (that is, CW-Net) against no explanation (that is, our control) to help generalizability of the results across the myriad concept-based explainability techniques in the literature.

CW-Net training

We used two datasets:

Dataset 1: 500,000 scenarios and 8 concept labels (Supplementary Table 1)
Dataset 2: 3,000,000 scenarios and 10 concept labels (Supplementary Table 2)

For a full list of the concept labels and their meanings, see section ‘Concept details’. Each scenario was associated with 146 trajectories, thus giving between 73–438 million training data points for the concept classifier, each with multiple concept labels. Our algorithm assumes CW-Net training has access to the original dataset used to train the black-box ML planner, along with annotated human-understandable concept labels for each of these data points. The annotations can be multi-label, meaning that one datum can be associated with as many concepts as desired or useful. The experiments with CLOSE and ASV concepts used models trained on dataset 1. The experiments with BIKE and PEDESTRIAN concepts used models trained on dataset 2.

During training, the parameters of the trajectory generator G, the scene encoder H and the scene-trajectory encoder E are frozen, and only the concept classifier C and the new reward model R′ are trainable. Two separate losses are optimized jointly.

First, a concept classification loss ${{\mathcal{L}}}_{\mathrm{concept}}$ is used to train C to predict the correct concept label(s). In our setting, this loss combines cross-entropy with binary cross-entropy for different concepts, depending on the semantics of the corresponding scenario types. For example, in dataset 1, we use cross-entropy to model the steering concepts of the car (LEFT, RIGHT and STRAIGHT), and the speed concepts (STOP and SLOW), while also using binary cross-entropy to predict the presence of other concepts such as ASV, INTERSECTION and CLOSE. These losses are then averaged into one:

$${{\mathcal{L}}}_{\mathrm{concept}}=\frac{1}{2k}\mathop{\sum }\limits_{i=1}^{k}\left(\frac{1}{{M}_{\mathrm{CCE}}}\mathop{\sum }\limits_{j=1}^{{M}_{\mathrm{CCE}}}{{\mathcal{L}}}_{\mathrm{CCE}}({c}_{i,j},{\hat{c}}_{i,j})+\frac{1}{{M}_{\mathrm{BCE}}}\mathop{\sum }\limits_{l=1}^{{M}_{\mathrm{BCE}}}{{\mathcal{L}}}_{\mathrm{BCE}}({c}_{i,l},{\hat{c}}_{i,l})\right)$$

where M_CCE is the number of concepts modelled using categorical cross-entropy (for example, steering and speed), and M_BCE is the number of concepts modelled using binary cross-entropy (for example, presence of features such as ASV, INTERSECTION and CLOSE). c_i,j and ${\widehat{c}}_{i,j}$ represent the true and predicted labels for the jth concept under CCE for the ith data point, whereas c_i,l and ${\widehat{c}}_{i,l}$ represent the true and predicted labels for the lth concept under BCE for the ith data point. On dataset 2, we take a different approach and model everything, including the speed concepts (STOP, SLOW and FAST), with binary cross-entropy. In general, these parameters can be tuned to fit the task at hand.

Second, a cross-entropy loss ${{\mathcal{L}}}_{\mathrm{trajectory}}$ is used to train the network to predict the correct trajectory, which we define as the original trajectory chosen by the black-box planner. Both losses are averaged:

$${{\mathcal{L}}}_{\mathrm{total}}=\frac{1}{2}({{\mathcal{L}}}_{\mathrm{concept}}+{{\mathcal{L}}}_{\mathrm{trajectory}})$$

A focal loss⁵² is applied to counteract data imbalances, just as in the original DriveIRL planner^1,2. Computationally, our networks were trained on a large distributed setup using PyTorch Lightning.

Concept separation

When adding interpretability modules post hoc, as we have, there is the possibility that the network will not have learned to separate the concepts of interest, and thus fail to be able to predict them accurately⁵³. We observed this in the experimental prototype we tested (Supplementary Table 2), when certain concepts such as CLOSE and PEDESTRIAN had poor precision and high recall, relatively speaking. There are two important points to note here. First, the better trained and more sophisticated an architecture is, the more it naturally learns to separate an impressive number of concepts in an unsupervised manner^54,55,56, so this is unlikely to be an issue for most companies with the flagship models in the future. Second, even if CW-Net has not learned to separate certain concepts (for example, red traffic lights compared with green ones), this could highlight the reason why the AV fails to act appropriately—such as stop compared with go—in a given situation (for example, because there are insufficient traffic light scenarios in the training data). Hence, from an explainability perspective, concepts with low accuracy are often particularly useful, as we demonstrate in the paper.

Alternative architecture

Alongside our primary causal architecture shown in Fig. 1, we also developed an alternative that gave post hoc justifications for AV behaviour (Extended Data Fig. 1). Specifically, we froze the weights of the pretrained black-box ML planner and trained a concept classifier head C to work in parallel to the reward layer R. Similarly to the causal architecture (Fig. 1c), C used the scene-trajectory embeddings z_i to classify the concepts. This approach is beneficial because of its relative simplicity and accessibility, although the drawback is that it may be less faithful to the model’s reasoning process, as the concept classifications are not directly used by the model to rank state-trajectory pairs. However, there is ample evidence that these explanations are often capable⁵⁷, so we include both as an option and demonstrate their utility. Specifically, this parallel architecture was used in the real-world scenarios with the CLOSE concept (section ‘Unexpected stopping for nearby vehicles’).

Concept details

Dataset 1 concepts were as follows:

LEFT, RIGHT, STRAIGHT: Classification of driving direction concepts, trained with cross-entropy loss. For example, the concept LEFT represents training scenarios in which the car was turning left.
STOP, SLOW: Classification of car speed concepts, trained with cross-entropy loss. The concept of, for example, STOP represents training scenarios in which the car was stopped.
ASV (approaching stopped vehicle): Scenarios in which the car was approaching a stopped vehicle. Trained with binary cross-entropy.
INTERSECTION: Scenarios in which the car was at an intersection. Trained with binary cross-entropy.
CLOSE: Scenarios in which the car was within 3 m of another vehicle. Trained with binary cross-entropy.

Dataset 2 had the following concepts:

SLOW: Scenarios in which the car was driving at 1–2 m s⁻¹.
STOP: Scenarios in which the car was stationary.
FAST: Scenarios in which the car was driving faster than 2 m s⁻¹.
STOP SIGN: Scenarios in which the car was close to a stop sign.
TRAFFIC LIGHT: Scenarios in which the car was close to a traffic light.
INTERSECTION: Scenarios in which the car was at an intersection.
PEDESTRIAN: Scenarios in which the car was close to a pedestrian.
FOLLOWING: Scenarios in which the car was following another vehicle.
BIKE: Scenarios in which the car was close to a cyclist.
PUDO (pedestrian pickup/drop-off): Scenarios in which the car was in a pedestrian pickup–drop-off zone.

All concepts in dataset 2 were trained with a binary cross-entropy loss.

Although the AV was trained on 8–10 concepts (depending on the dataset), we focus our study on a subset of concepts (CLOSE, ASV, BIKE and PEDESTRIAN) because they are the only ones that relate to other road users and are easy to test counterfactually. The concept FOLLOWING involves other road users too, but no notable occurrences happened during real-world deployment.

Simulation results

We tested our CW-Net model across the entire nuPlan validation dataset to see how its driving performance compares with the original black-box ML planner it was trained from. The dataset is a large-scale planning benchmark for autonomous driving³⁹ and measures how close a trained AV is to a human expert in L₂ distance, progress along the route and safety (no collisions). In the black-box model, when following the lane or decelerating from high speed, the planner was able to make progress along the route (>93% of human driving distance), while avoiding collisions (>90% collision-free) and staying close to the ground-truth human expert trajectory (<1 m displacement at 5 s). Performance was worse when starting from a stop, with less progress (74% of human driving distance), more collisions (81% collision-free) and greater deviation from the human expert (1.2 m displacement at 5 s). Overall, the results showed our variation of the AV architecture had less than 0.01 L₂ difference to the original black-box agent on average across all measurements, and not meaningfully different, showing that it is possible to train our more interpretable model in Fig. 1 without sacrificing performance. The full results are in Extended Data Table 1.

For concept accuracy verification, we used 5% holdout data from our training datasets; the results are given in Supplementary Tables 1 and 2. Across both datasets, the mean accuracy was 0.54, precision 0.23, recall 0.77 and F1 score 0.31. Overall, the results indicated that CW-Net did not separate all concepts equally well, which suits our purposes as the explanations will highlight when and how this happens, and how it relates to driving performance, thus helping with mental model refinement (see section ‘Concept separation’). Notable results include an F1 score of 0.82 for detecting the SLOW concept, and close to zero for detecting the BIKE concept, showing the latter is perhaps not well encoded or understood by the car.

Mental model elicitation studies

These studies were designed to replicate the driver’s experiences in the private track tests (section ‘Mental model improvement in deployment’) with a larger cohort, to help demonstrate the robustness of our findings regarding the drivers’ mental models (Extended Data Fig. 2). The same study was run separately with experts (other drivers and test engineers from Motional) and non-experts (randomly sampled users from Prolific.com). Specifically, we were interested in testing the following hypotheses:

Hypothesis 1: The participants’ responses before observing the explanation (in the video replay) would be more consistent with the belief of the safety driver before observing the explanation (in the car).
Hypothesis 2: The participants’ responses after observing the explanation would be more consistent with the belief of the safety driver after observing the explanation.
Hypothesis 3: The participants’ responses after observing the explanation would be more consistent with the ground-truth reason for AV behaviour.
Hypothesis 4: Counterfactual prediction ability would correlate with user mental model goodness, the latter as measured by the nearest-neighbour tasks and the free-form text response in ref. ⁴³.
Hypothesis 5: The two measures of mental model goodness would correlate with each other and have similar distributions across groups.

Design and materials

We focused on the same three surprising events observed during the private track tests. The experiment was designed to measure mental models through a combination of nearest-neighbour and prediction tasks, accompanied by confidence scores and free-form text responses, in which we could probe the participants’ mental models⁴³ (Extended Data Fig. 2). First, users saw the respective video and were asked to rate two possible reasons for AV behaviour, along with confidence scores (that is, the nearest-neighbour mental model elicitation task). Then, users were asked to make the same counterfactual prediction as the driver (that is, the prediction task), along with a confidence score, and a free-form text rationale explaining their reasoning. Then, users saw the same video with the explanation and repeated the questions. This within-participant design mirrors the experience of the safety drivers during the on-road tests.

Participants

For the expert group, we recruited nine safety drivers, test engineers and test specialists from Motional (aged between 18 years and 80 years, 8 male and 1 female). All participants volunteered to participate and were not paid. For non-experts, we sampled users sourced from Prolific.com (www.prolific.com), which is known for its high-quality user base. Thirty users were randomly sampled US citizens aged between 18 years and 80 years, native English speakers and an even number of male and female. To ensure high-quality text responses, we paid users above the average rate with USD15 per hour and stressed that they should take the study only if they were certain they understood the instructions. The study received MIT IRB approval from the Committee on the Use of Humans as Experimental Subjects, exempt ID: E-5903, start date 1 July 2024, end date 31 August 2026.

Metrics

We used the nearest-neighbour task and the free-form text rationale as direct measures of mental models. For the nearest-neighbour task, we measured mental model goodness as a combination of choice accuracy and confidence. Specifically, we formalized mental model improvement as shifting from an incorrect to a correct belief (with respect to the ground-truth reasons for AV behaviour), or increasing confidence in the correct belief, or decreasing confidence in the incorrect belief (Extended Data Fig. 3). For the free-form text rationale, we used an LLM-as-a-judge (GPT-5) to determine which text response (before or after observing the CW-Net explanation) is closer to the ground-truth reason for AV behaviour (see Supplementary Methods for details). The LLM prompts were tuned over three iterations on the expert responses (Extended Data Fig. 4a) and then evaluated once on the non-expert responses (Extended Data Fig. 4b). Mental model improvement was formalized as instances where the free-form response after observing the CW-Net explanation is closer to the ground truth, compared with the free-form response before observing the explanation.

We used the prediction task as a measure of downstream performance that relies on mental models, thus measuring them indirectly^44,58. We measured prediction improvement as a combination of choice accuracy and confidence in the same way as for the nearest-neighbour task (Extended Data Fig. 5).

To analyse the relationship between direct (nearest neighbour, free-form text rationale) and indirect (prediction) measures of mental models, we used a linear mixed-effects model (LME) to evaluate the effect of the mental model change (improve or worsen) on the prediction change variable (that is, the delta in confidence change in the prediction), while controlling for individual participant variation as a random effect (Extended Data Fig. 5). To collapse accuracy and confidence on a single scale for computing the confidence deltas, we simply flipped the sign of confidence values for inaccurate beliefs to negative. All code for this analysis will be available on publication (see section ‘Code availability’ below).

Study conclusions

We found evidence favouring all of our hypotheses.

Hypothesis 1: The participants’ initial belief (before observing the explanation) was often more similar to the safety driver’s initial belief (P < 10⁻⁹, exact binomial test).
Hypothesis 2: The participants’ final belief (after observing the explanation) was often times more similar to the safety driver’s final belief (P < 0.0002, exact binomial test).
Hypothesis 3: The participants’ final belief (after observing the explanation) was often times more similar to the ground truth (P < 10⁻⁵, exact binomial test).
Hypothesis 4: There was a clear relationship between mental model category and predictive ability of users, with only free-form text response categories failing to reach significance with experts (Extended Data Fig. 5).
Hypothesis 5: Interaction analysis showed no significant difference in the relationship between nearest-neighbour score improvement and text rationale improvement across the two groups (P = 0.842; Extended Data Fig. 4c), indicating a consistent underlying mechanism for both experts and non-experts. Non-experts demonstrated a significant positive correlation (β = 0.27, P < 0.001), suggesting that improved scores were strong predictors of improved mental model rationale in this group. Although experts exhibited a nearly identical positive coefficient (β = 0.23), the relationship did not reach statistical significance, probably because of a smaller sample size (P = 0.243).

Public roads evaluation using SAGAT

We deployed CW-Net in manual mode on Las Vegas public roads to collect complex, naturalistic scenarios analogous to those already discovered during the private track tests (Extended Data Fig. 6). We used these as materials for an online SAGAT study (n = 100). This setup allows us to validate the robustness of the CW-Net explanations using a well-established rigorous framework (Extended Data Fig. 2). We used a between-participant design in which we compared CW-Net explanations (experimental group) against baseline explanations describing speed and steering (control group)⁴⁹ (Extended Data Fig. 7).

Gathering materials

We collected data for ASV (01:02:55), CLOSE (00:50:42), and BIKE ( > 3 hours). We labelled sequences where human driving mimicked the ML planner in surprising scenarios that resembled those already discovered in the private track tests (for example, CLOSE: stuck beside vehicles; ASV: braking for hallucinations; Extended Data Fig. 6). PEDESTRIAN replaced BIKE because there were no naturally occurring cyclists in the new video data. We sampled two surprising events per concept (6 total) and six corresponding ‘unsurprising’ events to ensure explanations did not degrade situational awareness in regular driving (Supplementary Table 5). Activation thresholds were 0.5 for ASV/PEDESTRIAN and 0.94 for CLOSE, derived from prior private track data (Fig. 3).

Note the distribution of concept activations did not change significantly between the private-track tests and the public-road tests (Supplementary Table 7), despite the tests being performed more than a year apart, on different AVs, with different software stacks and under completely different conditions. This demonstrates the robustness and reliability of the algorithm.

Study design

We used a between-participants design (n = 100) assessing SAGAT perception, comprehension, and projection. Participants were split into experimental (CW-Net explanations) and control (speed/steering placeholder) groups. Attention checks based on material content and viewing times reduced the pool to 99 participants, 51 in the experimental and 48 in the control.

Materials

Stimuli included 13 videos (6 surprising, 6 unsurprising and 1 attention check). The experimental group saw concept activations, whereas the control saw speed and steering data (Extended Data Fig. 8). Following each video, a ‘blackout’ screen presented six binary questions: four related to perception, one to comprehension and one to projection.

Participants

We recruited gender-balanced US residents (18+ years, native English speakers) by Prolific.com. Participants were paid $12 per hour. The study received MIT IRB approval from Committee on the Use of Humans as Experimental Subjects, exempt ID: E-5903, start date 1 July 2024, end date 31 August 2026.

Metrics

We analysed the average of each participant on each question type using two-tailed t-tests, splitting data by surprising and unsurprising events for each situational awareness dimension.

Study conclusions

After collection and attention check filtering, we collected 99 responses out of the target 100. The results are shown in Extended Data Fig. 7. In surprising events, explanations significantly improved situational awareness after Bonferroni correction, showing large effect sizes for perception (Cohen’s d = 1.290) and comprehension (d = 0.996), alongside a medium effect for projection (d = 0.606). In unsurprising events, no significant differences occurred (perception d = 0.085, projection d = −0.142, comprehension d = −0.514), confirming that explanations provided benefit in anomalous situations without adversely affecting situational awareness during routine operations. Furthermore, we confirmed feature robustness by comparing concept distributions to previous private track tests using Wasserstein distance (Supplementary Table 7), finding no meaningful changes.