Better average accuracy can coexist with worse inequalities. Research on prediction targets, clinical interaction and patient participation shows why fairness needs to be assessed through the decisions an AI system changes.
A healthcare algorithm can become more accurate while leaving a particular group of patients less well served. That possibility challenges a familiar assumption in medical technology: that improving the average result automatically improves the service for everyone. The relevant question is not only how often a model is correct, but whose needs it recognises and what happens after its recommendation.
For European health systems, fairness is consequently a practical design and evaluation problem. It involves the outcome a model is trained to predict, the people represented in its evidence and the way clinicians and patients respond to it. Research is showing that those elements can pull in different directions.
The wrong target can produce a convincing prediction
A landmark US study published in Science in 2019 examined an algorithm used to identify people for additional healthcare support. At the same risk score, Black patients were considerably sicker than White patients. The system predicted expenditure rather than illness, allowing unequal access to care and differences in spending to distort the identification of health needs.
The problem was not simply that the model could not predict its chosen outcome. It was that the chosen outcome was an imperfect substitute for the purpose of the service. Predicting who would generate higher costs did not reliably identify who needed additional help.
That distinction matters beyond the original US setting. A European service considering an algorithm for outreach could ask whether previous appointments, recorded diagnoses or healthcare use adequately reflect unmet need. A person with little recorded activity might be healthy, but might also face barriers to obtaining care.
The development brief should therefore explain why the prediction target is appropriate for the decision. Patients and frontline staff may identify reasons why an apparently convenient measure misses important needs. Their contribution can change what the model is asked to learn, before researchers begin comparing its performance.
A fair model does not guarantee a fair clinical partnership
A 2024 Nature Medicine experiment involved 389 dermatologists and 459 primary-care physicians from 39 countries. They assessed images in a simulated teledermatology setting. Decision support improved diagnostic accuracy, but the study also found that it widened the accuracy gap between light and dark skin for generalists. An overall improvement did not eliminate the disparity.
This was an image-based experiment with limited clinical context, rather than a trial of patient outcomes in routine care. It also included experimentally varied decision support to examine how clinicians responded. Its value is to show that fairness needs to be assessed in the combined human and software process, not inferred from the model alone.
A hospital could therefore compare decisions made before and after assistance, examining when clinicians accept, modify or reject suggestions. If the pattern differs between patient groups, investigators would need to understand why. The interface, available information and clinicians’ familiarity with particular presentations could all be relevant questions.
The outcome should be expressed in terms meaningful to the service. Missing a condition, making an unnecessary referral and delaying a decision have different consequences. Combining them into one score may conceal the inequality that patients actually experience.
Explanations need testing with the people who receive them
An August 2026 study examined explainable AI in dermatological decision-making through complementary digital experiments with members of the public and clinicians. It compared ways of presenting assistance and examined their effects on decisions, including differences across skin tones. The findings showed that the effect of an explanation depends on who uses it and how the decision is organised.
For developers, an explanation is therefore another part of the intervention to evaluate. A detailed rationale could help a user identify a mistake, but it could also make an incorrect recommendation sound more convincing. Greater confidence is not itself evidence that the resulting decision is better.
A practical test would include cases in which the AI is wrong and examine whether users can recognise the problem. It could also compare what happens when a person forms an initial judgement before viewing the recommendation. Those experiments would help establish whether an interface supports independent assessment or encourages uncritical acceptance.
Patient-facing explanations need their own assessment. A description that is useful to a specialist may not help somebody deciding whether to attend an appointment. Testing comprehension and the action people take is more informative than simply asking whether the explanation appears clear or trustworthy.
Fairness must be reassessed when the setting changes
Research published in 2024 on the limits of fair medical imaging AI examined what happens when models move between datasets. The investigators found that models adjusted to reduce disparities in one setting could retain high overall accuracy while developing substantial fairness gaps elsewhere. The research team’s account describes testing across additional hospital datasets.
The implication is specific: an evaluation at one institution cannot establish equal performance in every other institution. Patient populations, disease patterns, equipment and recording practices may differ. A reassuring headline result should therefore prompt a closer examination of the deployment population, rather than close the question.
Local testing also needs enough information to support its conclusions. A subgroup with very few relevant cases can produce an uncertain estimate even when the overall dataset is large. Reporting that uncertainty is more useful than presenting a precise-looking percentage without its denominator.
Monitoring could focus on a small set of clinically important disparities agreed before launch. Where a difference emerges, the response might involve additional evaluation, changes to the service or restrictions on use. The point is to link the finding to a decision and an accountable team, rather than collect subgroup figures that nobody acts upon.
Dataset documentation should make missing people visible
The STANDING Together initiative developed 29 consensus recommendations on diversity, inclusion and the use of health datasets. Published in The Lancet Digital Health, the work addresses how data are documented and used so that limitations are more visible to people developing healthcare AI. The recommendations are a research and practice resource, not a legal certification of fairness.
Documentation can help a purchaser distinguish a genuinely examined population from one whose presence is simply assumed. It should also make clear when relevant characteristics are unavailable or recorded inconsistently. Absence of information about a group is not evidence that the system performs equally well for that group.
The useful response is not to collect every possible attribute without a defined purpose. Teams need to explain which questions the information will answer, how it will be protected and whether the resulting analysis is sufficiently reliable. Statistical assessment and appropriate data governance have to be developed together.
This can also expose a mismatch between the dataset and the service. A model built from patients who already reached specialist care may have limited evidence about people who struggle to obtain a referral. Better documentation allows that limitation to enter the deployment decision instead of remaining hidden in a methods appendix.
Patient participation should change something identifiable
A companion account of STANDING Together describes patient and public participation in developing the recommendations, including contributors with experience of different healthcare systems. It provides an example of involving people in the work behind a technical framework, rather than only asking them to endorse its final presentation.
For a healthcare AI project, meaningful participation could shape which errors matter most, how uncertainty is communicated and what alternatives remain available. Participants should be able to see which decisions their contributions influenced. A record of changes would provide stronger evidence of involvement than the number of meetings held.
The process also needs to accommodate people who cannot easily attend conventional workshops. Timing, accessible materials, language support and payment for time can affect whose experience is heard. No single contributor can represent every patient, and disagreement may reveal a real trade-off that the service must address openly.
Fairness becomes more concrete when an organisation can explain the benefit it is pursuing, the groups it has examined and the action it will take if a disparity appears. Better models are part of that work. So are better questions about access, clinical judgement and patient priorities. The meaningful endpoint is a service whose improvements reach the people it is intended to help.
