Nathaniel Sharadin (University of Hong Kong), "Predicting and Preferring"
Forthcoming, Inquiry: An Interdisciplinary Journal of Philosophy
Most people don’t have advance directives. You probably don’t have one. You should get one. If most people had advance directives, I probably wouldn’t have written my new paper. More on that in a bit.
Because most people don’t have advance directives, clinicians regularly face a difficult moral problem. Barring exceptional circumstances, a patient’s care should, on grounds of autonomy, reflect their preferences. But in the event of a patient’s incapacitation, it’s impossible to directly interrogate preferences concerning care. Hence, the moral problem: care morally should match a patient’s preferences, but, absent an advance directive, the content of an incapacitated patient’s preferences is often opaque.
We know how best to solve -- or at least how best to mitigate -- this moral problem. People should get (i.e., make) advance directives. Policymakers, in turn, should take reasonable steps to incentivize the widespread adoption of advance directives. For instance, in the US context, it is fiscally wasteful, medically foolish, and, as I’m right now pointing out, morally careless to allow seniors -- a group far more likely to become incapacitated than others -- to enroll in Medicare without completing an advance directive. It is morally careless because for no good reason it risks a well-understood, widely recognized moral harm, viz. the harm associated with a patient’s receiving care they prefer not to receive. Other countries -- some with better healthcare systems than the US (as measured by outcomes) — have equally bad adoption rates for advance directives.
This is a classic example of a public policy failure enabling an ongoing moral disaster. Failed housing policy enabling record levels of homelessness is another. In the case of housing, we know what to do: give each person a home (thus averting homelessness); and we know what public policy should do: incentivize building homes (thus reducing the shortage of affordable shelter). In the case of caring for incapacitated patients, we know what to do: get advance directives (thus averting the moral problem); and we know what public policy should do: strongly incentivize advance directives (thus reducing the problem’s incidence rate).
But people systematically don’t do what they should, and public policy systematically fails to incentivize the right things. So we’re left with suboptimal solutions to our moral problem. One suboptimal solution is to use surrogates to decide on behalf of incapacitated patients. But surrogates are famously epistemically unreliable. Another idea is to simply pick the cheapest medically acceptable course of treatment for those without advance directives. This is likely to be unpopular. On the upside, a pick-the-cheapest regime would incentivize people to simply do what they should and make an advance directive.
Recently, philosophers and biomedical ethicists have proposed a more interesting suboptimal solution: we can use modern machine learning (ML) techniques to train a predictive model of patients’ preferences (a “PPP”). A PPP then takes as input (e.g.) demographic descriptors and other known facts about a particular patient and produces as output a probability distribution over medically acceptable courses of treatment, where accuracy for the prediction is measured by calibration to the particular patient’s actual preferences over the relevant courses of treatment.
In earlier work I argued that the use of PPPs in medicine runs the same kinds of moral risks associated with the use of so-called “naked” statistical evidence in the justice system (e.g. in predictive policing and the law). I’ve also argued that there are systematic barriers to implementing PPPs, even if we think they are a morally good idea (they aren’t). There are also enormous practical barriers to (legally, one hopes) acquiring data sufficient to train such a model, let alone data required to validate it. And there are relatively unexplored problems with PPPs involving patient’s incomplete or “gappy” preferences. These are all fertile ground for new work.
Despite these existing challenges, PPPs remain a live proposal with active support. My new paper presents another challenge for the use of PPPs in medicine. That challenge can be summarized in a sentence: patients are owed assurance that models trained to predict their preferences are not also incentivized to shape those preferences.
What does it mean to say that a predictive model could be incentivized to shape a patient’s preferences?
When it comes to a model for predicting anything -- the weather, the news, whatever -- there are two fundamentally different methods for improving that model’s predictive accuracy (as measured by Brier Score, say). On the one hand, we can improve the accuracy of a predictive model by holding the world fixed and changing the model. This is what happens when we adjust a parameter, add a variable, etc. This is familiar stuff; think of Nate Silver tweaking his election forecasts in the runup to a US election cycle, or climate scientists adjusting their predictions about the planet’s warming.
On the other hand, we can improve the accuracy of a predictive model by holding the model fixed and changing the world. This method for improving accuracy is less familiar not because it’s less common, but because it’s less common among the most visible, or most salient forecasting models. Nate Silver isn’t rigging elections in order to have his forecasts beat the market. Climate scientists aren’t conspiring to heat the planet as a way of calibrating their models.
But some predictive models, in particular the kinds of content recommendation engines that are ubiquitous (though largely invisible) online, regularly do exactly this. Here’s a simple example: you are predicted by some digital platform to prefer content with feature A to content with feature B. Let’s say that prediction is inaccurate: you actually slightly prefer content with feature B to content with feature A. But as a result of the prediction, you are shown more content with feature A. And then as a result of the mere exposure effect, you come to like content with feature A more than you previously did. Since your initial preference for B over A was very slight, you now prefer A to B. The prediction of the content recommendation engine is thereby made more accurate by a change to the world, rather than a change to the prediction.
Machine learning researchers call this phenomenon “performative prediction,” an unwieldy name I begrudgingly agree to use. In broad strokes, performative prediction is what happens when a model’s prediction makes a difference to the actual distribution of the probabilities in the world that it is aiming to accurately model. Performative prediction (sometimes called “auto-induced distributional shift”) is possible using many different kinds of models with different architectures (and training regimes) and in many different kinds of deployment environments.
Maybe some risk of some kinds of performative prediction doesn't really matter. It’s arguably very bad for me only in one specific sense if I end up preferring to consume content concerning annoying NIMBYs to content comprising good philosophy because of content recommendation engines run by Reddit, Google, TikTok, and the like. And maybe that’s not so bad. And anyway, I should take more responsibility for my content consumption preferences. And anyway anyway, what content I consume probably isn’t (within a wide range) a big (moral) deal.
Things are different when it comes to a patient’s preferences regarding care -- their (as it were) medical content consumption preferences. It is (pro tanto) morally wrong to shape patients’ preferences in order to improve the accuracy of a prediction about what care they’d prefer. Patients’ preferences are rightly and widely regarded as sacred, on grounds of autonomy. They are of course not inviolable. So, this doesn’t mean it’s always (even pro tanto) morally wrong to actually shape patients’ preferences. Physicians do this all the time, and while some of those interactions are likely morally suspect, certainly not all are.
But it is (pro tanto) morally wrong to shape patient preferences in order to improve the accuracy of predictions about those preferences. To see this clearly, consider the difference between two hospital-run bounty programs. The first program, Wealth for Health, pays physicians a bonus when their patients systematically prefer A to B at a greater rate than the relevant baseline. This is done for the sake of improving health outcomes (say, because it is in some way medically good to prefer A to B). The second program, Acquire for Brier, works exactly the same, but is done for the sake of improving the hospital’s predictions about patients’ preferences (say, because it’s easier to predict the other preferences of patients who prefer A to B than it is to predict the preferences of patients who prefer B to A). I submit that physician participation in the Wealth for Health bounty program is (probably) morally fine, whereas joining Acquire for Brier clearly isn’t.
If it’s morally wrong to shape patient preferences in order to improve the accuracy of a prediction about those preferences, then it is also (pro tanto) morally wrong to develop and deploy an ML-based system for predicting (incapacitated) patients’ preferences unless we can be assured that this system is not incentivized to do just this. Consider: it is pro tanto wrong to do X. So, it is (pro tanto) wrong to develop and deploy an ML-based system without any assurance that it won’t do X. That general form of moral reasoning seems very difficult to deny. It is pro tanto wrong to punch people in the face. So, it is (pro tanto) wrong to develop and deploy an ML-based system without any assurance that it won’t punch people in the face.
Luckily, almost no one is developing and deploying models without assurance they won’t punch people in the face. But many people are continuing to advocate for developing and deploying patient preference prediction models without any assurance that they won’t shape the preferences of the patients whose preferences they are designed to predict. In my view, that is a very bad outcome. Worse, it risks a serious moral harm for no good reason. We should all just make advance directives.