Modeling and Learning Dynamic Moral Judgments towards Allocations in High-stakes Domains

Download SSRN version here

Keywords: moral judgment modeling, moral preference learning, ethical principles

Resource allocation in high-stakes domains, such as public health, education and employment, is critical for promoting long-term justice and equity. As AI tools are increasingly applied to assist these decisions, it is essential to model peoples’ moral judgments to ensure their views on who and how allocation policies should prioritize and benefit are incorporated into AI systems. In this paper, we propose a framework for modeling and learning dynamic moral judgments in sequential resource allocations. Our approach is based on a Markov Decision Process (MDP) model, where the state reward function characterizes a stakeholder’s moral preferences over allocation policies. To reflect shifts in ethical principles of allocations, we define piecewise linear reward functions to account for the changing moral priorities. We design a reward learning algorithm that actively queries stakeholders for feedback, using which the approximate reward function is iteratively updated with Bayesian inference. We illustrate our model through simulation experiment and human subject study focused on the allocation of scarce medical resource distributions during a hypothetical viral epidemic. The simulations demonstrate the effectiveness and validity of our approach, while the human subject study illustrates its applicability in interacting with participants to describe their moral judgments.