Back to AI Research

AI Research

Participatory Moral AI Is Not Neutral: The Invisibl... | AI Research

Key Takeaways

  • Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers examines how the process of gathering public input for AI decision-making is influenc...
  • As AI systems make more morally loaded decisions across society, one response has been moral preference elicitation.
  • In this approach, researchers poll participants on hypothetical dilemmas and use the aggregated votes to train a policy that an AI model then applies at scale.
  • Before any vote is cast, developers make three key choices in the moral AI elicitation pipeline: feature scoping, voter sampling, and question framing.
  • In other words, they decide which features go to a vote, which voters to include, and how to present the question.
Paper AbstractExpand

As AI systems make more morally loaded decisions across society, one response has been moral preference elicitation. In this approach, researchers poll participants on hypothetical dilemmas and use the aggregated votes to train a policy that an AI model then applies at scale. Before any vote is cast, developers make three key choices in the moral AI elicitation pipeline: feature scoping, voter sampling, and question framing. In other words, they decide which features go to a vote, which voters to include, and how to present the question. These choices are often opaque, undocumented, and treated as technical details rather than normative ones. We examine each of these choices within a common empirical study and show that each can shape the preferences produced by moral AI elicitation. Across two phases (N = 809) in three deployment contexts (i.e., AI kidney allocation, AI agents simulating absent workers, and generative AI depictions of the deceased), we examine the three main stages of the moral AI elicitation pipeline. First, morally relevant features shift across contexts. This suggests that feature schemas should not be assumed to transfer across deployment domains. Second, preferences differ by political ideology for roughly one-third of features, with some differences reversing direction. The ideological composition of the voter pool can therefore affect the resulting aggregated preference profile. Third, the wording of the elicitation question can narrow or widen ideological gaps by up to a full scale point. The framing conditions also change how moral foundations are associated with participants' judgments. Taken together, these findings suggest that voting-based alignment cannot deliver fair or transparent AI by aggregation alone; at minimum, each stage of the moral AI elicitation pipeline should be audited and disclosed.

Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers examines how the process of gathering public input for AI decision-making is influenced by the hidden design choices of developers. While researchers often use "moral preference elicitation"—polling people on hypothetical dilemmas to create AI policies—this paper argues that the resulting policies are not neutral reflections of public values. Instead, they are shaped by three upstream developer decisions: which features are put to a vote, who is invited to vote, and how the questions are phrased.

The Three Stages of Elicitation

The authors, Taenyun Kim, Edyta Bogucka, and Daniele Quercia, identify three stages where developer influence enters the pipeline:

  • Feature Scoping: Developers decide which factors are relevant for an AI to consider. The study found that these features do not transfer across contexts. For example, kidney allocation relies on medical utility, while workplace AI focuses on accountability, and generative AI for the deceased centers on consent.

  • Voter Sampling: Developers choose the participant pool. The researchers found that political ideology significantly influences preferences for roughly one-third of features. Because different groups prioritize different moral foundations, the specific composition of the voter pool directly changes the final aggregated policy.

  • Question Framing: Developers decide how to word the dilemma. The study showed that framing can shift preferences by up to a full scale point and can either narrow or widen ideological gaps between participants.

Study Methodology

To test these effects, the authors conducted a two-phase study (N = 809) across three distinct deployment contexts: AI kidney allocation, AI agents simulating absent workers, and generative AI depictions of the deceased.
In Phase 1, participants identified which features should or should not matter for each scenario. In Phase 2, a new group of participants evaluated these features using a 7-point scale under three different framing conditions: a control, a "World-You-Want" prompt focusing on societal consequences, and a "Could-Be-You" prompt based on the Rawlsian veil of ignorance. The researchers then used moderated mediation analysis to determine how political ideology and framing interacted to produce specific moral preferences.

Key Findings

The evidence suggests that voting-based alignment cannot be considered a neutral or purely democratic process. Because the output is conditioned by the initial design choices, the authors conclude that aggregation alone is insufficient to ensure fairness or transparency.
The study found that:

  • Moral feature relevance is highly context-specific, meaning developers cannot assume a "one-size-fits-all" set of features for different AI applications.

  • Political ideology causes some preferences to reverse direction, making the selection of voters a consequential normative choice.

  • Question framing is not a neutral tool; it acts as a design intervention that changes how participants weigh moral foundations.

Implications for AI Development

The authors argue that because these choices are often treated as technical details rather than moral ones, they remain opaque and undocumented. To address this, they propose that each stage of the moral AI elicitation pipeline should be subject to a sensitivity audit. By disclosing these upstream choices, developers can make the process more transparent and contestable, acknowledging that the resulting AI policy is a product of both public input and the developer's own design framework.

Comments (0)

No comments yet

Be the first to share your thoughts!