Sarah E. Carter

Integrating AI systems into workflows risks undermining the competence of the people supported by them, specifically due to a loss of meta-cognitive competence. We discuss a recent suggestion to mitigate this through better uncertainty quantification. While this is certainly a step in the right direction, there is a question whether users are sufficiently supported to engage in critical reflection with literacy and tools alone. We therefore suggest that socio-technical system design focused on the role of AI systems is crucial to preserving autonomy, even when supported by uncertainty quantification.

AI systems increasingly support human decision-making across domains of professional, skill-based, and personal activity. While previous work has examined how AI might affect human autonomy globally, the effects of AI on domain-specific autonomy—the capacity for self-governed action within defined realms of skill or expertise—remain understudied. We analyze how AI decision-support systems affect two key components of domain-specific autonomy: skilled competence (the ability to make informed judgments within one's domain) and authentic value-formation (the capacity to form genuine domain-relevant values and preferences). By engaging with prior investigations and analyzing empirical cases across medical, financial, and educational domains, we demonstrate how the absence of reliable failure indicators and the potential for unconscious value shifts can erode domain-specific autonomy both immediately and over time. We then develop a constructive framework for autonomy-preserving AI support systems. We propose specific socio-technical design patterns—including careful role specification, implementation of defeater mechanisms, and support for reflective practice—that can help maintain domain-specific autonomy while leveraging AI capabilities. This framework provides concrete guidance for developing AI systems that enhance rather than diminish human agency within specialized domains of action.

AI systems increasingly support human decision-making across domains of professional, skill-based, and personal activity. While previous work has examined how AI might affect human autonomy globally, the effects of AI on domain-specific autonomy -- the capacity for self-governed action within defined realms of skill or expertise -- remain understudied. We analyze how AI decision-support systems affect two key components of domain-specific autonomy: skilled competence (the ability to make informed judgments within one's domain) and authentic value-formation (the capacity to form genuine domain-relevant values and preferences). By engaging with prior investigations and analyzing empirical cases across medical, financial, and educational domains, we demonstrate how the absence of reliable failure indicators and the potential for unconscious value shifts can erode domain-specific autonomy both immediately and over time. We then develop a constructive framework for autonomy-preserving AI support systems. We propose specific socio-technical design patterns -- including careful role specification, implementation of defeater mechanisms, and support for reflective practice -- that can help maintain domain-specific autonomy while leveraging AI capabilities. This framework provides concrete guidance for developing AI systems that enhance rather than diminish human agency within specialized domains of action.

While many of us claim to value privacy, we often share data online or on our smartphones. The privacy literature is rich with discussions around how our own cognitive biases and heuristics are exploited to encourage such incongruous actions. However, further research is required to understand how we value data privacy—in particular, how personal values influence our decisions and why we may act against our values. Here, we discuss recent findings from a semi-structured interview investigation exploring how we value data privacy. Interview questions probed the relationship between privacy and values when selecting smartphone applications and privacy decisions more generally. To elucidate often-unconscious values, questions aimed to obtain statements pertaining to self-governing commitments representing values. Reflexive thematic analysis was used to interrogate instances where participants felt unable to act according to their values or where values appeared to conflict. From the interviews, we identified six major value themes that were motivationally related to privacy decisions, shaped by an individual’s understanding of the value and the context. Tensions between values were also linked to context with individual variation. However, values Use, Community, and Control were frequent and in tension with each other. We discuss the implications of these findings for smartphone app developers and stores, for designing privacy technologies, and for regulators. We also point to areas of future research into how we value privacy, particularly engagement with traditionally underrepresented groups.

There are a host of data privacy decisions we must make every day – and it is exceedingly difficult, if not impossible, for us to make meaningful decisions about all of them. In this thesis, I define, conceptualize, interrogate, and design for value-centered privacy decision making – that is, decisions that are focused on who we are and what we value – as a means of respecting and promoting user autonomy. To achieve this, this work utilizes philosophical theory to understand value-centered privacy decisions and translates this theory into a system that promotes such decisions. In summary, this work has two major contributions. Firstly, I conceptualize and define value-centered privacy decision-making using a value-centered theory of autonomy. I explore how we can create the space for value-centered privacy decisions by applying the Four-Dimensional Theory of Self Governance (4DT). I first conceptualize privacy decisions in terms of these four dimensions – self-definition, self-realization, self-unification, and self-constitution – and explore existing data privacy challenges through this lens. In particular, I conceptualize notice fatigue in terms self-realization, self-unification, and self-constitution; a lack of relevant privacy controls in terms of self-realization and self-unification; and nudges in terms of self-realization and self-unification. I then present and discuss results from a mixed-methods investigation into how values are involved in privacy decisions – in particular, app choice. We found that they were related in a highly individualized, context specific manner, observing different values that were more relevant based on the app in question. This suggests that the value-privacy relationship is largely informed by individual preferences and understandings of values. However, the values of Use, Control, and Community were quite prevalent, with Use and Control in particular spanning contexts and individual participants. They were also frequently perceived as in conflict with each other. This suggests that these three values are the most relevant to consider when designing for value-centered privacy decisions. The participants’ experiences can also be explained using 4DT, providing empirical support for our conceptualization of value centered privacy. However, the study results also provide insights into how existing systems – such as surveillance capitalism and the attention economy – frustrate value centered privacy decisions. Secondly, I use the 4DT-based understanding of value-centered privacy decisions to establish the usability and effectiveness of the value-centered approach, designing a privacy assistant to help users make app choices that are in more accordance with their personal values. To inform the design of a smartphone assistant that creates this space for users, I examine an existing technology – personalized privacy assistants (PPAs) – using the 4DT lens. Using insights from this examination, I propose a value-centered, smartphone privacy assistant (VcPA) to help users make more value-centered decisions at one privacy decision point: smartphone app choices. This VcPA consists of three features: selective notices, exploratory notices, and a “suggest alternative apps” feature. I then present the results from testing a prototype VcPA system with users, serving as a proof-of concept that a value-centered privacy assistant, designed using privacy preferences and values, could help users when making privacy decisions such as choosing apps. In particular, we found that the VcPA prototype helped users download value-consistent apps, with the “suggest alternatives” feature especially well-received. We also identified places where the VcPA could be improved – for example, profiles could be improved by being made more customizable; VcPA notices could be made easier to understand; and the “suggest alternatives” feature could be more streamlined. This thesis lays the groundwork for future researchers to design systems that promote value-centered privacy decisions. To guide this future work, I lastly present prospective research avenues to advance the value-centered approach to data privacy decision-making. In particular, I discuss limitations of the studies in this work, including engagement with a wider range of demographic groups; touch upon how the identified VcPA improvements, such as improved VcPA profiles, might be accomplished; briefly explore the possibility of applying the value-centered understanding to other privacy contexts; and consider how both system-wide regulation and individual autonomy enhancing interventions, such as the VcPA, can empower us to shape a technological future based on our values.

In the current post-GDPR landscape, privacy notices have become ever more prevalent on our phones and online. However, these notices are not well suited to their purpose of helping users make informed decisions. I suggest that instead of utilizing notice to elicit informed consent, we could repurpose privacy notices to create the space for more meaningful, value-centered user decisions. Value-centered privacy decisions, or those that accurately reflect who we are and what we value, encapsulate the intuitive role of personal values in data privacy decisions. To explore how we could design for such decisions, I utilize Suzy Killmister’s Four-Dimensional Theory of Autonomy (4DT) to operationalize value-centered privacy decisions. I then utilize 4DT to help design a system—called a value-centered privacy assistant (VcPA)—that could help create the space for value-centered data privacy decisions using privacy notices. Using this 4DT lens, I further assess the degree that an existing technology, personalized privacy assistants (PPAs), use notices in a manner that allows for value-centered decision-making. I lastly utilize insights from the PPA assessment to inform the design of a VcPA, concluding that a VcPA could utilize notices to assist users in value-centered app selection and in other data privacy decisions.