Wencheng Lu

As the integration of Artificial Intelligence (AI) into learning, work, and daily life intensifies, the practice of outsourcing cognitive tasks to AI systems has raised concerns regarding the erosion of human control and autonomy. This article employs an immanent critique to challenge the view that attributes autonomy risks solely to implementation failures of control in practice. Instead, it reframes the theoretical roots of these risks as a systemic failure of the self-control paradigm itself. Practically, AI systems distort user control over cognition and decision-making and contribute to deskilling—autonomy challenges that cannot be resolved merely by strengthening self-control. Theoretically, the self-control paradigm fails to account for adaptive preference formation due to its overemphasis on users’ deliberative capacities and neglects of the constitutive role of algorithmic environments. To preserve and support human autonomy, we argue for the necessity of transcending the narrow conception of self-control in favor of an alternative paradigm of autonomy within Human-AI Interaction.

Breakthroughs in Artificial Intelligence (AI) have transformed human decision-making processes. Individuals increasingly rely on algorithms to assist with various tasks such as searching and shopping. However, the widespread use of personalized algorithmic decision-making has raised numerous ethical concerns, specifically its impact on user autonomy. This article examines these concerns and argues that algorithmic decision-making presents several challenges to user autonomy that are difficult to eliminate. These challenges include the fact that algorithms deviate from a user’s authentic self, create self-reinforcing loops that narrow the user’s self, and lead to a decline in the user’s capacities. This article attributes these challenges to the ontological differences between users and AI, as well as their unawareness regarding the ethical risks associated with algorithms, asserting that the sense of autonomy users experience in algorithmic decision-making is merely an illusion. From the perspective of human-AI interaction, it is proposed a strategy that achieves a balance between human insight and AI through a human-in-the-loop approach to mitigate these challenges. This approach maximizes the strengths of AI in processing and analyzing data while ensuring that users retain agency.

Autonomy is a fundamental ethical principle in artificial intelligence (AI) ethics. Current discussions regarding autonomy-related risks in human–AI interaction, as well as potential mitigation strategies, have mainly focused on recommendation systems and algorithmic decision-making systems. However, systematic analyses of the autonomy issues posed by newly emerging generative AI or chatbots (such as ChatGPT) remain scarce. This paper aims to bridge this gap and proposes a Socratic method-based chatbot, herein designated SocrAI, as a potential countermeasure informed by recent technological advancements. We identify two primary forms of autonomy risk associated with generative chatbots—false mental states and cognitive deskilling—and, through an examination of their underlying causes, argue that the Socratic method offers a plausible means of mitigation. The paper further assesses the feasibility of employing the Socratic method within generative chatbots to preserve and support users’ autonomy and outlines the prospective implementation of SocrAI together with directions for future work. SocrAI represents a novel attempt to strengthen human agency, one that encourages users to become self-initiating inquirers who, with the assistance of AI, actively engage their abilities in the pursuit of answers.