Generative AI systems are increasingly authorized as sources of moral content in education, positioned to generate ethical dilemmas and structure moral discussions. This paper argues that GenAI, in principle, cannot legitimately perform this function in a substitution capacity. The central claim: when GenAI systems are positioned to furnish moral reasoning rather than scaffold it, they create a structural asymmetry—instructors cannot verify what trained the system, cannot guarantee its contents,…
Read moreGenerative AI systems are increasingly authorized as sources of moral content in education, positioned to generate ethical dilemmas and structure moral discussions. This paper argues that GenAI, in principle, cannot legitimately perform this function in a substitution capacity. The central claim: when GenAI systems are positioned to furnish moral reasoning rather than scaffold it, they create a structural asymmetry—instructors cannot verify what trained the system, cannot guarantee its contents, yet remain accountable for what it teaches. This is not a transparency problem. It is a problem of institutional accountability in moral formation. The paper distinguishes two platform categories: those that scaffold human reasoning (MIT Moral Machine, FCAI Ethics Exercise Tool) and those that position AI outputs as moral starting points (YesChat's Ethical Dilemma Generator, Taskade's AI-powered generator and many other ones). The latter creates a structural problem: instructors cannot know what trained the system, cannot verify its contents, yet remain accountable for what it teaches. In theory the distinction is clear. In practice, almost nobody recognizes it. Instructors and institutions treat these platforms interchangeably—as tools that accomplish the work. Most of us, educators in the humanities (philosophy, literature, religion, linguistics and etc.), do not ask which category the platform falls into, not out of carelessness but because we are not trained to analyze computational systems or machine learning processes. We came to academia to work with texts, meaning, and interpretation, and we are neither prepared nor compensated to develop technical literacy in these areas. Under institutional pressure to perform efficiently, the distinction collapses. This erasure by convenience and structural conditions masks a fundamental asymmetry that applies specifically to substitution platforms. The argument unfolds in five movements: (I) the black box as structural opacity in substitution platforms; (II) empirical harm through bias and epistemological exclusion; (III) displacement of moral formation; (IV) political dimensions through Tolstoy and Arendt; (V) epistemological argument through Husserl, Hume, and Camus. Moral judgment requires intentionality, sentiment, and genuine inquiry—conditions AI cannot possess. At stake is the capacity of human beings to form themselves as moral agents.
