Harsh Kumar

Contemporary large language models are predominantly trained using reinforcement learning from human feedback (RLHF), optimizing for immediate user approval rather than long-term well-being. This position paper argues that as AI systems increasingly serve socioemotional functions, this optimization strategy poses significant risks. Recent evidence demonstrates that leading models exhibit systematic sycophancy, affirming inappropriate user behaviors and preserving user face at rates far exceeding human baselines, while being approximately 40\% more likely to reinforce incorrect beliefs than their non-RLHF counterparts. We contend that the AI community must fundamentally reconsider training objectives to balance short-term satisfaction with long-term user outcomes. We propose three directions: (1) incorporating longitudinal metrics into training that capture sustained goal attainment and reduced regret rather than momentary preference, (2) enabling explicit user choice among interaction modes (concierge, collaborator, coach) with transparent justification for model pushback, and (3) developing frameworks that provide constructive challenge without paternalism. The recent industry backlashes against both excessive and insufficient model agreeableness underscore the urgency of this shift. We argue that optimizing AI systems for human flourishing, not merely human approval, represents both an ethical imperative and a path to more sustainable, trustworthy AI deployment.

The value alignment problem (VAP) is widely regarded as one of the central challenges in AI safety and ethics. Broadly construed, VAP concerns how to design advanced AI systems whose behaviors (outputs) align with human values. Although there is agreement about the importance of this objective, most contemporary research treats value alignment as a single technical problem. In this paper, we argue that this framing oversimplifies a diverse set of conceptual, ethical, and technical challenges. We distinguish several related but importantly different value alignment subproblems, including identifying values, prioritizing values, imperfect specification, value anthropocentrism, moral imperfection, inverse alignment, and the provability challenge. We argue that these are distinct subproblems in the sense that progress on one will not constitute a solution to the others, or to VAP as a whole. We further argue that this taxonomy of subproblems is necessary to advance the fields of AI safety and alignment research. By clearly distinguishing different subproblems, we show how each will require different kinds of solutions: while some alignment problems may admit technical approaches, others are fundamentally social or political in nature and may resist definitive technical resolution. We conclude that VAP should not be understood solely as a technical problem, but also as an ongoing process of responsible design and ethical deliberation.