Constitutional AI Research in Production Deployment

Title

Constitutional AI Research in Production Deployment

Content

Constitutional AI: Harmlessness from AI Feedback — Anthropic Research Paper (arXiv:2212.08073) Authors: Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Jared Kaplan Abstract: As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Introduction: We would like to train AI systems that remain helpful, honest, and harmless, even as some AI capabilities reach or exceed human-level performance. This suggests that we will need to develop techniques that do not rely on humans to supervise all aspects of AI behavior, and that can be used to automatically test and enhance robustness to harmful behaviors. Constitutional AI involves two stages. In the Supervised Stage: Critique → Revision → Supervised Learning. We generate responses to harmfulness prompts using a helpful-only AI assistant, then ask the model to critique its response according to a principle in the constitution, and then revise the original response in light of the critique. In the RL Stage: AI Comparison Evaluations → Preference Model → Reinforcement Learning. This stage mimics RLHF, except that human preferences for harmlessness are replaced with 'AI feedback' (RLAIF). Results: We demonstrate that RL-CAI models are significantly more harmless than RLHF and SL-CAI models. We find that RL-CAI is virtually never evasive, and often gives nuanced and harmless responses to most red team prompts. As language model capabilities improve, AI identification of harms improves significantly. Chain-of-thought reasoning improves this ability and leads to evaluations that are becoming competitive with preference models trained on human feedback labels.