OpenAI 用人类反馈对 774M 参数的 GPT-2 语言模型做了若干任务的微调，成功匹配上了外部人类标注者的偏好——尽管那些偏好有时并不与 OpenAI 自己的偏好一致。具体地，在摘要任务上标注者偏好整句从输入里照搬过来的句子（OpenAI 只要求他们确保准确性），于是模型就学到了照搬；摘要任务需要 6 万个人类标注，而以多种风格续写文本这类更简单的任务只需要 5 千个。其动机是把安全技术推

正文

We’ve fine-tuned the 774M parameter GPT-2 language model using human feedback for various tasks, successfully matching the preferences of the external human labelers, though those preferences did not always match our own. Specifically, for summarization tasks the labelers preferred sentences copied wholesale from the input (we’d only asked them to ensure accuracy), so our models learned to copy. Summarization required 60k human labels; simpler tasks which continue text in various styles required only 5k. Our motivation is to move safety techniques closer to the general task of “machines talking to humans,” which we believe is key to extracting information about human values.