RLHF taught GPT-3 to reply using rankings from about 40 people

Why does ChatGPT answer your question, instead of carrying on writing it? A 2022 OpenAI paper asked one silly question. Why is it important to eat socks after meditating? GPT-3, OpenAI's 2020 model, replied with a question of its own. What can you learn from socks? Then OpenAI's team trained GPT-3 further, and named the new model InstructGPT. InstructGPT gave the sock question a long, polite reply, full of theories. That extra training is called RLHF, reinforcement learning from human feedback. Reinforcement learning means a model learns by trying things and getting a score. Human feedback means the score traces back to what people preferred. ChatGPT was trained with the same method, with some changes to how the data was collected.

Picture a cook who has read every cookbook in the library. That cook is GPT-3 after its reading stage. The cook can make anything, and the cook has no idea what a diner wants. A recipe for delicious is impossible to write down. Tasting is easy. Anyone can taste two plates and point at the better one. Diners taste a few plates and put them in order. A food critic studies thousands of those orders, until the critic can predict them and give any plate a score. Then the cook practises against the critic, because the critic can score every plate the cook makes, and forty diners can't. In this kitchen, the cook is the chatbot, a plate is one answer, and the diners are about forty people OpenAI hired.

Before any tasting, the cook copies. A prompt is the request a person types to the model. OpenAI's hired team wrote an answer by hand for about thirteen thousand prompts. Then training nudged GPT-3's dials toward writing those answers. A dial is one of the numbers inside a model that training turns. This first step gets the model replying like an assistant. The next two steps teach the model which answers people like better.

Now the model writes several answers to one prompt. A person puts those answers in order, from best to worst. Each person ranked between four and nine answers for every prompt. The paper's diagram shows one ranking. The prompt says, explain the moon landing to a six year old. Answer A starts by explaining gravity. Answer B starts by explaining war. Answer C says the moon is a natural satellite. Answer D says people went to the moon. The person ranked D first and C second, and called A and B a tie.

That one ranking splits into pairs. D beat C, D beat A, and D beat B. C beat A, and C beat B. That makes five pairs with a winner, plus one tie, between A and B. The critic is a second model, called a reward model. The critic's only job is to read a prompt and an answer, and give back one number. Training shows the critic both answers from a pair. Whenever the critic scores the loser above the winner, training nudges the critic's dials until the winner scores higher. The gap between two scores becomes the critic's guess at how likely a person is to prefer one answer. OpenAI trained the critic on rankings for about thirty-three thousand prompts.

Now the cook practises. The model writes an answer to a new prompt. The critic scores that answer. Training nudges the model's dials toward answers that score higher, one small nudge at a time. This step used about thirty-one thousand prompts, all sent in by OpenAI's customers. The people step out here, and the critic does all the scoring. The cook wears one leash. When the model's writing drifts too far from how it wrote after the copying step, training pulls the model back.

People judged the answers, and preferred the answers of a trained model with one point three billion dials over the answers of GPT-3, with a hundred and seventy-five billion dials. GPT-3 has more than a hundred times as many dials. The paper's chart measures every model against the largest copying model from step one. People preferred the small trained model's answer over the copying model's answer about fifty-three percent of the time. People preferred GPT-3's answer over the copying model's answer only twenty-four percent of the time.

This picture is roughly right. Here is where the picture breaks. First, the cook practises against the critic, and the critic is only a guess at the diners' taste. So the cook can learn to please the critic in ways the diners wouldn't like. The authors added the leash to limit that. Second, people can prefer a wrong answer. Go back to the socks. The paper files InstructGPT's sock reply under simple mistakes. InstructGPT went along with a question built on a false idea, and InstructGPT hedged, wrapping the reply in maybes. The authors chose the sock question to show those mistakes. The reply is what the model really wrote. The team told the people ranking answers to reward humility. The authors suspect the critic picked up that taste for hedging.

Third, the taste belongs to a small group. The diners were about forty contractors, mostly English speakers living in the United States or Southeast Asia. The contractors followed instructions OpenAI wrote, as a paid job. The contractors agreed with each other about seventy-three percent of the time. The authors write that they tuned GPT-3 to the preferences of these contractors, and of OpenAI's researchers, who wrote the instructions.

First, people write answers by hand, and the model copies them. Then the model writes several answers to one prompt, and a person puts them in order. Every ranking splits into pairs with a winner. A critic trains on those pairs until the critic gives winners the higher score. Last, the model practises against the critic, on a leash, over thirty-one thousand prompts. Those three steps are RLHF.