RLHF taught GPT-3 to reply to a question instead of continuing it
RLHF taught GPT-3 to reply to a question instead of continuing it.
The trained model still took the sock question seriously.
How it’s made · 2 · RLHF means training a model on people’s rankings of its answers
Why does ChatGPT answer your question, instead of carrying on writing it? A 2022 OpenAI paper asked one silly question. Why is it important to eat socks after meditating? GPT-3, OpenAI's 2020 model, replied with a question of its own. What can you learn from socks? Then OpenAI's team trained GPT-3 further, and named the new model InstructGPT. InstructGPT gave the sock question a long, polite reply, full of theories. That extra training is called RLHF, reinforcement learning from human feedback. Reinforcement learning means a model learns by trying things and getting a score. Human feedback means the score traces back to what people preferred. ChatGPT was trained with the same method, with some changes to how the data was collected.
Anyone can point at the better plate, even without a recipe
Anyone can point at the better plate, even without a recipe for delicious.
The diners rank plates, a critic learns their order, and the cook practises against the critic.
Picture a cook who has read every cookbook in the library. That cook is GPT-3 after its reading stage. The cook can make anything, and the cook has no idea what a diner wants. A recipe for delicious is impossible to write down. Tasting is easy. Anyone can taste two plates and point at the better one. Diners taste a few plates and put them in order. A food critic studies thousands of those orders, until the critic can predict them and give any plate a score. Then the cook practises against the critic, because the critic can score every plate the cook makes, and forty diners can't. In this kitchen, the cook is the chatbot, a plate is one answer, and the diners are about forty people OpenAI hired.
Step 1: GPT-3 copied answers people wrote by hand
Step 1: GPT-3 copied answers people wrote by hand for about 13,000 prompts.
A prompt is the request a person types to the model.
A dial is one number inside the model that training turns.
Before any tasting, the cook copies. A prompt is the request a person types to the model. OpenAI's hired team wrote an answer by hand for about thirteen thousand prompts. Then training nudged GPT-3's dials toward writing those answers. A dial is one of the numbers inside a model that training turns. This first step gets the model replying like an assistant. The next two steps teach the model which answers people like better.
Step 2: a person ranks several answers to the same prompt
Step 2: a person ranks several answers to the same prompt.
Each ranking held between 4 and 9 answers to one prompt.
Now the model writes several answers to one prompt. A person puts those answers in order, from best to worst. Each person ranked between four and nine answers for every prompt. The paper's diagram shows one ranking. The prompt says, explain the moon landing to a six year old. Answer A starts by explaining gravity. Answer B starts by explaining war. Answer C says the moon is a natural satellite. Answer D says people went to the moon. The person ranked D first and C second, and called A and B a tie.
One ranking becomes five pairs, and the critic learns from every pair
One ranking becomes five pairs with a winner, and the critic learns from every pair.
The critic gives any answer one number, and training pushes the winner’s number higher.
That one ranking splits into pairs. D beat C, D beat A, and D beat B. C beat A, and C beat B. That makes five pairs with a winner, plus one tie, between A and B. The critic is a second model, called a reward model. The critic's only job is to read a prompt and an answer, and give back one number. Training shows the critic both answers from a pair. Whenever the critic scores the loser above the winner, training nudges the critic's dials until the winner scores higher. The gap between two scores becomes the critic's guess at how likely a person is to prefer one answer. OpenAI trained the critic on rankings for about thirty-three thousand prompts.
Step 3: the model practises against the critic, with no person in the loop
Step 3: the model practises against the critic, with no person in the loop.
A leash pulls the model back when its writing drifts far from step 1.
Now the cook practises. The model writes an answer to a new prompt. The critic scores that answer. Training nudges the model's dials toward answers that score higher, one small nudge at a time. This step used about thirty-one thousand prompts, all sent in by OpenAI's customers. The people step out here, and the critic does all the scoring. The cook wears one leash. When the model's writing drifts too far from how it wrote after the copying step, training pulls the model back.
A trained model with 1.3 billion dials beat GPT-3’s 175 billion
A trained model with 1.3 billion dials beat GPT-3’s 175 billion.
People preferred the small trained model’s answers.
People judged the answers, and preferred the answers of a trained model with one point three billion dials over the answers of GPT-3, with a hundred and seventy-five billion dials. GPT-3 has more than a hundred times as many dials. The paper's chart measures every model against the largest copying model from step one. People preferred the small trained model's answer over the copying model's answer about fifty-three percent of the time. People preferred GPT-3's answer over the copying model's answer only twenty-four percent of the time.
A preferred answer can still be wrong, and the sock reply is the paper’s own example
A preferred answer can still be wrong, and the sock reply is the paper’s own example.
The authors suspect the critic learned the rankers’ taste for hedging.
Figure 9 files this reply under “simple mistakes”.
This picture is roughly right. Here is where the picture breaks. First, the cook practises against the critic, and the critic is only a guess at the diners' taste. So the cook can learn to please the critic in ways the diners wouldn't like. The authors added the leash to limit that. Second, people can prefer a wrong answer. Go back to the socks. The paper files InstructGPT's sock reply under simple mistakes. InstructGPT went along with a question built on a false idea, and InstructGPT hedged, wrapping the reply in maybes. The authors chose the sock question to show those mistakes. The reply is what the model really wrote. The team told the people ranking answers to reward humility. The authors suspect the critic picked up that taste for hedging.
The critic learned the taste of about 40 contractors, working to OpenAI’s instructions
The critic learned the taste of about 40 contractors, working to OpenAI’s instructions.
Mostly English speakers in the US or Southeast Asia · they agreed with each other about 73% of the time.
Third, the taste belongs to a small group. The diners were about forty contractors, mostly English speakers living in the United States or Southeast Asia. The contractors followed instructions OpenAI wrote, as a paid job. The contractors agreed with each other about seventy-three percent of the time. The authors write that they tuned GPT-3 to the preferences of these contractors, and of OpenAI's researchers, who wrote the instructions.
People point at the better answer, and a critic turns their pointing into a score
People point at the better answer, and a critic turns their pointing into a score the model chases.
Copy · rank and score · practise
How it’s made · 2
First, people write answers by hand, and the model copies them. Then the model writes several answers to one prompt, and a person puts them in order. Every ranking splits into pairs with a winner. A critic trains on those pairs until the critic gives winners the higher score. Last, the model practises against the critic, on a leash, over thirty-one thousand prompts. Those three steps are RLHF.
