Pretraining taught GPT-3 to write with one fill-in-the-blank game
Pretraining taught GPT-3 to write with one fill-in-the-blank game.
300 billion rounds, each one hiding the next piece of real text under a card.
How it’s made · 1 · a piece is a chunk of a word
How did ChatGPT learn to write? GPT-3, OpenAI's 2020 model, learned from a fill-in-the-blank game. The game hides the next piece of a sentence, and GPT-3 bets on what sits under the card. A piece is a chunk of a word, the unit from the tokens episode. GPT-3 played that game three hundred billion times. That game is called pretraining, the first stage of making a model like GPT-3. In pretraining the model reads, before anyone teaches it to answer questions.
Every sentence is a quiz with the answer printed right after the blank
Every sentence is a quiz with the answer printed right after the blank.
Lift the card and the text marks the guess.
Take one sentence. I poured myself a cup of. Put a card over the piece that comes next, and guess what the card is hiding. Then lift the card. The sentence itself shows the real next piece. We picked this sentence, and our sentence continues with coffee. The text does the marking. Every book, every web page and every Wikipedia article works the same way. Each text is a long stack of these quizzes, one quiz per piece, with every answer already printed. That is why GPT-3 could learn from raw text, with zero labels added by people.
The model spreads one bet across all 50,257 pieces it knows
The model spreads one bet across all 50,257 pieces it knows.
The whole bet always adds up to 100%.
The model answers with a bet. Every piece the model knows gets a share, and the shares always add up to a hundred percent. OpenAI never released GPT-3's dials, the numbers inside the model that training turns. So we ran this sentence through GPT-2 Large instead. GPT-2 Large is OpenAI's 2019 model, built on the same design as GPT-3 at a smaller size. GPT-2 Large put forty-four percent of its bet on tea. GPT-2 Large put twenty-one percent on coffee and six percent on hot. The rest of the bet spreads thin over the other fifty thousand or so pieces.
Coffee held 21% of the bet, so a wrong favorite still earns partial credit
Coffee held 21% of the bet, so a wrong favorite still earns partial credit.
Less of the bet on the real piece means a bigger penalty.
GPT‑1’s goal: make the bet on each real next piece as large as it can be.
Now lift the card. The real piece is coffee. GPT-2 Large's favorite was tea, and the model still earns partial credit, because twenty-one percent of the bet sat on coffee. The score comes from that share alone. A model that puts all of its bet on the real piece earns a perfect score. A model that puts a sliver on the real piece takes a big penalty. That penalty measures how surprised the model was by the real piece. OpenAI wrote this goal down as one line in its 2018 paper on GPT-1, the first model in the family. That line asks for the biggest possible bet on each real next piece, given the pieces before it. GPT-3 trains on that same goal.
GPT-3 ran the round of bet, score, nudge and slide 300 billion times
GPT-3 ran the round of bet, score, nudge and slide 300 billion times.
One round per piece of text, over the whole reading list.
A dial is one number inside the model that training turns.
After the score, training nudges every dial a hair. Each nudge makes the bet on the real piece a little bigger next time. That nudge is the return trip from the training versus inference lesson, the walk back through the dials that works out which dial to turn. Then the card slides one piece to the right, and the game starts again on the next piece. GPT-3 played three hundred billion rounds of this game, one round for every piece of text it trained on.
GPT-3 read from five shelves, and OpenAI chose each shelf’s share
GPT-3 read from five shelves, and OpenAI chose each shelf’s share.
Common Crawl held 410 billion pieces and got 60% of the reading.
GPT-3's paper prints its reading list as a table of five shelves. The biggest shelf is Common Crawl, a free, public archive of pages copied from the web. After filtering, the Common Crawl shelf held four hundred and ten billion pieces, and it got sixty percent of the reading. Next come web pages gathered by following links, and two book collections from the internet. The smallest shelf is English Wikipedia, at three billion pieces. Wikipedia got three percent of the reading. OpenAI's team gave the shelves it judged higher quality more of the reading.
Wikipedia got read 3.4 times, and the web archive less than half of once
Wikipedia got read 3.4 times, and the web archive less than half of once.
The drawn line marks one full read of a shelf.
Those shares decide how often GPT-3 saw each shelf. Across the three hundred billion pieces, GPT-3 read Wikipedia about three point four times over. GPT-3 got through the Common Crawl shelf less than half of one time, zero point four four. The authors wrote that this trade accepts a small amount of overfitting, in exchange for higher quality training data. Overfitting here means learning the repeated shelves a little too closely. OpenAI filtered the web shelf before training. OpenAI started from forty-five terabytes of compressed web text and kept five hundred and seventy gigabytes after filtering. A terabyte is a thousand gigabytes.
More computing bought GPT-3’s models less surprise, on a line you can predict
More computing bought GPT-3’s models less surprise, on a line you can predict.
14 models, from 100,000 dials up to GPT-3.
GPT-3's paper checks whether the game kept paying off, with a chart. The chart plots surprise against the computing spent on training, for fourteen models, from a hundred thousand dials up to GPT-3 itself. Each model's surprise falls, then levels off. The lowest surprise any model reached, at each amount of computing, lands on one smooth line. An earlier OpenAI study drew that line from smaller models. GPT-3's models stayed on the line for a hundred times more computing, with only small misses.
Pretraining alone makes a model that continues text, even a question
Pretraining alone makes a model that continues text, even a question.
Asked about socks, GPT-3 replied with a question of its own.
This picture is roughly right. Here is where the picture breaks. First, the sliding card is slow motion. Real training puts a card over every position in a passage at once, two thousand and forty-eight pieces long, and scores all of those bets together. Second, the game teaches a model to continue text. OpenAI's 2022 paper on teaching GPT-3 to follow instructions shows the gap. The authors asked GPT-3 why it is important to eat socks after meditating. GPT-3 replied with another question. What can you learn from socks? The authors picked that question to show the habit. On a coding question in the same paper, GPT-3 gave a real answer about half the time. So a model trained only on this game may continue your question instead of answering it. Turning that continuer into an assistant is a separate, later step, and that step gets its own lesson.
One sentence, one card and a hand-picked library taught GPT-3 to write
One sentence, one card and a hand-picked library taught GPT-3 to write.
Every answer in the game came from the text itself.
How it’s made · 1
A card covers the next piece of a real sentence. The model spreads a bet over every piece it knows. Lifting the card gives the score. The score comes from the share of the bet on the real piece. Every dial turns a hair toward a better bet, and the card slides one piece right. GPT-3 played that round three hundred billion times, over five shelves, with Wikipedia read three point four times and the web archive less than half of once.
