GPT-3 learned to write by guessing the next piece 300 billion times

How did ChatGPT learn to write? GPT-3, OpenAI's 2020 model, learned from a fill-in-the-blank game. The game hides the next piece of a sentence, and GPT-3 bets on what sits under the card. A piece is a chunk of a word, the unit from the tokens episode. GPT-3 played that game three hundred billion times. That game is called pretraining, the first stage of making a model like GPT-3. In pretraining the model reads, before anyone teaches it to answer questions.

Take one sentence. I poured myself a cup of. Put a card over the piece that comes next, and guess what the card is hiding. Then lift the card. The sentence itself shows the real next piece. We picked this sentence, and our sentence continues with coffee. The text does the marking. Every book, every web page and every Wikipedia article works the same way. Each text is a long stack of these quizzes, one quiz per piece, with every answer already printed. That is why GPT-3 could learn from raw text, with zero labels added by people.

The model answers with a bet. Every piece the model knows gets a share, and the shares always add up to a hundred percent. OpenAI never released GPT-3's dials, the numbers inside the model that training turns. So we ran this sentence through GPT-2 Large instead. GPT-2 Large is OpenAI's 2019 model, built on the same design as GPT-3 at a smaller size. GPT-2 Large put forty-four percent of its bet on tea. GPT-2 Large put twenty-one percent on coffee and six percent on hot. The rest of the bet spreads thin over the other fifty thousand or so pieces.

Now lift the card. The real piece is coffee. GPT-2 Large's favorite was tea, and the model still earns partial credit, because twenty-one percent of the bet sat on coffee. The score comes from that share alone. A model that puts all of its bet on the real piece earns a perfect score. A model that puts a sliver on the real piece takes a big penalty. That penalty measures how surprised the model was by the real piece. OpenAI wrote this goal down as one line in its 2018 paper on GPT-1, the first model in the family. That line asks for the biggest possible bet on each real next piece, given the pieces before it. GPT-3 trains on that same goal.

Slide text

Coffee held 21% of the bet, so a wrong favorite still earns partial credit.

Less of the bet on the real piece means a bigger penalty.

I poured myself a cup ofcoffeereal bets · GPT-2 Large (OpenAI 2019)tea 44%coffee 21%50,000+ other pieceshot 6%water 1%warm 1%21% of the bet sat here

GPT‑1’s goal: make the bet on each real next piece as large as it can be.

Radford et al. 2018, GPT-1 · equation 1 · page 3

After the score, training nudges every dial a hair. Each nudge makes the bet on the real piece a little bigger next time. That nudge is the return trip from the training versus inference lesson, the walk back through the dials that works out which dial to turn. Then the card slides one piece to the right, and the game starts again on the next piece. GPT-3 played three hundred billion rounds of this game, one round for every piece of text it trained on.

GPT-3's paper prints its reading list as a table of five shelves. The biggest shelf is Common Crawl, a free, public archive of pages copied from the web. After filtering, the Common Crawl shelf held four hundred and ten billion pieces, and it got sixty percent of the reading. Next come web pages gathered by following links, and two book collections from the internet. The smallest shelf is English Wikipedia, at three billion pieces. Wikipedia got three percent of the reading. OpenAI's team gave the shelves it judged higher quality more of the reading.

Those shares decide how often GPT-3 saw each shelf. Across the three hundred billion pieces, GPT-3 read Wikipedia about three point four times over. GPT-3 got through the Common Crawl shelf less than half of one time, zero point four four. The authors wrote that this trade accepts a small amount of overfitting, in exchange for higher quality training data. Overfitting here means learning the repeated shelves a little too closely. OpenAI filtered the web shelf before training. OpenAI started from forty-five terabytes of compressed web text and kept five hundred and seventy gigabytes after filtering. A terabyte is a thousand gigabytes.

GPT-3's paper checks whether the game kept paying off, with a chart. The chart plots surprise against the computing spent on training, for fourteen models, from a hundred thousand dials up to GPT-3 itself. Each model's surprise falls, then levels off. The lowest surprise any model reached, at each amount of computing, lands on one smooth line. An earlier OpenAI study drew that line from smaller models. GPT-3's models stayed on the line for a hundred times more computing, with only small misses.

Slide text

More computing bought GPT-3’s models less surprise, on a line you can predict.

14 models, from 100,000 dials up to GPT-3.

surpriselower is better

best reachable for eachamount of computing

Brown et al. 2020, GPT-3 · Figure 3.1 · page 11

This picture is roughly right. Here is where the picture breaks. First, the sliding card is slow motion. Real training puts a card over every position in a passage at once, two thousand and forty-eight pieces long, and scores all of those bets together. Second, the game teaches a model to continue text. OpenAI's 2022 paper on teaching GPT-3 to follow instructions shows the gap. The authors asked GPT-3 why it is important to eat socks after meditating. GPT-3 replied with another question. What can you learn from socks? The authors picked that question to show the habit. On a coding question in the same paper, GPT-3 gave a real answer about half the time. So a model trained only on this game may continue your question instead of answering it. Turning that continuer into an assistant is a separate, later step, and that step gets its own lesson.

A card covers the next piece of a real sentence. The model spreads a bet over every piece it knows. Lifting the card gives the score. The score comes from the share of the bet on the real piece. Every dial turns a hair toward a better bet, and the card slides one piece right. GPT-3 played that round three hundred billion times, over five shelves, with Wikipedia read three point four times and the web archive less than half of once.