Training GPT-3 cost 900 billion times one piece of an answer

Building GPT-3 took three point one four times ten to the twenty-third operations. Answering a question with Claude Haiku 4.5 costs a fraction of a cent. GPT-3 is the 2020 OpenAI model whose own paper prints that first number. Claude Haiku 4.5 is one of Anthropic's models, listed at one dollar per million pieces in and five dollars per million pieces out. A piece is a chunk of a word, the unit from the tokens episode. A two hundred word answer comes to two hundred and eighty-three pieces, at about a seventh of a cent. One machine produced both numbers, and the arithmetic between them fits on one diagram.

Picture a wall of dials. Each dial is one number the model can turn, and researchers call one of those dials a parameter. GPT-3 has one hundred seventy-four thousand six hundred million dials. When you ask GPT-3 something, your question runs forward along that wall once. Every dial multiplies the number passing through it. The model adds that result into the running answer. One multiply and one add is two operations per dial. An operation is one piece of arithmetic the chip does. That one forward trip is the whole cost of an answer. Reading a page works the same way. Your eyes cross every line once, and then you are done.

Learning uses the same wall of dials, and learning adds a return trip. After the forward trip, the model checks how wrong its guess was. Then the model walks back along the wall working out which dial to nudge and by how much. Kaplan and colleagues at OpenAI measured that return trip in 2020. Those authors write that the return trip costs approximately twice what the forward trip costs. So the forward trip is two operations per dial, the return trip is four, and learning from one piece costs about six operations per dial. That is three times what answering one piece costs, and that factor of three is the entire difference in kind.

That six is not our arithmetic. The six sits in the appendix of GPT-3's paper, on the row for the largest of the eight models the authors trained. The three printed next to that six is the easy one to misread. The three is the total multiplier against the forward trip, so two times three gives six. The return trip on its own costs four operations per dial, not three.

The millions come from the repetition. GPT-3 was trained on three hundred billion pieces of text, and GPT-3 ran the forward trip and the return trip once for every one of those pieces. Multiply it out. One hundred seventy-four thousand six hundred million dials, times three hundred billion pieces, times six operations each, gives three point one four times ten to the twenty-third operations. That is a three with twenty-three more digits after it. That total is the one the paper prints on its own row, so you can check the arithmetic against the table.

Put one answer next to one training run. Answering one piece costs two operations per dial, so two times one hundred seventy-four thousand six hundred million dials comes to about three hundred and forty-nine billion operations. Divide the training total by that, and you get about nine hundred billion. That ratio is exactly the two factors multiplied: three for the return trip, three hundred billion for the repetition. Both bars fit in one picture only because each step up the scale is ten times the step below.

A second lab checks the rule. Llama 3 is Meta's 2024 model. Meta says Llama 3 has four hundred and five billion dials, trained on fifteen point six trillion pieces, and that the run took three point eight times ten to the twenty-fifth operations. Do the same multiply. Six operations, times four hundred and five billion dials, times fifteen point six trillion pieces, gives three point seven nine times ten to the twenty-fifth. Meta built Llama 3 at a different lab on a different design, and the same six still reproduces the total Meta published. Llama 3 also read fifty-two times as many pieces as GPT-3 read, which is why the headline numbers keep climbing. Meta ran that training on up to sixteen thousand NVIDIA H100 graphics chips, each drawing seven hundred watts.

This picture is roughly right. Here is where it breaks. First, the six leaves out the attention work. Attention is the step where the model compares every piece in your question against every other piece. GPT-3's paper says that step usually uses under ten percent of the total. For this exact model the dropped share works out to about one point four percent, so the six is a good floor here. At today's long context windows that share grows, which is why every number here carries the word about. Second, operations are not dollars. The published figure counts arithmetic. That figure leaves out the failed runs, the people, and the power behind sixteen thousand chips at seven hundred watts. Third, this arithmetic covers learning the language. Meta says a further stage follows, where the team tunes the model to follow instructions, and that stage sits on top of this number.

The two numbers from the start now have the multiplication drawn between them. Answering runs one trip forward through the dials, about two operations per dial per piece. Learning runs that same trip plus a return trip costing twice as much, about six. Then the training run does that pair once for every one of three hundred billion pieces. Three times for the return trip, three hundred billion times for the repetition. That is the whole gap between the millions it took to build GPT-3 and the seventh of a cent Claude Haiku 4.5 charges for an answer. What instruction training adds on top of this number is the next Foundations lesson.