Building GPT-3 cost a fortune, answering costs a fraction of a cent
Building GPT-3 cost a fortune. Asking Claude Haiku 4.5 a question costs a fraction of a cent.
Both numbers below are published by the people who own the models.
Building GPT-3 took three point one four times ten to the twenty-third operations. Answering a question with Claude Haiku 4.5 costs a fraction of a cent. GPT-3 is the 2020 OpenAI model whose own paper prints that first number. Claude Haiku 4.5 is one of Anthropic's models, listed at one dollar per million pieces in and five dollars per million pieces out. A piece is a chunk of a word, the unit from the tokens episode. A two hundred word answer comes to two hundred and eighty-three pieces, at about a seventh of a cent. One machine produced both numbers, and the arithmetic between them fits on one diagram.
Answering runs one trip forward through the dials
Answering runs one trip forward through 174,600 million dials.
Two operations per dial, for every piece of text in play.
A dial is one number the model can turn.
Picture a wall of dials. Each dial is one number the model can turn, and researchers call one of those dials a parameter. GPT-3 has one hundred seventy-four thousand six hundred million dials. When you ask GPT-3 something, your question runs forward along that wall once. Every dial multiplies the number passing through it. The model adds that result into the running answer. One multiply and one add is two operations per dial. An operation is one piece of arithmetic the chip does. That one forward trip is the whole cost of an answer. Reading a page works the same way. Your eyes cross every line once, and then you are done.
Learning walks the wall a second time
Learning walks the wall a second time to find which dial to nudge.
Two operations out, four back, six in total for one piece.
Learning uses the same wall of dials, and learning adds a return trip. After the forward trip, the model checks how wrong its guess was. Then the model walks back along the wall working out which dial to nudge and by how much. Kaplan and colleagues at OpenAI measured that return trip in 2020. Those authors write that the return trip costs approximately twice what the forward trip costs. So the forward trip is two operations per dial, the return trip is four, and learning from one piece costs about six operations per dial. That is three times what answering one piece costs, and that factor of three is the entire difference in kind.
GPT-3’s appendix prints the 2, the 3 and the 6
GPT-3’s appendix prints the 2, the 3 and the 6 on one row.
The 3 is the total multiplier against the forward trip, so the return trip alone costs 4.
| forward operations per dial per piece | 2 |
| multiplier for the return trip | 3 |
| operations per dial per piece to learn | 6 |
| dials | 174,600 million |
| training pieces | 300 billion |
| total training operations | 3.14 × 1023 |
That six is not our arithmetic. The six sits in the appendix of GPT-3's paper, on the row for the largest of the eight models the authors trained. The three printed next to that six is the easy one to misread. The three is the total multiplier against the forward trip, so two times three gives six. The return trip on its own costs four operations per dial, not three.
The multiply lands back on the total the paper prints
The multiply lands back on 3.14 × 1023, the total the paper prints.
300 billion pieces of text, each one through the forward trip and the return trip.
174,600 million dials × 300 billion pieces × 6 operations = 3.14 × 1023
The millions come from the repetition. GPT-3 was trained on three hundred billion pieces of text, and GPT-3 ran the forward trip and the return trip once for every one of those pieces. Multiply it out. One hundred seventy-four thousand six hundred million dials, times three hundred billion pieces, times six operations each, gives three point one four times ten to the twenty-third operations. That is a three with twenty-three more digits after it. That total is the one the paper prints on its own row, so you can check the arithmetic against the table.
One training run against one answered piece
One training run cost about 900 billion times one answered piece.
Three times for the return trip, 300 billion times for the repetition.
Each step up the scale is ten times the step below.
Put one answer next to one training run. Answering one piece costs two operations per dial, so two times one hundred seventy-four thousand six hundred million dials comes to about three hundred and forty-nine billion operations. Divide the training total by that, and you get about nine hundred billion. That ratio is exactly the two factors multiplied: three for the return trip, three hundred billion for the repetition. Both bars fit in one picture only because each step up the scale is ten times the step below.
Meta’s Llama 3 lands on the same rule
Meta’s Llama 3 lands on the same rule four years later.
Six operations per dial per piece reproduces the figure Meta states.
6 × 405 billion × 15.6 trillion = 3.79 × 1025
52× the training pieces of GPT-3A second lab checks the rule. Llama 3 is Meta's 2024 model. Meta says Llama 3 has four hundred and five billion dials, trained on fifteen point six trillion pieces, and that the run took three point eight times ten to the twenty-fifth operations. Do the same multiply. Six operations, times four hundred and five billion dials, times fifteen point six trillion pieces, gives three point seven nine times ten to the twenty-fifth. Meta built Llama 3 at a different lab on a different design, and the same six still reproduces the total Meta published. Llama 3 also read fifty-two times as many pieces as GPT-3 read, which is why the headline numbers keep climbing. Meta ran that training on up to sixteen thousand NVIDIA H100 graphics chips, each drawing seven hundred watts.
The six is a floor, and the bill is more than the operations
The six is a floor, and the training bill is more than the operations.
Attention work, dollars and instruction training all sit outside this number.
This picture is roughly right. Here is where it breaks. First, the six leaves out the attention work. Attention is the step where the model compares every piece in your question against every other piece. GPT-3's paper says that step usually uses under ten percent of the total. For this exact model the dropped share works out to about one point four percent, so the six is a good floor here. At today's long context windows that share grows, which is why every number here carries the word about. Second, operations are not dollars. The published figure counts arithmetic. That figure leaves out the failed runs, the people, and the power behind sixteen thousand chips at seven hundred watts. Third, this arithmetic covers learning the language. Meta says a further stage follows, where the team tunes the model to follow instructions, and that stage sits on top of this number.
The gap between the millions and the cents
The gap between the millions and the cents is 3 × 300 billion.
2 operations per dial to answer · 6 to learn · 300 billion pieces to repeat
Foundations 101 · 5 · next: what instruction training adds on top
The two numbers from the start now have the multiplication drawn between them. Answering runs one trip forward through the dials, about two operations per dial per piece. Learning runs that same trip plus a return trip costing twice as much, about six. Then the training run does that pair once for every one of three hundred billion pieces. Three times for the return trip, three hundred billion times for the repetition. That is the whole gap between the millions it took to build GPT-3 and the seventh of a cent Claude Haiku 4.5 charges for an answer. What instruction training adds on top of this number is the next Foundations lesson.
