LoRA fine-tunes GPT-3 by training 4.7 million new numbers

How do you teach a giant AI model a new job, without building a second giant? In 2021, a team at Microsoft published an answer called Lora. Last lesson, GPT-3, OpenAI's 2020 model, copied answers people wrote by hand. That copying step was fine-tuning. Fine-tuning means more training on a narrow set of examples, so a general model gets good at one job. A dial is one of the numbers inside a model that training turns. The ordinary way nudges all hundred and seventy-five billion of GPT-3's dials. Each tuned job then gets saved as a whole new copy, three hundred and fifty gigabytes. Lora, short for low-rank adaptation, leaves every one of those dials alone. Lora trains four point seven million new numbers instead, and those numbers kept up with the ordinary way.

Picture GPT-3's dials printed on a huge sheet of paper, one number in every square. Ordinary fine-tuning reprints the whole sheet for every job. Lora locks the printed sheet and lays a clear sheet over the top. Training writes only on the clear sheet. The model reads the two sheets together, so every square counts as the printed number plus whatever the clear sheet adds there.

GPT-3 keeps its dials in grids. The grids Lora works on are twelve thousand, two hundred and eighty-eight squares wide, and the same number of squares tall. That makes about a hundred and fifty-one million dials in one grid. GPT-3 stacks ninety-six layers, and each layer holds several grids of dials.

Lay one clear sheet over one of those grids. If training wrote the clear sheet one square at a time, the clear sheet would need about a hundred and fifty-one million numbers of its own. Do that for every grid in every layer, and the clear sheets weigh as much as GPT-3. The saving has to come from how Lora writes on the clear sheet.

Lora writes the clear sheet like a times table. One column of numbers runs down the left edge. One row of numbers runs across the top. Every square gets its column number times its row number. On a small sheet, four squares by four, a column of four numbers and a row of four numbers fill all sixteen squares. On a GPT-3 grid, one column and one row hold twenty-four thousand, five hundred and seventy-six numbers, and those numbers fill all hundred and fifty-one million squares. The numbers you store grow with the edge of the sheet, and the squares they fill grow with its area. Every row on a times-table sheet is the same row, scaled up or down. So Lora can stack a few column-and-row pairs and add their sheets together. The count of pairs is called the rank, and a small count is the low rank in low-rank adaptation.

Training starts with the row full of random numbers and the column full of zeros. Zero times any number is zero, so every square on the clear sheet starts blank. Before training, the model behaves exactly like the original GPT-3. Training then nudges only the numbers in the column and the row. In each of GPT-3's ninety-six layers, the authors used one pair on each of two grids, or two pairs on one grid. Either way, the total came to four point seven million trained numbers. The other hundred and seventy-five billion dials stayed locked.

The authors tested both ways on three jobs. The first job turns a question into a lookup in a database. Lora got seventy-three point four percent of those right, and ordinary fine-tuning got seventy-three point eight. The authors say their scores on that job wobble by about half a percent, so the first job is a tie. The second job judges whether one sentence follows from another. Lora got ninety-one point seven percent right, against eighty-nine point five. The third job sums up a chat, scored by how many words the summary shares with one a person wrote. Lora scored fifty-three point eight, against fifty-two. A bigger Lora setup, with four pairs on the same grids, shrank the saved file for each job from three hundred and fifty gigabytes to thirty-five megabytes. After training, Lora presses each clear sheet onto its printed grid, so the finished model answers as fast as an ordinary fine-tuned one.

This picture is roughly right, and here is where the picture breaks. A times-table sheet can only write a simple change. The authors tried from one pair up to sixty-four pairs on the same two grids. On the database job, one pair got seventy-three point four percent right, and sixty-four pairs got seventy-three point five. The authors also say where they expect a few pairs to stop being enough. Their example is a job in a different language from the one GPT-3 learned in. For a job like that, the authors write that retraining the whole model could certainly do better than a thin sheet.

Second, a clear sheet only works on top of the printed one. The thirty-five megabyte file holds only the columns and rows. To use that file, you still load the full three hundred and fifty gigabyte GPT-3 underneath. Lora shrinks every extra job instead. For a hundred jobs, a hundred full copies of GPT-3 take about thirty-five terabytes. That is thirty-five thousand gigabytes. One shared GPT-3 plus a hundred clear sheets take about three hundred and fifty-four gigabytes.

First, lock GPT-3's dials. Lay a clear sheet over some of its grids. Write each clear sheet as a times table, from one column and one row, with the column starting at zero. Train only the columns and the rows. When training ends, press each sheet onto its grid. That is how Lora taught GPT-3 new jobs with four point seven million trained numbers.