Cat and kitten sit equally far apart
Cat and kitten sit equally far apart as numbered slots.
A chatbot still treats those two words as the same request.
Cat and kitten barely share letters. A chatbot still treats those two words as the same request. ChatGPT and Claude read a sentence as numbered pieces. Those pieces are numbered slots. Slot forty-seven and slot one thousand nine hundred three sit equally far apart as numbers. Cat can sit in one slot. Kitten can sit in the other. The numbers carry no closeness.
Words that keep the same company
Words that keep the same company sit near each other.
Streets are company · who shows up next to whom.
Treat each word as a pin on a city map. The streets are company. Company means who shows up next to whom. Cat drops on the map. Kitten drops nearby. Car drops far away. Cat and kitten keep the same neighbors. Car keeps different neighbors. The numbered chips become pins. Slot forty-seven is now a place. Slot one thousand nine hundred three is now a place. The letters of cat and kitten barely overlap. The pins sit close. A chatbot can treat those two spellings as related.
A 2013 Google paper learned each place
A 2013 Google paper learned each place by guessing neighbors.
Word2Vec trained on about six billion words of Google News.
Guess the middle word · or guess the neighbors.
A 2013 Google paper learned those places by guessing neighbors. The paper is called Word2Vec. Mikolov, Chen, Corrado, and Dean wrote Word2Vec. Those authors trained on Google News. Google News held about six billion words. The word list was capped at the million most common words. Each word became a list of numbers. That list is a place on the map. Researchers call that list an embedding. Two guessing games taught the places. Look at the neighbors and guess the middle word. Or look at the middle word and guess the neighbors. On this grid, a window of neighbors sits around cat. One game looks at that window and guesses cat. The other game looks at cat and guesses the window. Either game pushes cat toward kitten. Words that keep the same company end up near each other.
King minus man plus woman
King minus man plus woman lands near queen.
Closest other pin · search skips king, man, and woman.
King, man, and woman already sit on this map. King sits near man. Queen sits near woman. Take the pin for king. Subtract the place for man. Add the place for woman. The walk lands at a new spot on the map. The computer searches the rest of the pins for the closest other pin. The search skips king, man, and woman. Those three pins started the walk. The closest leftover pin is queen. Each kind of pair has its own step on the map. The step from man to woman is one of those steps. Add that step to king. The new spot sits near queen. Mikolov, Yih, and Zweig wrote this walk in a second 2013 paper. Those authors say the walk lands very close to queen.
Paris minus France plus Italy
Paris minus France plus Italy lands near Rome.
The same walk, Paris minus France plus Japan, lands near Tokyo.
The same walk moves cities. A gallery in the Word2Vec paper, printed as Table 8, shows the city walks. Paris sits near France. Rome sits near Italy. Tokyo sits near Japan. The country-to-capital step is the same kind of reusable step as the man-to-woman step. Start at Paris. Subtract the place for France. Add the place for Italy. The closest other pin is Rome. Start at Paris. Subtract the place for France. Add the place for Japan. The closest other pin is Tokyo. The same arrow walks new cities.
The full analogy test hit 53.3%
The full analogy test hit the exact word 53.3% of the time.
Three hundred numbers per word · 783 million words of Google News.
| kind of question | exact-word hits |
|---|---|
| meaning | 50.0% |
| grammar | 55.9% |
| whole test | 53.3% |
A hit is the exact closest pin. Synonyms count as misses.
The computer takes the closest other pin as the answer. Word2Vec ran that search on an analogy test. An analogy test asks for the missing fourth word. King minus man plus woman is one of those questions. Queen is the missing word. One run used three hundred numbers per word. That run trained on seven hundred eighty-three million words of Google News. That run scored fifty-three point three percent of the questions. Fifty point zero percent of the meaning questions hit the exact word. Fifty-five point nine percent of the grammar questions hit the exact word. The test counted a hit only when the closest pin was the exact word. Synonyms counted as misses.
Table 8’s showcase would score about 60%
Table 8’s showcase would score about 60% on exact match.
Big minus bigger plus small lands on larger.
Bank is one pin for rivers and money.
53.3% is the full analogy test · about 60% is Table 8’s showcase under the same exact-word rule.
This picture is roughly right. Here is where the picture breaks. Table 8 is a showcase. Mikolov, Chen, Corrado, and Dean say those showcase rows would score only about sixty percent if the test demands the exact word. Fifty-three point three percent measured the full analogy test. About sixty percent is how Table 8's showcase rows would score under the same exact-word rule. The same gallery walks big minus bigger plus small. The closest pin is larger. The map stores one pin per spelling. Bank is one spelling. Rivers and money share that one pin. Bank is a teaching example of that design. Today's chat models still drop a pin on each piece. Those models learn the pin while they guess the next piece.
Slot forty-seven and slot 1903 now sit as pins
Foundations 101 · 3 · next: how a model decides which earlier pins matter
Slot forty-seven and slot one thousand nine hundred three started as numbered pieces. Those pieces now sit as pins with neighbors. A word is a pin on a map of company. Company means who shows up next to whom. The finished map holds every pin from the earlier slides. Cat sits near kitten. Car sits far away. King sits with man, woman, and queen. Paris sits with France, Italy, Rome, Japan, and Tokyo. Bank sits as one pin. Word2Vec learned those places by guessing neighbors on Google News. King minus man plus woman lands near queen. Paris minus France plus Italy lands near Rome. The computer takes the closest other pin. The search skips the pins that started the walk. Each spelling gets one pin. Today's chat models still drop a pin on each piece. Those models learn the pin while they guess the next piece. How a model decides which earlier pins matter is the next Foundations lesson.
