Week 6 of 8 · Language half

From counts to meaning

Last week we took the first steps of turning language into math. This week we take the counts from last week and ask what those numerical representations can tell us about similarity and meaning, first for documents and then for words.

A quick status. You have now learned to represent language with numbers using tokens, counts and document-term matrices. These tools turned language into something a computer can store and compare. But we ended on two problems.

The first one you saw in the lookalikes explorable. With raw counts, the, and and of dominated every comparison, and the pages only started to find their real neighbors once we threw a stopword list at them by hand. The second was the dog, puppy and democracy example. When every word has its own coordinate, every word is exactly equally far from every other word. A count vector does not know what a word means, and a Bag-of-Words representation can miss that two documents are expressing almost the same underlying concept ... simply because they use different words.

One way to get closer to meaning is to look at context. Words that occur in similar surroundings often have something in common, and documents that use language in similar ways may also be related even when they do not contain exactly the same words.

This week we use that idea at two levels. First, we represent entire documents and compare which pages are similar, which terms distinguish groups of pages, and which themes recur across a corpus. Then we change the object and represent individual words through the contexts in which they occur.

A useful way to remember the split is that documents are represented by their words; later, words are represented by their contexts.

As in week 5, most sections today end with a short Python example. They are there to be run: put them in Jupyter, change a number, and watch what happens.

1 · Documents need better weights

Raw word counts reward frequent words. That sounds reasonable until the same boring words are frequent everywhere. A term can appear many times on one page and still tell us very little about what makes that page distinctive (if the same term also appears on almost every other page).

That was exactly what we saw in the lookalikes explorable, and removing stopwords by hand was a blunt fix: somebody has to write the list, and it only knows about English, not about Marvel.

TF-IDF tackles the same problem, but using the data itself. It combines two pieces of information: how frequent a term is within each document, and how unusual that term is relative to the full collection of documents.

The TF-IDF idea comes from information retrieval. Imagine searching for the superpower. A document should not rank highly simply because it contains the hundreds of times. The word superpower is much more informative because it occurs in fewer documents. TF-IDF therefore gives a term a high weight when it occurs often in a particular document, but discounts it when it also occurs in many other documents. (If you built last week's 20-line search engine, this is the fix it was missing.)

Read the quick overview below, focusing on the sections about TF-IDF, but feel free to read more if you find it interesting:

There are several (closely related) ways to define TF-IDF, all implementing subtly different versions of the same general idea. Here we use a simple version where TF (term frequency) is measured relative to document length and the IDF (inverse document frequency) is defined based on how many documents contain the term in question. Let us first look at each part separately. Term frequency measures how common a term $t$ is inside a particular document $d$:

$$ \mathrm{tf}(t,d)=\frac{\mathrm{count}(t,d)}{|d|} $$

Here, $\mathrm{count}(t,d)$ is simply the number of times term $t$ appears in document $d$, while $|d|$ is the total number of tokens in that document. Dividing by document length means that seeing a word 10 times in a short page counts for more than seeing it 10 times in a very long page.

The second part asks how common the term is across the entire corpus:

$$ \mathrm{idf}(t)=\log\frac{N}{\mathrm{df}(t)} $$

Here, $N$ is the total number of documents in the corpus, and $\mathrm{df}(t)$ is the number of those documents that contain the term $t$. If a word appears in almost every document, $\mathrm{df}(t)$ is large and its IDF becomes small. If it appears in only a few documents, its IDF becomes larger.

Take three tiny documents:

• D1: The mutant superpower was wild and dangerous.
• D2: The superpower felt wild and very dangerous.
• D3: The Avengers assembled before the battle.

The word the appears in all three documents, so its document frequency is 3. Notice that document frequency counts documents, not total occurrences: the occurs four times across the corpus, but it is still present in 3 of 3 documents. Its IDF is therefore

$$ \mathrm{idf}(\mathrm{the})=\log\frac{3}{3}=\log(1)=0. $$

So the receives no weight from IDF. superpower appears in two of the three documents, so

$$ \mathrm{idf}(\mathrm{superpower})=\log\frac{3}{2}\approx 0.405. $$

mutant appears only in D1, so it receives the largest IDF of the three:

$$ \mathrm{idf}(\mathrm{mutant})=\log\frac{3}{1}=\log(3)\approx 1.099. $$

Here and in the explorable, log is the natural logarithm. Other log bases rescale the values but do not change their ordering.

Now we can calculate the actual TF-IDF weights. D1 and D2 each contain 7 tokens, and superpower occurs once in each, so

$$ \mathrm{tf}(\mathrm{superpower},D1)=\mathrm{tf}(\mathrm{superpower},D2)=\frac{1}{7}\approx 0.143. $$

Multiplying TF by IDF gives

$$ \mathrm{tfidf}(\mathrm{superpower},D1) =\mathrm{tfidf}(\mathrm{superpower},D2) =\frac{1}{7}\log\frac{3}{2} \approx 0.058. $$

mutant also occurs once in the 7-token D1, but because it appears in only one document its IDF is larger:

$$ \mathrm{tfidf}(\mathrm{mutant},D1) =\frac{1}{7}\log(3) \approx 0.157. $$

For the, IDF is 0, so its TF-IDF weight is 0 in every document regardless of how often it occurs there. This is exactly what we want: the representation can distinguish a word that is common in one particular document from a word that is common everywhere.

In general, we multiply the two parts:

$$ \mathrm{tfidf}(t,d)=\mathrm{tf}(t,d)\,\mathrm{idf}(t). $$

In summary: A term has a high TF-IDF score when it appears relatively often in one document but is relatively uncommon across the rest of the corpus. Common words that occur everywhere receive little weight, while words that help distinguish one document from the others receive more.

I created the explorable below so you can visualize and play with these quantities directly.

Start with the superpower and move from counts to TF-IDF. Then switch to mutant superpower. Watch how the term weights and the document ranking change.

With the Marvel corpus, the setup is the same, only larger. Each of the 303 character pages is one document, and each vocabulary term gets a weight that depends both on how often it appears on that page and on how widely it appears across the full collection.

Cosine similarity, a second look. You met cosine similarity last week in the lookalikes explorable. Now that the vectors are full of weights rather than counts, it is worth slowing down to see why cosine similarity is the right way to compare them.

One option would be ordinary Euclidean distance, but that is awkward for text because the length of a document can dominate the comparison. A long article and a short paragraph about the same thing can end up far apart simply because one contains more words.

Cosine similarity compares the direction of two vectors. For vectors $A$ and $B$,

$$ \cos(\theta)=\frac{A\cdot B}{\lVert A\rVert\lVert B\rVert}. $$

The dot product $A\cdot B$ is large when the same dimensions contain large values in both vectors. Dividing by the lengths of the two vectors removes the effect of their overall magnitude, so what remains depends only on the angle between them. This is why multiplying a document vector by 10 does not change its cosine similarity with any other vector. Its length changes, but its direction does not. Two vectors pointing in exactly the same direction have cosine similarity 1. For the non-negative TF-IDF vectors we use here, the cosine similarity is never below 0, and vectors with no weighted vocabulary in common have similarity 0.

The figure below uses the TF-IDF vectors from the three sentences I constructed above.

The arrows show the angle between the full high-dimensional vectors. They are drawn in two dimensions only so that we can see that angle.

Compare D1 with D2, then D1 with D3. Finally scale one vector without changing its direction and watch what happens to the cosine.

Python example. The definitions on this page are short enough to write yourself, and doing so once is the best way to both understand and trust them. Here they are, run on the three sentences:

import math, re
import numpy as np

documents = [
    "The mutant superpower was wild and dangerous.",
    "The superpower felt wild and very dangerous.",
    "The Avengers assembled before the battle.",
]
docs = [re.findall(r"[a-z]+", d.lower()) for d in documents]
N = len(docs)

def tf(t, d):
    return d.count(t) / len(d)

def idf(t):
    df = sum(t in d for d in docs)
    return math.log(N / df)

def tfidf(t, d):
    return tf(t, d) * idf(t)

[round(tfidf("superpower", d), 3) for d in docs]
# [0.058, 0.058, 0.0]

Those are the numbers from the worked example above. Now build the full vectors and compare them:

vocab = sorted({t for d in docs for t in d})
V = np.array([[tfidf(t, d) for t in vocab] for d in docs])

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

print(round(cosine(V[0], V[1]), 3), round(cosine(V[0], V[2]), 3))
# 0.214 0.0

D1 and D3 share only the, and the has weight 0, so their cosine is exactly 0. Now do what everybody actually does, and ask scikit-learn:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(documents)

print(cosine_similarity(X).round(3))
# [[1.    0.571 0.139]
#  [0.571 1.    0.139]
#  [0.139 0.139 1.   ]]

scikit-learn says D1 and D3 are at 0.139, not 0. Nobody made a mistake. Print vectorizer.idf_ and you will see that the gets an IDF of 1, not 0, because scikit-learn uses a slightly different definition of both TF and IDF. Exercise 6.2 asks you to find out exactly what it does, and the Goodies have the answer when you are done. The lesson is the same as with CountVectorizer last week: a library makes choices for you, and you only see them if you check its output against a case you can do by hand.

Documentation: scikit-learn's text feature extraction.

Back to the lookalikes. Now I present to you last week's explorable with one more button. Tadaaa! The TF-IDF version keeps every stopword and uses the definitions above.

Start with Wolverine, where last week left off, and switch from stopwords removed to TF-IDF. Then pick Storm and look hard at her closest page. Finally, switch to closest unlinked pairs.

Even with no stopword list at all, TF-IDF does better than the hand-made list did. On average, 4.0 of each page's ten textual neighbors are also linked to it in the network, against 2.8 with stopwords removed, 1.9 with raw counts, and 0.3 for ten random pages. The IDF solved the stopword issue, and it also caught stopwords that only exist in our specific context. The words Marvel and Comics occur on all 303 pages, so their weight is exactly 0, just like the, although no English stopword list has ever heard of them.

But it's not perfect. Look at Storm. Her closest page is the Human Torch, and the words behind the match are storm and storm's. The Human Torch is Johnny Storm. TF-IDF rewards rare words, and a surname is a very rare word, so it cannot tell a name from a meaning. The closest unlinked pair in the whole corpus makes the same point (Emma Frost and Jack Frost). Just under it sits a cluster of short pages about the members of Strikeforce: Morituri, a team whose members share a comic, its creators, and a rare word. The network knows the team, but many pairs of teammates never link to each other directly, and the text puts them together anyway. That one is real structure. Telling those two kinds of match apart is your job, not the vector's.

What makes a page distinctive. TF-IDF can also be read one page at a time. The highest-weighted words on a page are its fingerprint: the words it uses a lot and the rest of the corpus rarely does. With names kept, the fingerprints are mostly names, just as Storm's match was. Take the names out and what is left starts to describe the character.

Start with Wolverine and switch to names removed. Click adamantium and follow it to the other pages that use it. Then pick Storm, remove the names, and ask why her and she make her list.

2 · Compare groups of documents

The network part of the course gave us structural communities without reading a single page. Text lets us ask what those communities are actually talking about.

"Scattertext" is a tool for comparing two labeled collections of text term-by-term. It counts how often words or phrases are used in each corpus and allows the user to see shared and distinctive language at the same time. The original Scattertext package and examples contain several ways of scoring and displaying those differences.

I made the teaching view below from the 2012 US convention speech sample corpus that comes with the Scattertext package, following the same two-corpus comparison that Scattertext is built for. Each dot is one term. The horizontal axis shows Republican uses per 10,000 words and the vertical axis shows Democratic uses per 10,000 words. A term used at about the same rate in both groups sits near the diagonal. A term that is relatively more common in one group moves away from the diagonal, toward that group's axis. The axes are log-scaled so the many less frequent terms do not all collapse into the same corner, while the tick labels still show uses per 10,000 words.

The package saves you a lot of bookkeeping and gives you useful interactive tools. But the basic idea is something you could easily recreate yourself, especially with agentic help: count terms in two corpora, turn the counts into comparable rates, plot one rate against the other, and remember to go back to the underlying text and have a closer look before you decide what a term means.

Search for a term or click a point. Terms near the diagonal are used at similar rates; terms above it are relatively more common in the Democratic speeches, while terms below it are relatively more common in the Republican speeches.

Extra feature: Scattertext compares how often two groups use each word. With my group I built a variation, the dialectogram, that compares how two groups use the same word, based on the word embeddings we look at in the second half of this page.

Python example. The convention speeches ship with the package, so the teaching figure takes a handful of lines to rebuild (pip install scattertext first).

import scattertext as st

df = st.SampleCorpora.ConventionData2012.get_data()   # 189 speeches: party, text, speaker
corpus = st.CorpusFromPandas(
    df, category_col="party", text_col="text",
    nlp=st.whitespace_nlp_with_sentences,
).build()

html = st.produce_scattertext_explorer(
    corpus, category="democrat",
    category_name="Democratic", not_category_name="Republican",
    metadata=df["speaker"], width_in_pixels=1000,
)
open("conventions.html", "w").write(html)

Open conventions.html in a browser. For Marvel, the only thing that changes is the DataFrame: one row per page, a text column from marvel_pages, and a category column holding each page's network community. If the install fails in your existing environment, a clean virtual environment with Python 3.11 or 3.12 is usually faster than untangling old dependencies.

Documentation: the Scattertext repository and examples.

3 · Find recurring themes

TF-IDF and Scattertext describe documents mainly through individual terms. Sometimes several different words are part of a broader recurring pattern. election, voter, parliament, minister and government, for example, can all contribute to a linguistic theme we might interpret as politics even if no single politics-related document contains all of them.

Topic modeling is what we call a large family of methods designed to find patterns of words that belong together in a collection of documents. The basic goal of topic modeling is to move away from the dependence on individual words and instead find recurring groups of related words: words that tend to appear in the same documents.

These topics (groups of related words) can help us summarize what's going on in a large corpus, compare documents, or ask whether different parts of the corpus tend to discuss different things.

There are many different topic-modeling methods. They all have strengths and weaknesses, and different methods define a "topic" in different ways. Some work directly on the document-term matrix: non-negative matrix factorization (NMF), for example, splits it into a small set of word groups and records how strongly each document uses each group. Newer methods start from document embeddings instead, cluster similar documents, and then describe each cluster by its most characteristic words; next week's BERTopic is one of these. Here we go deeper with Latent Dirichlet Allocation, LDA, because it keeps two things especially clear: what a topic is, and what a document is made of.

In LDA, a topic is a probability distribution over words, and a document is a mixture of topics. Imagine a topic whose high-probability words are mutant, school, gene, telepath and sentinel. We might decide to call it mutants, but that label comes from us. The model only gives us the probabilities. A particular document might then be 80% of that topic and 20% of another topic.

LDA is a generative model: it tells a step-by-step story of how the documents could have been produced. The generative story behind LDA is useful in order to understand what the model assumes. For every word position in a document, we basically imagine that we first choose a topic according to that document's topic mixture and then choose a word according to that topic's word distribution. In real data we only observe the words. LDA works backwards and estimates the hidden topic-word distributions and document-topic mixtures that could plausibly have produced them.

The two levels are easy to mix up at first, so the explorable below puts them next to each other. Choose a document and first look at its mixture across topics. Those probabilities tell you how strongly each topic contributes to that document. Then select one of the topics and inspect its distribution over words. Those probabilities tell you which words are most characteristic of the topic. Keep track of what each distribution sums over: topics for a document, or words for a topic.

Change the document and read both levels of the model: first the document's mixture over topics, then the selected topic's distribution over words. Remember that the topic names are human interpretations of the word distributions.

This also means that topics are not facts waiting inside the corpus to be discovered. You choose how many topics the model should look for (this is called model selection and for my money one of the hardest things in the universe, so don't spend too much time on it). And preprocessing, random initialization and the corpus itself can all affect the result. A list of top words is therefore only the beginning of the interpretation. You should also inspect the documents with high weight on that topic before you decide what it means.

Python example. scikit-learn fits LDA on the same kind of count matrix you built with CountVectorizer last week. LDA wants counts, not TF-IDF weights, because its generative story is about drawing words.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

names = list(marvel_pages)
vectorizer = CountVectorizer(stop_words="english", min_df=5, max_df=0.5)
X = vectorizer.fit_transform(marvel_pages[n] for n in names)

lda = LatentDirichletAllocation(n_components=8, random_state=0)
theta = lda.fit_transform(X)          # one row per page: its mixture over the 8 topics

vocab = vectorizer.get_feature_names_out()
for k, topic in enumerate(lda.components_):
    top = topic.argsort()[::-1][:8]
    print(k, " ".join(vocab[i] for i in top))

min_df=5 drops words that occur on fewer than five pages, and max_df=0.5 drops words that occur on more than half of them, so both are modeling choices of exactly the kind this section warns about. Each row of theta sums to 1, since it is a document's mixture, and theta[names.index("Spider-Man")] is the document half of the explorable above.

When we ran this, most of the eight topics came out as lists of character names: a Spider-Man topic (spider, venom, symbiote, peter, gwen), an X-Men topic (jean, phoenix, cyclops, frost), and so on. That is not a bug. Wikipedia pages about characters are full of names, and the model has no idea that we were hoping for themes. Change random_state from 0 to 1 and the Spider-Man and X-Men topics survive while most of the others reshuffle, which is the stability check that exercise 6.5 asks you to do properly.

Take the names out. The fix is a preprocessing choice, not a better model. Treat every word that is capitalized in most of its uses as a name, add those to the stopwords, and keep only tokens made of letters:

import re
from collections import Counter
from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS

seen, capitalized = Counter(), Counter()
for text in marvel_pages.values():
    for w in re.findall(r"[A-Za-z]+", text):
        seen[w.lower()] += 1
        capitalized[w.lower()] += w[0].isupper()
proper = {w for w in seen if capitalized[w] > seen[w] / 2}    # mostly capitalized: a name

vectorizer = CountVectorizer(stop_words=list(ENGLISH_STOP_WORDS | proper), min_df=5, max_df=0.5,
                             token_pattern=r"(?u)\b[a-zA-Z]{2,}\b")    # letters only
X = vectorizer.fit_transform(marvel_pages[n] for n in names)

Refit with the same settings and themes start to appear: vampires and gods (blood, vampire, god, gods), mutants (mutant, mutants, energy), symbiotes (symbiote, suit, host) and combat (shield, government, combat, trained). A new artifact appears too: a topic made of Wikipedia's media boilerplate (voiced, playable, list, best), because the pages of popular characters end with sections on their films and video games. Removing the names did not reveal the "true" topics; it changed what the model could see. And here's the deep wisdom that we draw from this example: the preprocessing is part of the model.

Documentation: scikit-learn's LatentDirichletAllocation.

4 · Now let's think about words

So far the rows we cared about represented documents. Now the question changes. How can a vector represent the meaning of a word?

Last week's answer was disappointing. In last week's one-hot vectors a word gets its own coordinate and nothing else, so dog is exactly as close to puppy as it is to democracy: cosine 0 in both cases. To do better, a word's vector has to be built from something other than its identity.

One way to make the intuition visible is to give you a word you may not know and let the surrounding sentences do the work. Take tezgüino.

I think you have a pretty good feel for tezgüino from these examples. Pretty cool, right?

This is a classic example in distributional semantics. Lin (1998) uses a slightly modified version of an example from Nida (1975, p. 167) to show how the meaning of an unfamiliar word can be inferred from context. Nobody gave you a dictionary definition here, but you inferred that tezgüino is some kind of alcoholic drink from the company the word keeps.

That is the intuition behind the distributional hypothesis: words that occur in similar contexts tend to have similar meanings. J. R. Firth's famous formulation is “You shall know a word by the company it keeps.”

To turn the intuition into cold, hard math, choose a context window around every occurrence of a target word and count the neighboring words. Stack those rows and you have a word-context matrix. In that matrix each row is a word, while each column records how often a particular context word appeared near it.

I made a small version of that matrix below. It uses the tezgüino sentences together with a few similar sentences about wine and bourbon. Change the context window and watch which words enter the rows. Then compare the rows for the three target words.

Change the context window and select a target row. Each cell counts how often that context word appears near the target in this tiny corpus. Similar rows mean that the words have appeared in similar surroundings.

Importantly, the size of the context window is part of the model. A narrow window pays more attention to immediate linguistic context. A wider window tends to pick up broader topical associations. Two words with similar rows appeared in similar surroundings, and we can once again use our old friend, the cosine similarity, to compare them.

The object has changed, but the recipe should feel familiar: define the dimensions, build a vector for each object, then compare the vectors.

Python example. A word-context row is a Counter over the words inside the window, added up over every occurrence of the target:

import re
from collections import Counter

sentences = [
    "A bottle of tezgüino sits on the table.", "Everybody likes tezgüino.",
    "Tezgüino makes you drunk.", "We make tezgüino from corn.",
    "A bottle of wine sits on the table.", "Everybody likes wine.",
    "Wine makes you drunk.", "We make bourbon from corn.", "Bourbon makes you drunk.",
]
tokenized = [re.findall(r"\w+", s.lower()) for s in sentences]

def context_row(target, window):
    row = Counter()
    for tokens in tokenized:
        for i, word in enumerate(tokens):
            if word == target:
                row.update(tokens[max(0, i - window):i] + tokens[i + 1:i + 1 + window])
    return row

for w in ["tezgüino", "wine", "bourbon"]:
    print(w, dict(context_row(w, 1)))
# tezgüino {'of': 1, 'sits': 1, 'likes': 1, 'makes': 1, 'make': 1, 'from': 1}
# wine {'of': 1, 'sits': 1, 'likes': 1, 'makes': 1}
# bourbon {'make': 1, 'from': 1, 'makes': 1}

Every context word in the tezgüino row also turns up around wine or bourbon, which is the matrix version of the guess you made from the four sentences. Try a window of 2 and watch bottle, you and corn arrive; the overlap still holds.

5 · Raw co-occurrence is not enough

Raw co-occurrence has the same problem as raw document counts. A word such as the occurs near almost everything. Pointwise mutual information, PMI, asks whether a word-context pair occurs more often than we would expect, if the two were independent:

$$ \mathrm{PMI}(w,c)=\log_2\frac{P(w,c)}{P(w)P(c)}. $$

PMI is conventionally written with a base-2 logarithm because of the connection to information theory (see also Jurafsky & Martin and the explorable below). But the base isn't really important. As is the case with IDF, the base only rescales the values: zero stays zero and the signs do not change.

If $w$ and $c$ occur together exactly as often as independence would predict, the ratio in the PMI equation is 1 and PMI is 0. A positive value means the pair occurs together more often than expected. A negative value means it occurs together less often than expected. If you start from a word-context count matrix, you can estimate these probabilities from the counts: the cell gives the joint count for $w$ and $c$, the row and column totals give the marginal counts, and the sum of the whole matrix gives the total number of counted word-context pairs.

Positive PMI, PPMI, keeps the positive values and replaces negative ones with zero:

$$ \mathrm{PPMI}(w,c)=\max(\mathrm{PMI}(w,c),0). $$

When we use PPMI, common but uninformative contexts lose influence, while unexpectedly strong associations stand out.

Select a word, then switch the matrix from counts to PPMI. Watch which context dimensions shrink, which grow, and whether the nearest neighbor changes.

Python example. Here is the explorable's matrix in NumPy. The three probabilities are one line each, exactly as described above:

import numpy as np

words = ["dog", "puppy", "cat", "bank", "finance", "democracy", "election"]
contexts = ["the", "bark", "leash", "pet", "vote", "money", "river"]
C = np.array([
    [30, 12,  9, 10,  0,  0,  0],
    [25,  8, 10, 12,  0,  0,  0],
    [28,  0,  1, 13,  0,  0,  0],
    [30,  0,  0,  0,  2, 15, 13],
    [22,  0,  0,  0,  3, 14,  0],
    [24,  0,  0,  0, 16,  2,  0],
    [20,  0,  0,  0, 15,  1,  0],
])

P = C / C.sum()                        # P(w, c): each cell over the grand total
Pw = P.sum(axis=1, keepdims=True)      # P(w): row totals
Pc = P.sum(axis=0, keepdims=True)      # P(c): column totals
with np.errstate(divide="ignore"):     # log2(0) = -inf for the empty cells
    pmi = np.log2(P / (Pw * Pc))
ppmi = np.maximum(pmi, 0)

def cos(M, a, b):
    va, vb = M[words.index(a)], M[words.index(b)]
    return va @ vb / (np.linalg.norm(va) * np.linalg.norm(vb))

print(round(cos(C, "dog", "finance"), 2), round(cos(ppmi, "dog", "finance"), 2))
# 0.72 0.0

In terms of raw counts, dog and finance have a cosine similarity of 0.72, almost entirely because both co-occur with the. But after PPMI, their cosine similarity is 0. Compare that with last week's one-hot vectors, where dog and puppy were at 0 too. In our new framework, dog and puppy stay close (0.96), and the meaningless similarity disappears. Pretty neat.

Folks talk about embeddings, but you don't need big machinery to get them. What we've just put together is a count-based word embedding. Each row is a vector whose geometry reflects context. The vectors are large and sparse, and we use cosine similarity to compare words (rather than documents).

6 · Learn dense word embeddings

OK. So I was just proudly exclaiming that we'd gotten to embeddings in a simple way. But now we're getting to the juicy part. We're going to learn about and play around with the word embedding that changed the world and in many ways kicked off the impressive new era of language technology that we're currently living through.

Count-based word vectors are high-dimensional and sparse because every possible context word is a dimension. Word2Vec takes a different route and learns much shorter, dense vectors without explicitly keeping that whole matrix.

The basic trick is to turn context into a prediction task. I'm not going to give you the whole explanation here, Jurafsky and Martin do it better. But I'll tell you the two key terms to keep an eye out for:

• Skip-gram gives the model a center word and train it to recognize words that occur nearby.
• CBOW runs the prediction in the other direction: the surrounding words are used to predict the center word.

As this happens again and again across a corpus, words that are useful in similar contexts tend to end up with similar vectors.

Negative sampling makes the training manageable. Instead of comparing an observed word-context pair with every word in a large vocabulary, the model compares it to a small number of sampled words that did not occur in that context. That is the intuition you need here; the reading below goes further into how the vectors are actually learned.

If you prefer to build intuition visually first (I do), the StatQuest video below is the right start. StatQuest is an excellent statistics and machine-learning resource, and this video gives a very concrete picture of what Word2Vec is trying to learn. Watch it, then read.

Watch for the training task rather than the finished vectors. Which word is given to the model, which word is predicted, and where do the learned numbers live?

The explorable below strips the training setup down to the examples the model sees. Move the center word and the context window, switch between Skip-gram and CBOW, and resample the negative words. The point is to understand how ordinary text is turned into training examples before worrying about the neural network that learns from them.

Move the context window and switch between Skip-gram and CBOW. Watch which training examples each setting creates.

GloVe is another static embedding method. It learns dense vectors from global word co-occurrence statistics rather than the local prediction task used by Word2Vec. For this course, the idea it shares with Word2Vec is most important: both methods place words in a learned vector space where distance and direction can carry regularities in how words are used.

It's important to realize that count-based and prediction-based embeddings are closer than they first appear. Skip-gram with negative sampling has a mathematical connection to a shifted PMI matrix. You do not need that derivation for Test 2 (the curious will find it in the Goodies). The broader point is enough: counting contexts and predicting contexts can recover related structure from the same corpus. We once again compare the dense vectors from Word2Vec and GloVe using cosine similarity.

A famous intuition-building example. Aligned vectors often correspond to related words, and some relations can even appear as directions in the space.

One famous example is

king - man + woman ≈ queen

In some trained static embedding spaces, the direction from man to king is similar to the direction from woman to queen. If we start at the vector for king, subtract the vector for man, and add the vector for woman, the resulting point can therefore land close to queen. The example is memorable because it turns an abstract vector space into something we can reason about geometrically. It does not mean that every analogy works, or that the model has learned a neat human definition of royalty or gender.

The explorable below draws the idea in two dimensions. A real embedding contains many more dimensions, so focus on the subtraction, addition and relative directions rather than the exact coordinates on the page.

Step through the subtraction and addition. Treat the two-dimensional drawing as a picture of the relationship, not as the real coordinates of a trained embedding.

Python example. gensim trains Word2Vec in one line once the corpus is a list of tokenized sentences. On the full Marvel pages it takes about ten seconds on a laptop.

import re
from gensim.models import Word2Vec
from gensim.utils import simple_preprocess

sentences = [
    simple_preprocess(s)                          # lowercase, strip punctuation, tokenize
    for page in marvel_pages.values()
    for s in re.split(r"(?<=[.!?])\s+", page)     # a crude sentence splitter
]

model = Word2Vec(
    sentences, vector_size=100, window=5, min_count=5,
    sg=1,                                         # 1 = Skip-gram, 0 = CBOW
    seed=0, workers=1,                            # one worker makes the run reproducible
)
model.wv.most_similar("hammer", topn=6)

We got mjolnir and odinson first, which is exactly right, and then truck, hide, torn and inevitably, which is not. (Your list may differ a little depending on your version of gensim.) That is what a corpus of 700,000 tokens gives you: the strongest associations come through, and the rest of the list is noise that looks just as confident. The cosine next to each neighbor will not warn you, which is why every neighbor list in this week's exercises ends with a trip back to the text.

Pretrained static embeddings are one download away, and trying the famous analogy yourself takes one line:

import gensim.downloader as api

glove = api.load("glove-wiki-gigaword-100")      # 128 MB, trained on Wikipedia + news
glove.most_similar(positive=["king", "woman"], negative=["man"], topn=3)
# [('queen', 0.770), ('monarch', 0.684), ('throne', 0.676)]

Now ask GloVe about hammer and you get stick, bolt, nail, thrower and discus: tools and athletics, not Thor. Neither answer is wrong. Each vector space describes the language it was trained on.

Documentation: gensim's Word2Vec tutorial.

A map of the Marvel universe. Word vectors can also place whole characters in a space we can read. Take each character's fingerprint from Section 1 (its most distinctive words, names removed) and average the GloVe vectors of those words, weighted by TF-IDF. Then build a direction the way king - man does: the average vector of magic, spell, sorcerer and friends minus the average vector of science, technology, experiment and friends. Projecting every character onto that direction gives a scale from science to magic, and two such scales give a map.

Find Doctor Strange, Machine Man and Spider-Man, and read the words that pull each one. Switch the vertical axis to body ↔ mind and look for the telepaths. Then type your own axis: which ones give a map you can read, and which only give noise? Finally, color the points by network community: do the communities from the first half of the course also live in different parts of this map?

None of these positions is a fact about the characters. Each axis is a direction somebody chose, measured with vectors trained on Wikipedia and news, applied to how Wikipedia happens to describe each character. The recipe comes from sociology: Kozlowski, Taddy and Evans used it to measure how Americans understand social class in The Geometry of Culture (2019).

Bias is always there. Static embeddings also learn associations we may not want. If occupations or social groups occur in stereotyped contexts, the vectors can encode those regularities too. The map above shows it: color it by gender and switch to the war ↔ home axis, and the home end is almost all women, pulled there mostly by the words her and she. GloVe has learned that female pronouns sit closer to family and love than to soldier and mission. A single static vector also has trouble with polysemy. bank gets one location whether the sentence is about money or a river.

7 · Next week: from meaning to measurement

This week started with documents and ended with words. In both cases we turned patterns in text into vectors and used geometry to compare them, and in both cases the vectors were blind to one thing: context inside the sentence. Every bank got the same vector, river or money.

Next week the vectors become contextual, and whole texts get a single embedding that captures more than their word counts. Then we put the representations to work as measurement tools: walking through a corpus by meaning, measuring sentiment, turning names in prose into structured data, and searching by meaning rather than by keyword. And because a measurement is only as good as its validation, we will spend real time on how bias can enter before the model ever runs.

Goodies

Everything above is the course. Everything below is for the student who finished the page and wants more. Nothing here is on the test; all of it is worth your time.

← All weeks