From raw documents to a Bag-of-Words matrix
The general process
One course example
Corpus
Start with a collection of documents.
Corpus
D1: brains predict the future
D2: models predict the future
D3: brains build models
↓
↓
Tokenize
Split each document into the units you want to count.
Tokenized documents
D1: [brains] [predict] [the] [future]
D2: [models] [predict] [the] [future]
D3: [brains] [build] [models]
↓
↓
Build the vocabulary
Collect the distinct terms that define the matrix columns.
Vocabulary
brains · build · future · models · predict · the
↓
↓
Count
Count every vocabulary term inside every document.
Counts inside each document
D1: brains 1 · future 1 · predict 1 · the 1
D2: future 1 · models 1 · predict 1 · the 1
D3: brains 1 · build 1 · models 1
↓
↓
Encode
Write the counts in the same vocabulary order for every document.
Document vectors
D1: [1, 0, 1, 0, 1, 1]
D2: [0, 0, 1, 1, 1, 1]
D3: [1, 1, 0, 1, 0, 0]
↓
↓
Document-term matrix
Stack the document vectors as rows in one numerical matrix.
Document-term matrix
doc
brains
build
future
models
predict
the
D1
1
0
1
0
1
1
D2
0
0
1
1
1
1
D3
1
1
0
1
0
0