Word Vector Clustering

Vector clustering, with words

Same pipeline, real NLP: each word is embedded by GloVe (50 dimensions, learned from Wikipedia co-occurrence). Words are 50-D, so the cube below is a PCA shadow — the dimensionality reduction the color demo didn't need.

1 Corpus

2 Embed

Embedding model: GloVe 50d — learned from which words appear near which. Loading vectors (2 MB)…

Looks each word up in the learned space, then projects 50-D → 3-D for display.

Learn with it — five experiments

See learned similarity. Embed, then find dog and cat sitting together — no one told GloVe they're both pets; it learned it from the company words keep in text.
Cluster the corpus. Run k-means or DBSCAN: animals, foods, countries and jobs separate — the clusters existed in the geometry before any algorithm ran, exactly like the colors.
Distrust the shadow. The panel's similarity numbers use all 50 dimensions, but the picture is a 3-D projection — check the explained-variance figure above the cube. Two dots can look close while their true similarity is mediocre; click them and see.
Mix words. Select two words: there's no way to "un-embed" their midpoint back into a word (unlike colors!), so the panel shows the nearest real words to the midpoint instead — which is literally how vector search works.
Probe the model's world-view. Add words like chicken (animal or food?), bank, or apple and see which cluster claims them. A single static vector per word has to average all its meanings — the limitation that pushed NLP toward contextual embeddings.

Teaching notes

The colors are coordinates. Words have no natural color, so each dot is painted with its own position — (PC1, PC2, PC3) read as (R, G, B). That's Part 1's trick run in reverse: there, color determined position; here, position determines color. It adds no new information (color and position are the same three numbers), but it makes neighborhoods pop — a semantic cluster shows up as a color family before any algorithm runs. And note what it implies: the color belongs to the projection, not the word — re-fit the PCA on a different corpus and every word would change color.
These vectors were learned. Colors got their coordinates for free (the data was already numeric). Words as strings share almost nothing — dog and puppy have no letters in common — so GloVe had to be trained to place them nearby, using millions of sentences of co-occurrence statistics.
The cube is a shadow. PCA keeps the 3 directions with the most variance and throws away the other 47. All similarity numbers in the panel are computed in the full 50-D space; the layout is only a viewing aid — the honest version of what every embedding visualization does.
Clustering here runs on the 3-D shadow so what you see is exactly what the algorithm saw. Real systems cluster in the full space — and their results can differ from what any 3-D picture suggests.
No inverse exists. hex → rgb could be run backwards; word → vector cannot. Between vectors there are no words, so any decoded result must snap to the nearest real vocabulary entry — the fundamental move behind semantic search.

3 Cluster

Similarity computed in full 50-D

Merge tree height = merge distance; dashed line = current cut