Word Embeddings

Dense vectors that capture semantic meaning and relationships. Words with similar meanings have similar vector representations, allowing machines to perform "math" on language.

High-Fidelity Vector Space

Drag to rotate • Scroll to zoom

The "Real World" Context: High-Dimensionality

Real embeddings (like Word2Vec or BERT) have 300 to 1024 dimensions. This 3D view is a mathematically reduced projection (like PCA) for human visualization. Real conceptual clusters exist amidst thousands of other seemingly random words.

Select Relationship

Core Concepts

Advantages

  • Dimensionality Reduction: Compresses a vocabulary of 100,000+ sparse words into dense 300-dimensional continuous vectors.
  • Semantic Capture: Automatically learns synonyms, antonyms, and relationships through distributional semantics (words used in similar contexts have similar vectors).
  • Context-Awareness (Modern): Transformer models like BERT create dynamic embeddings, recognizing polysemy (e.g., 'bank' as a river vs. financial institution).

Limitations

  • Static Context (Traditional): Models like Word2Vec map one strict vector per word, entirely missing nuanced context differences.
  • Corpus Bias: Vector relationships blindly reflect the societal and historical biases present in the training data text.