Introduction

Welcome back, fellow explorers of the digital word jungle! If you thought our last adventure through the wilds of text vectorization was a blast, get ready for the sequel that promises even more excitement. In this part, we’re diving deeper into the art of transforming words into numbers—because let’s face it, as much as we love language, machines prefer their snacks in numeric form. We’ll uncover the magic of advanced embeddings like GloVe and FastText, which are as revolutionary as discovering your favorite pizza topping on a Monday. So, buckle up and prepare to level up your understanding of text representation!

4. Word Embeddings: Capturing Meaning and Relationships

Unlike the previous methods, word embeddings are dense vectors that can capture semantic relationships between words. While techniques like Bag of Words (BoW) and TF-IDF treat words as independent entities, word embeddings recognize that words with similar meanings will have similar vectors. This is where machine learning models come into play.

Word2Vec Visualized in Matrix Form:

To grasp the concept of word embeddings, let’s delve into Word2Vec, a powerful technique that transforms words into numerical vectors. We’ll illustrate this process using a simple vocabulary.

Vocabulary and One-Hot Encoding

Consider the following vocabulary:

Vocabulary={“the”,”cat”,”sat”,”on”,”mat”}

Each word can be represented using one-hot encoding:

• “the” → [1,0,0,0,0]
• “cat” → [0,1,0,0,0]
• “sat” → [0,0,1,0,0]
• “on” → [0,0,0,1,0]
• “mat” → [0,0,0,0,1]

One-Hot Encoding Matrix

This can be organized into a One-Hot Encoding Matrix as follows:

Feeding Data into the Neural Network

Once we have the one-hot encoded vectors, we feed them into a neural network structured as follows:

• Input Layer: Accepts the one-hot encoded vector of the target or context words.
• Hidden Layer: A dense layer that processes the input through randomly initialized weights.
• Output Layer: Produces a probability distribution over the vocabulary for context words (Skip-Gram) or the target word (CBOW).

Neural Network Training

During training, the network adjusts its weights based on the loss derived from the predicted probabilities compared to the actual context words. The commonly used loss function here is cross-entropy loss.

Key Steps in Training:

• Forward Propagation: The input one-hot vector is multiplied by the weights of the hidden layer, yielding hidden layer activations.
• Softmax Activation: The output layer applies the softmax function to produce probabilities for each word in the vocabulary.
• Backpropagation: The weights are updated based on the gradient of the loss with respect to each weight.

Output Word Vectors

After training, we extract the adjusted weights from the hidden layer, which become the word embeddings. The final vector representation for each word is obtained by multiplying the one-hot vector of the word by the hidden weights matrix:

Example of Final Word Vector Calculation

Let’s calculate the final word vector for “sat,” represented by the one-hot vector [0,0,1,0,0]. Assume the hidden weights matrix is:

Calculating the final word vector:

Advantages of Word2Vec:

• Captures Word Magic: Word2Vec’s ability to figure out that “king” minus “man” plus “woman” equals “queen” is like solving a word puzzle. It captures semantic relationships in ways that feel almost magical!
• Efficient and Scalable:Need to train on an entire library of digital books? Word2Vec is here for it. It’s built to handle large datasets quickly and doesn’t break a sweat when you throw a massive corpus at it.
• Learns from Context:Word2Vec has a knack for learning from the words around it. It captures local word context like an eavesdropper at a coffee shop, figuring out relationships between words without ever asking for permission.

Limitations of Word2Vec:

• One Meaning to Rule Them All:Much like how my GPS sometimes leads me astray, Word2Vec doesn’t always get context right. Whether it’s “bank” by the river or your bank account, Word2Vec gives the same vector for both. Not ideal if you’re looking for nuanced meanings.
• OOV (Out of Vocabulary) Blues:Got a word that didn’t show up during training? Sorry, Word2Vec has no idea what that is. It’s like when someone starts talking about the latest TikTok trend, and you’re completely out of the loop.
• Can’t Handle Morphology:Word2Vec treats “running” and “run” like two completely different creatures, so it sometimes struggles with words that share roots or forms. It’s like forgetting that “cat” and “cats” are practically the same thing.

GloVe (Global Vectors for Word Representation)

Overview:

GloVe, developed by Stanford, is a method that combines two powerful ideas: global matrix factorization and local context windows. Unlike Word2Vec, which focuses solely on local context by training on sliding windows of words, GloVe leverages the global co-occurrence statistics of the entire corpus. The basic idea is to create a co-occurrence matrix that counts how often pairs of words appear together within a certain context window across a large text corpus.

Key Steps in GloVe:

• Building the Co-occurrence Matrix:

  GloVe first constructs a co-occurrence matrix X, where each entry Xij represents how often word i appears in the context of word j. This matrix captures global statistics, meaning it records how often words co-occur across the entire corpus, not just in the immediate context.

  Example: Vocabulary and Co-occurrence Matrix

  Consider a simple vocabulary from a corpus consisting of the following sentences:
  1. “The cat sat on the mat.”
  2. “The cat sat.”
  3. “The mat sat on the floor.”
  4. “The cat lay on the mat.”

  The corresponding vocabulary would be:

  Vocabulary = {“the”, “cat”, “sat”, “on”, “mat”, “floor”}

  Based on the corpus, we create the co-occurrence matrix by counting how often each word appears in the context of another word within the sentences. The matrix might look something like this:(window size = 1)

  

  This matrix shows how many times each word in the vocabulary appears in the context of another word. For example, “the” appears four times near “cat” and three times near “sat” across the sentences.

• Co-Occurrence Matrix Factorization:

  GloVe factorizes the co-occurrence matrix into two lower-dimensional matrices, where each word is represented as a dense vector. This factorization process is where the model learns the word embeddings by capturing the relationships between words.

• Weighted Least Squares Objective:

  Once the co-occurrence matrix is created, GloVe defines a cost function that models the logarithm of word co-occurrences, aiming to learn word embeddings that minimize this objective. The idea is that the dot product of the word vectors wi and wj should approximate the log of their co-occurrence:

  

  where wi and wj are the word vectors, and bi and bj are biases.

• Training Process:

  The model is trained by minimizing the squared difference between the predicted and actual co-occurrences, weighted by frequency. This helps the model focus on more frequent word pairs, which tend to carry more meaningful relationships.

• Final Word Vectors in GloVe:

  After training, each word is represented by a dense vector derived from the factorization of the co-occurrence matrix. These word vectors capture semantic relationships between words, much like Word2Vec.

Example:

Final Word Vector for “cat” After factorizing the co-occurrence matrix, we might get the following word vector for “cat”:

Final Word Vector(“cat”)=[0.45,0.12,0.78]

This vector captures the position of “cat” in relation to other words in the vocabulary.

Advantages of GloVe:

• Global ContextUnlike Word2Vec, which focuses on local context, GloVe captures global word relationships through the co-occurrence matrix, giving a broader view of word associations.
• Advantages of GloVe:GloVe benefits from efficient matrix factorization techniques, making it scalable for large corpora.

Limitations of GloVe:

• Lack of Contextual SensitivityLike Word2Vec, GloVe produces static embeddings, meaning that words like “bank” will have the same vector representation in “river bank” and “bank account,” failing to capture multiple meanings of words.

FastText (Developed by Facebook AI Research)

Overview:

FastText, created by Facebook AI Research, is a word embedding method that builds on the strengths of Word2Vec but takes things one step further. One of the key limitations of models like Word2Vec and GloVe is that they treat each word as a single, indivisible unit. That’s fine most of the time, but what about words that are rare, or even worse, words the model hasn’t seen before (out-of-vocabulary or OOV words)? FastText addresses this by breaking words down into smaller parts, called character n-grams. This approach makes it much better at handling rare words and even words that weren’t in the training data.

FastText is especially handy in morphologically rich languages where word forms can change drastically with prefixes, suffixes, etc. This model lets us represent words like “run,” “runner,” and “running” in a way that preserves their similarities because it looks at the pieces that make up the word.

Key Steps in FastText:

• Breaking Words Into Subwords (Character N-Grams):FastText doesn’t just look at entire words; it breaks them into smaller subword units (character n-grams). So, for example, the word “cat” might be split into the following 3-grams: ““. This gives FastText a better understanding of the word structure and makes it much more flexible.
• Building Word Vectors from N-Grams:Instead of representing a word with a single vector like in Word2Vec or GloVe, FastText represents a word by adding up the vectors of its character n-grams. For the word “cat,” the final vector would be a combination of the vectors for ““.
• Training the Model (Skip-Gram or CBOW):Like Word2Vec, FastText can be trained using two main approaches: Skip-Gram (predicting context words given a target word) or Continuous Bag of Words (CBOW, predicting the target word from context). But instead of just learning word embeddings, FastText also learns embeddings for each n-gram. This allows the model to handle rare words and out-of-vocabulary words better.

Example: How FastText Works with Subwords

To make this clearer, let’s consider a small vocabulary:
Vocabulary = {“cat,” “mat,” “cats,” “mats”}

For the word “cat,” FastText will break it into the following 3-grams:

For a word like “mats,” FastText generates the 3-grams:

What happens next is that FastText builds the word vector for “cat” by adding up the vectors for its subwords: “<ca,” “cat,” and “at>.” Similarly, for “mats,” the final vector would come from “<ma,” “mat,” “ats,” and “ts>.”

This way, even if a word like “cats” was rare or not seen during training, FastText can still create a meaningful vector for it by breaking it into n-grams and combining their vectors.

Training Process:

FastText trains using the same ideas as Word2Vec. The model tries to predict the context words given a target word (Skip-Gram) or predict the target word from context (CBOW). But here’s the twist: it doesn’t just learn word embeddings, it also learns n-gram embeddings, making the model more flexible.

Example: Subword Representation of “cats”

Let’s say after training, we have the following vectors for each n-gram in “cat”:

• “<ca”: [0.25, 0.13, 0.70]
• “cat”: [0.40, 0.12, 0.65]
• “at>”: [0.18, 0.22, 0.45]

The final word vector for “cat” would be the sum of these:

Final Word Vector(“cat”)=[0.25+0.40+0.18,0.13+0.12+0.22,0.70+0.65+0.45]=[0.83,0.47,1.80]

If we had “cats,” we would also include vectors for “ats” and “ts,” giving it a slightly different representation than just “cat.”

Why FastText Stands Out:

• Handles Unseen WordsOne of the coolest things about FastText is how it handles words it hasn’t seen before. Even if a word like “mats” wasn’t in the training data, FastText can still generate a vector for it using the character n-grams.
• Understands Word Morphology:Since it breaks down words into smaller parts, FastText gets a much better understanding of things like prefixes, suffixes, and the structure of words. This makes it great for languages where words change form a lot.
• Efficient Training:Despite dealing with more complex subword information, FastText remains pretty fast and efficient when training on large datasets.

FastText’s Limitations:

• Static EmbeddingsJust like Word2Vec and GloVe, FastText produces static embeddings. So even though it handles rare and unseen words well, it doesn’t capture different meanings of the same word in different contexts. For example, “bank” will have the same representation whether we’re talking about a riverbank or a financial institution.
• Memory Usage:Since FastText stores embeddings for both words and n-grams, it can use up more memory compared to simpler models.

5. Advanced Contextualized Word Embeddings

Recent advancements in NLP have introduced contextualized word embeddings, which take the context of a word into account. This means that the same word can have different meanings depending on its surrounding words.

Examples:

• ELMo (Embeddings from Language Models)ELMo generates word embeddings based on the entire sentence, allowing it to capture context. For instance, the word “bank” in “river bank” versus “bank account” will have different representations.
• BERT (Bidirectional Encoder Representations from Transformers):BERT takes this further by looking at both the left and right context of a word simultaneously. It understands that “the bank was crowded” has a different meaning than “he deposited money at the bank.”

Pros and Cons:

• Pros:These models provide a rich understanding of word meanings based on context, significantly improving performance in various NLP tasks. It’s like having a chat with a friend who knows exactly what you mean, even when you use vague references!
• Cons:They require substantial computational resources and data to train, making them more complex to implement. It’s like bringing a whole film crew for your birthday party—it’s fabulous but requires a lot of planning!

In summary, contextualized word embeddings are like a new level of empathy in conversations. They allow the model to respond appropriately based on context—think of it as a friend who remembers your inside jokes and doesn’t mix up your favourite films. Imagine a world where every text you send gets interpreted perfectly, leaving no room for misunderstandings!

Summary Table of Text Vectorization Techniques

Conclusion: The Wordy Wonders of Vectorization

As we wrap up this deep dive into text vectorization, we’ve uncovered the secrets of transforming words into numerical magic. From the clever simplicity of Word2Vec, which taught us that similar words can vibe together in the numerical world, to the global gaze of GloVe, capturing relationships across vast corpora, and the adaptable brilliance of FastText, which takes a scalpel to words and lets us peek beneath the surface.

It’s been quite the journey, akin to finding the perfect slice of pizza (with just the right amount of toppings, of course)! Now you’re equipped with the knowledge to navigate this landscape, and who knows? Maybe your next project will be the one to harness these wordy wonders into a groundbreaking application.

So, as you venture forth, remember: words might be the heart of communication, but with the right tools, you can give them a numerical voice that resonates in the machine’s world. Stay tuned for more exciting explorations in the realm of AI and text processing!