Vector Space Models

Introduction

In the realm of natural language processing (NLP) and machine learning, the concept of Vector Space Models (VSMs) has been pivotal. This guide aims to unravel the complexities of VSMs, delving into their core principles and applications.

What are Vector Space Models?

At its heart, a Vector Space Model is a mathematical model that represents text as vectors of identifiers. Imagine each word or document as a point in a multi-dimensional space. This approach transforms linguistic information into a form that computers can process, leading to groundbreaking advancements in information retrieval and NLP.

Euclidean Distance: Measuring Textual Proximity

One fundamental concept in VSMs is the Euclidean distance. It’s the “straight line” distance between two points in vector space. Let us assume that you want to compute the distance between two points: $A, B$ . To do so, you can use the euclidean distance defined as

Here’s a Python function to calculate the Euclidean distance between two points in a vector space:

import numpy as np

def euclidean_distance(vec1, vec2):
    return np.sqrt(np.sum((vec1 - vec2) ** 2))

# Example Usage
vec1 = np.array([1, 2, 3])
vec2 = np.array([4, 5, 6])
distance = euclidean_distance(vec1, vec2)
print("Euclidean Distance:", distance)

Cosine Similarity: Beyond Distance

While Euclidean distance is useful, it’s not always sufficient, especially in high-dimensional spaces. Enter Cosine Similarity – a measure that calculates the cosine of the angle between two vectors. This approach focuses on the orientation rather than the magnitude of vectors, making it more robust for text comparison.

This snippet shows how to calculate cosine similarity between two vectors:

from numpy.linalg import norm

def cosine_similarity(vec1, vec2):
    return np.dot(vec1, vec2) / (norm(vec1) * norm(vec2))

# Example Usage
vec1 = np.array([1, 2, 3])
vec2 = np.array([4, 5, 6])
similarity = cosine_similarity(vec1, vec2)
print("Cosine Similarity:", similarity)

Why Cosine Similarity?

Cosine similarity is preferred in text analysis because it is less affected by the length of the documents. Longer documents may appear more dissimilar when using Euclidean distance, even if they share the same topics. Cosine similarity overcomes this by normalizing the dot product of the vectors to the length of the vectors.

Manipulating Words in Vector Spaces

In vector spaces, words can be manipulated mathematically. For instance, consider word embeddings like Word2Vec. These models map words into a high-dimensional space where semantically similar words are closer together.

An Example

Imagine vector representations for “King,” “Queen,” “Man,” and “Woman.” In a well-structured vector space, if you take the vector for “King,” subtract “Man,” and add “Woman,” the resulting vector would be closest to the vector for “Queen.”

For this example, you need pre-trained word vectors like Word2Vec or GloVe. Here’s a conceptual example using Word2Vec:

import gensim.downloader as api

# Load pre-trained Word2Vec model
model = api.load('word2vec-google-news-300')

# Example: King - Man + Woman ≈ Queen
king = model['king']
man = model['man']
woman = model['woman']
queen = model['queen']

# Perform vector arithmetic
result_vector = king - man + woman
similar_words = model.similar_by_vector(result_vector)

print("Words similar to 'king - man + woman':", similar_words[0])

Visualization and PCA: Making Sense of Complexity

High-dimensional vector spaces are hard to visualize. This is where techniques like Principal Component Analysis (PCA) come into play.

What is PCA?

PCA is a statistical technique that reduces the dimensionality of data while retaining most of the variation in the dataset. It does this by finding new axes (principal components) along which the data varies the most.

PCA in Action

For example, if you have a 100-dimensional word vector space, PCA can help reduce it to a 2D or 3D space. This reduced representation can then be plotted, providing visual insights into the relationships and clusters among words.

Understanding the PCA Algorithm

The PCA algorithm involves the following steps:

Standardization: The data is standardized to have a mean of 0 and a variance of 1.
Covariance Matrix Computation: This matrix captures the covariance between every pair of dimensions.
Eigenvalue Decomposition: The covariance matrix is decomposed into eigenvalues and eigenvectors.
Selecting Principal Components: Eigenvectors with the highest eigenvalues are selected as the principal components.
Transforming Data: The original data is transformed into this new subspace.

Here’s how to use PCA for reducing the dimensionality of word vectors and then visualizing them:

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

# Example word vectors (usually obtained from a model)
word_vectors = np.array([model[w] for w in ['king', 'queen', 'man', 'woman']])

# Applying PCA
pca = PCA(n_components=2)
result = pca.fit_transform(word_vectors)

# Plotting
plt.scatter(result[:, 0], result[:, 1])
for i, word in enumerate(['king', 'queen', 'man', 'woman']):
    plt.annotate(word, xy=(result[i, 0], result[i, 1]))
plt.show()

PCA implementation from scratch

import numpy as np
import matplotlib.pyplot as plt

# PCA implementation from scratch
def pca(X, num_components):
    # Centering the data
    X_meaned = X - np.mean(X, axis=0)

    # Calculating the covariance matrix
    cov_mat = np.cov(X_meaned, rowvar=False)

    # Calculating Eigenvalues and Eigenvectors of the covariance matrix
    eigen_values, eigen_vectors = np.linalg.eigh(cov_mat)

    # Sorting the eigenvalues and eigenvectors
    sorted_index = np.argsort(eigen_values)[::-1]
    sorted_eigenvalue = eigen_values[sorted_index]
    sorted_eigenvectors = eigen_vectors[:, sorted_index]

    # Selecting a subset from the rearranged Eigenvalue matrix
    eigenvector_subset = sorted_eigenvectors[:, 0:num_components]

    # Transforming the data
    X_reduced = np.dot(eigenvector_subset.transpose(), X_meaned.transpose()).transpose()

    return X_reduced

# Example data for PCA
np.random.seed(0) # Seed for reproducibility
X_example = np.random.rand(10, 3) # Random data (10 samples, 3 features)

# Example usage of the functions
vec1 = np.array([1, 2, 3])
vec2 = np.array([4, 5, 6])

# Apply PCA
pca_result = pca(X_example, 2)

# Print the results
pca_result

Conclusion

Vector Space Models are a cornerstone of modern NLP and machine learning. By understanding Euclidean distance, cosine similarity, word manipulation, and PCA, we unlock powerful tools for text analysis and representation. These concepts not only enhance our computational prowess but also provide deeper insights into the complex world of language and information.

Remember to install the necessary Python packages (numpy, gensim, matplotlib, and scikit-learn) to run these snippets. Also, for the Word2Vec example, you need an internet connection to download the pre-trained model. These examples are basic and meant for educational purposes. In a real-world application, you might need more complex code and error handling.

!pip install numpy
!pip install gensim
!pip install matplotlib
!pip install scikit-learn

Natural Language Processing