15 mins read
## Word vectors, what’s the big idea? It’s all about that context

## Impact and implications of Word2vec

## From raw text to word vectors: high-level approach

# Creating structured data from raw text

## Defining the embedding layer and the neural network

## Embedding layer: stores all the word vectors

## Neural network: maps word vectors to outputs

## Fitting all together: inputs to model to outputs

## Defining a loss function: to optimize the model

# TensorFlow implementation: Skip-gram algorithm

### Data generator

### Defining the skip-gram model

### Running the code

## Conclusion

This article is going to be about Word2vec algorithms. Word2vec algorithms output word vectors. Word vectors, underpin many of the natural language processing (NLP) systems, that have taken the world by a storm (Amazon Alexa, Google translate, etc.). We will talk about the details in the upcoming sections.

Without further due, let us stick our feet in. Word vectors are numerical representations of words, that preserve the semantic relationship between words. For example, the vector of the word ** cat** will be very similar to the vector of the word

I don’t think I need to spell out the odd one in the above sentences, and obviously the ones with ** pencil**, as the missing word. Why do you feel it as the odd sentence? The spelling is fine, the grammar is right, then why? It is because the

To get an idea about the implication of the Word2vec techniques, try the following. Go ahead and bring up google scholar. Type in some NLP-related tasks (e.g. question answering, chatbots, machine translation, etc). Filter the papers published after 2013 (that’s when Word2vec methods came out). Get the proportion of the papers using word vectors to the total number of papers. I bet this proportion is going to be quite high. To make the statement concrete, word vectors are used for:

- Language modeling
- Chatbots
- Machine translation
- Question answering
- … and many more

You can see all the exciting NLP frontiers actually depend on word vectors heavily. Now let us discuss what sort of implications word vectors have to make the model better. When using word vectors, semantically close words will appear to be similar computations within the model, whereas other words will appear to do different-looking computations. This is a desirable property to have, as encoding such information in the input itself, leads to the model performing well, with lesser data.

Now, with a solid intuition in the bag, we’re going first to discuss the high-level mechanics of the Word2vec algorithms. We’ll polish the details up in later sections until we can be sure that we know how to implement a Word2vec algorithm. In order to learn word vectors in an unsupervised manner (i.e. without human labeling data), we have to define and complete certain tasks. Here’s a high-level list of these tasks.

- Create data tuples of the format [input word, output word], where each word is represented as one-hot vectors, from raw text
- Define a model that can take in the one-hot vectors as inputs and outputs, to be trained
- Define a loss function that predicts the correct word, which is actually in the context of the input word, to optimize the model
- Evaluate the model by making sure similar words have similar word vectors

This seemingly simple procedure will lead to learning very powerful word vectors. Let us move on to the nitty-gritty of each step of the above pipeline.

This is not a very difficult task. This is a simple manipulation of raw text to put that into a certain structure. Think of the following sentence.

*The cat pushed the glass off the table*

The data created from this sentence would look as follows. Each row after the sentence represents a single data point. The blue box represents the one-hot input word (** the middle word, called the target word**), and the red box represents the one-hot output word (

The neural network is used to learn from the structured data defined above. However, it comes with a twist! To make it clear, you have the following components.

- A batch of inputs represented as one-hot vectors
- A batch of outputs represented as one-hot vectors (only at the training phase)
- An embedding layer
- A neural network

No need to be scared if you did not get a feeling of what and how the last two components perform. We will probe each of these components to understand what they do.

First on our agenda is the ** embedding layer**. The embedding layer stores the word vectors of all the words found in the vocabulary. As you can imagine this is an enormous matrix (of size

Next in line is the last LEGO block of our model; the neural network. During the training, the neural network takes an input word and attempts to predict the output word. Then using a loss function, we penalize the model for incorrect classifications and reward the model for correct classifications. We will be limiting our conversation to processing a single input and a single output at a time. However, in reality, you process data in batches (say, 64 data points). Let’s delineate the exact process used during training:

- For a given input word (the target word), find the corresponding word vector from the embedding layer
- Feed the word vector to the neural network, then try to predict the correct output word (a context word)
- By comparing the prediction and true context word, compute the loss
- Use the loss along with a stochastic optimizer to optimize the neural network and the embedding layer

One thing to note is, that when computing the prediction, we use a softmax activation to normalize the predictions to a valid probability distribution.

Knowing all the nuts and bolts of the Word2vec algorithm, we can put all the parts together. So that once this model is trained, all we have to do is ** save the embedding layer to the disk**. Then we can enjoy semantic preserved word vectors at any time of the day. Below we see what the full picture looks like.

This particular arrangement of data and the model layout is known as the skip-gram algorithm; a Word2vec algorithm. And this is what we’re going to focus on. The other algorithm is called a continuous bag-of-words (CBOW) model.

One crucial bit of information we haven’t discussed so far but is essential is the loss function. Normally, standard softmax cross-entropy loss is a good loss function for a classification task. Using this loss is not very practical for a Word2vec model, as for a simpler task like sentiment analysis (where you have 2 possible outputs: positive or negative). Here things can get funky. In a real-world task, that consumes billions of words, the vocabulary size can grow up to 100,000 or beyond easily. This makes the computation of the softmax normalization heavy. This is because the full computation of softmax requires calculating cross-entropy loss with respect to all the output nodes.

So we are going with a smarter alternative, called ** sampled softmax loss**. In sampled softmax loss, you do the following. Note that there are quite a lot of changes from the standard softmax cross-entropy loss. First, you compute the cross-entropy loss between the true context word ID for a given target word and the prediction value corresponding to the true context word ID. Then to that, we add the cross-entropy loss of

`K`

negative samples we sampled according to some noise distribution. On a high level, we define the loss as follows:The `SigmoidCrossEntropy`

is a loss we can define on a single output node, independent of the rest of the nodes. This makes it ideal for our problem, as our vocabulary can grow quite large. I’m not going to dive into the very details of this loss. You don’t need to understand how exactly this is implemented, as these are available as a built-in function in TensorFlow. But understanding the parameters involved in the loss (e.g. K) is important. The takeaway message is that the sampled softmax loss computes the loss by considering two types of entities:

- The index is given by the true context word ID in the prediction vector (words within the context window)
`K`

indices that indicate word IDs, and are considered to be noise (words outside the context window)

I further visualize this by illustrating an example.

Here we will regurgitate what we just discussed into an implementation. This is available as an exercise **here**. In this section, we’re going to implement the following.

- A data generator
- The skip-gram model (with TensorFlow)
- Running the skip-gram algorithm

First, let us understand how to generate data. We are not going to go into details of this code, as we have already discussed the internal mechanics of the data generation. This is just converting that logic to an implementation.

```
def generate_batch(batch_size, window_size):
global data_index
# two numpy arras to hold target words (batch)
# and context words (labels)
batch = np.ndarray(shape=(batch_size), dtype=np.int32)
labels = np.ndarray(shape=(batch_size, 1), dtype=np.int32)
# span defines the total window size
span = 2 * window_size + 1
# The buffer holds the data contained within the span
queue = collections.deque(maxlen=span)
# Fill the buffer and update the data_index
for _ in range(span):
queue.append(data[data_index])
data_index = (data_index + 1) % len(data)
for i in range(batch_size // (2*window_size)):
k=0
# Avoid the target word itself as a prediction
for j in list(range(window_size))+list(range(window_size+1,2*window_size+1)):
batch[i * (2*window_size) + k] = queue[window_size]
labels[i * (2*window_size) + k, 0] = queue[j]
k += 1
# Everytime we read num_samples data points, update the queue
queue.append(data[data_index])
# If end is reached, circle back to the beginning
data_index = (data_index + np.random.randint(window_size)) % len(data)
return batch, labels
```

First, we will define some hyperparameters required for the model.

```
batch_size = 128
embedding_size = 64
window_size = 4
num_sampled = 32 # Number of negative examples to sample.
```

The `batch_size`

defines the number of data points we process at a given time. Then the `embedding_size`

is the size of a word vector. The next hyperparameter `window_size`

defines the size of the context window we visualized above. Finally `num_sampled`

defines the number of negative samples in the loss function (`K`

). Then we define TensorFlow placeholders for inputs and outputs.

```
tf.reset_default_graph()
# Training input data (target word IDs).
train_dataset = tf.placeholder(tf.int32, shape=[batch_size]) # Training input label data (context word IDs)
train_labels = tf.placeholder(tf.int32, shape=[batch_size, 1])
```

Here the `train_dataset`

takes a list of word IDs of `batch_size`

that represents a selected set of target words. Finally, `train_labels`

represents a `batch_size`

list of corresponding context words for the selected target words. Next, we define the model parameters required to define the model: the embedding layer and the weights and biases of the neural network.

```
################################################
# Model variables #
################################################
# Embedding layer
embeddings = tf.Variable(tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0))
# Neural network weights and biases
softmax_weights = tf.Variable(
tf.truncated_normal([vocabulary_size, embedding_size],
stddev=0.1 / math.sqrt(embedding_size))
)
softmax_biases = tf.Variable(tf.random_uniform([vocabulary_size], -0.01, 0.01))
```

We have defined the embedding layer as a TensorFlow variable: `embeddings`

. Then we define the neural network weights (`softmax_weights`

) and biases (`softmax_biases`

). Thereafter we define a key operation required to connect the embedding layer to the neural network to jointly optimize the embedding layer and the neural network.

```
# Look up embeddings for a batch of inputs.
embed = tf.nn.embedding_lookup(embeddings, train_dataset)
```

The `tf.nn.embedding_lookup`

function takes our embedding layer as the input and a set of word IDs (`train_dataset`

) and outputs the corresponding word vectors to the variable `embed`

. With the embedding lookup function defined, we can define the sampled softmax loss function we discussed above.

```
################################################
# Computes loss #
################################################
loss = tf.reduce_mean(tf.nn.sampled_softmax_loss(weights=softmax_weights, biases=softmax_biases, inputs=embed, labels=train_labels, num_sampled=num_sampled, num_classes=vocabulary_size) )
```

Here the `tf.nn.sampled_softmax_loss`

function takes in a set of weights (`softmax_weights`

), biases (`softmax_biases`

), a set of word vectors corresponding to the word IDs found in `train_dataset`

, IDs of the correct context words (`train_labels`

), number of noise samples (`num_sampled`

) and the size of the vocabulary (`vocabulary_size`

). With the output calculation operations and the loss defined, we can define an optimizer to optimize the loss with respect to the parameters of the embeddings layer and the neural network.

```
################################################
# Optimization #
################################################
optimizer = tf.train.AdamOptimizer(0.001).minimize(loss)
```

Then we get the normalized embedding layer by making the vector magnitude equal to 1.

```
########## For evaluation ##########
norm = tf.sqrt(tf.reduce_sum(tf.square(embeddings), 1, keepdims=True))
normalized_embeddings = embeddings / norm
```

Here we are going to discuss the details on how to run the previously defined TensorFlow model. First, we define a `session`

and then initialize all the TensorFlow variables randomly.

```
num_steps = 250001
session = tf.InteractiveSession()
# Initialize the variables in the graph
tf.global_variables_initializer().run()
print('Initialized')
average_loss = 0
```

Now for a pre-defined number of steps, we generate batches of data: target words (`batch_data`

) and context words (`batch_labels`

).

```
for step in range(num_steps):
# Generate a single batch of data
batch_data, batch_labels = generate_batch(batch_size, window_size)
```

Then for each generated batch, we optimize the embedding layer and the neural network by running `session.run([optimize, loss],...)`

. We also get the resulting loss out, to make sure it is decreasing over time.

```
# Optimize the embedding layer and neural network
# compute loss
feed_dict = {train_dataset:batch_data, train_labels:batch_labels}
_, l = session.run([optimizer, loss], feed_dict=feed_dict)
```

Here, every 5000 steps, we print the average loss, as a visual aid.

```
if (step+1) % 5000 == 0:
if step > 0:
average_loss = average_loss / 5000
print('Average loss at step %d: %f' % (step+1, average_loss))
average_loss = 0
```

Finally, we get the final embeddings out, which we later use for visualization of certain words.

```
sg_embeddings = normalized_embeddings.eval()
session.close()
```

Finally, if you visualize the embeddings using a manifold learning algorithm like t-SNE, you will get the following.

As you can see the words related to cat are found in a certain direction, and words related to the dog are found in a different direction. And words that fall between (e.g. animal or pet) fall between these two directions, which is pretty much what we needed.

This brings us to the end of our conversation. Word vectors are a very powerful representation of words that helps the machine learning models to perform better. We went through the data generation process as well as different components found in a Word2vec model. Then we discussed one specific variant of a Word2vec algorithm; a skip-gram model. We went through the implementation of the algorithm in TensorFlow. Finally, we visualized the embeddings and saw that the learned embeddings actually depict some useful semantics. You can find the exercise file **here**.

Source: