!jupyter nbconvert --to markdown 15_7_4_Exercises.ipynb
[NbConvertApp] WARNING | Config option `kernel_spec_manager_class` not recognized by `NbConvertApp`. [NbConvertApp] Converting notebook 15_7_4_Exercises.ipynb to markdown [NbConvertApp] Writing 4100 bytes to 15_7_4_Exercises.md
The computational complexity for calculating each gradient depends on the word embedding method and the optimization technique used. For example, for the skip-gram model with negative sampling, the complexity is O(kd), where k is the number of negative samples and d is the dimension of the word vectors¹. For the continuous bag-of-words model with hierarchical softmax, the complexity is O(log(V)d), where V is the size of the vocabulary².
If the dictionary size is huge, then the word embedding methods may face some issues, such as:
The softmax function becomes computationally expensive, as it requires summing over all the words in the vocabulary for each input word².
The word vectors may not be expressive enough to capture all the semantic and syntactic nuances of the language, as they are constrained by a fixed dimensionality³.
The training time and memory requirements may increase significantly, as the number of parameters grows linearly or quadratically with the vocabulary size⁴.
(1) On the Dimensionality of Word Embedding - arXiv.org. https://arxiv.org/pdf/1812.04224.pdf.
(2) The Ultimate Guide to Word Embeddings - neptune.ai. https://neptune.ai/blog/word-embeddings-guide.
(3) Learning Word Embedding | Lil'Log - GitHub Pages. https://lilianweng.github.io/posts/2017-10-15-word-embedding/.
(4) arXiv:1411.2738v4 [cs.CL] 5 Jun 2016. https://arxiv.org/pdf/1411.2738.pdf.
According to the word2vec paper by Mikolov et al. (2013), one way to train word vectors for fixed phrases in English is to use a simple data-driven approach to find the phrases¹. The idea is to first train a word2vec model on the original data, and then score all possible bigrams using a formula that measures how frequently the words co-occur together compared to their individual frequencies¹. The bigrams with scores above a certain threshold are then treated as single tokens in the next iteration of training¹. This process can be repeated multiple times to find longer phrases¹. For example, the phrase "new york" may be merged into a single token "new_york" after the first iteration, and then combined with other words to form phrases like "new_york_times" or "new_york_city" in subsequent iterations¹. This way, the word vectors for phrases can capture more information than the sum of their parts.
The relationship between the dot product of two word vectors in the skip-gram model and the cosine similarity is that they are proportional to each other, as long as the word vectors are normalized to have unit length. This is because the cosine similarity is defined as the dot product divided by the product of the magnitudes, and if the magnitudes are both 1, then the cosine similarity is equal to the dot product. Therefore, maximizing the dot product is equivalent to maximizing the cosine similarity in the skip-gram model¹⁵.
For a pair of words with similar semantics, the cosine similarity of their word vectors (trained by the skip-gram model) may be high because the skip-gram model tries to maximize the probability of predicting the context words given a center word, and words that have similar meanings tend to appear in similar contexts. Therefore, the word vectors that are trained by the skip-gram model tend to capture the semantic and syntactic similarities between words, and words that are more similar will have higher cosine similarity²⁴.