[1901.09813] Analogies Explained: Towards Understanding Word Embeddings

Word embeddings generated by neural network methods such as word2vec (W2V) are well known to exhibit seemingly linear behaviour, e.g. the embeddings of analogy "woman is to queen as man is to king" approximately describe a parallelogram. This property is particularly intriguing since the embeddings are not trained to achieve it. Several explanations have been proposed, but each introduces assumptions that do not hold in practice. We derive a probabilistically grounded definition of paraphrasing that we re-interpret as word transformation, a mathematical description of "$w_x$ is to $w_y$". From these concepts we prove existence of linear relationships between W2V-type embeddings that underlie the analogical phenomenon, identifying explicit error terms.

2 mentions: @pliang279@thegautamkamath
Keywords: embedding
Date: 2019/06/11 17:15

Referring Tweets

@pliang279 2. on NLP: several papers at #ICML2019 attempting to explain biases and analogies in word representations: Understanding the Origins of Bias in Word Embeddings (https://t.co/MrKPZxAjNd ), Analogies Explained: Towards Understanding Word Embeddings (https://t.co/PK76NiqH5Y ), and
@thegautamkamath There were also 7 honorable mentions. I'll list and link them below. Analogies Explained: Towards Understanding Word Embeddings, by Carl Allen and @tmh31 (https://t.co/TxetP4snuH). 4/n

Related Entries

Read more Neural Text Embeddings for Information Retrieval (WSDM 2017)
Read more [DL輪読会]Learning an Embedding Space for Transferable Robot Skills
Read more Variational Autoencoder in Tensorflow - facial expression low dimensional embedding - Machine learni...
Read more A Neural Network in PyTorch for Tabular Data with Categorical Embeddings – Let the Machines Learn
Read more [論文紹介] Field-aware Probabilistic Embedding Neural Network for CTR Prediction [RecSys 2018] / Paper s...