doc2vec¶

This is an experimental code developed by Tomas Mikolov found in the word2vec google group: https://groups.google.com/d/msg/word2vec-toolkit/Q49FIrNOQRo/J6KG8mUj45sJ

This is not yet available on Pypi you need the latest master branch from git.

The input format for doc2vec is still one big text document but every line should be one document prepended with an unique id, for example:

_*0 This is sentence 1
_*1 This is sentence 2

Requirements¶

nltk
Download some data: http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz
Untar that data: tar -xvf aclImdb_v1.tar.gz

Preprocess¶

Merge data into one big document with an id per line and do some basic preprocessing: word tokenizer.

In [1]:

from __future__ import unicode_literals

In [2]:

import os
import nltk

In [3]:

directories = ['train/pos', 'train/neg', 'test/pos', 'test/neg', 'train/unsup']

In [4]:

input_file = open('/Users/drodriguez/Downloads/alldata.txt', 'w')

In [5]:

id_ = 0
for directory in directories:
    rootdir = os.path.join('/Users/drodriguez/Downloads/aclImdb', directory)
    for subdir, dirs, files in os.walk(rootdir):
        for file_ in files:
            with open(os.path.join(subdir, file_), 'r') as f:
                doc_id = '_*%i' % id_
                id_ = id_ + 1

                text = f.read()
                text = text.decode('utf-8')
                tokens = nltk.word_tokenize(text)
                doc = ' '.join(tokens).lower()
                doc = doc.encode('ascii', 'ignore')
                input_file.write('%s %s\n' % (doc_id, doc))

In [6]:

input_file.close()

doc2vec¶

In [1]:

%load_ext autoreload
%autoreload 2

In [2]:

import word2vec

In [3]:

word2vec.doc2vec('/Users/drodriguez/Downloads/alldata.txt', '/Users/drodriguez/Downloads/vectors.bin', cbow=0, size=100, window=10, negative=5, hs=0, sample='1e-4', threads=12, iter_=20, min_count=1, verbose=True)

Starting training using file /Users/drodriguez/Downloads/alldata.txt
Vocab size: 355046
Words in train file: 28300990
Alpha: 0.000002  Progress: 100.01%  Words/thread/sec: 92.57k

Predictions¶

Is possible to load the vectors using the same wordvectors class as a regular word2vec binary file.

In [1]:

%load_ext autoreload
%autoreload 2

In [2]:

import word2vec

In [3]:

model = word2vec.load('/Users/drodriguez/Downloads/vectors.bin')

In [5]:

model.vectors.shape

Out[5]:

(355046, 100)

The documents vector are going to be identified by the id we used in the preprocesing section, for example document 1 is going to have vector:

In [7]:

model['_*1']

Out[7]:

array([-0.09961915, -0.02504692, -0.00935447, -0.06283784,  0.0496435 ,
        0.08408608,  0.07249928,  0.02332756,  0.04755085,  0.14972655,
       -0.13693954,  0.01296212, -0.05615778,  0.05408363, -0.01922397,
        0.00903398,  0.11205222,  0.02491457,  0.04302743, -0.06734619,
       -0.2004746 , -0.10970256, -0.04777983, -0.05336951, -0.10399633,
       -0.06500414,  0.0393892 , -0.08285502,  0.05692215,  0.01362013,
        0.0013779 , -0.24589944, -0.16099831, -0.11000603, -0.08007748,
       -0.05447361,  0.10116527,  0.06073807,  0.00416331,  0.00434075,
       -0.02536621,  0.12531835, -0.0312396 , -0.03754066,  0.10542928,
       -0.01937485,  0.03270554,  0.03367785, -0.31589472,  0.00840659,
       -0.09368768, -0.11164349, -0.02970047, -0.11497822,  0.06357043,
       -0.16664146,  0.02935979,  0.25292322,  0.01335857, -0.19644944,
        0.08630948,  0.05118916, -0.08062234,  0.03329093, -0.13994266,
        0.07419056, -0.0284326 , -0.04101218, -0.01186225,  0.10280388,
        0.00699921,  0.07681306,  0.0986157 , -0.06155488, -0.17678751,
       -0.0433546 , -0.1698599 ,  0.00764652, -0.11591533, -0.12973167,
       -0.01140277, -0.02404138, -0.06018848, -0.02115276, -0.14684282,
       -0.18135296, -0.03216174, -0.02125036,  0.2539596 ,  0.16910006,
       -0.11961638,  0.03961169,  0.07747228,  0.02761923,  0.07856126,
        0.06564176,  0.05922704,  0.10623101,  0.04387141, -0.14101151])

We can ask for similarity words or documents on document 1

In [10]:

indexes, metrics = model.cosine('_*1')

In [11]:

model.generate_response(indexes, metrics).tolist()

Out[11]:

[(u'houselessness', 0.9697854490765448),
 (u'_*62909', 0.8435915200187546),
 (u'_*92297', 0.8382383325156331),
 (u'_*62902', 0.8354628520801568),
 (u'_*31249', 0.8321578405132342),
 (u'_*20758', 0.8302829157776485),
 (u'_*12342', 0.8263513274559964),
 (u'_*32435', 0.8237210585123108),
 (u'_*67836', 0.823590134267539),
 (u'_*31245', 0.8230957438394273)]

Now its just a case of matching the id to the data created on the preprocessing step

In [ ]: