Joblib as serialization of TF data

Joblib is a data serialization format used in Big Data use cases.

pip3 install joblib

It looks attractive from the specifactions, but how will it perform?

As a simple test, we take the feature data for g_word_utf8. It is a map from the numbers 1 to 426584 to Hebrew word occurrences (Unicode strings).

In Text-Fabric we have a representation in plain text and a compressed, pickled representation.

Outcome

Text-Fabric is much faster in loading this kind of data.

The size of the joblib uncompressed serialization is much bigger than the TF text representation.

The size of the gzipped joblib serialization is still 30% more than gzipped, pickled TF serialization.

Detailed comparison

name kind size load time
g_word_utf8.tf tf: plain unicode text 5.4 MB 1.6 s
g_word_utf8.tfx tf: gzipped binary 3.2 MB 0.2 s
g_word_utf8.joblib joblib: uncompressed 11.2 MB 2.0 s
g_word_utf8.joblib.gz joblib: gzipped 3.2 MB 2.7 s

Conclusion

We do not see reasons to replace the TF feature data serialization by joblib.

In [1]:
import os
import gzip
import joblib

from tf.fabric import Fabric

GZIP_LEVEL = 2 # same as used in Text-Fabric

Load from the textual data

In [3]:
VERSION = 'c'
BHSA = f'BHSA/tf/{VERSION}'
PARA = f'parallels/tf/{VERSION}'
In [3]:
TF = Fabric(locations='~/github/etcbc', modules=[BHSA, PARA])
api = TF.load('')
api.makeAvailableIn(globals())
This is Text-Fabric 5.5.22
Api reference : https://annotation.github.io/text-fabric/Api/Fabric/
Tutorial      : https://github.com/annotation/text-fabric/blob/master/docs/tutorial.ipynb
Example data  : https://github.com/annotation/text-fabric-data

117 features found and 0 ignored
  0.00s loading features ...
   |     1.60s T g_word_utf8          from /Users/dirk/github/etcbc/BHSA/tf/c
  6.02s All features loaded/computed - for details use loadLog()

The load time is ~ 1.6 seconds.

But during this time, the textual data has been compiled and written to a binary form. Let's load again.

Load from binary data

In [4]:
TF = Fabric(locations='~/github/etcbc', modules=[BHSA, PARA])
api = TF.load('')
api.makeAvailableIn(globals())
This is Text-Fabric 5.5.22
Api reference : https://annotation.github.io/text-fabric/Api/Fabric/
Tutorial      : https://github.com/annotation/text-fabric/blob/master/docs/tutorial.ipynb
Example data  : https://github.com/annotation/text-fabric-data

117 features found and 0 ignored
  0.00s loading features ...
  5.27s All features loaded/computed - for details use loadLog()
In [5]:
loadLog()
   |     0.03s B otype                from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.53s B oslots               from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.01s B book                 from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.01s B chapter              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.01s B verse                from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.13s B g_cons               from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.18s B g_cons_utf8          from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.14s B g_lex                from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.19s B g_lex_utf8           from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.16s B g_word               from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.33s B g_word_utf8          from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.16s B lex0                 from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.25s B lex_utf8             from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B qere                 from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B qere_trailer         from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B qere_trailer_utf8    from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.01s B qere_utf8            from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.08s B trailer              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.09s B trailer_utf8         from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B __levels__           from otype, oslots, otext
   |     0.03s B __order__            from otype, oslots, __levels__
   |     0.04s B __rank__             from otype, __order__
   |     1.65s B __levUp__            from otype, oslots, __rank__
   |     0.88s B __levDown__          from otype, __levUp__, __rank__
   |     0.30s B __boundary__         from otype, oslots, __rank__
   |     0.01s B __sections__         from otype, oslots, otext, __levUp__, __levels__, book, chapter, verse
   |     0.00s B book@am              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@ar              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@bn              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@da              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@de              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@el              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@en              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@es              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@fa              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@fr              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@he              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@hi              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@id              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@ja              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@ko              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@la              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@nl              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@pa              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@pt              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@ru              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@sw              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@syc             from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@tr              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@ur              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@yo              from /Users/dirk/github/etcbc/BHSA/tf/c
   |     0.00s B book@zh              from /Users/dirk/github/etcbc/BHSA/tf/c

The load time of the feature g_word_utf8 is ~ 0.2 seconds.

Make an joblib feature data file

In [6]:
tempDir = os.path.expanduser('~/github/annotation/text-fabric/_temp/joblib')
os.makedirs(tempDir, exist_ok=True)
In [9]:
feature = 'g_word_utf8'
data =  TF.features[feature].data
print(len(data))
print(data[2])
426584
רֵאשִׁ֖ית

We write the feature data to an Avro data file.

In [10]:
dataFile = f'{tempDir}/{feature}.joblib'
In [16]:
indent(reset=True)
info('start writing')
joblib.dump(data, dataFile)
info('done')
  0.00s start writing
  3.20s done

We make also a gzipped data file.

In [17]:
indent(reset=True)
info('start writing')
dataFileZ = f'{dataFile}.gz'
joblib.dump(data, dataFileZ, compress=('gzip', GZIP_LEVEL))
info('done')
  0.00s start writing
  3.70s done

Load from joblib file

In [18]:
indent(reset=True)
info('start reading')
rData = joblib.load(dataFile)
info('done')
print(rData[2])
  0.00s start reading
  2.06s done
רֵאשִׁ֖ית

Load time ~ 2.0 seconds.

Load from joblib file (gzipped)

In [19]:
indent(reset=True)
info('start reading')
rData = joblib.load(dataFileZ)
info('done')
print(rData[2])
  0.00s start reading
  2.73s done
רֵאשִׁ֖ית

Load time ~ 2.7 seconds.

In [ ]: