First, let's import the Python bindings for MeTA:
import metapy
If you don't have metapy
installed, you can install it with a
pip install metapy
on the command line on Linux, macOS, or Windows for either Python 2.7 or Python 3.x. (I will be using Python 3.6 in this tutorial.)
Double-check that you are running the latest version. Right now, that should be 0.2.10
.
metapy.__version__
'0.2.10'
Now, let's set MeTA to log to standard error so we can see progress output for long-running commands. (Only do this once, or you'll get double the output.)
metapy.log_to_stderr()
Now, let's download all of the files we need for the tutorial.
import urllib.request
import os
import tarfile
if not os.path.exists('sigir18-tutorial.tar.gz'):
urllib.request.urlretrieve('https://meta-toolkit.org/data/2018-06-25/sigir18-tutorial.tar.gz',
'sigir18-tutorial.tar.gz')
if not os.path.exists('data'):
with tarfile.open('sigir18-tutorial.tar.gz', 'r:gz') as files:
files.extractall()
Let's index our data using the InvertedIndex
format. In a search engine, we want to quickly determine what documents mention a specific query term, so the InvertedIndex
stores a mapping from term to a list of documents that contain that term (along with how many times they do).
inv_idx = metapy.index.make_inverted_index('cranfield.toml')
> Counting lines in file: [=================================] 100% ETA 00:00:00 1529953996: [info] Creating index: cranfield-idx/inv (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/inverted_index.cpp:119) > Tokenizing Docs: [> ] 0% ETA 00:00:00 1529953996: [warning] Empty document (id = 470) generated! (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/inverted_index.cpp:228) 1529953996: [warning] Empty document (id = 994) generated! (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/inverted_index.cpp:228) > Tokenizing Docs: [========================================] 100% ETA 00:00:00 > Merging: [================================================] 100% ETA 00:00:00 1529953996: [info] Created uncompressed postings file cranfield-idx/inv/postings.index (197.770000 KB) (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/inverted_index.cpp:148) > Compressing postings: [===================================] 100% ETA 00:00:00 1529953996: [info] Created compressed postings file (168.060000 KB) (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/inverted_index.cpp:279) 1529953996: [info] Done creating index: cranfield-idx/inv (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/inverted_index.cpp:166)
This may take a minute at first, since the index needs to be built. Subsequent calls to make_inverted_index
with this config file will simply load the index, which will not take any time.
Here's how we can interact with the index object:
inv_idx.num_docs()
1400
inv_idx.unique_terms()
4137
inv_idx.avg_doc_length()
87.17857360839844
inv_idx.total_corpus_terms()
122050
Let's search our index. We'll start by creating a ranker:
ranker = metapy.index.DirichletPrior()
Now we need a query. Let's create an example query.
query = metapy.index.Document()
query.content("flow equilibrium")
Now we can use this to search our index like so:
top_docs = ranker.score(inv_idx, query, num_results=5)
top_docs
[(235, 1.2931444644927979), (1251, 1.256299614906311), (316, 1.1081531047821045), (655, 1.0878994464874268), (574, 1.076568841934204)]
We are returned a ranked list of (doc_id, score) pairs. The scores are from the ranker, which in this case was Okapi BM25. Since the tutorial.toml
file we created for the cranfield dataset has store-full-text = true
, we can verify the content of our top documents by inspecting the document metadata field "content".
for num, (d_id, _) in enumerate(top_docs):
content = inv_idx.metadata(d_id).get('content')
print("{}. {}...\n".format(num + 1, content[0:250]))
1. criteria for thermodynamic equilibrium in gas flow . when gases flow at high velocity, the rates of internal processes may not be fast enough to maintain thermodynamic equilibrium . by defining quasi-equilibrium in flow as the condition in which the... 2. on the approach to chemical and vibrational equilibrium behind a strong normal shock wave . the concurrent approach to chemical and vibrational equilibrium of a pure diatomic gas passing through a strong normal shock wave is investigated . it is dem... 3. non-equilibrium flow of an ideal dissociating gas . the theory of an'ideal dissociating'gas developed by lighthill/1957/for conditions of thermodynamic equilibrium is extended to non-equilibrium conditions by postulating a simple rate equation for th... 4. departure from dissociation equilibrium in a hypersonic nozzle . the equations of motion for the flow of an ideal dissociating gas through a nearly conical nozzle have been solved numerically, assuming a simple equation for the rate of dissociation, ... 5. atomic recombination in a hypersonic wind tunnel nozzle . the flow of an ideal dissociating gas through a nearly conical nozzle is considered . the equations of one-dimensional motion are solved numerically assuming a simple rate equation together wi...
Since we have the queries file and relevance judgements, we can do an IR evaluation.
ev = metapy.index.IREval('cranfield.toml')
We will loop over the queries file and add each result to the IREval
object ev
.
def evaluate_ranker(ranker, ev, num_results):
ev.reset_stats()
with open('data/cranfield/cranfield-queries.txt') as query_file:
for query_num, line in enumerate(query_file):
query.content(line.strip())
results = ranker.score(inv_idx, query, num_results)
avg_p = ev.avg_p(results, query_num + 1, num_results)
print("Query {} average precision: {}".format(query_num + 1, avg_p))
evaluate_ranker(ranker, ev, 10)
Query 1 average precision: 0.19 Query 2 average precision: 0.5433333333333332 Query 3 average precision: 0.6541666666666666 Query 4 average precision: 0.5 Query 5 average precision: 0.35 Query 6 average precision: 0.0625 Query 7 average precision: 0.10666666666666666 Query 8 average precision: 0.0 Query 9 average precision: 0.6984126984126983 Query 10 average precision: 0.0625 Query 11 average precision: 0.028571428571428574 Query 12 average precision: 0.18 Query 13 average precision: 0.0 Query 14 average precision: 0.5 Query 15 average precision: 0.7 Query 16 average precision: 0.08333333333333333 Query 17 average precision: 0.07142857142857142 Query 18 average precision: 0.3333333333333333 Query 19 average precision: 0.0 Query 20 average precision: 0.2685185185185185 Query 21 average precision: 0.0 Query 22 average precision: 0.0 Query 23 average precision: 0.04722222222222222 Query 24 average precision: 0.3333333333333333 Query 25 average precision: 0.6507936507936507 Query 26 average precision: 0.13888888888888887 Query 27 average precision: 0.0 Query 28 average precision: 0.0 Query 29 average precision: 0.14166666666666666 Query 30 average precision: 0.11904761904761904 Query 31 average precision: 0.0 Query 32 average precision: 0.023809523809523808 Query 33 average precision: 0.5317460317460317 Query 34 average precision: 0.20555555555555557 Query 35 average precision: 0.0 Query 36 average precision: 0.25 Query 37 average precision: 0.05555555555555555 Query 38 average precision: 0.01111111111111111 Query 39 average precision: 0.12857142857142856 Query 40 average precision: 0.07222222222222222 Query 41 average precision: 0.7666666666666666 Query 42 average precision: 0.1638095238095238 Query 43 average precision: 0.48333333333333334 Query 44 average precision: 0.0 Query 45 average precision: 0.155 Query 46 average precision: 0.5380952380952382 Query 47 average precision: 0.15083333333333332 Query 48 average precision: 0.1433333333333333 Query 49 average precision: 0.0 Query 50 average precision: 0.0 Query 51 average precision: 0.16666666666666666 Query 52 average precision: 0.027777777777777776 Query 53 average precision: 0.15857142857142856 Query 54 average precision: 0.05555555555555555 Query 55 average precision: 0.11499999999999999 Query 56 average precision: 0.05 Query 57 average precision: 0.01 Query 58 average precision: 0.15873015873015872 Query 59 average precision: 0.03571428571428571 Query 60 average precision: 0.4666666666666667 Query 61 average precision: 0.45999999999999996 Query 62 average precision: 0.0 Query 63 average precision: 0.0 Query 64 average precision: 0.5 Query 65 average precision: 0.0 Query 66 average precision: 0.0 Query 67 average precision: 0.0365079365079365 Query 68 average precision: 0.1 Query 69 average precision: 0.06666666666666667 Query 70 average precision: 0.1 Query 71 average precision: 0.03125 Query 72 average precision: 0.0 Query 73 average precision: 0.2861111111111111 Query 74 average precision: 0.020833333333333332 Query 75 average precision: 0.03333333333333333 Query 76 average precision: 0.028571428571428574 Query 77 average precision: 0.28888888888888886 Query 78 average precision: 0.6666666666666666 Query 79 average precision: 0.0 Query 80 average precision: 0.0 Query 81 average precision: 0.25 Query 82 average precision: 0.09 Query 83 average precision: 0.0625 Query 84 average precision: 0.3 Query 85 average precision: 0.05 Query 86 average precision: 0.41666666666666663 Query 87 average precision: 0.0 Query 88 average precision: 0.594047619047619 Query 89 average precision: 0.075 Query 90 average precision: 0.19 Query 91 average precision: 0.0873015873015873 Query 92 average precision: 0.4726984126984126 Query 93 average precision: 0.5 Query 94 average precision: 0.4 Query 95 average precision: 0.5 Query 96 average precision: 0.4463095238095239 Query 97 average precision: 0.15416666666666665 Query 98 average precision: 0.0 Query 99 average precision: 0.19642857142857142 Query 100 average precision: 0.2185185185185185 Query 101 average precision: 0.6180555555555555 Query 102 average precision: 0.08333333333333333 Query 103 average precision: 0.05555555555555555 Query 104 average precision: 0.1 Query 105 average precision: 0.35333333333333333 Query 106 average precision: 0.24666666666666667 Query 107 average precision: 0.10476190476190476 Query 108 average precision: 0.6261904761904761 Query 109 average precision: 0.0 Query 110 average precision: 0.0 Query 111 average precision: 0.07619047619047618 Query 112 average precision: 0.3611111111111111 Query 113 average precision: 0.10416666666666666 Query 114 average precision: 0.0 Query 115 average precision: 0.0 Query 116 average precision: 0.03333333333333333 Query 117 average precision: 0.0 Query 118 average precision: 0.041666666666666664 Query 119 average precision: 0.5 Query 120 average precision: 0.17724867724867724 Query 121 average precision: 0.4768707482993197 Query 122 average precision: 0.05925925925925926 Query 123 average precision: 0.0 Query 124 average precision: 0.0 Query 125 average precision: 0.02 Query 126 average precision: 0.20833333333333331 Query 127 average precision: 0.025 Query 128 average precision: 0.0 Query 129 average precision: 0.4773809523809524 Query 130 average precision: 0.3933333333333333 Query 131 average precision: 0.09375 Query 132 average precision: 0.6592063492063491 Query 133 average precision: 0.12976190476190477 Query 134 average precision: 0.5 Query 135 average precision: 0.38690476190476186 Query 136 average precision: 0.03333333333333333 Query 137 average precision: 0.1875 Query 138 average precision: 0.25 Query 139 average precision: 0.0 Query 140 average precision: 0.10833333333333334 Query 141 average precision: 0.05555555555555555 Query 142 average precision: 0.0 Query 143 average precision: 0.6111111111111112 Query 144 average precision: 0.14722222222222223 Query 145 average precision: 0.07142857142857142 Query 146 average precision: 0.41666666666666663 Query 147 average precision: 0.14666666666666667 Query 148 average precision: 0.125 Query 149 average precision: 0.18285714285714286 Query 150 average precision: 1.0 Query 151 average precision: 0.0 Query 152 average precision: 0.0 Query 153 average precision: 0.19999999999999998 Query 154 average precision: 1.0 Query 155 average precision: 0.06481481481481481 Query 156 average precision: 0.6365476190476189 Query 157 average precision: 0.43809523809523804 Query 158 average precision: 0.175 Query 159 average precision: 0.015625 Query 160 average precision: 0.2 Query 161 average precision: 0.43333333333333335 Query 162 average precision: 0.020833333333333332 Query 163 average precision: 0.21666666666666667 Query 164 average precision: 0.40208333333333335 Query 165 average precision: 0.16666666666666666 Query 166 average precision: 0.0 Query 167 average precision: 0.325 Query 168 average precision: 0.08333333333333333 Query 169 average precision: 0.125 Query 170 average precision: 0.4611111111111111 Query 171 average precision: 0.4888888888888889 Query 172 average precision: 0.5416666666666666 Query 173 average precision: 0.8333333333333333 Query 174 average precision: 0.03333333333333333 Query 175 average precision: 0.02222222222222222 Query 176 average precision: 0.0 Query 177 average precision: 0.475 Query 178 average precision: 0.49166666666666664 Query 179 average precision: 0.125 Query 180 average precision: 0.34523809523809523 Query 181 average precision: 0.2 Query 182 average precision: 0.7 Query 183 average precision: 0.5308730158730158 Query 184 average precision: 0.06938775510204082 Query 185 average precision: 0.6388888888888888 Query 186 average precision: 0.016666666666666666 Query 187 average precision: 0.16666666666666666 Query 188 average precision: 0.09 Query 189 average precision: 0.015873015873015872 Query 190 average precision: 0.1 Query 191 average precision: 0.014285714285714285 Query 192 average precision: 0.5 Query 193 average precision: 0.6481481481481481 Query 194 average precision: 0.08888888888888889 Query 195 average precision: 0.1111111111111111 Query 196 average precision: 0.03333333333333333 Query 197 average precision: 0.6666666666666666 Query 198 average precision: 0.3125 Query 199 average precision: 0.05208333333333333 Query 200 average precision: 0.24074074074074073 Query 201 average precision: 0.3 Query 202 average precision: 0.26 Query 203 average precision: 0.07222222222222222 Query 204 average precision: 0.0 Query 205 average precision: 0.75 Query 206 average precision: 0.3333333333333333 Query 207 average precision: 0.15 Query 208 average precision: 0.6829365079365081 Query 209 average precision: 0.02 Query 210 average precision: 0.27777777777777773 Query 211 average precision: 0.0125 Query 212 average precision: 0.36666666666666664 Query 213 average precision: 0.5583333333333333 Query 214 average precision: 0.08333333333333333 Query 215 average precision: 0.0 Query 216 average precision: 0.0 Query 217 average precision: 0.05833333333333333 Query 218 average precision: 0.0 Query 219 average precision: 0.014285714285714285 Query 220 average precision: 0.05333333333333333 Query 221 average precision: 0.24333333333333332 Query 222 average precision: 0.5148148148148147 Query 223 average precision: 0.44375 Query 224 average precision: 0.0 Query 225 average precision: 0.13333333333333333
Afterwards, we can get the mean average precision of all the queries.
dp_map = ev.map()
print("MAP: {}".format(dp_map))
MAP: 0.21512203955656342
Now, let's use the two-component mixture model we discussed as an implementation of pseudo-feedback for retrieval and see if it helps improve performance. The actual ranking function used here is KL-divergence, where the query model is adjusted to include pseudo-feedback from the retrieved documents.
In order to work, the ranker needs to be able to quickly determine what words were used in the feedback document set. The InvertedIndex
does not provide fast access to this (since it is a mapping from term to documents, rather than from documents to terms), so we will want to first create a ForwardIndex
to get the document -> terms mapping.
fwd_idx = metapy.index.make_forward_index('cranfield.toml')
> Counting lines in file: [=================================] 100% ETA 00:00:00 1529954010: [info] Creating forward index: cranfield-idx/fwd (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/forward_index.cpp:239) > Tokenizing Docs: [> ] 0% ETA 00:00:00 1529954010: [warning] Empty document (id = 470) generated! (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/forward_index.cpp:335) 1529954010: [warning] Empty document (id = 994) generated! (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/forward_index.cpp:335) > Tokenizing Docs: [========================================] 100% ETA 00:00:00 > Merging: [================================================] 100% ETA 00:00:00 1529954010: [info] Done creating index: cranfield-idx/fwd (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/forward_index.cpp:278)
Now we can construct the KL-divergence pseudo-feedback ranker. The main components are:
DirichletPrior
)max_terms
, the number of terms from the feedback model to incorporate into the new query model (default 50)feedback = metapy.index.KLDivergencePRF(fwd_idx, metapy.index.DirichletPrior())
evaluate_ranker(feedback, ev, 10)
Query 1 average precision: 0.13999999999999999 Query 2 average precision: 0.524047619047619 Query 3 average precision: 0.6642857142857143 Query 4 average precision: 0.5 Query 5 average precision: 0.6875 Query 6 average precision: 0.0625 Query 7 average precision: 0.11666666666666665 Query 8 average precision: 0.0 Query 9 average precision: 0.49999999999999994 Query 10 average precision: 0.0625 Query 11 average precision: 0.023809523809523808 Query 12 average precision: 0.15714285714285714 Query 13 average precision: 0.0 Query 14 average precision: 0.5 Query 15 average precision: 0.6428571428571428 Query 16 average precision: 0.05555555555555555 Query 17 average precision: 0.05 Query 18 average precision: 0.16666666666666666 Query 19 average precision: 0.0 Query 20 average precision: 0.33201058201058203 Query 21 average precision: 0.0 Query 22 average precision: 0.0 Query 23 average precision: 0.075 Query 24 average precision: 0.38888888888888884 Query 25 average precision: 0.7530864197530864 Query 26 average precision: 0.061111111111111116 Query 27 average precision: 0.0 Query 28 average precision: 0.0 Query 29 average precision: 0.091005291005291 Query 30 average precision: 0.19999999999999998 Query 31 average precision: 0.0 Query 32 average precision: 0.023809523809523808 Query 33 average precision: 0.6666666666666666 Query 34 average precision: 0.21031746031746032 Query 35 average precision: 0.03333333333333333 Query 36 average precision: 0.6 Query 37 average precision: 0.1111111111111111 Query 38 average precision: 0.02 Query 39 average precision: 0.12857142857142856 Query 40 average precision: 0.07222222222222222 Query 41 average precision: 0.7916666666666666 Query 42 average precision: 0.14523809523809522 Query 43 average precision: 0.3333333333333333 Query 44 average precision: 0.0 Query 45 average precision: 0.13999999999999999 Query 46 average precision: 0.49071428571428566 Query 47 average precision: 0.14027777777777778 Query 48 average precision: 0.1433333333333333 Query 49 average precision: 0.0 Query 50 average precision: 0.0 Query 51 average precision: 0.22999999999999998 Query 52 average precision: 0.08125 Query 53 average precision: 0.15 Query 54 average precision: 0.037037037037037035 Query 55 average precision: 0.11833333333333333 Query 56 average precision: 0.05 Query 57 average precision: 0.0125 Query 58 average precision: 0.16666666666666666 Query 59 average precision: 0.03571428571428571 Query 60 average precision: 0.45999999999999996 Query 61 average precision: 0.4083333333333333 Query 62 average precision: 0.0 Query 63 average precision: 0.0 Query 64 average precision: 0.5 Query 65 average precision: 0.0 Query 66 average precision: 0.0 Query 67 average precision: 0.03111111111111111 Query 68 average precision: 0.06666666666666667 Query 69 average precision: 0.2 Query 70 average precision: 0.1 Query 71 average precision: 0.041666666666666664 Query 72 average precision: 0.0 Query 73 average precision: 0.38916666666666666 Query 74 average precision: 0.05555555555555555 Query 75 average precision: 0.02222222222222222 Query 76 average precision: 0.02040816326530612 Query 77 average precision: 0.34027777777777773 Query 78 average precision: 0.7916666666666666 Query 79 average precision: 0.0 Query 80 average precision: 0.0 Query 81 average precision: 0.25 Query 82 average precision: 0.07333333333333333 Query 83 average precision: 0.05 Query 84 average precision: 0.2375 Query 85 average precision: 0.05 Query 86 average precision: 0.30952380952380953 Query 87 average precision: 0.0 Query 88 average precision: 0.594047619047619 Query 89 average precision: 0.10833333333333334 Query 90 average precision: 0.1875 Query 91 average precision: 0.1111111111111111 Query 92 average precision: 0.5747619047619048 Query 93 average precision: 0.5 Query 94 average precision: 0.4 Query 95 average precision: 0.5 Query 96 average precision: 0.6042063492063492 Query 97 average precision: 0.24166666666666664 Query 98 average precision: 0.0 Query 99 average precision: 0.25 Query 100 average precision: 0.25555555555555554 Query 101 average precision: 0.5444444444444444 Query 102 average precision: 0.08333333333333333 Query 103 average precision: 0.1 Query 104 average precision: 0.1 Query 105 average precision: 0.4333333333333333 Query 106 average precision: 0.3746031746031746 Query 107 average precision: 0.1619047619047619 Query 108 average precision: 0.5488095238095239 Query 109 average precision: 0.0 Query 110 average precision: 0.0 Query 111 average precision: 0.014285714285714287 Query 112 average precision: 0.3611111111111111 Query 113 average precision: 0.10555555555555556 Query 114 average precision: 0.0 Query 115 average precision: 0.0 Query 116 average precision: 0.02857142857142857 Query 117 average precision: 0.0 Query 118 average precision: 0.037037037037037035 Query 119 average precision: 0.3333333333333333 Query 120 average precision: 0.1527777777777778 Query 121 average precision: 0.42857142857142855 Query 122 average precision: 0.07777777777777778 Query 123 average precision: 0.0 Query 124 average precision: 0.0 Query 125 average precision: 0.03333333333333333 Query 126 average precision: 0.20833333333333331 Query 127 average precision: 0.02222222222222222 Query 128 average precision: 0.0 Query 129 average precision: 0.6738095238095239 Query 130 average precision: 0.55 Query 131 average precision: 0.18333333333333335 Query 132 average precision: 0.7254365079365078 Query 133 average precision: 0.18010204081632653 Query 134 average precision: 0.5 Query 135 average precision: 0.37351190476190477 Query 136 average precision: 0.047619047619047616 Query 137 average precision: 0.1125 Query 138 average precision: 0.5 Query 139 average precision: 0.0 Query 140 average precision: 0.10833333333333334 Query 141 average precision: 0.041666666666666664 Query 142 average precision: 0.0 Query 143 average precision: 0.39285714285714285 Query 144 average precision: 0.20555555555555557 Query 145 average precision: 0.07142857142857142 Query 146 average precision: 0.30952380952380953 Query 147 average precision: 0.14999999999999997 Query 148 average precision: 0.075 Query 149 average precision: 0.17666666666666667 Query 150 average precision: 1.0 Query 151 average precision: 0.0 Query 152 average precision: 0.0 Query 153 average precision: 0.2571428571428571 Query 154 average precision: 1.0 Query 155 average precision: 0.06666666666666667 Query 156 average precision: 0.7532142857142856 Query 157 average precision: 0.40555555555555556 Query 158 average precision: 0.19375 Query 159 average precision: 0.020833333333333332 Query 160 average precision: 0.2 Query 161 average precision: 0.3611111111111111 Query 162 average precision: 0.013888888888888888 Query 163 average precision: 0.19444444444444442 Query 164 average precision: 0.40208333333333335 Query 165 average precision: 0.125 Query 166 average precision: 0.0 Query 167 average precision: 0.5 Query 168 average precision: 0.08333333333333333 Query 169 average precision: 0.25 Query 170 average precision: 0.46990740740740744 Query 171 average precision: 0.5555555555555555 Query 172 average precision: 0.5583333333333333 Query 173 average precision: 0.7 Query 174 average precision: 0.03333333333333333 Query 175 average precision: 0.02857142857142857 Query 176 average precision: 0.0 Query 177 average precision: 0.475 Query 178 average precision: 0.75 Query 179 average precision: 0.25 Query 180 average precision: 0.3880952380952381 Query 181 average precision: 0.1 Query 182 average precision: 0.7 Query 183 average precision: 0.6842063492063492 Query 184 average precision: 0.05215419501133787 Query 185 average precision: 0.6296296296296297 Query 186 average precision: 0.04722222222222222 Query 187 average precision: 0.125 Query 188 average precision: 0.19 Query 189 average precision: 0.022222222222222223 Query 190 average precision: 0.05 Query 191 average precision: 0.05357142857142857 Query 192 average precision: 0.575 Query 193 average precision: 0.6565255731922398 Query 194 average precision: 0.19999999999999998 Query 195 average precision: 0.3333333333333333 Query 196 average precision: 0.05 Query 197 average precision: 0.6666666666666666 Query 198 average precision: 0.1875 Query 199 average precision: 0.048611111111111105 Query 200 average precision: 0.40740740740740744 Query 201 average precision: 0.33999999999999997 Query 202 average precision: 0.22666666666666666 Query 203 average precision: 0.07333333333333333 Query 204 average precision: 0.0 Query 205 average precision: 0.7 Query 206 average precision: 0.38888888888888884 Query 207 average precision: 0.3 Query 208 average precision: 0.7345238095238096 Query 209 average precision: 0.03333333333333333 Query 210 average precision: 0.27777777777777773 Query 211 average precision: 0.025 Query 212 average precision: 0.36666666666666664 Query 213 average precision: 0.49309523809523814 Query 214 average precision: 0.08333333333333333 Query 215 average precision: 0.0 Query 216 average precision: 0.0 Query 217 average precision: 0.07857142857142857 Query 218 average precision: 0.0 Query 219 average precision: 0.014285714285714285 Query 220 average precision: 0.07333333333333333 Query 221 average precision: 0.24333333333333332 Query 222 average precision: 0.5238095238095238 Query 223 average precision: 0.4583333333333333 Query 224 average precision: 0.0 Query 225 average precision: 0.15
fb_map = ev.map()
print("Feedback MAP: {}".format(fb_map))
print("DP MAP: {}".format(dp_map))
Feedback MAP: 0.22816526133086987 DP MAP: 0.21512203955656342