To get started: consult start
Search in Text-Fabric is a template based way of looking for structural patterns in your dataset.
It is inspired by the idea of topographic query.
Within Text-Fabric we have the unique possibility to combine the ease of formulating search templates for complicated patterns with the power of programmatically processing the results.
This notebook will show you how to get up and running.
Search is a powerful feature for a wide range of purposes.
Quite a bit of the implementation work has been dedicated to optimize performance. Yet I do not pretend to have found optimal strategies for all possible search templates. Some search tasks may turn out to be somewhat costly or even very costly.
That being said, I think search might turn out helpful in many cases, especially by reducing the amount of hand-coding needed to work with special subsets of your data.
Search is as simple as saying (just an example)
results = A.search(template)
A.show(results)
See all ins and outs in the search template docs.
%load_ext autoreload
%autoreload 2
from tf.app import use
A = use("Nino-cunei/oldbabylonian", hoist=globals())
This is Text-Fabric 9.2.0 Api reference : https://annotation.github.io/text-fabric/tf/cheatsheet.html 67 features found and 0 ignored
We start with the most simple form of issuing a query. Let's look for the numerals with a repeat greater than 3. We also want to show the words in which they occur.
All work involved in searching takes place under the hood.
query = """
word
sign type=numeral repeat>3
"""
results = A.search(query)
0.17s 546 results
A.table(results, end=10)
n | p | word | sign |
---|---|---|---|
1 | P509373 obverse:11 | 7(disz) | 7(disz) |
2 | P509376 obverse:4 | 4(disz)-kam_ | 4(disz)- |
3 | P509377 reverse:10 | 6(disz) | 6(disz) |
4 | P389958 obverse:7' | _5(disz) | _5(disz) |
5 | P510527 reverse:11 | _igi-6(disz)-gal2 | 6(disz)- |
6 | P510530 reverse:17 | 4(disz)-kam_ | 4(disz)- |
7 | P510534 reverse:5' | 4(disz) | 4(disz) |
8 | P510534 reverse:10' | _5(disz)-ta-am3_ | _5(disz)- |
9 | P510536 obverse:5 | 6(disz) | 6(disz) |
10 | P510539 obverse:7 | 4(disz) | 4(disz) |
We can show them in unicode representation as well:
A.table(results, end=10, fmt="text-orig-unicode")
n | p | word | sign |
---|---|---|---|
1 | P509373 obverse:11 | 𒐌 | 𒐌 |
2 | P509376 obverse:4 | 𒐉𒄭𒁁 | 𒐉 |
3 | P509377 reverse:10 | 𒐋 | 𒐋 |
4 | P389958 obverse:7' | 𒐊 | 𒐊 |
5 | P510527 reverse:11 | 𒅆𒐋𒅅 | 𒐋 |
6 | P510530 reverse:17 | 𒐉𒄭𒁁 | 𒐉 |
7 | P510534 reverse:5' | 𒐉 | 𒐉 |
8 | P510534 reverse:10' | 𒐊𒋫𒀀𒀭 | 𒐊 |
9 | P510536 obverse:5 | 𒐋 | 𒐋 |
10 | P510539 obverse:7 | 𒐉 | 𒐉 |
The hyperlinks take us all to the CDLI archival page of the document (tablet) in question.
Note that we can choose start and/or end points in the results list.
A.table(results, start=500, end=503, fmt="text-orig-rich")
n | p | word | sign |
---|---|---|---|
500 | P305753 case - obverse:10 | 5(diš) | 5(diš) |
501 | P305753 case - reverse:10 | 5(diš) | 5(diš) |
502 | P305753 case - reverse:14 | 4(u) | 4(u) |
503 | P305753 case - lower edge:1 | igi-6(diš)-gal₂ | 6(diš)- |
We can show the results more fully with show()
.
That gives us pretty displays of tablet lines with the results highlighted.
A.show(results, end=3)
result 1
result 2
result 3
There are two fundamentally different ways of presenting the results: condensed and uncondensed.
In uncondensed view, all results are listed individually. You can keep track of which parts belong to which results. The display can become unwieldy.
This is the default view, because it is the straightest, most logical, answer to your query.
In condensed view all nodes of all results are grouped in containers first (e.g. verses), and then presented container by container. You loose the information of what parts belong to what result.
As an example of is the difference, we look for all numerals.
query = """
% we choose a tablet with several numerals
document pnumber=P510556
sign type=numeral repeat>3
"""
Note that you can have comments in a search template. Comment lines start with a %
.
results = A.search(query)
A.table(results, end=10)
0.17s 5 results
n | p | document | sign |
---|---|---|---|
1 | P510556 obverse:6 | P510556 | 4(disz)- |
2 | P510556 obverse:12 | P510556 | 5(disz) |
3 | P510556 reverse:4 | P510556 | _5(gesz2) |
4 | P510556 reverse:8 | P510556 | 6(gesz2) |
5 | P510556 reverse:8 | P510556 | 5(ban2) |
Let's expand the results display:
A.show(results, end=2, skipCols="1")
result 1
result 2
As you see, the results are listed per result tuple, even if they occur all in the same verse. This way you can keep track of what exactly belongs to each result.
Now in condensed mode:
A.show(results, condensed=True, withNodes=True)
line 1
line 2
line 3
line 4
The last line has two results, and both results are highlighted in the same line display.
We can modify the container in which we see our results.
By default, it is line
, but we can make it face
as well:
A.show(results, end=2, condensed=True, condenseType="face")
face 1
face 2
We now see the the displays of two faces, one with two numerals in it and one with three.
Let us make a new search where we look for two different things in the same line.
We can apply different highlight colours to different parts of the result. The signs in the pair are member 0 and 1 of the result tuples. The members that we do not map, will not be highlighted. The members that we map to the empty string will be highlighted with the default color.
NB: Choose your colours from the CSS specification.
query = """
line
sign missing=1
sign question=1
sign damage=1
"""
results = A.search(query)
A.table(results, end=10, baseTypes="sign")
0.61s 776 results
n | p | line | sign | sign | sign |
---|---|---|---|---|---|
1 | P510530 obverse:5 | {disz}[{d}x]-ra#?-bi il-qe2-a-am-ma | [{d} | ra#?- | ra#?- |
2 | P510530 obverse:5 | {disz}[{d}x]-ra#?-bi il-qe2-a-am-ma | x]- | ra#?- | ra#?- |
3 | P510530 reverse:11 | [szu-bu-lim?] sza# ta-asz-pu-ra-am | [szu- | lim?] | sza# |
4 | P510530 reverse:11 | [szu-bu-lim?] sza# ta-asz-pu-ra-am | bu- | lim?] | sza# |
5 | P510530 reverse:11 | [szu-bu-lim?] sza# ta-asz-pu-ra-am | lim?] | lim?] | sza# |
6 | P510534 reverse:15' | li#-iq#-bi-ma da-ba-ab#-szu li#?-in-na-me-[er] | [er] | li#?- | li#- |
7 | P510534 reverse:15' | li#-iq#-bi-ma da-ba-ab#-szu li#?-in-na-me-[er] | [er] | li#?- | iq#- |
8 | P510534 reverse:15' | li#-iq#-bi-ma da-ba-ab#-szu li#?-in-na-me-[er] | [er] | li#?- | ab#- |
9 | P510534 reverse:15' | li#-iq#-bi-ma da-ba-ab#-szu li#?-in-na-me-[er] | [er] | li#?- | li#?- |
10 | P510534 reverse:20' | li#-il#-qe2#?-[szu-ma?] | [szu- | qe2#?- | li#- |
A.table(
results,
end=10,
baseTypes="sign",
colorMap={0: "", 2: "cyan", 3: "magenta", 4: "lightsalmon"},
)
n | p | line | sign | sign | sign |
---|---|---|---|---|---|
1 | P510530 obverse:5 | {disz}[{d}x]-ra#?-bi il-qe2-a-am-ma | [{d} | ra#?- | ra#?- |
2 | P510530 obverse:5 | {disz}[{d}x]-ra#?-bi il-qe2-a-am-ma | x]- | ra#?- | ra#?- |
3 | P510530 reverse:11 | [szu-bu-lim?] sza# ta-asz-pu-ra-am | [szu- | lim?] | sza# |
4 | P510530 reverse:11 | [szu-bu-lim?] sza# ta-asz-pu-ra-am | bu- | lim?] | sza# |
5 | P510530 reverse:11 | [szu-bu-lim?] sza# ta-asz-pu-ra-am | lim?] | lim?] | sza# |
6 | P510534 reverse:15' | li#-iq#-bi-ma da-ba-ab#-szu li#?-in-na-me-[er] | [er] | li#?- | li#- |
7 | P510534 reverse:15' | li#-iq#-bi-ma da-ba-ab#-szu li#?-in-na-me-[er] | [er] | li#?- | iq#- |
8 | P510534 reverse:15' | li#-iq#-bi-ma da-ba-ab#-szu li#?-in-na-me-[er] | [er] | li#?- | ab#- |
9 | P510534 reverse:15' | li#-iq#-bi-ma da-ba-ab#-szu li#?-in-na-me-[er] | [er] | li#?- | li#?- |
10 | P510534 reverse:20' | li#-il#-qe2#?-[szu-ma?] | [szu- | qe2#?- | li#- |
A.show(results, end=5, colorMap={0: "", 2: "cyan", 3: "magenta", 4: "lightsalmon"})
result 1
result 2
result 3
result 4
result 5
Color mapping works best for uncondensed results. If you condense results, some nodes may occupy different positions in different results. It is unpredictable which color will be used for such nodes:
A.show(
results,
condensed=True,
end=5,
colorMap={0: "", 2: "cyan", 3: "magenta", 4: "lightsalmon"},
)
line 1
line 2
line 3
line 4
line 5
You can specify to what container you want to condense. By default, everything is condensed to lines.
Let's change that to faces.
Note that the end
parameter counts the number of faces now.
A.show(
results,
end=2,
condensed=True,
condenseType="face",
colorMap={0: "", 2: "cyan", 3: "magenta", 4: "lightsalmon"},
)
face 1
face 2
You can stipulate an order on the things in your template. You only have to put a relational operator between them. Say we want only results where the damage follows the missing.
query = """
line
sign question=1
sign missing=1
< sign damage=1
"""
results = A.search(query)
A.table(results, end=10, baseTypes="sign")
0.61s 372 results
n | p | line | sign | sign | sign |
---|---|---|---|---|---|
1 | P510530 obverse:5 | {disz}[{d}x]-ra#?-bi il-qe2-a-am-ma | ra#?- | [{d} | ra#?- |
2 | P510530 obverse:5 | {disz}[{d}x]-ra#?-bi il-qe2-a-am-ma | ra#?- | x]- | ra#?- |
3 | P510530 reverse:11 | [szu-bu-lim?] sza# ta-asz-pu-ra-am | lim?] | [szu- | sza# |
4 | P510530 reverse:11 | [szu-bu-lim?] sza# ta-asz-pu-ra-am | lim?] | bu- | sza# |
5 | P510530 reverse:11 | [szu-bu-lim?] sza# ta-asz-pu-ra-am | lim?] | lim?] | sza# |
6 | P510550 obverse:9 | _ha-za-nu-um{sar}_ i-ka-am#? [x] x-al#-la-x-ma | am#? | [x] | al#- |
7 | P510565 reverse:4 | [a-na?] _e2_ {d}utu szu-ri-ba-am# | na?] | [a- | am# |
8 | P510565 reverse:4 | [a-na?] _e2_ {d}utu szu-ri-ba-am# | na?] | na?] | am# |
9 | P510567 reverse:2 | [...] ia#? _sag-us2_ | ia#? | [...] | ia#? |
10 | P510586 obverse:11 | [{disz}]i-na-e2-ul-masz-numun# a-na#? [ma-ah-ri-ka?] | na#? | [{disz}] | numun# |
We can also require the things to be adjacent.
query = """
line
sign question=1
sign missing=1
<: sign damage=1
"""
results = A.search(query)
A.table(results, end=10, baseTypes="sign")
A.show(
results,
end=10,
baseTypes="sign",
colorMap={0: "", 2: "cyan", 3: "magenta", 4: "lightsalmon"},
)
0.60s 84 results
n | p | line | sign | sign | sign |
---|---|---|---|---|---|
1 | P510530 obverse:5 | {disz}[{d}x]-ra#?-bi il-qe2-a-am-ma | ra#?- | x]- | ra#?- |
2 | P510530 reverse:11 | [szu-bu-lim?] sza# ta-asz-pu-ra-am | lim?] | lim?] | sza# |
3 | P510567 reverse:2 | [...] ia#? _sag-us2_ | ia#? | [...] | ia#? |
4 | P510586 obverse:27 | [asz-szum ha]-bil2#?-we?-du-um sza asz-pu-ra#-[ak-kum] | bil2#?- | ha]- | bil2#?- |
5 | P510586 obverse:27 | [asz-szum ha]-bil2#?-we?-du-um sza asz-pu-ra#-[ak-kum] | we?- | ha]- | bil2#?- |
6 | P510588 left:1':2 | [i?]-na# zimbir{ki#} | [i?]- | [i?]- | na# |
7 | P510597 obverse:12 | [a?-na] babila2#[{ki}] | [a?- | na] | babila2# |
8 | P510657 envelope - obverse:1 | [tup]-pi#? a-hi-i3-li2-ia | pi#? | [tup]- | pi#? |
9 | P510660 reverse:7 | [t,e4?]-em# _a-sza3_ ma-la a-di i-na-an-na | [t,e4?]- | [t,e4?]- | em# |
10 | P510662 reverse:1 | [1(disz)?] _tug2#?_ a-la-qe2-am | [1(disz)?] | [1(disz)?] | _tug2#?_ |
result 1
result 2
result 3
result 4
result 5
result 6
result 7
result 8
result 9
result 10
Finally, we make the three things fully adjacent in fixed order:
query = """
line
sign question=1
<: sign missing=1
<: sign damage=1
"""
results = A.search(query)
A.table(results, end=10, baseTypes="sign")
A.show(results, end=10, colorMap={0: "", 2: "cyan", 3: "magenta", 4: "lightsalmon"})
0.59s 7 results
n | p | line | sign | sign | sign |
---|---|---|---|---|---|
1 | P510597 obverse:12 | [a?-na] babila2#[{ki}] | [a?- | na] | babila2# |
2 | P292756 obverse:21 | [sza? (...)] it#-ti-ia u2-ul in-nam-ru-ma | [sza? | (...)] | it#- |
3 | P313393 obverse:3 | um-ma ni?-[...]-ma# | ni?- | [...]- | ma# |
4 | P386470 reverse:15' | [u2?-sze]-et#-bu-u2-[szu?] | [u2?- | sze]- | et#- |
5 | P305755 reverse:3 | u3 KA#?-[x]-UM#? a-hu-u2-szu qa2-ta-ti-szu-ma | KA#?- | [x]- | UM#? |
6 | P307155 obverse:11 | mi-im#-ma# sza ta#-GA?-[x]-BU#? [x] | GA?- | [x]- | BU#? |
7 | P308015 reverse:2 | {disz}UD?-[x] {ki#}-li-ib-lu-ut, | UD?- | [x] | {ki#}- |
result 1
result 2
result 3
result 4
result 5
result 6
result 7
We would like to see the original ATF and the flags for signs.
The way to do that, is to perform a A.prettySetup(features)
first.
We concentrate on one specific result.
A.displaySetup(extraFeatures="atf flags")
A.show(
results,
start=4,
end=4,
baseTypes="sign",
colorMap={0: "", 2: "cyan", 3: "magenta", 4: "lightsalmon"},
)
result 4
The features without meaningful values have been left out. We can also change that by passing a set of values we think are not meaningful. The default set is
{None, 'NA', 'none', 'unknown'}
A.displaySetup(noneValues=set())
A.show(
results,
start=4,
end=4,
baseTypes="sign",
colorMap={0: "", 2: "cyan", 3: "magenta", 4: "lightsalmon"},
)
result 4
This makes clear that it is convenient to keep None
in the noneValues
:
A.displaySetup(noneValues={None})
A.show(
results,
start=4,
end=4,
baseTypes="sign",
colorMap={0: "", 2: "cyan", 3: "magenta", 4: "lightsalmon"},
)
result 4
We can even choose to suppress other values, e.g. the value 1.
That will remove all the features such as question
, missing
.
A.displaySetup(noneValues={None, "NA", "unknown", 1})
A.show(
results,
start=4,
end=4,
baseTypes="sign",
colorMap={0: "", 2: "cyan", 3: "magenta", 4: "lightsalmon"},
)
result 4
In the rest of the notebook we stick to our normal setup, so we reset the extra features.
A.displayReset()
A.show(
results,
start=4,
end=4,
baseTypes="sign",
colorMap={0: "", 2: "cyan", 3: "magenta", 4: "lightsalmon"},
)
result 4
In earlier displays we saw the types of signs, because the query mentioned it.
Suppose we want to display the type also here, then we can modify the query by mentioning the feature type
.
But we do not want to impose extra limitations, so we say type*
, meaning: no conditions on type whatsoever.
query = """
line
sign question=1 type*
<: sign missing=1
<: sign damage=1
"""
results = A.search(query)
A.show(
results, start=4, end=4, colorMap={0: "", 2: "cyan", 3: "magenta", 4: "lightsalmon"}
)
We do not see the features, because they are sign
features, and our display stops at the word
level.
But we can improve on that:
A.show(
results,
start=4,
end=4,
baseTypes="sign",
colorMap={0: "", 2: "cyan", 3: "magenta", 4: "lightsalmon"},
)
result 4
So far we have show()
n the results of searches.
But you can also construct your own tuples and show them.
Whereas you can use search to get a pretty good approximation of what you want, most of the times you do not arrive precisely at your destination.
Here is an example where we use search to come close, and then work our way to produce the end result.
We look for lines that have more missing signs than damaged signs.
In our search templates we cannot formulate that a feature has different values on two nodes in the template. We could spell out all possible combinations of values and make a search template for each of them, but that is needlessly complex.
Let's first use search to find all clauses containing missing and damaged signs.
query = """
line
sign missing
sign damage
"""
results = A.search(query)
0.33s 9900 results
Now the hand coding begins. We are going to extract the tuples we want.
lines = {}
for (l, m, d) in results:
lines.setdefault(l, (set(), set()))
lines[l][0].add(m)
lines[l][1].add(d)
print(f"{len(lines)} lines")
3031 lines
Now we have all lines with both missing and damaged signs, without duplicates.
For each line we have a set with its missing signs and one with its damaged signs.
We filter in order to retain the lines with more missing than damaged signs. We put all missing signs in one big set and all damaged signs in one big set.
answer = []
missing = set()
damage = set()
for (l, (m, d)) in lines.items():
if len(m) > len(d):
answer.append((l, *m, *d))
missing |= m
damage |= d
len(answer)
1345
answer[0]
(230894, 955, 956, 954)
We are going to make a dictionary of highlights: one color for the missing signs and one for the damaged.
highlights = {}
colorM = "lightsalmon"
colorD = "mediumaquamarine"
for s in missing:
highlights[s] = colorM
for s in damage:
highlights[s] = colorD
And now we can show them:
A.table(answer, start=1, end=10, highlights=highlights)
n | p | line | sign | sign | sign | ||||||
---|---|---|---|---|---|---|---|---|---|---|---|
1 | P509377 reverse:2 | _sze_ u3 _ku3-babbar_ ad-di-na-ak#-[kum-ma] | [kum- | ma] | ak#- | ||||||
2 | P481192 obverse:10' | _a-sza3_ u2-ul e-ri#-[isz ...] | [isz | ...] | ri#- | ||||||
3 | P481192 obverse:13' | [x (x)] x-a-tum it-ta#-za-az-za i-mu-ut-ta | [x | (x)] | ta#- | ||||||
4 | P481192 obverse:14' | [...] x tup-pu# [x x] a-na# ma#-har a-[wi-le-e] | [...] | [x | x] | [wi- | le- | e] | na# | ma#- | pu# |
5 | P481192 reverse:13' | [x (x)] x-ma a-na ma-ah-ri#-ka# il-li-kam x x [(x) (x)] | (x)] | [(x) | [x | (x)] | ri#- | ka# | |||
6 | P389256 obverse:4' | uz-ni-ia li#-[x ...] | [x | ...] | li#- | ||||||
7 | P389256 left:2 | [...] i-di-in# [...] | [...] | [...] | in# | ||||||
8 | P510527 reverse:1 | [a-wi]-lu#-u2 i-na mu-uh2-hi ip-qu2-i3-li2-szu _di-ku5_ | [a- | wi]- | lu#- | ||||||
9 | P510528 reverse:1 | [x x] x x ra# [...] [tup]-pi2#-im# lu x [x x] | [x | x] | [...] | [tup]- | [x | x] | pi2#- | im# | ra# |
10 | P510530 obverse:3 | [um-ma {d}na-bi]-um#-na-s,i-ir-ma | [um- | ma | {d} | na- | bi]- | um#- |
As you see, you have total control.
All chapters:
See the cookbook for recipes for small, concrete tasks.
CC-BY Dirk Roorda