Search in Text-Fabric is a template based way of looking for structural patterns in your dataset.
It is inspired by the idea of topographic query, as worked out in MQL which has been implemented in Emdros. See also pitfalls of MQL
Within Text-Fabric we have the unique possibility to combine the ease of formulating search templates for complicated syntactical patterns with the power of programmatically processing the results.
This notebook will show you how to get up and running.
See the notebook searchFromMQL for examples how MQL queries can be expressed in Text-Fabric search.
Search is a powerful feature for a wide range of purposes.
Quite a bit of the implementation work has been dedicated to optimize performance. Yet I do not pretend to have found optimal strategies for all possible search templates. Some search tasks may turn out to be somewhat costly or even very costly.
That being said, I think search might turn out helpful in many cases, especially by reducing the amount of hand-coding needed to work with special subsets of your data.
Search is as simple as saying (just an example)
results = B.search(template)
B.show(results)
See all ins and outs in the search template docs.
For the BHSA, you use the B
API, which relies on the more generic
S
api
of Text-Fabric.
Later on, when we look under the hood, we shall occasionally fall back on the S
API.
The ins and outs of installing Text-Fabric, getting the corpus, and initializing a notebook are explained in the start tutorial.
%load_ext autoreload
%autoreload 2
from tf.fabric import Fabric
from tf.extra.bhsa import Bhsa
We also load the module with parallels between verses, because its main feature, crossref
, is an edge feature that contains values. The BHSAA itself does not have such edge features.
In order to get this data, in a terminal navigate to ~/github/etcbc
and say
git clone https://github.com/etcbc/parallels
VERSION = '2017'
DATABASE = '~/github/etcbc'
BHSA = f'bhsa/tf/{VERSION}'
PARA = f'parallels/tf/{VERSION}'
TF = Fabric(locations=[DATABASE], modules=[BHSA, PARA], silent=False )
This is Text-Fabric 5.6.3 Api reference : https://dans-labs.github.io/text-fabric/Api/General/ Tutorial : https://github.com/Dans-labs/text-fabric/blob/master/docs/tutorial.ipynb Example data : https://github.com/Dans-labs/text-fabric-data 118 features found and 0 ignored
Let us just not load any specific features.
api = TF.load('', silent=True)
api.makeAvailableIn(globals())
Let's add extra power:
B = Bhsa(api, 'search', version=VERSION)
Documentation: BHSA Feature docs BHSA API Text-Fabric API 5.6.3 Search Reference
We start with the most simple form of issuing a query. Let's look for the proper nouns in 1 Samuel. We also want to show the clauses in which they occur.
All work involved in searching takes place under the hood.
query = '''
book book=Samuel_I
clause
word sp=nmpr
'''
results = B.search(query)
B.table(results, end=10)
0.79s 1868 results
n | book | clause | word |
---|---|---|---|
1 | 1_Samuel | וַיְהִי֩ אִ֨ישׁ אֶחָ֜ד מִן־הָרָמָתַ֛יִם צֹופִ֖ים מֵהַ֣ר אֶפְרָ֑יִם | אֶפְרָ֑יִם |
2 | 1_Samuel | וּשְׁמֹ֡ו אֶ֠לְקָנָה בֶּן־יְרֹחָ֧ם בֶּן־אֱלִיה֛וּא בֶּן־תֹּ֥חוּ בֶן־צ֖וּף אֶפְרָתִֽי׃ | אֶ֠לְקָנָה |
3 | 1_Samuel | וּשְׁמֹ֡ו אֶ֠לְקָנָה בֶּן־יְרֹחָ֧ם בֶּן־אֱלִיה֛וּא בֶּן־תֹּ֥חוּ בֶן־צ֖וּף אֶפְרָתִֽי׃ | יְרֹחָ֧ם |
4 | 1_Samuel | וּשְׁמֹ֡ו אֶ֠לְקָנָה בֶּן־יְרֹחָ֧ם בֶּן־אֱלִיה֛וּא בֶּן־תֹּ֥חוּ בֶן־צ֖וּף אֶפְרָתִֽי׃ | אֱלִיה֛וּא |
5 | 1_Samuel | וּשְׁמֹ֡ו אֶ֠לְקָנָה בֶּן־יְרֹחָ֧ם בֶּן־אֱלִיה֛וּא בֶּן־תֹּ֥חוּ בֶן־צ֖וּף אֶפְרָתִֽי׃ | תֹּ֥חוּ |
6 | 1_Samuel | וּשְׁמֹ֡ו אֶ֠לְקָנָה בֶּן־יְרֹחָ֧ם בֶּן־אֱלִיה֛וּא בֶּן־תֹּ֥חוּ בֶן־צ֖וּף אֶפְרָתִֽי׃ | צ֖וּף |
7 | 1_Samuel | שֵׁ֤ם אַחַת֙ חַנָּ֔ה | חַנָּ֔ה |
8 | 1_Samuel | וְשֵׁ֥ם הַשֵּׁנִ֖ית פְּנִנָּ֑ה | פְּנִנָּ֑ה |
9 | 1_Samuel | לִפְנִנָּה֙ יְלָדִ֔ים | פְנִנָּה֙ |
10 | 1_Samuel | וּלְחַנָּ֖ה אֵ֥ין יְלָדִֽים׃ | חַנָּ֖ה |
The hyperlinks take us all to the beginning of the book of 1 Samuel, but it is more convenient to jump to the precise verse in which the result occurs. It is better to put hyperlinks underneath the words instead. That is column 3.
Note in passing that we can choose start and/or end points in the results list.
B.table(results, start=8, end=13, linked=3)
n | book | clause | word |
---|---|---|---|
8 | 1_Samuel | וְשֵׁ֥ם הַשֵּׁנִ֖ית פְּנִנָּ֑ה | פְּנִנָּ֑ה |
9 | 1_Samuel | לִפְנִנָּה֙ יְלָדִ֔ים | פְנִנָּה֙ |
10 | 1_Samuel | וּלְחַנָּ֖ה אֵ֥ין יְלָדִֽים׃ | חַנָּ֖ה |
11 | 1_Samuel | וְלִזְבֹּ֛חַ לַיהוָ֥ה צְבָאֹ֖ות בְּשִׁלֹ֑ה | יהוָ֥ה |
12 | 1_Samuel | וְלִזְבֹּ֛חַ לַיהוָ֥ה צְבָאֹ֖ות בְּשִׁלֹ֑ה | שִׁלֹ֑ה |
13 | 1_Samuel | וְשָׁ֞ם שְׁנֵ֣י בְנֵֽי־עֵלִ֗י חָפְנִי֙ וּפִ֣נְחָ֔ס כֹּהֲנִ֖ים לַיהוָֽה׃ | עֵלִ֗י |
We can show the results more fully with show()
.
B.show(results, start=1, end=3)
As you see, when a verse has multiple results inside, they are shown in the same verse display.
This is not always desirable, and we can display the list in a result-by-result as well.
Here is an example.
Note that you can have comments in a search template. Comment lines start with a #
.
query = '''
book book=Genesis
chapter chapter=1
verse verse=1
sentence
# order is not important!
word nu=sg
word nu=pl
'''
results = B.search(query)
B.table(results, linked=3)
1.43s 6 results
n | book | chapter | verse | sentence | word | word |
---|---|---|---|---|---|---|
1 | Genesis | Genesis 1 | Genesis 1:1 בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | רֵאשִׁ֖ית | אֱלֹהִ֑ים |
2 | Genesis | Genesis 1 | Genesis 1:1 בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | רֵאשִׁ֖ית | שָּׁמַ֖יִם |
3 | Genesis | Genesis 1 | Genesis 1:1 בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בָּרָ֣א | אֱלֹהִ֑ים |
4 | Genesis | Genesis 1 | Genesis 1:1 בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בָּרָ֣א | שָּׁמַ֖יִם |
5 | Genesis | Genesis 1 | Genesis 1:1 בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | אָֽרֶץ׃ | אֱלֹהִ֑ים |
6 | Genesis | Genesis 1 | Genesis 1:1 בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | אָֽרֶץ׃ | שָּׁמַ֖יִם |
There are two plural and three singular words in Genesis 1:1. Search templates do not specify order, so all six combinations qualify as results.
Let's show them:
B.show(results)
verse 1
You see one verse display and all words highlighted. However, the exact results, which are pairs of words, cannot be readily read off. You might want to show the 6 individual results.
Note that we can apply different highlight colors to different parts of the result. The words in the pair are member 5 and 6 of the result tuples. The members that we do not map, will not be highlighted. The members that we map to the empty string will be highlighted with the default color.
NB: Choose your colors from the CSS specification.
B.show(results, condensed=False, colorMap={4: '', 5: 'cyan', 6: 'magenta'})
book 1
book 2
book 3
book 4
book 5
book 6
Color mapping works best for uncondensed results. If you condense results, some nodes may occupy different positions in different results. It is unpredictable which color will be used for such nodes:
B.show(results, condensed=True, colorMap={4: '', 5: 'cyan', 6: 'magenta'})
verse 1
You can specify to what container you want to condense. By default, everything is condensed to verses.
Let's change that to phrases:
B.show(results, condensed=True, condenseType='phrase', colorMap={4: '', 5: 'cyan', 6: 'magenta'})
phrase 1
phrase 2
phrase 3
phrase 4
You can stipulate an order on the words in your template. You only have to put a relational operator between them. Say we want only results where the plural follows the singular.
query = '''
book book=Genesis
chapter chapter=1
verse verse=1
sentence
word nu=sg
< word nu=pl
'''
results = B.search(query)
B.table(results)
1.28s 4 results
n | book | chapter | verse | sentence | word | word |
---|---|---|---|---|---|---|
1 | Genesis | Genesis 1 | Genesis 1:1 בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | רֵאשִׁ֖ית | אֱלֹהִ֑ים |
2 | Genesis | Genesis 1 | Genesis 1:1 בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | רֵאשִׁ֖ית | שָּׁמַ֖יִם |
3 | Genesis | Genesis 1 | Genesis 1:1 בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בָּרָ֣א | אֱלֹהִ֑ים |
4 | Genesis | Genesis 1 | Genesis 1:1 בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בָּרָ֣א | שָּׁמַ֖יִם |
We can also require the words to be adjacent.
query = '''
book book=Genesis
chapter chapter=1
verse verse=1
sentence
word nu=sg
<: word nu=pl
'''
results = B.search(query)
B.table(results)
B.show(results, condensed=False, colorMap={5: 'lightsalmon', 6: 'mediumaquamarine'})
1.33s 1 result
n | book | chapter | verse | sentence | word | word |
---|---|---|---|---|---|---|
1 | Genesis | Genesis 1 | Genesis 1:1 בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בְּרֵאשִׁ֖ית בָּרָ֣א אֱלֹהִ֑ים אֵ֥ת הַשָּׁמַ֖יִם וְאֵ֥ת הָאָֽרֶץ׃ | בָּרָ֣א | אֱלֹהִ֑ים |
book 1
We would like to see the gender, number and person for words.
The way to do that, is to perform a B.prettySetup(features)
first.
B.prettySetup(features='ps gn nu')
B.show(results, condensed=False, colorMap={5: 'lightsalmon', 6: 'mediumaquamarine'})
The features without meaningful values have been left out. We can also change that by passing a set of values we think are not meaningful. The default set is
{None, 'NA', 'none', 'unknown'}
B.prettySetup(features='ps gn nu', noneValues=set())
B.show(results, condensed=False, colorMap={5: 'lightsalmon', 6: 'mediumaquamarine'})
book 1
This makes clear that it is convenient to keep None
in the noneValues
:
B.prettySetup(features='ps gn nu', noneValues={None})
B.show(results, condensed=False, colorMap={5: 'lightsalmon', 6: 'mediumaquamarine'})
book 1
We can even choose to suppress other values, e.g. the male gender values and the singular number values.
B.prettySetup(features='ps gn nu', noneValues={None, 'NA', 'unknown', 'none', 'm', 'sg'})
B.show(results, condensed=False, colorMap={5: 'lightsalmon', 6: 'mediumaquamarine'})
In the rest of the notebook we stick to our normal setup, so we reset the extra features.
B.prettySetup()
B.show(results, condensed=False, colorMap={5: 'lightsalmon', 6: 'mediumaquamarine'})
book 1
So far we have show()
n the results of searches.
But you can also construct your own tuples and show them.
Whereas you can use search to get a pretty good approximation of what you want, most of the times you do not arrive precisely at your destination.
Here is an example where we use search to come close, and then work our way to produce the end result.
We look for clauses with a one-word subject that does not agree in number with its predicate.
In our search templates we cannot formulate that a feature has different values on two nodes in the template. We could spell out all possible combinations of values and make a search template for each of them, but that is needlessly complex.
Let's first use search to find all clauses containing a one word subject and a predicate. And, to narrow down it further, we require that the word in the subject and the verb in the predicate are marked for number.
(You may want to consult the feature docs, see the link at the start of the notebook, where Bhsa()
is called).
Note that the order of the phrases does not matter.
query = '''
clause
phrase function=Subj
=: word nu=sg|pl
:=
phrase function=Pred|PreO
word sp=verb
nu=sg|pl
'''
results = B.search(query)
2.61s 10639 results
Now the hand coding begins. We are going to extract the tuples we want.
wantedResults = tuple(
(subj, pred)
for (clause, phraseS, subj, phraseV, pred) in results
if F.nu.v(subj) != F.nu.v(pred)
)
print(f'{len(wantedResults)} filtered results')
469 filtered results
And now we can show them:
B.show(wantedResults, start=1, end=3, colorMap={1: 'lightsalmon', 2: 'mediumaquamarine'})
Now suppose that we want to highlight the non-qal verb forms with a different color.
We have to assing colors to the members of our tuples:
highlights = {}
for (subj, pred) in wantedResults:
highlights[subj] = 'lightsalmon'
highlights[pred] = 'mediumaquamarine' if F.vs.v(pred) == 'qal' else 'yellow'
Now we can call show with the highlights
parameter instead of the colorMap
parameter.
B.show(wantedResults, start=2, end=3, highlights=highlights)
As you see, you have total control.