You might want to consider the start of this tutorial.

Short introductions to other TF datasets:

• Dead Sea Scrolls,
• Old Babylonian Letters,

or the

• Quran

Gaps and spans¶

Searches often do not deliver the results you expect. Besides typos, lack of familiarity with the template formalism and bugs in the system, there is another cause: difficult semantics of the data.

Most users reason about phrases, clauses and sentences as if they are consecutive blocks of words. But in the BHSA this is not the case: each of these objects may have gaps.

Most of the time, verse boundaries coincide with the boundaries of sentences, clauses, and phrases. But not always, there are verse spanning sentences.

Note

These phenomena may wreak havoc with your intuitive reasoning about what search templates should deliver. Query templates do not require the objects to be consecutive and still they make sense. But that might not be your sense, unless you Mind the gap!

We are going to show these issues in depth.

Gaps¶

TF-search has no primitives to deal with gaps directly. Nodes correspond to textual objects such as words, phrases, clauses, verses, books. Usually these are consecutive sequences of one or more words, but in theory they can be arbitrary sets of slots.

And, as far as the BHSA corpus is concerned, in practice too. If we look at phrases, then the overwhelming majority is consecutive, without gaps, But there is also a substantial amount of phrases with gaps.

People that are familiar with MQL (see from MQL) may remember that in MQL you can search for a gap. The MQL query

SELECT ALL OBJECTS WHERE

[phrase FOCUS
    [word lex='L']
    [gap]
]

looks for a phrase with a gap in it (i.e. one or more consecutive words between the start and the end of the phrase that do not belong to the phrase). The query then asks additionally for those gap-containing phrases that have a certain word in front of the gap.

We want this too!

Find the gap¶

We start with a query that aims to get the same results as the MQL query above.

In our template, we require that there is a word wPreGap in the phrase that is just before the gap, a word wGap that comes right after, so it is in the gap, and hence does not belong to the phrase. But this all must happen before the last word wLast of the phrase.

Nice and quick. Let's see the results.

Let's color the word in the gap differently.

All gapped phrases¶

These were particular gaps. Now we want to get all gapped phrases.

We can just lift the special requirement that the preGapWord has to satisfy a special lexical condition.

Not too bad! We could wait for it. Here are some results.

If a phrase has multiple gaps, we encounter it multiple times in our results.

We show the two results in Genesis 7:21.

If we want just the phrases, and only once, we can run the query in shallow mode, see advanced:

Excursion¶

Sometimes there are two subphrases with exactly the same words in it. They only differ in their values for the feature rela. Let's find such a case.

We do have to show the subphrases, though.

Some types are hidden, let's find out which ones:

Let's pass a different set of hidden types when showing the results:

More research has shown that these subphrases differ in the presence of the rela feature.

The red one has a feature rela='par', the blue one not.

Two nodes with the same node type and the same slots. Yet: different nodes, different feature annotations.

At the moment I do not know why the encoders of the BHSA have chosen to do this.

A different query¶

We can make an equivalent query to get the gaps.

Experience has shown that this is a slow query, so we handle it with care.

This is a good example of a query that is slow to deliver even its first result. And that is bad, because it is such a straightforward query.

Why is this one so slow, while the previous one went so smoothly?

The crucial thing is the wGap word. In the latter template, wGap is not embedded in anything. It is constrained by wFirst < wGap and wGap < wLast. However, the way the search strategy works is by examining all possibilities for wFirst < wGap and only then checking whether wGap < wLast. The algorithm cannot check both conditions at the same time.

With embedding relations, things are better. Text-Fabric is heavily optimized to deal with embedding relationships.

In the former template, we see that the wGap is required to be adjacent to wPreGap, and this one is embedded in the phrase. Hence there are few cases to consider for wPreGap, and per instance there is only one wGap.

Lesson

Try to prevent the use of free floating nodes in your template that become constrained by other spatial relationships than embedding.

To the rescue¶

The former template had it right. Can we rescue the latter template?

We can assume that the phrase and the gap each contain a word in one and the same verse. Note that phrase and gap may belong to different clauses and sentences. We assume that a phrase cannot belong to more than two verses, so either the first or the last word of the phrase is in the same verse as a word in the gap.

We are going to run this query in shallow mode.

Shallow mode tends to be quicker, but that does not always materialize. The number of results agrees with the first query. Yet we have been lucky, because we required the word in the gap to be in the same verse as the first word in the phrase. What if we require if it is the last word in the phrase?

Then we would not have found all results.

So, this road, although doable, is much less comfortable, performance-wise and logic-wise.

Check the gaps¶

In this misty landscape of gaps we need some corroboration that we found the right results.

1. is every node in gapQueryResults a phrase?
2. does every phrase in the gapQueryResults have a gap?
3. is every gapped phrase contained in gapQueryResults?

We check all this by hand coding.

Here is a function that checks whether a phrase has a gap. If the distance between its end points is greater than the number of words it contains, it must have a gap.

Now we can perform the checks.

Note that by hand coding we can get the gapped phrases much more quickly and securely!

Custom sets for (non-)gapped phrases¶

We have obtained a set with all gapped phrases, and we have paid a price:

• either an expensive query,
• or an inconvenient bit of hand coding.

It would be nice if we could kick-start our queries using this set as a given. And that is exactly what we are going to do now.

We make two custom sets and give them a name, gapphrase for gapped phrases and conphrase for non-gapped phrases (consecutive phrases).

Suppose we want all verbs that occur in a gapped phrase.

Note that we have used the foreign name gapphrase in our search template, instead of phrase.

But we can still run search(), provided we tell it what we mean by gapphrase. We do that by passing the sets parameter to search(), which should be a dictionary of sets. Search will look up gapphrase in this dictionary, and will use its value, which should be a node set. That way, it understands that the expression gapphrase stands for the nodes in the given node set.

Here we go:

That looks good.

We can also apply feature conditions to gapphrase:

We reduce the details by setting the baseType to phrase. The highlighted phrases will now get a yellow background.

We reduce the details by setting the baseType to phrase_atom. The highlighted phrases will not get a yellow background now.

Two-phrase clauses¶

We can find the gaps, but do our minds always reckon with gaps? Gaps cause unexpected semantics. Here is a little puzzle.

Suppose we want to count the clauses consisting of exactly two phrases.

Here follows a little journey. We use a query to find the clauses, check the result with hand-coding, scratch our heads, refine the query, the hand-coding and our question until we are satisfied.

Attempt 1¶

By query¶

The following template should do it: a clause, starting with a phrase, followed by an adjacent phrase, which terminates the clause.

If we want to have the clauses only, we run it in shallow mode:

Note result 3 above: it seems we have 3 phrases. Yet there are only 2. We take a closer look:

One phrase is chunked into two phrase atoms, which are hidden by default. Let's make that more clear:

By hand¶

Let us check this with a piece of hand-written code. We want clauses that consist of exactly two phrases.

The difference¶

Strange, we end up with more cases. What is happening? Let us compare the results. We look at the first result where both methods diverge.

We put the difference finding in a little function.

Lo and behold:

• the hand-written code is right in a sense: this is a clause that consists exactly of two phrases.
• the query is also right in a sense: the two phrases are not adjacent: there is a gap in the clause between them!

Attempt 2¶

By hand¶

We modify the hand-written code such that only clauses qualify if the two phrases are adjacent.

The difference¶

Now we have less cases. What is going on?

Observe:

This clause has three phrases, but the third one lies inside the second one.

• the hand-written code is right in a sense: this clause has three phrases.
• the query is right in a sense: it contains two adjacent phrases that together span the whole clause.

Attempt 3¶

By query¶

Can we adjust the pattern to exclude cases like this? Yes, with custom sets, see advanced.

Instead of looking through all phrases, we can just consider non gapped phrases only.

Earlier in this notebook we have constructed the set of non-gapped phrases and put it under the name conphrase in the custom sets.

The difference¶

There is still a difference.