You might want to consider the start of this tutorial.

Gaps and spans¶

Searches often do not deliver the results you expect. Besides typos, lack of familiarity with the template formalism and bugs in the system, there is another cause: difficult semantics of the data.

Most users reason about phrases, clauses and sentences as if they are consecutive blocks of words. But in the BHSA this is not the case: each of these objects may have gaps.

Most of the time, verse boundaries coincide with the boundaries of sentences, clauses, and phrases. But not always, there are verse spanning sentences.

These phenomena may wreak havoc with your intuitive reasoning about what search templates should deliver.

We are going to show these issues in depth.

Gaps¶

Search has no direct primitives to deal with gaps. Nodes correspond to textual objects such as word, phrases, clauses, verses, book. Usually these are consecutive sequences of words, but in theory they can be arbitrary sets of slots.

And, as far as the BHSA corpus is concerned, in practice too. If we look at phrases, then the overwhelming majority is consecutive, so without gaps, but gaps in phrases do occur and are not even exceptional.

People that are familiar with MQL (see fromMQL may remember that in MQL you can search for a gap. The MQL query

SELECT ALL OBJECTS WHERE

[phrase FOCUS
    [word lex='L']
    [gap]
]

looks for a phrase with a gap in it (i.e. one or more consecutive words between the start and the end of the phrase that do not belong to the phrase). The query then asks additionally for those gap-containing phrases that have a certain word in front of the gap.

We want this too!

Note

The fact that sentences, clauses, and phrases may not be coherent chunks wreaks havoc with your basic intuitions about textual objects. Our query templates do not require the objects to be consecutive and still they make sense. But that might not be your sense, unless you Mind the gap!

Find the gap¶

We start with a query that aims to get the same results as the MQL query above.

A phrase p is gapped, if there is a word between two of its words that does not belong to it.

In our template, we require that there is a word wPreGrap in the phrase that is just before the gap, a word wGap that comes right after, so it is in the gap, and hence does not belong to the phrase. But this all must happen before the last word wLast of the phrase.

Nice and quick. Let's see the results.

Let's color the word in the gap differently.

All gapped phrases¶

These were particular gaps. Now we want to get all gapped phrases.

We can just lift the special requirement that the preGapWord has to satisfy a special lexical condition.

Not too bad! We could wait for it. Here are some results.

If a phrase has multiple gaps, we encounter it multiple times in our results.

We show the two condensed results in Genesis 7:21.

If we want just the phrases, and only once, we can run the query in shallow mode, see advanced:

A different query¶

We can make an equivalent query to get the gaps.

Experience has shown that this is a slow query, so we handle it with care.

This is a good example of a query that is slow to deliver even its first result. And that is bad, because it is such a straight-forward query.

Why is this one so slow, while the previous one went so smoothly?

The crucial thing is the wGap word. In the latter template, wGap is not embedded in anything. It is constrained by wFirst < wGap and wGap < wLast. However, the way the search strategy works is by examining all possibilities for wFirst < wGap and only then checking whether wGap < wLast. The algorithm cannot check both conditions at the same time.

With embedding relations, things are better. Text-Fabric is heavily optimized to deal with embedding relationships.

In the former template, we see that the wGap is required to be adjacent to wPreGap, and this one is embedded in the phrase. Hence there are few cases to consider for wPreGap, and per instance there is only one wGap.

Lesson

Try to prevent the use of free floating nodes in your template that become constrained by other spatial relationships than embedding.

To the rescue¶

The former template had it right. Can we rescue the latter template?

We can assume that the phrase and the gap both contain a word that belongs to the same verse. Note that phrase and gap may belong to different clauses and sentences. We assume that a phrase cannot belong to more than two verses, so either the first or the last word of the phrase is in the same verse as a word in the gap.

We are going to run this query in shallow mode.

Shallow mode tends to be quicker, but that does not always materialize. The number of results agrees with the first query. Yet we have been lucky, because we required the word in the gap to be in the same verse as the first word in the phrase. What if we require if it is the last word in the phrase?

Then we would not have found all results.

So, this road, although doable, is much less comfortable, performance-wise and logic-wise.

Check the gaps¶

In this misty landscape of gaps we need some corroboration that we found the right results.

1. is every node in gapQueryResults a phrase?
2. does every phrase in the gapQueryResults have a gap?
3. is every gapped phrase contained in gapQueryResults?

We check all this by hand coding.

Here is a function that checks whether a phrase has a gap. If the distance between its end points is greater than the number of words it contains, it must have a gap.

Now we can perform the checks.

Note that by hand coding we can get the gapped phrases much more quickly and securely!

Custom sets for (non-)gapped phrases¶

We have obtained a set with all gapped phrases, and we have paid a price:

• either an expensive query,
• or an inconvenient bit of hand coding.

It would be nice if we could kick-start our queries using this set as a given. And that is exactly what we are going to do now.

We make to custom sets and give them a name, one for gapped phrases and one for non-gapped phrases.

Suppose we want all verbs that occur in a gapped phrase.

Note that we have used the foreign name gapphrase in our search template, instead of phrase.

But we can still run search(), provided we tell it what we mean by gapphrase. We do that by passing the sets parameter to search(), which should be a dictionary of sets. Search will look up gapphrase in this dictionary, and will use its value, which should be a node set. That way, it understands that the expression gapphrase stands for the nodes in the given node set.

Here we go:

That looks good.

We can also apply feature conditions to gapphrase:

Two-phrase clauses¶

We can find the gaps, but do our minds always reckon with gaps? Gaps cause unexpected semantics. Here is a little puzzle.

Suppose we want to count the clauses consisting of exactly two phrases.

Here follows a little journey. We use a query to find the clauses, check the result with hand-coding, scratch our heads, refine the query, the hand-coding and our question until we are satisfied.

Attempt 1¶

By query¶

The following template should do it: a clause, starting with a phrase, followed by an adjacent phrase, which terminates the clause.

If we want to have the clauses only, we run it in shallow mode:

By hand¶

Let us check this with a piece of hand-written code. We want clauses that consist of exactly two phrases.

The difference¶

Strange, we end up with too many cases. What is happening? Let us compare the results. We look at the first result where both methods diverge.

We put the difference finding in a little function.

Lo and behold:

• the hand-written code is right in a sense: this is a clause that consists exactly of two phrases.
• the query is also right in a sense: the two phrases are not adjacent: there is a gap in the clause between them!

Attempt 2¶

By hand¶

We modify the hand-written code such that only clauses qualify if the two phrases are adjacent.

The difference¶

Now we have too few cases. What is going on?

Observe:

This clause has three phrases, but the third one lies inside the second one.

• the hand-written clause is right in a sense: this clause has three phrases.
• the query is right in a sense: it contains two adjacent phrases that together span the whole clause.

Attempt 3¶

By query¶

Can we adjust the pattern to exclude cases like this? Yes, with custom sets, see advanced.

Instead of looking through all phrases, we can just consider non gapped phrases only.

Earlier in this notebook we have constructed the set of non-gapped phrases and put it under the name conphrase in the custom sets.

The difference¶

There is still a difference.

Observe:

This clause has two phrases, the second one has a gap, which coincides with a gap in the clause.

• the hand-written clause is right in a sense: this clause has two phrases, consecutive, and they span the whole clause, nothin left out.
• the query is right in a sense: the second phrase is not consecutive.

Attempt 4¶

By hand¶

We modify the hand-written code, so that only consecutive clauses qualify.

The difference¶

Now the number of results agree. But are they really the same?

Conclusion¶

It took four attempts to arrive at the final concept of things that we were looking for.

Sometimes the search template had to be modified, sometimes the hand-written code.

The interplay and systematic comparison between the attempts helped to spot all relevant configurations of phrases within clauses.

Spans¶

Here is another cause of wrong query results: there are sentences that span multiple verses. Such sentences are not contained in any verse. That makes that they are easily missed out in queries.

We describe a scenario where that happens.

Mother clauses¶

A clause and its mother do not have to be in the same verse. We are going to fetch are the cases where they are in different verses.

All mother clauses¶

But first we fetch all pairs of clauses connected by a mother edge.

Mother in another verse¶

Now we modify the query to the effect that mother and daughter must sit in distinct verses.

Mother in same verse¶

As a check, we modify the latter query and require v1 and v2 to be the same verse, to get the mother pairs of which both members are in the same verse.

The difference¶

Let's check if the numbers add up:

• the first query asked for all pairs
• the second query asked for pairs with members in different verses
• the third query asked for pairs with members in the same verse

Then the results of the second and third query combined should equal the results of the first query.

That makes sense.

Still, let's check:

The numbers do not add up. We are missing cases. Why?

Clauses may cross verse boundaries. In that case they are not part of a verse, and hence our latter two queries do not detect them. Let's count how many verse boundary crossing clauses there are.

Some of these verse spanning clauses do not have mothers or are not mothers. Let's count the cases where two clauses are in a mother relation and at least one of them spans a verse.

We need two queries for that. These queries are almost similar. One retrieves the clause pairs where the mother crosses verse boundaries, and the other where the daughter does so.

But we are programmers. We do not have to repeat ourselves:

We may find the mother clause pairs in which it least one member is verse spanning by hand-coding in an easier way:

Starting with the set of all mother pairs, we filter out any pair that has a verse spanner.

And, to be completely sure:

By custom sets¶

If we are content with the clauses that do not span verses, we can put them in a set, and modify the queries by replacing clause by conclause and bind the right set to it.

Here we go. In one cell we run the queries to get all pairs, the mother-daughter-in-separate-verses pairs, and the mother-daughter-in-same-verses pair and we do the math of checking.

Lessons¶

• mix programming with composing queries;
• a good way to do so is custom sets;
• use programming for processing results;
• find the balance between queries and hand-coding.

Next¶

You have now finished the search tutorial.

If you are interested in reproducing MQL queries in Text-Fabric search templates, see fromMQL.

basic advanced relations quantifiers rough gaps