MQL versus TF-Query¶

Introduction¶

Bas Meeuse has written an excellent primer for exegetes. In a series of increasingly difficult examples he shows how one can use template based search to explore syntactic patterns in the text.

The ETCBC database makes this possible, especially by its incarnation as BHSA data.

In that context, there are two ways to perform template based search:

• the website SHEBANQ offers MQL queries that can be run and saved without installing anything;
• Text-Fabric offers complete computational control over the data, including template based queries, inspired by, but different from MQL

Both MQL and TF-Query have been developed in close connection with the ETCBC, MQL by Ulrik Sandborg-Petersen, as part of his database system Emdros, and Text-Fabric by Dirk Roorda.

MQL is older, dating from 2005, and it is implemented in the programming language C. Text-Fabric dates from 2016 and is implemented in Python(3). This time difference shows not only in various aspects of the MQL and TF-Query syntax, but also in other aspects of usability.

Comparison¶

The power of MQL and TF-Query are not comparable in that one is stronger than the other. Here are a few points.

Weighing¶

Here is how we weigh the differences:

• MQL has the Kleene-star, which TF-Query lacks. That is a serious advantage of MQL over TF-Query.
• TF-Query has more control over quantifiers, i.e. extra conditions on an object in terms of the existence or absence of related objects.
• TF has expressions for spatial relationships between objects, and offers better control; that said, MQL has a relative notion of adjacency (which holds around gaps) that is difficult to reproduce in TF-Query
• For non-programmers, the mileage with MQL surpasses that of TF-Query, in practice. But, the further you go in MQL, the more difficult and convoluted the queries become. You tend to get long queries in which subqueries are repeated over and over again.
• For programmers TF-Query offers several extras, so that results that cannot be obtained by MQL can be achieved by a mixture of TF-Query and pieces of hand-coding.
• The display of query results in TF surpasses SHEBANQ, which does not have a notion of individual result.
• As to performance: Emdros is implemented in C, and has probably a very good performance. However, Emdros relies on an underlying database engine, such as SQLite or MySQL, which creates overhead. TF-Query is done in pure Python (no Cython yet!) but with lots of care that Python is used at its maximum efficiency. Much data access happens at C speed, in fact. There is no other software between the data and the metal. But we should definitely measure and compare the performances of MQL and TF-Query for a few equivalent queries.

Summing up¶

Research questions usually go further than what can be expressed in queries, no matter how sophisticated the query language. Writing a query is just a first step, and usually more than one query is needed to gather in the evidence that can confirm or reject a hypothesis.

Queries tend to be difficult and error prone. Therefore we need checking and cross-checking. When we check a query by means of another query, we have to make sure that the other query is right as well. We need good visual access to query result, and we need the tools to combine and filter query results.

In my opinion, TF with TF-Query supports this complex practice significantly better than SHEBANQ with MQL. SHEBANQ is often a good start to get an idea, and it can also be used to advertise aspects of a research question after the work has been done, but it does not support all the intermediate steps. The big bonus of SHEBANQ is that it does not require local installation nor programming skills.

But when it comes to the nitty-gritty in the middle of research, there comes a moment to pay the price: if the computer is your tool, then you need to acquire the number-one skill to control information: programming.

Examples¶

The purpose of this primer is to help people to transition from MQL to TF-Query. To that end we reproduce the MQL queries in the SHEBANQ tutorial by Bas Meeuse by means of Text-Fabric queries, and discuss the subtle and not so subtle distinctions that we will encounter.

Reference¶

• MQL Query Guide (PDF)
• TF Query Guide (HTML)
• Other MQL-TF queries (notebook)

Author¶

Dirk Roorda (dirk.roorda@dans.knaw.nl)

Copyright: No restrictions. Attribution appreciated. CC-BY.

Loading¶

We load the Text-Fabric program and the BHSA data.

The example queries¶

We will compare the results of MQL queries, performed by SHEBANQ with results obtained from TF-Queries.

To that end we have downloaded the CSV results of all example queries in the tutorial by Bas. We have written a convenience function to get the statistics of the SHEBANQ query: the number of results, verses, and words.

The number of results cannot be determined from the SHEBANQ data. I have taken it from the SHEBANQ interface and stored it. The number of verses and words are read from the exported files from SHEBANQ.

It may be possible that in the meanwhile queries have changed on SHEBANQ. This tutorial may not reflect that. It is based on what was in SHEBANQ around 2021-03-21.

Example 1¶

Bas Meeuse: Example 1: Moses starts the speech

[clause FOCUS
  [word lex = '>MR[' OR lex = 'DBR[']
  [phrase function = Subj
    [word lex = 'MCH=/']
  ]
  ..
  [phrase function IN (Cmpl)
    [word lex = 'JHWH/' OR lex = '>LHJM/']
  ]
]

We need to do some effort to count the distinct verses and focused words in the results.

The convenience function getTfVerse() takes care of this.

The words that SHEBANQ displays are the words that belong to the objects marked by the keyword FOCUS. In TF-Query, the results are tuples where each member corresponds to an object in the query, in that order.

So when we count the words from TF-Query results, we must specify which members of the tuple have FOCUS. We do that by giving the index numbers of the focused objects. (Index numbers start at 0).

The numbers agree exactly.

This is not a rigorous proof that the MQL results are identical to the TF results, but it is a very good indication.

However, we will also check that all words and verses in the shebanq results are exactly the words and verses in the TF results. We try to find the first difference.

Finally, here are the results of the TF query.

Example 2¶

This is a difficult case. We do it in a separate notebook: example 2.

Example 3¶

Bas Meeuse: Example 3: Whose people?

[phrase_atom FOCUS
  [word AS P sp = prps]
  ..
  [word lex = "W" OR lex = ">W"]
  ..
  [word prs !~ "a" AND prs_ps = P.ps]
]

Note that in TF-Query we do not have the value !~ regex comparison, i.e. value is not a match of regex. We do have the value ~ regex comparison, so we have to find a regex that says: "value does not contain a". That is the same as: every character in value is not an a:

^[^a]*$

is a regex that matches:

• the beginning of the value
• zero or more times a character which is not an a
• the end of the value.

See the Python re docs.

Example 4¶

Willem van Peursen: Judges 5.1 (Sample query)

[clause
  [phrase FOCUS function=Pred
    [word sp=verb AND nu=sg AND gn=f]
  ]
  ..
  [phrase FOCUS function=Subj
    [word sp=conj]
  ]
]

Note that the focus in the query is on both of the phrase objects. In the TF-Query results, the members of the result tuples that correspond to these phrases are at index 1 and 3 (counting from 0).

Let's show that. Here is the first result tuple:

These are nodes. Nodes are just numbers through which all information about objects kan be retrieved, like bar codes.

Here are the types of these nodes:

So the focused objects are 1 and 3, and this is what we pass to getTfVerses():

Example 5¶

Oliver Glanz: DHQ article: taking a woman

[clause
  [phrase FOCUS function IN (Pred,PreC)
    [word lex = "LQX["]
  ]
  ..
  [phrase function = Objc
    [word FOCUS gn =  f AND nu=sg AND sp=nmpr]
  ]
]

Example 6¶

Bas Meeuse: Example 6: Turning point in Deut. 29,3?

[clause focus
  [phrase function = Nega]
  [phrase typ = VP
    [word ps = p3]
  ]
  [phrase function = Subj]
  ..
  [phrase function = Time
    [word first lex = "<D"]
  ]
]

Example 7¶

This is a case where we need two TF-queries. We do it in a separate notebook: example 7.

Example 8¶

Reinoud Oosting: to withhold + from

[clause
  [phrase function = Pred OR function = PreC
    [word FOCUS sp = verb AND vs = qal AND lex = "MN<[" ]
  ]
  ..
  [phrase function = Cmpl
    [word FOCUS sp = prep ]
  ]
]
OR
[clause
  [phrase function = Cmpl
    [word FOCUS sp = prep ]
  ]
  ..
  [phrase function = Pred OR function = PreC
    [word FOCUS sp = verb AND vs = qal AND lex = "MN<["]
  ]
]

Note, that as in example 7, the OR is used to specify different orders of the same objects. Since in TF-Query the order is not important, by default, we do not need the OR, leading to a much simpler query.

Example 9¶

This is a difficult case. We do it in a separate notebook: example 9.

Example 10a¶

This is a difficult case. We do it in a separate notebook: example 10a.

Example 10b¶

This is a case where we need two TF-queries. We do it in a separate notebook: example 10b.