<img align="right"src="images/dans-small.png"/>

Trees - for BHSA data (Hebrew)¶

This notebook composes syntax trees out of the BHSA data. To this end, it imports the module etcbc.trees, which contains the logic to synthesize trees from the data as it lies encoded in an EMDROS database and has been translated to LAF.

etcbc.trees can also work for other data, such as the CALAP database of parts of the Syriac Peshitta.

This notebook invokes the functions of etcbc.trees to generate trees for every sentence in the Hebrew Bible. After that it performs a dozen sanity checks on the trees. There are 13 sentences that do not pass some of these tests (out of 71354).

They will be excluded from further processing, since they probably have been coded wrong in the first place.

BHSA data and syntax trees¶

The process of tree construction is not straightforward, since the BHSA data have not been coded as syntax trees. Rather they take the shape of a collection of features that describe observable characteristics of the words, phrases, clauses and sentences. Moreover, if a phrase, clause or sentence is discontinuous, it is divided in phrase_atoms, clause_atoms, or sentence_atoms, respectively, which are by definition continuous.

There are no explicit hierarchical relationships between these objects. But there is an implicit hierarchy: embedding. Every object carries with it the set of word occurrences it contains.

The module etcbc.trees constructs a hierarchy of words, subphrases, phrases, clauses and sentences based on the embedding relationship.

But this is not all. The BHSA data contains a mother relationship, which in some cases denotes a parent relationship between objects. The module etcbc.trees reconstructs the tree obtained from the embedding by using the mother relationship as a set of instructions to move certain nodes below others. In some cases extra nodes will be constructed as well.

The embedding relationship¶

Objects:¶

The BHSA data is coded in such a way that every object is associated with a type and a monad set.

The type of an object, $T(O)$, determines which features an object has. ETCBC types are sentence, sentence_atom, clause, clause_atom, phrase, phrase_atom, subphrase, word.

There is an implicit ordering of object types, given by the sequence above, where word comes first and sentence comes last. We denote this ordering by $<$.

The monad set of an object, $m(O)$, is the set of word occurrences contained by that object. Every word occurrence in the source has a unique sequence number, so monad sets are sets of sequence numbers.

Note that when a sentence contains a clause which contains a phrase, the sentence, clause, and phrase contain words directly as monad sets. The fact that a sentence contains a clause is not marked directly, it is a consequence of the monad set embedding.

Definition (monad set order):¶

There is a natural order on monad sets, following the intuition that monad sets with smaller elements come before monad set with bigger elements, and embedding monad sets come before embedded monad sets. Hence, if you enumerate a set of objects that happens to constitute a tree hierarchy based on monad set embedding, and you enumerate those objects in the monad set order, you will walk the tree in pre-order.

This order is a modification of the one as described in (Doedens 1994, 3.6.3).

> Doedens, Crist-Jan (1994), *Text Databases. One Database Model and Several Retrieval Languages*, number 14 in Language and Computers, Editions Rodopi, Amsterdam, Netherlands and Atlanta, USA. ISBN: 90-5183-729-1, http://books.google.nl/books?id=9ggOBRz1dO4C. The order as defined by Doedens corresponds to walking trees in post-order.

For a lot of processing, it is handy to have a the stack of embedding elements available when working with an element. That is the advantage of pre-order over post-order. It is very much like SAX parsing.

Here is as the formal definition of my order:

$m_1 < m_2$ if either of the following holds:

1. $m_2 \subset m_1$ (note that the embedder comes first!)
2. $m_1 \not\subset m_2 \wedge m_2 \not\subset m_1 \wedge \min(m_1 \setminus m_2) < \min(m_2 \setminus m_1)$

We will not base our trees on all object types, since in the BHSA data they do not constitute a single hierarchy. We will restrict ourselves to the set $\cal O = \{$ sentence, clause, phrase, word $\}$.

Definition (directly below):¶

Object type $T_1$ is directly below $T_2$ ( $T_1 <_1 T_2 $ ) in $\cal O$ if $T_1 < T_2$ and there is no $T$ in $\cal O$ with $T_1 < T < T_2$.

Now we can introduce the notion of (tree) parent with respect to a set of object types $\cal O$ (e.g. ):

Definition (parent)¶

Object $A$ is a parent of object $B$ if the following are true:

1. $m(A) \subseteq\ m(B)$
2. $T(A) <_1 T(B)$ in $\cal O$.

The mother relationship¶

While using the embedding got us trees, using the mother relationship will give us more interesting trees. In general, the mother in the ETCBC points to an object on which the object in question is, in some sense, dependent. The nature of this dependency is coded in a specific feature on clauses, the clause_constituent_relation. For a list of values this feature can take and the associated meanings, see the notebook clause_phrase_types in this directory.

Here is a description of what we do with the mother relationship.

If a clause has a mother, there are three cases for the clause_constituent_relation of this clause:

1. its value is in $\{$ Adju, Objc, Subj, PrAd, PreC, Cmpl, Attr, RgRc, Spec $\}$
2. its value is Coor
3. its value is in $\{$ Resu, ReVo, none $\}$

In case 3 we do nothing.

In case 1 we remove the link of the clause to its parent and add the clause as a child to either the object that the mother points to, or to the parent of the mother. We do the latter only if the mother is a word. We will not add children to words.

In the diagrams, the red arrows represent the mother relationship, and the black arrows the embedding relationships, and the fat black arrows the new parent relationships. The gray arrows indicated severed parent links.

In case 2 we create a node between the mother and its parent. This node takes the name of the mother, and the mother will be added as child, but with name Ccoor, and the clause which points to the mother is added as a sister.

This is a rather complicated case, but the intuition is not that difficult. Consider the sentence:

John thinks that Mary said it and did it

We have a compound object sentence, with Mary said it and did it as coordinated components. The way this has been marked up in the BHSA database is as follows:

Mary said it, clause with clause_constituent_relation=Objc, mother=John thinks(clause)

and did it, clause with clause_constituent_relation=Coor, mother=Mary said it(clause)

So the second coordinated clause is simply linked to the first coordinated clause. Restructuring means to create a parent for both coordinated clauses and treat both as sisters at the same hierarchical level. See the diagram.

Note on order¶

When we add nodes to new parents, we let them occupy the sequential position among its new sisters that corresponds with the monad set ordering.

Note on discontinuity¶

Sentences, clauses and phrases are not always continuous. Before restructuring it will not always be the case that if you walk the tree in pre-order, you will end up with the leaves (the words) in the same order as the original sentence. Restructuring generally improves that, because it often puts an object under a non-continuous parent object precisely at the location that corresponds with the a gap in the parent.

However, there is no guarantee that every discontinuity will be resolved in this graceful manner. When we create the trees, we also output the list of monad numbers that you get when you walk the tree in pre-order. Whenever this list is not monotonic, there is an issue with the ordering.

Note on incest¶

If a mother points to itself or a descendant of itself, we have a grave form of incest. In these cases, the restructuring algorithm will disconnect a parent link without introducing a new link to the tree above it: a whole fragment of the tree becomes disconnected and will get lost.

Sanity check 6 below reveals that this occurs in fact 4 times in the BHSA version 4 (it occurred 13 times in the BHSA 3 version). We will exclude these trees from further processing.

Note on adultery¶

If a mother points outside the sentence of the clause on which it is specified we have a form of adultery. This should not happen. Mothers may point outside their sentences, but not in the cases that trigger restructuring. Yet, the sanity checks below reveal that this occurs twice. We will exclude these cases from further processing.

The engines are fired up now, all ETCBC data we need is accessible through the fabric and tr objects.

Next we select the information we want and load it into memory.

Before we can apply tree construction, we have to specify the objects and features we work with, since the tree constructing algorithm can also work on slightly different data, with other object types and different feature names.

We also construct an index that tells to which verse each sentence belongs.

Now we can actually construct the tree by initializing a tree object. After that we call its restructure_clauses() method.

Then we have two tree structures for each sentence:

• the etree, i.e. the tree obtained by working out the embedding relationships and nothing else
• the rtree, i.e. the tree obtained by restructuring the etree

We have several tree relationships at our disposal:

• eparent and its inverse echildren
• rparent and its inverse rchildren
• eldest_sister going from a mother clause of kind Coor (case 2) to its daughter clauses
• sisters being the inverse of eldest_sister

where eldest_sister and sisters only occur in the rtree.

This will take a while (25 seconds approx on a MacBook Air 2012).

Checking for sanity¶

Let us see whether the trees we have constructed satisfy some sanity constraints. After all, the algorithm is based on certain assumptions about the data, but are those assumptions valid? And restructuring is a tricky operation, do we have confidence that nothing went wrong?

1. How many sentence nodes? From earlier queries we know what to expect.
2. Does any sentence have a parent? If so, there is something wrong with our assumptions or algorithm.
3. Is every top node a sentence? If not, we have material outside a sentence, which contradicts the assumptions.
4. Do you reach all sentences if you go up from words? If not, some sentences do not contain words.
5. Do you reach all words if you go down from sentences? If not, some words have become disconnected from their sentences.
6. Do you reach the same words in reconstructed trees as in embedded trees? If not, some sentence material has got lost during the restructuring process.
7. From what object types to what object types does the parent relationship link? Here we check that parents do not link object types that are too distant in the object type ranking.
8. How many nodes have mothers and how many mothers can a node have? We expect at most one.
9. From what object types to what object types does the mother relationship link?
10. Is the mother of a clause always in the same sentence? If not, foreign sentences will be drawn in, leading to (very) big chunks. This may occur when we use mother relationships in cases where clause_constituent_relation has different values than the ones that should trigger restructuring.
11. Has the max/average tree depth increased after restructuring? By how much? This is meant as an indication by how much our tree structures improve in significant hierarchy when we take the mother relationship into account.

Writing Trees¶

After all these checks we can proceed to print out the tree structures as plain, bracketed text strings.

Per tree we also print a string of the monad numbers that you get when you walk the tree in pre-order. And we produce object ids from the EMDROS database and node ids from the LAF version.

First we apply our algorithms to a limited set of interesting trees and a random sample. For those cases we also apply a debug_write() method that outputs considerably more information.

This output has been visually checked by Constantijn Sikkel and Dirk Roorda.

Finally, here is the production of the whole set of trees.

Preview¶

Here are the first lines of the output.