%load_ext autoreload
%autoreload 2
from tf.app import use
from tf.browser.ner.annotate import Annotate
It is often the case that the same names in different books refer to different people/locations/other entities.
Here we show how to mark up the occurrences of a name but only in as far they occur in the same book.
More concretely, we mark up all occurrences of אלהים in Genesis.
We use the basic Annotate API for this.
A = use("ETCBC/bhsa")
Locating corpus resources ...
Name | # of nodes | # slots / node | % coverage |
---|---|---|---|
book | 39 | 10938.21 | 100 |
chapter | 929 | 459.19 | 100 |
lex | 9230 | 46.22 | 100 |
verse | 23213 | 18.38 | 100 |
half_verse | 45179 | 9.44 | 100 |
sentence | 63717 | 6.70 | 100 |
sentence_atom | 64514 | 6.61 | 100 |
clause | 88131 | 4.84 | 100 |
clause_atom | 90704 | 4.70 | 100 |
phrase | 253203 | 1.68 | 100 |
phrase_atom | 267532 | 1.59 | 100 |
subphrase | 113850 | 1.42 | 38 |
word | 426590 | 1.00 | 100 |
3
ETCBC/bhsa
/Users/me/text-fabric-data/github/ETCBC/bhsa/app
gb112c161cfd21eae403d51a2733740d8743460e7
''
<code>Genesis 1:1</code> (use <a href="https://github.com/{org}/{repo}/blob/master/tf/{version}/book%40en.tf" target="_blank">English book names</a>)
g_uvf_utf8
g_vbs
kq_hybrid
languageISO
g_nme
lex0
is_root
g_vbs_utf8
g_uvf
dist
root
suffix_person
g_vbe
dist_unit
suffix_number
distributional_parent
kq_hybrid_utf8
crossrefSET
instruction
g_prs
lexeme_count
rank_occ
g_pfm_utf8
freq_occ
crossrefLCS
functional_parent
g_pfm
g_nme_utf8
g_vbe_utf8
kind
g_prs_utf8
suffix_gender
mother_object_type
absent
n/a
none
unknown
NA
{docRoot}/{repo}
''
''
https://{org}.github.io
0_home
{}
True
local
/Users/me/text-fabric-data/github/ETCBC/bhsa/_temp
BHSA = Biblia Hebraica Stuttgartensia Amstelodamensis
10.5281/zenodo.1007624
ner
Phonetic Transcriptions
https://nbviewer.jupyter.org/github/etcbc/phono/blob/master/programs/phono.ipynb
10.5281/zenodo.1007636
ETCBC
/tf
phono
Parallel Passages
https://nbviewer.jupyter.org/github/ETCBC/parallels/blob/master/programs/parallels.ipynb
10.5281/zenodo.1007642
ETCBC
/tf
parallels
ETCBC
/tf
bhsa
2021
https://shebanq.ancient-data.org/hebrew
Show this on SHEBANQ
la
True
{webBase}/text?book=<1>&chapter=<2>&verse=<3>&version={version}&mr=m&qw=q&tp=txt_p&tr=hb&wget=v&qget=v&nget=vt
{webBase}/word?version={version}&id=<lid>
v1.8.1
{typ} {rela}
''
True
{code}
1
''
True
{label}
''
True
gloss
{voc_lex_utf8}
word
orig
{voc_lex_utf8}
{typ} {function}
''
True
{typ} {rela}
1
''
{number}
''
True
{number}
1
''
True
{number}
''
pdp vs vt
lex:gloss
hbo
We look for the words whose lexeme is >LHJM/
in Genesis.
words = [result[1] for result in A.search("""
book book=Genesis
word lex=>LHJM/
""")]
0.16s 219 results
NE = A.makeNer()
We set up a set to add the entities to:
NE.setSet("elohim")
Annotation set elohim has 215 annotations
NE.addEntity(("elohim", "per"), tuple((word,) for word in words), silent=False)
Already present: 0 x Added: 219 x
Now we can retrieve the verses where these entities occur:
content = NE.filterContent(eVals=("elohim", "per"))
190 verses
Let's show the first and last 5 verses:
NE.showContent(content, end=5)
NE.showContent(content, start=185)
If we want to change the entity kind from per
to div
, we remove these entities first:
NE.delEntity(("elohim", "per"), tuple((word,) for word in words), silent=False)
Not present: 0 x Deleted: 219 x
And then add them with the right kind:
NE.addEntity(("elohim", "div"), tuple((word,) for word in words), silent=False)
Already present: 215 x Added: 4 x
With a check:
content = NE.filterContent(eVals=("elohim", "div"))
190 verses
NE.showContent(content, end=2)
If you start the TF browser and switch to the annotate tool, you'll find the new set elohim
there, and you
can make manual adjustments to individual occurrences.
!tf --chrome --tool=ner ETCBC/bhsa
This is Text-Fabric 12.1.2 appName='ETCBC/bhsa' slug='ETCBC/bhsa' Loading TF corpus data. Please wait ... Setting up TF browser for ETCBC/bhsa **Locating corpus resources ...** Using app in ~/text-fabric-data/github/ETCBC/bhsa/app: rv1.8.1=#gb112c161cfd21eae403d51a2733740d8743460e7 offline under ~/text-fabric-data/github (local release) Using data in ~/text-fabric-data/github/ETCBC/bhsa/tf/2021: rv1.8.1=#gb112c161cfd21eae403d51a2733740d8743460e7 offline under ~/text-fabric-data/github (local release) Using data in ~/text-fabric-data/github/ETCBC/phono/tf/2021: rv2.1=#gaba4367b49750089e4e4122415a77cac43bd97bc offline under ~/text-fabric-data/github (local release) Using data in ~/text-fabric-data/github/ETCBC/parallels/tf/2021: rv2.1=#gf45f6cc3c4f933dba6e649f49cdb14a40dcf333f offline under ~/text-fabric-data/github (local release) TF setup done. WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on http://localhost:16650 Press CTRL+C to quit o-o-o Opening corpus in Chrome browser o-o-o Press <Ctrl+C> to stop the TF browser 127.0.0.1 - - [01/Nov/2023 11:23:49] "GET /ner/index HTTP/1.1" 200 - 127.0.0.1 - - [01/Nov/2023 11:23:52] "GET /browser/static/colors.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:52] "GET /browser/static/highlight.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:52] "GET /ner/static/index.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:52] "GET /browser/static/fonts.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:52] "GET /ner/static/base.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:52] "GET /browser/static/display.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:52] "GET /browser/static/jquery.js HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:52] "GET /ner/static/tool.js HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:59] "POST /ner/index HTTP/1.1" 200 - 127.0.0.1 - - [01/Nov/2023 11:23:59] "GET /browser/static/colors.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:59] "GET /browser/static/display.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:59] "GET /browser/static/highlight.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:59] "GET /browser/static/fonts.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:59] "GET /ner/static/base.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:59] "GET /ner/static/index.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:59] "GET /browser/static/jquery.js HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:23:59] "GET /ner/static/tool.js HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:02] "POST /ner/index HTTP/1.1" 200 - 127.0.0.1 - - [01/Nov/2023 11:24:02] "GET /browser/static/colors.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:02] "GET /browser/static/display.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:02] "GET /browser/static/highlight.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:02] "GET /browser/static/fonts.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:02] "GET /ner/static/base.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:02] "GET /ner/static/index.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:02] "GET /browser/static/jquery.js HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:02] "GET /ner/static/tool.js HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:22] "POST /ner/index HTTP/1.1" 200 - 127.0.0.1 - - [01/Nov/2023 11:24:22] "GET /browser/static/colors.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:22] "GET /browser/static/display.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:22] "GET /browser/static/highlight.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:22] "GET /browser/static/fonts.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:22] "GET /ner/static/base.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:22] "GET /ner/static/index.css HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:22] "GET /browser/static/jquery.js HTTP/1.1" 304 - 127.0.0.1 - - [01/Nov/2023 11:24:22] "GET /ner/static/tool.js HTTP/1.1" 304 - ^C TF web server has stopped
And then you see something like this (after selecting the elohim
set and clicking the elohim
entity
in the left column.
You can click and delete some of the occurrences, let's delete the first two and the last two occurrences ...
Now, back in our notebook, we retrieve the occurrences of the elohim
entity, after this change.
But first we have to stop the TF browser and reload the elohim
set.
The fact that we need to stop the TF browser before we can continue with our notebook is an indication that this is not the recommended way to start the TF browser.
Better start it independently from a command prompt! Anyway, you can stop the browser by interrupting the kernel (the black square button in the toolbar at the top of the notebook interface).
NE.setSet("elohim")
Annotation set elohim has 215 annotations
content = NE.filterContent(eVals=("elohim", "div"))
186 verses
Can we find the exact occurrences that we have deleted?
Yes, but there is no dedicated method (yet) to do it.
But we have to dig in the various representations of the entity data that the tool maintains under the hood, see processed data.
We want entityVal
.
occurrences = NE.getSetData().entityVal[("elohim", "div")]
type(occurrences)
set
len(occurrences)
215
Each occurrence is a tuple of slots. In general, occurrences of entities can consist of multiple words, hence tuples of slots and not individual slot. But in our case, the entities are single words, so we can get the words as follows:
currentWords = {occ[0] for occ in occurrences}
Finally, we can detect which entities we have deleted:
deletedWords = set(words) - currentWords
deletedWords
{4, 26, 28705, 28739}
That looks an awful lot like the first and last two occurrences of Elohim in Genesis.
We can show them in the usual Text-Fabric way:
A.table([(w,) for w in sorted(deletedWords)])
n | p | word |
---|---|---|
1 | Genesis 1:1 | אֱלֹהִ֑ים |
2 | Genesis 1:2 | אֱלֹהִ֔ים |
3 | Genesis 50:24 | אלֹהִ֞ים |
4 | Genesis 50:25 | אֱלֹהִים֙ |
We can also show these verses in the annotator view:
L = A.api.L
verses = {L.u(w, otype="verse")[0] for w in deletedWords}
content = NE.filterContent(buckets=verses)
NE.showContent(content)
4 verses
No entity is underlined, and in this case absence of evidence is evidence of absence.
Because if we show a verse with entities, we'll see them:
vInit = {A.nodeFromSectionStr(f"Genesis 1:{v}") for v in range(1, 6)}
content = NE.filterContent(buckets=vInit, qTokens=None)
NE.showContent(content)
5 verses
You have two interfaces to annotate named entities in your corpus, and both interfaces work with the same data.
The browser interface is good for mass-marking occurrences of single entities, and also for fine-tuning by making small sets of exceptions.
However, if you need to treat broad classes of occurrences of the same name in different ways, it is better to go over to a Jupyter Notebook and mark the entities programmatically.
Then you can still return to the browser interface and make individual exceptions.
What we did not show here is that in the programmatic interface you can also assign lots of different entities much more quickly than by hand in the browser interface.