#!/usr/bin/env python
# coding: utf-8
# # Differences in MT and SP in parasha #8: Vayishlach (Genesis 32:4-36:43)
# ## Table of Content (ToC)
#
# * 1 - Introduction
# * 2 - Load Text-Fabric app and data
# * 3 - Compare surface texts of SP and MT
# * 4 - Compare texts using minimum Levenshtein distance
# * 5 - Comparison of spelling of proper nouns between SP and MT
# * 6 - References and acknowledgement
# * 7 - Required libraries
# * 8 - Notebook version details
# # 1 - Introduction
# ##### [Back to ToC](#TOC)
#
# The Samaritan Pentateuch (SP) is a version of the Torah preserved by the Samaritan community, differing from the Masoretic Text (MT) in several aspects, including language, orthography, and occasionally theological emphasis. This notebook compares the text of the Masoretic Text, based on the BHSA dataset in Text-Fabric, with the Samaritan Pentateuch, also available as a Text-Fabric dataset.1
#
# In this analysis, we focus on comparing the text of the verses in a specific parasha, highlighting differences in wording and orthography. Additionally, special attention is given to spelling variations of proper nouns between the two traditions. This notebook draws inspiration from the notebook provided by Martijn Naaijer2 and aims to explore the textual nuances between these two important versions of the Torah.
# # 2 - Load Text-Fabric app and data
# ##### [Back to ToC](#TOC)
#
# The following code will load the Text-Fabric version of the [Samaritan Pentatuch](https://github.com/DT-UCPH/sp), the [Biblia Hebraica Stuttgartensia (Amstelodamensis)](https://etcbc.github.io/bhsa/) together with the additonal parasha related features from [tonyjurg/BHSaddons](https://github.com/tonyjurg/BHSaddons).
# In[1]:
from tf.app import use
# Load the SP data, and rename the node features class F,
# the locality class L and the text class T,
# then they cannot be overwritten while loading the MT.
SP = use('DT-UCPH/sp', version='3.4')
Fsp, Lsp, Tsp = SP.api.F, SP.api.L, SP.api.T
# Do the same for the MT dataset (BHSA) together with BHSaddons
MT = use('etcbc/bhsa', version='2021',mod="tonyjurg/BHSaddons/tf/")
Fmt, Lmt, Tmt = MT.api.F, MT.api.L, MT.api.T
# # 3 - Compare surface texts of SP and MT
# ##### [Back to ToC](#TOC)
#
# In this section, we compare the surface texts of the Samaritan Pentateuch (SP) and the Masoretic Text (MT) at the verse level. By analyzing the wording and structure of these texts, we aim to identify variations.
# In[2]:
# find all word nodes for this parasha (we can either use the transliterated name or the sequence number)
parashaQuery = '''
verse parashanum=8
'''
parashaResults = MT.search(parashaQuery)
# In[3]:
# Extract book, chapter, and verse information
bookChapterVerseList = [
Tmt.sectionFromNode(verse[0]) for verse in parashaResults
]
# Store parashname, start and end verse for future use
startNode=parashaResults[0][0]
endNode=parashaResults[-1][0]
parashaNameHebrew=Fmt.parashahebr.v(startNode)
parashaNameEnglish=Fmt.parashatrans.v(startNode)
bookStart,chapterStart,startVerse=Tmt.sectionFromNode(startNode)
parashaStart=f'{bookStart} {chapterStart}:{startVerse}'
bookEnd,chapterEnd,startEnd=Tmt.sectionFromNode(endNode)
parashaEnd=f'{chapterEnd}:{startEnd}'
htmlStart=''
htmlFooter=f'
'
)
return formattedDiff
# Gather differences into an HTML string
htmlContent = f'
Differences between MT and SP for parasha {parashaNameEnglish} ({parashaStart}-{parashaEnd})
'
for label, MTverseText in MTverses.items():
SPverseText = SPverses.get(label, '')
MTverseText = cleanText(MTverseText)
if MTverseText != SPverseText: # Check for differences
difference = formatAndHighlight(label, MTverseText, SPverseText)
htmlContent += difference
# Save the content to an HTML file
fileName = f"differences_MT_SP({parashaNameEnglish.replace(' ','%20')}).html"
with open(fileName, "w", encoding="utf-8") as file:
file.write(htmlContent)
# Display the content in the notebook
display(HTML(htmlContent))
# wrap html header and footer and display a download button
htmlContentFull = f'{htmlStart}{htmlContent}{htmlFooter}'
downloadButton = f"""
"""
display(HTML(downloadButton))
# # 4 - Compare texts using minimum Levenshtein distance
# ##### [Back to ToC](#TOC)
# The Levenshtein distance measures the minimum number of single-character edits (insertions, deletions, or substitutions) needed to transform one text into another, providing a quantitative way to compare textual differences. For comparing the Masoretic Text and Samaritan Pentateuch, it highlights variations in spelling, word order, or minor textual changes.
# In the context of the Levenshtein distance (in the script below `threshold`), a higher number indicates greater dissimilarity between two texts, meaning more edits (insertions, deletions, or substitutions) are needed to transform one text into the other.
# In[6]:
from Levenshtein import distance
from IPython.display import HTML, display
threshold = 20
# Create an HTML string to store the output
htmlContent = f'
Levenshtein distance >{threshold} between MT and SP for parasha {parashaNameEnglish} ({parashaStart}-{parashaEnd})
'
# Create header
MT.dm(f'### Levenshtein distance >{threshold} between MT and SP for parasha {parashaNameEnglish} ({parashaStart}-{parashaEnd})')
# Generate the HTML content
for label, MTverseText in MTverses.items():
SPverseText = SPverses.get(label, '')
levDistance = distance(MTverseText, SPverseText) # Calculate the distance
if levDistance > threshold:
formattedDiff = formatAndHighlight(label, MTverseText, SPverseText)
formattedDiff += f'
Levenshtein Distance: {levDistance}
' # Add the distance
MT.dm(formattedDiff)
htmlContent += formattedDiff # Append to the HTML content
# Save the content to an HTML file
fileName = f"levenshtein_differences_MT_SP({parashaNameEnglish.replace(' ','%20')}).html"
with open(fileName, "w", encoding="utf-8") as file:
file.write(htmlContent)
# wrap html header and footer and display a download button
htmlContentFull = f'{htmlStart}{htmlContent}{htmlFooter}'
downloadButton = f"""
"""
display(HTML(downloadButton))
# # 5 - Comparison of spelling of proper nouns between SP and MT
# ##### [Back to ToC](#TOC)
#
# This section focuses on comparing the spelling of proper nouns between the Samaritan Pentateuch (SP) and the Masoretic Text (MT). Proper nouns, including names of people, places, and unique terms, often exhibit variations in spelling
# In[7]:
import collections
def collectProperNounSpellings(F, L, T, inputList):
"""
Collect proper noun spellings and their associated word node numbers.
Ensures only one tuple is stored for each lexeme-to-spelling mapping.
"""
properNounsSpellings = {}
for bookChapterVerse in inputList:
verseNode = T.nodeFromSection(bookChapterVerse)
wordNodes = L.d(verseNode, 'word')
for wordNode in wordNodes:
if F.sp.v(wordNode) == 'nmpr': # Check if the word is a proper noun
lex = F.lex.v(wordNode) # Lexical form
spelling = F.g_cons.v(wordNode) # Spelling
# Store only the first occurrence for each lex-to-cons mapping
if lex not in properNounsSpellings or spelling not in {item[0] for item in properNounsSpellings[lex]}:
properNounsSpellings.setdefault(lex, []).append((spelling, wordNode))
return properNounsSpellings
SPspellingDict = collectProperNounSpellings(Fsp, Lsp, Tsp, bookChapterVerseList)
MTspellingDict = collectProperNounSpellings(Fmt, Lmt, Tmt, bookChapterVerseList)
# In[8]:
from IPython.display import HTML, display
# Initialize HTML content
htmlContent = f'
Spelling differences in proper nouns between SP and MT for parasha {parashaNameEnglish} ({parashaStart}-{parashaEnd})
'
# Generate the HTML output
for lex, MTspellings in MTspellingDict.items():
# Retrieve SP spellings, defaulting to an empty set if lex is not found
SPspellings = SPspellingDict.get(lex, set())
# Extract only the spellings (ignoring node numbers) for comparison
MTspellingSet = {spelling for spelling, _ in MTspellings}
SPspellingSet = {spelling for spelling, _ in SPspellings}
# Compare the sets of spellings
if MTspellingSet != SPspellingSet:
# Print MT spelling with reference
MTnode = list(MTspellings)[0][1] # Get first tuple's node number
book, chapter, verse = Tmt.sectionFromNode(MTnode)
MTgloss = Fmt.gloss.v(MTnode)
MTspelling = Fmt.g_cons_utf8.v(MTnode)
# Build HTML output
output = (
f'
'
)
# Print SP spellings with reference
if SPspellings:
SPnode = list(SPspellings)[0][1] # Get first tuple's node number
SPspelling = Fsp.g_cons_utf8.v(SPnode)
output += f'
SP Spelling: {SPspelling}
'
else:
output += '
SP Spelling: None
'
# Append the output to the HTML content
htmlContent += output
# Save the HTML content to a file
fileName = f"spelling_differences_SP_MT({parashaNameEnglish.replace(' ','%20')}).html"
with open(fileName, "w", encoding="utf-8") as file:
file.write(htmlContent)
# Display the HTML content in the notebook
display(HTML(htmlContent))
# wrap html header and footer and display a download button
htmlContentFull = f'{htmlStart}{htmlContent}{htmlFooter}'
downloadButton = f"""
"""
display(HTML(downloadButton))
# # 6 - References and acknowledgement
# ##### [Back to ToC](#TOC)
#
# 1 Christian Canu Højgaard, Martijn Naaijer, & Stefan Schorch. (2023). Text-Fabric Dataset of the Samaritan Pentateuch. Zenodo. https://doi.org/10.5281/zenodo.7734632
#
# 2 [Notebook created by Martijn Naaijer](https://github.com/DT-UCPH/sp/blob/main/notebooks/combine_sp_with_mt_data.ipynb)
# # 7 - Required libraries
# ##### [Back to ToC](#TOC)
#
# The scripts in this notebook require (beside `text-fabric`) the following Python libraries to be installed in the environment:
#
# collections
# difflib
# Levenshtein
#
# You can install any missing library from within Jupyter Notebook using either`pip` or `pip3`.
# # 8 - Notebook version details
# ##### [Back to ToC](#TOC)
#
#