Ideally, a comprehensive documentation set should be created as part of developing a Text-Fabric dataset. However, in practice, this is not always completed during the initial phase or after changes to features. This Jupyter Notebook contains Python code to automatically generate (and thus ensure consistency) a documentation set for any Text-Fabric dataset. It serves as a robust starting point for the development of a brand new documentation set or as validation for an existing one. One major advantage is that the resulting documentation set is fully hyperlinked, a task that can be laborious if done manually.
The main steps in producing the documentation set are:
The output format can be either Markdown, the standard for feature documentation stored on GitHub using its on-site processor, or HTML, which facilitates local storage and browsing with any web browser.
Your environment should (for obvious reasons) include the Python package Text-Fabric. If not installed yet, it can be installed using pip. Further it is required to be able to invoke the Text-Fabric data set (either from an online resource, or from a localy stored copy). There are no further requirements as the scripts basicly operate 'stand alone'.
At this step, the Text-Fabric dataset is loaded, which embedded data will be used to create a documentation set. For various options regarding other possible storage locations, see the documentation for function use.
%load_ext autoreload
%autoreload 2
# Loading the Text-Fabric code
# Note: it is assumed Text-Fabric is installed in your environment
from tf.fabric import Fabric
from tf.app import use
# load the app and data
A = use ("saulocantanhede/tfgreek2", version="0.5.5", hoist=globals())
Locating corpus resources ...
| 29s T sibling from ~/text-fabric-data/github/saulocantanhede/tfgreek2/tf/0.5.5
| Name | # of nodes | # slots / node | % coverage |
|---|---|---|---|
| book | 27 | 5102.93 | 100 |
| chapter | 260 | 529.92 | 100 |
| verse | 7944 | 17.34 | 100 |
| sentence | 19767 | 13.79 | 198 |
| group | 8964 | 7.02 | 46 |
| clause | 30479 | 7.19 | 159 |
| wg | 106868 | 6.88 | 533 |
| phrase | 66424 | 1.93 | 93 |
| subphrase | 119013 | 1.59 | 138 |
| word | 137779 | 1.00 | 100 |
3saulocantanhede/tfgreek2C:/Users/tonyj/text-fabric-data/github/saulocantanhede/tfgreek2/app8c4bc8e48e66e32f614b5966813104c0894ad822''[]noneunknownNA@text-orig-fullabout{docBase}/transcription.mdtranscriptiontext-orig-full}TruelocalC:/Users/tonyj/text-fabric-data/github/saulocantanhede/tfgreek2/_tempNestle 1904 Greek New Testament10.5281/zenodo.notyet[]saulocantanhede/tftfgreek20.5.5https://learner.bible/text/show_text/nestle1904/Show this on the websiteenhttps://learner.bible/text/show_text/nestle1904/<1>/<2>/<3>{webBase}/word?version={version}&id=<lid>0.5.4True#{num}: {cls} {rule} {junction}''#{num}: ''True#{num}: {function} {role} {rule} {type}''#{num}: {rule}''True{book} {chapter}:{verse}''True#{num}: {type} {role} {rule} {junction}''True}grc# If the following variable is set, it will be used as title for all pages. It is intended to the describe the dataset in one line
# customPageTitleMD="N1904 Greek New Testament [saulocantanhede/tfgreek2 - 0.5.4](https://github.com/saulocantanhede/tfgreek2)"
# customPageTitleHTML="N1904 Greek New Testament <a href=\"https://github.com/saulocantanhede/tfgreek2\">saulocantanhede/tfgreek2 - 0.5.4</a>"
# Specify the location to store the resulting files, relative to the location of this notebook (without a trailing slash).
resultLocation = "results"
# Type of output format ('html' for HTML, 'md' for Mark Down, or 'both' for both HTML and Mark Down)
typeOutput='both'
# HTML table style definition (only relevant for HTML output format)
htmlStyle='<style>\ntable {\nborder-collapse: collapse;\n}\n th, td {\nborder: 1px solid black;\n padding: 8px;\n}\nth {\nfont-weight: bold;\n}\n</style>'
# Limit the number of entries in the frequency tables per node type on each feature description page to this number
tableLimit=10
# This switch can be set to 'True' if you want additional information, such as dictionary entries and file details, to be printed. For basic output, set this switch to 'False'.
verbose=False
# The version number of the script
scriptVersion="0.3"
scriptDate="Jan. 24, 2024"
# Create the footers for MD and HTML, include today's date
from datetime import datetime
today = datetime.today()
formatted_date = today.strftime("%b. %d, %Y")
footerMD=f'\n\nCreated on {formatted_date} using [Doc4TF version {scriptVersion} ({scriptDate})](https://github.com/tonyjurg/Doc4TF)'
footerHTML=f'\n<p>Created on {formatted_date} using <a href=\"https://github.com/tonyjurg/Doc4TF\">Doc4TF - version {scriptVersion} ({scriptDate})</a></p></body></html>'
The following will create a dictionary containing all relevant information for the loaded node and edge features.
# Initialize an empty dictionary to store feature data
featureDict = {}
# Function to get feature description from metadata
def get_feature_description(metaData):
return metaData.get('description', "No feature description")
# Function to set data type based on 'valueType' in metadata
def set_data_type(metaData):
if 'valueType' in metaData:
return "String" if metaData["valueType"] == 'str' else "Integer"
return "Unknown"
# Function to process and add feature data to the dictionary
def process_feature(feature, featureType, featureMethod):
# Obtain the meta data
featureMetaData = featureMethod(feature).meta
featureDescription = get_feature_description(featureMetaData)
dataType = set_data_type(featureMetaData)
# Initialize dictionary to store feature frequency data
featureFrequencyDict = {}
# Skip for specific features based on type
if not (featureType == 'Node' and feature == 'otype') and not (featureType == 'Edge' and feature == 'oslots'):
for nodeType in F.otype.all:
frequencyLists = featureMethod(feature).freqList(nodeType)
if not isinstance(frequencyLists, int):
if len(frequencyLists)!=0:
featureFrequencyDict[nodeType] = {'nodetype': nodeType, 'freq': frequencyLists[:tableLimit]}
elif isinstance(frequencyLists, int):
if frequencyLists != 0:
featureFrequencyDict[nodeType] = {'nodetype': nodeType, 'freq': [("Link", frequencyLists)]}
# Add processed feature data to the main dictionary
featureDict[feature] = {'name': feature, 'descr': featureDescription, 'type': featureType, 'datatype': dataType, 'freqlist': featureFrequencyDict}
########################################################
# MAIN FUNCTION #
########################################################
########################################################
# Gather general information #
########################################################
print('Gathering generic details')
# Initialize default values
corpusName = A.appName
liveName = ''
versionName = A.version
# Trying to locate corpus information
if A.provenance:
for parts in A.provenance[0]:
if isinstance(parts, tuple):
key, value = parts[0], parts[1]
if verbose: print (f'General info: {key}={value}')
if key == 'corpus': corpusName = value
if key == 'version': versionName = value
# value for live is a tuple
if key == 'live': liveName=value[1]
if liveName is not None and len(liveName)>1:
# an URL was found
pageTitleMD = f'Doc4TF pages for [{corpusName}]({liveName}) (version {versionName})'
pageTitleHTML = f'<h1>Doc4TF pages for <a href="{liveName}">{corpusName}</a> (version {versionName})</h1>'
else:
# No URL found
pageTitleMD = f'Doc4TF pages for {corpusName} (version {versionName})'
pageTitleHTML = f'<h1>Doc4TF pages for {corpusName} (version {versionName})</h1>'
# Overwrite in case user provided a title
if 'customPageTitleMD_' in globals():
pageTitleMD = customPageTitleMD
if 'customPageTitleHTML' in globals():
pageTitleMD = customPageTitleHTML
########################################################
# Processing node features #
########################################################
print('Analyzing Node Features: ', end='')
for nodeFeature in Fall():
if not verbose: print('.', end='') # Progress indicator
process_feature(nodeFeature, 'Node', Fs)
if verbose: print(f'\nFeature {nodeFeature} = {featureDict[nodeFeature]}\n') # Print feature data if verbose
########################################################
# Processing edge features #
########################################################
print('\nAnalyzing Edge Features: ', end='')
for edgeFeature in Eall():
if not verbose: print('.', end='') # Progress indicator
process_feature(edgeFeature, 'Edge', Es)
if verbose: print(f'\nFeature {edgeFeature} = {featureDict[edgeFeature]}\n') # Print feature data if verbose
print('\nFinished')
Gathering generic details Analyzing Node Features: .................................................. Analyzing Edge Features: ..... Finished
Two types of pages will be created:
import os
# Initialize a counter for the number of files created
filesCreated = 0
# Get the current working directory and append a backslash for path building
pathFull = os.getcwd() + '\\'
# Iterating over each feature in the feature dictionary
for featureName, featureData in featureDict.items():
# Extracting various properties of each feature
featureDescription = featureData.get('descr')
featureType = featureData.get('type')
featureDataType = featureData.get('datatype')
# Initializing strings to accumulate HTML and Markdown content
nodeListHTML = nodeListMD = ''
tableListHTML = tableListMD = ''
frequencyData = featureData.get('freqlist')
# Processing frequency data for each node
for node in frequencyData:
# Building HTML and Markdown links for each node
nodeListHTML += f' <a href=\"featurebynodetype.htm#{node}\">{node}</a>'
nodeListMD += f' [`{node}`](featurebynodetype.md#{node}) '
# Starting HTML and Markdown tables for frequency data
tableListHTML += f'<h3>Frequency for nodetype <a href=\"featurebynodetype.htm#{node}\">{node}</a></h3><table><tr><th>Value</th><th>Occurences</th></tr>'
tableListMD += f'### Frequency for nodetype [{node}](featurebynodetype.md#{node})\nValue|Occurences\n---|---\n'
# Populating tables with frequency data
itemData = frequencyData.get(node).get('freq')
for item in itemData:
handleSpace = item[0] if item[0] != ' ' else 'space' # prevent garbling of tables where the value itself is a space
tableListHTML += f'<tr><td>{handleSpace}</td><td>{item[1]}</td></tr>'
tableListMD += f'{handleSpace}|{item[1]}\n'
tableListHTML += f'</table>\n'
# Creating info blocks for HTML and Markdown
infoBlockHTML = f'<table><tr><th>Data type</th><th>Feature type</th><th>Available for nodes</th></tr><tr><td><a href=\"featurebydatatype.htm#{featureDataType}\">{featureDataType}</a></td><td><a href="featurebytype.htm#{featureType}">{featureType}</a></td><td>{nodeListHTML}</td></tr></table>'
infoBlockMD = f'Data type|Feature type|Available for nodes\n---|---|---\n[`{featureDataType}`](featurebydatatype.md#{featureDataType.lower()})|[`{featureType}`](featurebytype.md#{featureType.lower()})|{nodeListMD}'
# Outputting in Markdown format
if typeOutput in ('md','both'):
pageMD = f'{pageTitleMD}\n# Feature: {featureName}\n{infoBlockMD}\n## Description\n{featureDescription}\n## Feature Values\n{tableListMD} {footerMD} '
fileNameMD = os.path.join(resultLocation, f"{featureName}.md")
try:
with open(fileNameMD, "w", encoding="utf-8") as file:
file.write(pageMD)
filesCreated += 1
# Log if verbose mode is on
if verbose: print(f"Markdown content written to {pathFull + fileNameMD}")
except Exception as e:
print(f"Exception: {e}")
break # Stops execution on encountering an exception
# Outputting in HTML format
if typeOutput in ('html','both'):
pageHTML = f'<html><head>{htmlStyle}</head><body><p>{pageTitleHTML}</p>\n<h1 id=\"start\">Feature: {featureName}</h1>\n{infoBlockHTML}\n<h2>Description</h2>\n<p>{featureDescription}</p>\n<h2>Feature Values</h2>\n{tableListHTML} {footerHTML}'
fileNameHTML = os.path.join(resultLocation, f"{featureName}.htm")
try:
with open(fileNameHTML, "w", encoding="utf-8") as file:
file.write(pageHTML)
filesCreated += 1
# Log if verbose mode is on
if verbose: print(f"HTML content written to {pathFull + fileNameHTML}")
except Exception as e:
print(f"Exception: {e}")
break # Stops execution on encountering an exception
# Reporting the number of files created
if filesCreated != 0:
print(f'Finished (written {filesCreated} {"html and md" if typeOutput == "both" else typeOutput} files to directory {pathFull + resultLocation})')
else:
print('No files written')
Finished (written 110 html and md files to directory C:\Users\tonyj\OneDrive\Documents\GitHub\Doc4TF\results)
# Initialize a counter for the number of files created
filesCreated = 0
# Example data function to create a list of examples for a given feature
def exampleData(feature):
# Check if the feature exists in featureDict and has non-empty freqlist.
if feature in featureDict and featureDict[feature]['freqlist']:
# Get the first value from the freqlist
freq_list = next(iter(featureDict[feature]['freqlist'].values()))['freq']
# Use list comprehension to create the example list.
example_list = ' '.join(f'`{item[0]}`' for item in freq_list[:4])
return example_list
else:
return "No values"
def writeToFile(fileName, content, fileType, verbose):
"""
Writes content to a file.
:param fileName: The name of the file to write to.
:param content: The content to write.
:param fileType: The type of file (e.g., 'md' for Markdown, 'html' for HTML).
:param verbose: If True, prints a message upon successful writing.
"""
global filesCreated
try:
with open(fileName, "w", encoding="utf-8") as file:
file.write(content)
filesCreated+=1
if verbose:
print(f"{fileType.upper()} content written to {fileName}")
except Exception as e:
print(f"Exception while writing {fileType.upper()} file: {e}")
# Set up some lists
nodeFeatureList = []
typeFeatureList = []
dataTypeFeatureList = []
for featureName, featureData in featureDict.items():
typeFeatureList.append((featureName,featureData.get('type')))
dataTypeFeatureList.append((featureName,featureData.get('datatype')))
for node in featureData.get('freqlist'):
nodeFeatureList.append((node, featureName))
###########################################################
# Create the page with overview per node type (e.g. word) #
###########################################################
pageMD=f'{pageTitleMD}\n# Overview features per nodetype\n'
pageHTML=f'<html><head>{htmlStyle}</head><body><p>{pageTitleHTML}</p>\n<h1>Overview features per nodetype</h1>'
# Sort the list alphabetically based on the second item of each tuple (featureName)
nodeFeatureList = sorted(nodeFeatureList, key=lambda x: x[1])
# Iterate over node types
for NodeType in F.otype.all:
NodeItemTextMD=f'## {NodeType}\n\nFeature|Featuretype|Datatype|Description|Examples\n---|---|---|---|---\n'
NodeItemTextHTML=f'<h2 id=\"{NodeType}\">{NodeType}</h2>\n<table><tr><th>Feature</th><th>Featuretype</th><th>Datatype</th><th>Description</th><th>Examples</th></tr>\n'
for node, feature in nodeFeatureList:
if node == NodeType:
featureData=featureDict[feature]
featureDescription=featureData.get('descr')
featureType=featureData.get('type')
featureDataType=featureData.get('datatype')
NodeItemTextMD+=f"[`{feature}`]({feature}.md#readme)|[`{featureType}`](featurebytype.md#{featureType})|[`{featureDataType}`](featurebydatatype.md#{featureDataType})|{featureDescription}|{exampleData(feature)}\n"
NodeItemTextHTML+=f"<tr><td><a href=\"{feature}.htm#start\">{feature}</a></td><td><a href=\"featurebytype.htm#{featureType}\">{featureType}</td><td><a href=\"featurebydatatype.htm#{featureDataType}\">{featureDataType}</a></td><td>{featureDescription}</td><td>{exampleData(feature)}</td></tr>\n"
NodeItemTextHTML+=f"</table>\n"
pageHTML+=NodeItemTextHTML
pageMD+=NodeItemTextMD
pageHTML+=f'{footerHTML}'
pageMD+=f'{footerMD}'
# Write to file by calling common function
if typeOutput in ('md','both'):
fileNameMD = os.path.join(resultLocation, "featurebynodetype.md")
writeToFile(fileNameMD, pageMD, 'md', verbose)
if typeOutput in ('html','both'):
fileNameHTML = os.path.join(resultLocation, "featurebynodetype.htm")
writeToFile(fileNameHTML, pageHTML, 'html', verbose)
####################################################################
# Create the page with overview per data type (string or integer) #
####################################################################
pageMD=f'{pageTitleMD}\n# Overview features per datatype\n'
pageHTML=f'<html><head>{htmlStyle}</head><body><p>{pageTitleHTML}</p>\n<h1>Overview features per datatype</hl>'
# Sort the list alphabetically based on the second item of each tuple (featureName)
dataTypeFeatureList = sorted(dataTypeFeatureList, key=lambda x: x[1])
DataItemTextMD=DataItemTextHTML=''
for DataType in ('Integer','String'):
DataItemTextMD=f'## {DataType}\n\nFeature|Featuretype|Available on nodes|Description|Examples\n---|---|---|---|---\n'
DataItemTextHTML=f'<h2 id=\"{DataType}\">{DataType}</h2>\n<table><tr><th>Feature</th><th>Featuretype</th><th>Available on nodes</th><th>Description</th><th>Examples</th></tr>\n'
for feature, featureDataType in dataTypeFeatureList:
if featureDataType == DataType:
featureDescription=featureDict[feature].get('descr')
featureType=featureDict[feature].get('type')
nodeListMD=nodeListHTML=''
for thisNode in featureDict[feature]['freqlist']:
nodeListMD+=f'[`{thisNode}`](featurebynodetype.md#{thisNode}) '
nodeListHTML+=f'<a href=\"featurebynodetype.htm#{thisNode}\">{thisNode}</a> '
DataItemTextMD+=f"[`{feature}`]({feature}.md#readme)|[`{featureType}`](featurebytype.md#{featureType.lower()})|{nodeListMD}|{featureDescription}|{exampleData(feature)}\n"
DataItemTextHTML+=f"<tr><td><a href=\"{feature}.htm#start\">{feature}</a></td><td><a href=\"featurebytype.htm#{featureType}\">{featureType}</a></td><td>{nodeListHTML}</td><td>{featureDescription}</td><td>{exampleData(feature)}</td></tr>\n"
DataItemTextHTML+=f"</table>\n"
pageMD+=DataItemTextMD
pageHTML+=DataItemTextHTML
pageHTML+=f'{footerHTML}'
pageMD+=f'{footerMD}'
# Write to file by calling common function
if typeOutput in ('md','both'):
fileNameMD = os.path.join(resultLocation, "featurebydatatype.md")
writeToFile(fileNameMD, pageMD, 'md', verbose)
if typeOutput in ('html','both'):
fileNameHTML = os.path.join(resultLocation, "featurebydatatype.htm")
writeToFile(fileNameHTML, pageHTML, 'html', verbose)
##################################################################
# Create the page with overview per feature type (edge or node) #
##################################################################
pageMD=f'{pageTitleMD}\n# Overview features per type\n'
pageHTML=f'<html><head>{htmlStyle}</head><body><p>{pageTitleHTML}</p>\n<h1 id=\"start\">Overview features per type</hl>'
# Sort the list alphabetically based on the second item of each tuple (nodetype)
typeFeatureList = sorted(typeFeatureList, key=lambda x: x[1])
for featureType in ('Node','Edge'):
ItemTextMD=f'## {featureType}\n\nFeature|Datatype|Available on nodes|Description|Examples\n---|---|---|---|---\n'
ItemTextHTML=f'<h2 id=\"{featureType}\">{featureType}</h2>\n<table><tr><th>Feature</th><th>Datatype</th><th>Available on nodes</th><th>Description</th><th>Examples</th></tr>\n'
for thisFeature, thisFeatureType in typeFeatureList:
if featureType == thisFeatureType:
featureDescription=featureDict[thisFeature].get('descr')
featureDataType=featureDict[thisFeature].get('datatype')
nodeListMD=nodeListHTML=''
for thisNode in featureDict[thisFeature]['freqlist']:
nodeListMD+=f'[`{thisNode}`](featurebynodetype.md#{thisNode}) '
nodeListHTML+=f'<a href=\"featurebynodetype.htm#{thisNode}\">{thisNode}</a> '
ItemTextMD+=f"[`{thisFeature}`]({thisFeature}.md#readme)|[`{featureDataType}`](featurebydatatype.md#{featureDataType.lower()})|{nodeListMD}|{featureDescription}|{exampleData(thisFeature)}\n"
ItemTextHTML+=f"<tr><td><a href=\"{thisFeature}.htm\">{thisFeature}</a></td><td><a href=\"featurebydatatype.htm#{featureDataType}\">{featureDataType}</a></td><td>{nodeListHTML}</td><td>{featureDescription}</td><td>{exampleData(thisFeature)}</td></tr>\n"
ItemTextHTML+=f"</table>\n"
pageMD+=ItemTextMD
pageHTML+=ItemTextHTML
pageHTML+=f'{footerHTML}'
pageMD+=f'{footerMD}'
# Write to file by calling common function
if typeOutput in ('md','both'):
fileNameMD = os.path.join(resultLocation, "featurebytype.md")
writeToFile(fileNameMD, pageMD, 'md', verbose)
if typeOutput in ('html','both'):
fileNameHTML = os.path.join(resultLocation, "featurebytype.htm")
writeToFile(fileNameHTML, pageHTML, 'html', verbose)
# Reporting the number of files created
if filesCreated != 0:
print(f'Finished (written {filesCreated} {"html and md" if typeOutput == "both" else typeOutput} files to directory {pathFull + resultLocation})')
else:
print('No files written')
Finished (written 6 html and md files to directory C:\Users\tonyj\OneDrive\Documents\GitHub\Doc4TF\results)
Licenced under Creative Commons Attribution 4.0 International (CC BY 4.0)