#!/usr/bin/env python # coding: utf-8 # # [Doc4TF](https://github.com/tonyjurg/Doc4TF) # #### *Automatic creation of feature documentation for existing Text-Fabric datasets* # # Version and release date: # In[1]: scriptVersion="0.5" scriptDate="May 13, 2024" # Additional notes on instal requirements added May 26, 2024. # ## Table of content # * 1 - Introduction # * 2 - Setting up the environment # * 3 - Load Text-Fabric data # * 4 - Creation of the dataset # * 4.1 - Setting up some global variables # * 4.2 - Store all relevant data into a dictionary # * 5 - Create the documentation pages # * 5.1 - Create the set of feature pages # * 5.2 - Create the index pages # * 6 - Changelog # * 7 - Licence # # 1 - Introduction # ##### [Back to TOC](#TOC) # # Ideally, a comprehensive documentation set should be created as part of developing a Text-Fabric dataset. However, in practice, this is not always completed during the initial phase or after changes to features. This Jupyter Notebook contains Python code to automatically generate (and thus ensure consistency) a documentation set for any [Text-Fabric](https://github.com/annotation/text-fabric) dataset. It serves as a robust starting point for the development of a brand new documentation set or as validation for an existing one. One major advantage is that the resulting documentation set is fully hyperlinked, a task that can be laborious if done manually. # # The main steps in producing the documentation set are: # * Load a Text-Fabric database # * Execute the code pressent in the subsequent cells. The code will: # * Construct the python dictionary stroring relevant data from the TF datase # * Create separate files for each feature # * Create a set of overview pages sorting the nodes accordingly # # The output format can be either Markdown, the standard for feature documentation stored on GitHub using its on-site processor, or HTML, which facilitates local storage and browsing with any web browser. # # 2. Setting up the environment # ##### [Back to TOC](#TOC) # Your environment should (for obvious reasons) include the Python package `Text-Fabric`. Text-Fabric requires at least Python version 3.7.0. If not installed yet, it can be installed using `pip`. More details on installing the Text-Fabric package can be found in [tf.about.install](https://annotation.github.io/text-fabric/tf/about/install.html). # # Further it is required to be able to invoke the Text-Fabric data set (either from an online resource, or from a localy stored copy). There are no further requirements as the scripts basicly operate 'stand alone'. # # 3 - Load Text-Fabric data # ##### [Back to TOC](#TOC) # At this step, the Text-Fabric dataset is loaded, which embedded data will be used to create a documentation set. # # Which dataset will be loaded is specified in the parameters as detailed below: # ``` # A = use ("{GitHub user name}/{repository name}", version="{version}", hoist=globals()) # ``` # For various options regarding other possible storage locations, and other load options, see the documentation for function [`use`](https://annotation.github.io/text-fabric/tf/app.html#tf.app.use). # In[3]: get_ipython().run_line_magic('load_ext', 'autoreload') get_ipython().run_line_magic('autoreload', '2') # In[5]: # Loading the Text-Fabric code # Note: it is assumed Text-Fabric is installed in your environment from tf.fabric import Fabric from tf.app import use # In[7]: # load the app and data A = use ("saulocantanhede/tfgreek2", version="0.5.7", hoist=globals()) # # 4 - Creation of the dataset # ## 4.1 - Setting up some global variables # ##### [Back to TOC](#TOC) # In[9]: # If the following variable is set, it will be used as title for all pages. It is intended to the describe the dataset in one line customPageTitleMD="N1904 Greek New Testament [saulocantanhede/tfgreek2 - 0.5.7](https://github.com/saulocantanhede/tfgreek2)" customPageTitleHTML="N1904 Greek New Testament saulocantanhede/tfgreek2 - 0.5.7" # Specify the location to store the resulting files, relative to the location of this notebook (without a trailing slash). resultLocation = "results" # Type of output format ('html' for HTML, 'md' for Mark Down, or 'both' for both HTML and Mark Down) typeOutput='md' # HTML table style definition (only relevant for HTML output format) htmlStyle='' # Limit the number of entries in the frequency tables per node type on each feature description page to this number tableLimit=10 # This switch can be set to 'True' if you want additional information, such as dictionary entries and file details, to be printed. For basic output, set this switch to 'False'. verbose=False # Create the footers for MD and HTML, include today's date from datetime import datetime today = datetime.today() formatted_date = today.strftime("%b. %d, %Y") footerMD=f'\n\nCreated on {formatted_date} using [Doc4TF version {scriptVersion} ({scriptDate})](https://github.com/tonyjurg/Doc4TF/blob/main/CreateFeatureDoc.ipynb)' footerHTML=f'\n

Created on {formatted_date} using Doc4TF - version {scriptVersion} ({scriptDate})

' # ## 4.2 - Store all relevant data into a dictionary # ##### [Back to TOC](#TOC) # The following will create a dictionary containing all relevant information for the loaded node and edge features. # In[11]: # Initialize an empty dictionary to store feature data featureDict = {} import time overallTime = time.time() def getFeatureDescription(metaData): """ This function looks for the 'description' key in the metadata dictionary. If the key is found, it returns the corresponding description. If the key is not present, it returns a default message indicating that no description is available. Parameters: metaData (dict): A dictionary containing metadata about a feature. Returns: str: The description of the feature if available, otherwise a default message. """ return metaData.get('description', "No feature description") def setDataType(metaData): """ This function checks for the 'valueType' key in the metadata. If the key is present, it returns 'String' if the value is 'str', and 'Integer' for other types. If the 'valueType' key is not present, it returns 'Unknown'. Parameters: metaData (dict): A dictionary containing metadata, including the 'valueType' of a feature. Returns: str: A string indicating the determined data type ('String', 'Integer', or 'Unknown'). """ if 'valueType' in metaData: return "String" if metaData["valueType"] == 'str' else "Integer" return "Unknown" def processFeature(feature, featureType, featureMethod): """ Processes a given feature by extracting metadata, description, and data type, and then compiles frequency data for different node types in a feature dictionary. Certain features are skipped based on their type. The processed data is added to a global feature dictionary. Parameters: feature (str): The name of the feature to be processed. featureType (str): The type of the feature ('Node' or 'Edge'). featureMethod (function): A function to obtain feature data. Returns: None: The function updates a global dictionary with processed feature data and does not return anything. """ # Obtain the meta data featureMetaData = featureMethod(feature).meta featureDescription = getFeatureDescription(featureMetaData) dataType = setDataType(featureMetaData) # Initialize dictionary to store feature frequency data featureFrequencyDict = {} # Skip for specific features based on type if not (featureType == 'Node' and feature == 'otype') and not (featureType == 'Edge' and feature == 'oslots'): for nodeType in F.otype.all: frequencyLists = featureMethod(feature).freqList(nodeType) if not isinstance(frequencyLists, int): if len(frequencyLists)!=0: featureFrequencyDict[nodeType] = {'nodetype': nodeType, 'freq': frequencyLists[:tableLimit]} elif isinstance(frequencyLists, int): if frequencyLists != 0: featureFrequencyDict[nodeType] = {'nodetype': nodeType, 'freq': [("Link", frequencyLists)]} # Add processed feature data to the main dictionary featureDict[feature] = {'name': feature, 'descr': featureDescription, 'type': featureType, 'datatype': dataType, 'freqlist': featureFrequencyDict} ######################################################## # MAIN FUNCTION # ######################################################## ######################################################## # Gather general information # ######################################################## print('Gathering generic details') # Initialize default values corpusName = A.appName liveName = '' versionName = A.version # Trying to locate corpus information if A.provenance: for parts in A.provenance[0]: if isinstance(parts, tuple): key, value = parts[0], parts[1] if verbose: print (f'General info: {key}={value}') if key == 'corpus': corpusName = value if key == 'version': versionName = value # value for live is a tuple if key == 'live': liveName=value[1] if liveName is not None and len(liveName)>1: # an URL was found pageTitleMD = f'Doc4TF pages for [{corpusName}]({liveName}) (version {versionName})' pageTitleHTML = f'

Doc4TF pages for {corpusName} (version {versionName})

' else: # No URL found pageTitleMD = f'Doc4TF pages for {corpusName} (version {versionName})' pageTitleHTML = f'

Doc4TF pages for {corpusName} (version {versionName})

' # Overwrite in case user provided a title if 'customPageTitleMD_' in globals(): pageTitleMD = customPageTitleMD if 'customPageTitleHTML' in globals(): pageTitleMD = customPageTitleHTML ######################################################## # Processing node features # ######################################################## print('Analyzing Node Features: ', end='') for nodeFeature in Fall(): if not verbose: print('.', end='') # Progress indicator processFeature(nodeFeature, 'Node', Fs) if verbose: print(f'\nFeature {nodeFeature} = {featureDict[nodeFeature]}\n') # Print feature data if verbose ######################################################## # Processing edge features # ######################################################## print('\nAnalyzing Edge Features: ', end='') for edgeFeature in Eall(): if not verbose: print('.', end='') # Progress indicator processFeature(edgeFeature, 'Edge', Es) if verbose: print(f'\nFeature {edgeFeature} = {featureDict[edgeFeature]}\n') # Print feature data if verbose print(f'\nFinished in {time.time() - overallTime:.2f} seconds.') # ## 5 - Create the documentation pages # # Two types of pages will be created: # * Feature description pages (one per feature) # * Set of index pages (linking to the feature pages) # ## 5.1 - Create the set of feature pages # ##### [Back to TOC](#TOC) # In[12]: import os import time overallTime = time.time() # Initialize a counter for the number of files created filesCreated = 0 # Get the current working directory and append a backslash for path building pathFull = os.getcwd() + '\\' # Iterating over each feature in the feature dictionary for featureName, featureData in featureDict.items(): # Extracting various properties of each feature featureDescription = featureData.get('descr') featureType = featureData.get('type') featureDataType = featureData.get('datatype') # Initializing strings to accumulate HTML and Markdown content nodeListHTML = nodeListMD = '' tableListHTML = tableListMD = '' frequencyData = featureData.get('freqlist') # Processing frequency data for each node for node in frequencyData: # Building HTML and Markdown links for each node nodeListHTML += f' {node}' nodeListMD += f' [`{node}`](featuresbynodetype.md#{node}) ' # Starting HTML and Markdown tables for frequency data tableListHTML += f'

Frequency for nodetype {node}

' tableListMD += f'### Frequency for nodetype [{node}](featuresbynodetype.md#{node})\nValue|Occurences\n---|---\n' # Populating tables with frequency data itemData = frequencyData.get(node).get('freq') for item in itemData: handleSpace = item[0] if item[0] != ' ' else 'space' # prevent garbling of tables where the value itself is a space tableListHTML += f'' tableListMD += f'{handleSpace}|{item[1]}\n' tableListHTML += f'
ValueOccurences
{handleSpace}{item[1]}
\n' # Creating info blocks for HTML and Markdown infoBlockHTML = f'
Data typeFeature typeAvailable for nodes
{featureDataType}{featureType}{nodeListHTML}
' infoBlockMD = f'Data type|Feature type|Available for nodes\n---|---|---\n[`{featureDataType}`](featuresbydatatype.md#{featureDataType.lower()})|[`{featureType}`](featuresbytype.md#{featureType.lower()})|{nodeListMD}' # Outputting in Markdown format if typeOutput in ('md','both'): pageMD = f'{pageTitleMD}\n# Feature: {featureName}\n{infoBlockMD}\n## Description\n{featureDescription}\n## Feature Values\n{tableListMD} {footerMD} ' fileNameMD = os.path.join(resultLocation, f"{featureName}.md") try: with open(fileNameMD, "w", encoding="utf-8") as file: file.write(pageMD) filesCreated += 1 # Log if verbose mode is on if verbose: print(f"Markdown content written to {pathFull + fileNameMD}") except Exception as e: print(f"Exception: {e}") break # Stops execution on encountering an exception # Outputting in HTML format if typeOutput in ('html','both'): pageHTML = f'{htmlStyle}

{pageTitleHTML}

\n

Feature: {featureName}

\n{infoBlockHTML}\n

Description

\n

{featureDescription}

\n

Feature Values

\n{tableListHTML} {footerHTML}' fileNameHTML = os.path.join(resultLocation, f"{featureName}.htm") try: with open(fileNameHTML, "w", encoding="utf-8") as file: file.write(pageHTML) filesCreated += 1 # Log if verbose mode is on if verbose: print(f"HTML content written to {pathFull + fileNameHTML}") except Exception as e: print(f"Exception: {e}") break # Stops execution on encountering an exception # Reporting the number of files created if filesCreated != 0: print(f'Finished in {time.time() - overallTime:.2f} seconds (written {filesCreated} {"html and md" if typeOutput == "both" else typeOutput} files to directory {pathFull + resultLocation})') else: print('No files written') # ## 5.2 - Create the index pages # ##### [Back to TOC](#TOC) # In[13]: import os import time overallTime = time.time() # Initialize a counter for the number of files created filesCreated = 0 def exampleData(feature): """ This function checks if the specified feature exists in the global `featureDict` and if it has a non-empty frequency list. If so, it extracts the first few values from this frequency list to create a list of examples. Parameters: feature (str): The name of the feature for which examples are to be created. Returns: str: A string containing the examples concatenated together. Returns "No values" if the feature does not exist in `featureDict` or if it has an empty frequency list. """ # Check if the feature exists in featureDict and has non-empty freqlist. if feature in featureDict and featureDict[feature]['freqlist']: # Get the first value from the freqlist freq_list = next(iter(featureDict[feature]['freqlist'].values()))['freq'] # Use list comprehension to create the example list. example_list = ' '.join(f'`{item[0]}`' for item in freq_list[:4]) return example_list else: return "No values" def writeToFile(fileName, content, fileType, verbose): """ Writes provided content to a specified file. If verbose is True, prints a confirmation message. This function attempts to write the given content to a file with the specified name. It handles any exceptions during writing and can optionally print a message upon successful writing. The function also increments a global counter `filesCreated` for each successful write operation. Parameters: fileName (str): The name of the file to write to. content (str): The content to be written to the file. fileType (str): The type of file (used for informational messages; e.g., 'md' for Markdown, 'html' for HTML). verbose (bool): If True, prints a message upon successful writing. Returns: None: The function does not return a value but writes content to a file and may print messages. """ global filesCreated try: with open(fileName, "w", encoding="utf-8") as file: file.write(content) filesCreated+=1 if verbose: print(f"{fileType.upper()} content written to {fileName}") except Exception as e: print(f"Exception while writing {fileType.upper()} file: {e}") # Set up some lists nodeFeatureList = [] typeFeatureList = [] dataTypeFeatureList = [] for featureName, featureData in featureDict.items(): typeFeatureList.append((featureName,featureData.get('type'))) dataTypeFeatureList.append((featureName,featureData.get('datatype'))) for node in featureData.get('freqlist'): nodeFeatureList.append((node, featureName)) ########################################################### # Create the page with overview per node type (e.g. word) # ########################################################### pageMD=f'{pageTitleMD}\n# Overview features per nodetype\n' pageHTML=f'{htmlStyle}

{pageTitleHTML}

\n

Overview features per nodetype

' # Sort the list alphabetically based on the second item of each tuple (featureName) nodeFeatureList = sorted(nodeFeatureList, key=lambda x: x[1]) # Iterate over node types for NodeType in F.otype.all: NodeItemTextMD=f'## {NodeType}\n\nFeature|Featuretype|Datatype|Description|Examples\n---|---|---|---|---\n' NodeItemTextHTML=f'

{NodeType}

\n\n' for node, feature in nodeFeatureList: if node == NodeType: featureData=featureDict[feature] featureDescription=featureData.get('descr') featureType=featureData.get('type') featureDataType=featureData.get('datatype') NodeItemTextMD+=f"[`{feature}`]({feature}.md#readme)|[`{featureType}`](featuresbytype.md#{featureType})|[`{featureDataType}`](featuresbydatatype.md#{featureDataType})|{featureDescription}|{exampleData(feature)}\n" NodeItemTextHTML+=f"\n" NodeItemTextHTML+=f"
FeatureFeaturetypeDatatypeDescriptionExamples
{feature}{featureType}{featureDataType}{featureDescription}{exampleData(feature)}
\n" pageHTML+=NodeItemTextHTML pageMD+=NodeItemTextMD pageHTML+=f'{footerHTML}' pageMD+=f'{footerMD}' # Write to file by calling common function if typeOutput in ('md','both'): fileNameMD = os.path.join(resultLocation, "featuresbynodetype.md") writeToFile(fileNameMD, pageMD, 'md', verbose) if typeOutput in ('html','both'): fileNameHTML = os.path.join(resultLocation, "featuresbynodetype.htm") writeToFile(fileNameHTML, pageHTML, 'html', verbose) #################################################################### # Create the page with overview per data type (string or integer) # #################################################################### pageMD=f'{pageTitleMD}\n# Overview features per datatype\n' pageHTML=f'{htmlStyle}

{pageTitleHTML}

\n

Overview features per datatype' # Sort the list alphabetically based on the second item of each tuple (featureName) dataTypeFeatureList = sorted(dataTypeFeatureList, key=lambda x: x[1]) DataItemTextMD=DataItemTextHTML='' for DataType in ('Integer','String'): DataItemTextMD=f'## {DataType}\n\nFeature|Featuretype|Available on nodes|Description|Examples\n---|---|---|---|---\n' DataItemTextHTML=f'

{DataType}

\n\n' for feature, featureDataType in dataTypeFeatureList: if featureDataType == DataType: featureDescription=featureDict[feature].get('descr') featureType=featureDict[feature].get('type') nodeListMD=nodeListHTML='' for thisNode in featureDict[feature]['freqlist']: nodeListMD+=f'[`{thisNode}`](featuresbynodetype.md#{thisNode}) ' nodeListHTML+=f'{thisNode} ' DataItemTextMD+=f"[`{feature}`]({feature}.md#readme)|[`{featureType}`](featuresbytype.md#{featureType.lower()})|{nodeListMD}|{featureDescription}|{exampleData(feature)}\n" DataItemTextHTML+=f"\n" DataItemTextHTML+=f"
FeatureFeaturetypeAvailable on nodesDescriptionExamples
{feature}{featureType}{nodeListHTML}{featureDescription}{exampleData(feature)}
\n" pageMD+=DataItemTextMD pageHTML+=DataItemTextHTML pageHTML+=f'{footerHTML}' pageMD+=f'{footerMD}' # Write to file by calling common function if typeOutput in ('md','both'): fileNameMD = os.path.join(resultLocation, "featuresbydatatype.md") writeToFile(fileNameMD, pageMD, 'md', verbose) if typeOutput in ('html','both'): fileNameHTML = os.path.join(resultLocation, "featuresbydatatype.htm") writeToFile(fileNameHTML, pageHTML, 'html', verbose) ################################################################## # Create the page with overview per feature type (edge or node) # ################################################################## pageMD=f'{pageTitleMD}\n# Overview features per type\n' pageHTML=f'{htmlStyle}

{pageTitleHTML}

\n

Overview features per type' # Sort the list alphabetically based on the second item of each tuple (nodetype) typeFeatureList = sorted(typeFeatureList, key=lambda x: x[1]) for featureType in ('Node','Edge'): ItemTextMD=f'## {featureType}\n\nFeature|Datatype|Available on nodes|Description|Examples\n---|---|---|---|---\n' ItemTextHTML=f'

{featureType}

\n\n' for thisFeature, thisFeatureType in typeFeatureList: if featureType == thisFeatureType: featureDescription=featureDict[thisFeature].get('descr') featureDataType=featureDict[thisFeature].get('datatype') nodeListMD=nodeListHTML='' for thisNode in featureDict[thisFeature]['freqlist']: nodeListMD+=f'[`{thisNode}`](featuresbynodetype.md#{thisNode}) ' nodeListHTML+=f'{thisNode} ' ItemTextMD+=f"[`{thisFeature}`]({thisFeature}.md#readme)|[`{featureDataType}`](featuresbydatatype.md#{featureDataType.lower()})|{nodeListMD}|{featureDescription}|{exampleData(thisFeature)}\n" ItemTextHTML+=f"\n" ItemTextHTML+=f"
FeatureDatatypeAvailable on nodesDescriptionExamples
{thisFeature}{featureDataType}{nodeListHTML}{featureDescription}{exampleData(thisFeature)}
\n" pageMD+=ItemTextMD pageHTML+=ItemTextHTML pageHTML+=f'{footerHTML}' pageMD+=f'{footerMD}' # Write to file by calling common function if typeOutput in ('md','both'): fileNameMD = os.path.join(resultLocation, "featuresbytype.md") writeToFile(fileNameMD, pageMD, 'md', verbose) if typeOutput in ('html','both'): fileNameHTML = os.path.join(resultLocation, "featuresbytype.htm") writeToFile(fileNameHTML, pageHTML, 'html', verbose) # Reporting the number of files created if filesCreated != 0: print(f'Finished in {time.time() - overallTime:.2f} seconds (written {filesCreated} {"html and md" if typeOutput == "both" else typeOutput} files to directory {pathFull + resultLocation})') else: print('No files written') # # 6 - Change log # ##### [Back to TOC](#TOC) # Changes to previous version ([0.4](https://github.com/tonyjurg/Doc4TF/blob/main/previous_versions/CreateFeatureDoc(0_4).ipynb)): # # * Cleaning up some spelling errors. # * Enhancement [#13](https://github.com/tonyjurg/Doc4TF/issues/13) (update names of index pages) # * Enhancement [#16](https://github.com/tonyjurg/Doc4TF/issues/16) (version number and date at start of Notebook) # # 7 - License # ##### [Back to TOC](#TOC) # Licenced under [Creative Commons Attribution 4.0 International (CC BY 4.0)](https://github.com/tonyjurg/Doc4TF/blob/main/LICENCE.md)