#!/usr/bin/env python
# coding: utf-8

# # [Doc4TF](https://github.com/tonyjurg/Doc4TF)
# #### *Automatic creation of feature documentation for existing Text-Fabric datasets*
# 
# Version: 0.3 (Jan. 24, 2024); fixing bug [10](https://github.com/tonyjurg/Doc4TF/issues/10) (Feb. 2, 2024)

# ## Table of content <a class="anchor" id="TOC"></a>
# * <a href="#bullet1">1 - Introduction</a>
# * <a href="#bullet2">2 - Setting up the environment</a>
# * <a href="#bullet3">3 - Load Text-Fabric data</a>
# * <a href="#bullet4">4 - Creation of the dataset</a>
#     * <a href="#bullet4x1">4.1 - Setting up some global variables</a>
#     * <a href="#bullet4x2">4.2 -  Store all relevant data into a dictionary</a>
# * <a href="#bullet5">5 - Create the documentation pages</a>
#     * <a href="#bullet5x1">5.1 - Create the set of feature pages</a>
#     * <a href="#bullet5x2">5.2 - Create the index pages</a>
# * <a href="#bullet6">6 - Licence</a>

# # 1 - Introduction <a class="anchor" id="bullet1"></a>
# ##### [Back to TOC](#TOC)
# 
# Ideally, a comprehensive documentation set should be created as part of developing a Text-Fabric dataset. However, in practice, this is not always completed during the initial phase or after changes to features. This Jupyter Notebook contains Python code to automatically generate (and thus ensure consistency) a documentation set for any [Text-Fabric](https://github.com/annotation/text-fabric) dataset. It serves as a robust starting point for the development of a brand new documentation set or as validation for an existing one. One major advantage is that the resulting documentation set is fully hyperlinked, a task that can be laborious if done manually.
# 
# The main steps in producing the documentation set are:
# * Load a Text-Fabric database
# * Execute the code pressent in the subsequent cells. The code will:
#    * Construct the python dictionarie stroring relevant data from the TF datase 
#    * Create separate files for each feature
#    * Create a set of overview pages sorting the nodes accordingly 
#    
# The output format can be either Markdown, the standard for feature documentation stored on GitHub using its on-site processor, or HTML, which facilitates local storage and browsing with any web browser.

# # 2. Setting up the environment<a class="anchor" id="bullet2"></a>
# ##### [Back to TOC](#TOC)

# Your environment should (for obvious reasons) include the Python package `Text-Fabric`. If not installed yet, it can be installed using `pip`. Further it is required to be able to invoke the Text-Fabric data set (either from an online resource, or from a localy stored copy). There are no further requirements as the scripts basicly operate 'stand alone'.  

# # 3 - Load Text-Fabric data <a class="anchor" id="bullet3"></a>
# ##### [Back to TOC](#TOC)

# At this step, the Text-Fabric dataset is loaded, which embedded data will be used to create a documentation set. For various options regarding other possible storage locations, see the documentation for function [`use`](https://annotation.github.io/text-fabric/tf/app.html#tf.app.use).

# In[1]:


get_ipython().run_line_magic('load_ext', 'autoreload')
get_ipython().run_line_magic('autoreload', '2')


# In[2]:


# Loading the Text-Fabric code
# Note: it is assumed Text-Fabric is installed in your environment
from tf.fabric import Fabric
from tf.app import use


# In[3]:


# load the app and data
A = use ("saulocantanhede/tfgreek2", version="0.5.5", hoist=globals())


# # 4 - Creation of the dataset<a class="anchor" id="bullet4"></a>

# ## 4.1 - Setting up some global variables<a class="anchor" id="bullet4x1"></a>
# ##### [Back to TOC](#TOC)

# In[4]:


# If the following variable is set, it will be used as title for all pages. It is intended to the describe the dataset in one line
# customPageTitleMD="N1904 Greek New Testament [saulocantanhede/tfgreek2 - 0.5.4](https://github.com/saulocantanhede/tfgreek2)"
# customPageTitleHTML="N1904 Greek New Testament <a href=\"https://github.com/saulocantanhede/tfgreek2\">saulocantanhede/tfgreek2 - 0.5.4</a>"

# Specify the location to store the resulting files, relative to the location of this notebook (without a trailing slash).
resultLocation = "results"

# Type of output format ('html' for HTML, 'md' for Mark Down, or 'both' for both HTML and Mark Down)
typeOutput='both'

# HTML table style definition (only relevant for HTML output format)
htmlStyle='<style>\ntable {\nborder-collapse: collapse;\n}\n th, td {\nborder: 1px solid black;\n padding: 8px;\n}\nth {\nfont-weight: bold;\n}\n</style>'

# Limit the number of entries in the frequency tables per node type on each feature description page to this number
tableLimit=10

# This switch can be set to 'True' if you want additional information, such as dictionary entries and file details, to be printed. For basic output, set this switch to 'False'.
verbose=False

# The version number of the script
scriptVersion="0.3"
scriptDate="Jan. 24, 2024"


# Create the footers for MD and HTML, include today's date
from datetime import datetime
today = datetime.today()
formatted_date = today.strftime("%b. %d, %Y")
footerMD=f'\n\nCreated on {formatted_date} using [Doc4TF  version {scriptVersion} ({scriptDate})](https://github.com/tonyjurg/Doc4TF)'
footerHTML=f'\n<p>Created on {formatted_date} using <a href=\"https://github.com/tonyjurg/Doc4TF\">Doc4TF - version {scriptVersion} ({scriptDate})</a></p></body></html>'


# ## 4.2 - Store all relevant data into a dictionary<a class="anchor" id="bullet4x2"></a>
# ##### [Back to TOC](#TOC)

# The following will create a dictionary containing all relevant information for the loaded node and edge features.

# In[5]:


# Initialize an empty dictionary to store feature data
featureDict = {}

# Function to get feature description from metadata
def get_feature_description(metaData):
    return metaData.get('description', "No feature description")

# Function to set data type based on 'valueType' in metadata
def set_data_type(metaData):
    if 'valueType' in metaData:
        return "String" if metaData["valueType"] == 'str' else "Integer"
    return "Unknown"

# Function to process and add feature data to the dictionary
def process_feature(feature, featureType, featureMethod):
    # Obtain the meta data
    featureMetaData = featureMethod(feature).meta
    featureDescription = get_feature_description(featureMetaData)
    dataType = set_data_type(featureMetaData)

    # Initialize dictionary to store feature frequency data
    featureFrequencyDict = {}

    # Skip for specific features based on type
    if not (featureType == 'Node' and feature == 'otype') and not (featureType == 'Edge' and feature == 'oslots'):
        for nodeType in F.otype.all:
            frequencyLists = featureMethod(feature).freqList(nodeType)
            if not isinstance(frequencyLists, int):
                if len(frequencyLists)!=0:
                    featureFrequencyDict[nodeType] = {'nodetype': nodeType, 'freq': frequencyLists[:tableLimit]}
            elif isinstance(frequencyLists, int):
                if frequencyLists != 0:
                    featureFrequencyDict[nodeType] = {'nodetype': nodeType, 'freq': [("Link", frequencyLists)]}

    # Add processed feature data to the main dictionary
    featureDict[feature] = {'name': feature, 'descr': featureDescription, 'type': featureType, 'datatype': dataType, 'freqlist': featureFrequencyDict}
    
########################################################
#                     MAIN FUNCTION                    #
########################################################

########################################################
#             Gather general information               #
########################################################

print('Gathering generic details')

# Initialize default values
corpusName = A.appName
liveName = ''
versionName = A.version

# Trying to locate corpus information
if A.provenance:
    for parts in A.provenance[0]: 
        if isinstance(parts, tuple):
            key, value = parts[0], parts[1]
            if verbose: print (f'General info: {key}={value}')
            if key == 'corpus': corpusName = value
            if key == 'version': versionName = value
            # value for live is a tuple
            if key == 'live': liveName=value[1]
if liveName is not None and len(liveName)>1:
    # an URL was found
    pageTitleMD = f'Doc4TF pages for [{corpusName}]({liveName}) (version {versionName})'
    pageTitleHTML = f'<h1>Doc4TF pages for <a href="{liveName}">{corpusName}</a> (version {versionName})</h1>'
else:
    # No URL found
    pageTitleMD = f'Doc4TF pages for {corpusName} (version {versionName})'
    pageTitleHTML = f'<h1>Doc4TF pages for {corpusName} (version {versionName})</h1>'

# Overwrite in case user provided a title
if 'customPageTitleMD_' in globals():
    pageTitleMD = customPageTitleMD
if 'customPageTitleHTML' in globals():
    pageTitleMD = customPageTitleHTML

    
########################################################
#             Processing node features                 #
########################################################

print('Analyzing Node Features: ', end='')
for nodeFeature in Fall():
    if not verbose: print('.', end='')  # Progress indicator
    process_feature(nodeFeature, 'Node', Fs)
    if verbose: print(f'\nFeature {nodeFeature} = {featureDict[nodeFeature]}\n')  # Print feature data if verbose

########################################################
#             Processing edge features                 #
########################################################

print('\nAnalyzing Edge Features: ', end='')
for edgeFeature in Eall():
    if not verbose: print('.', end='')  # Progress indicator
    process_feature(edgeFeature, 'Edge', Es)
    if verbose: print(f'\nFeature {edgeFeature} = {featureDict[edgeFeature]}\n')  # Print feature data if verbose

print('\nFinished')


# ## 5 - Create the documentation pages<a class="anchor" id="bullet5"></a>
# 
# Two types of pages will be created:
#    * Feature description pages (one per feature)
#    * Set of index pages (linking to the feature pages)

# ## 5.1 - Create the set of feature pages<a class="anchor" id="bullet5x1"></a>
# ##### [Back to TOC](#TOC)

# In[6]:


import os

# Initialize a counter for the number of files created
filesCreated = 0
# Get the current working directory and append a backslash for path building
pathFull = os.getcwd() + '\\'

# Iterating over each feature in the feature dictionary
for featureName, featureData in featureDict.items():
    # Extracting various properties of each feature
    featureDescription = featureData.get('descr')
    featureType = featureData.get('type')
    featureDataType = featureData.get('datatype')
    
    # Initializing strings to accumulate HTML and Markdown content
    nodeListHTML = nodeListMD = ''
    tableListHTML = tableListMD = ''
    frequencyData = featureData.get('freqlist')

    # Processing frequency data for each node
    for node in frequencyData:
        # Building HTML and Markdown links for each node
        nodeListHTML += f' <a href=\"featurebynodetype.htm#{node}\">{node}</a>'
        nodeListMD += f' [`{node}`](featurebynodetype.md#{node}) '

        # Starting HTML and Markdown tables for frequency data
        tableListHTML += f'<h3>Frequency for nodetype <a href=\"featurebynodetype.htm#{node}\">{node}</a></h3><table><tr><th>Value</th><th>Occurences</th></tr>'
        tableListMD += f'### Frequency for nodetype [{node}](featurebynodetype.md#{node})\nValue|Occurences\n---|---\n'

        # Populating tables with frequency data
        itemData = frequencyData.get(node).get('freq')
        for item in itemData:
            handleSpace = item[0] if item[0] != ' ' else 'space' # prevent garbling of tables where the value itself is a space
            tableListHTML += f'<tr><td>{handleSpace}</td><td>{item[1]}</td></tr>'
            tableListMD += f'{handleSpace}|{item[1]}\n'
        tableListHTML += f'</table>\n'

    # Creating info blocks for HTML and Markdown
    infoBlockHTML = f'<table><tr><th>Data type</th><th>Feature type</th><th>Available for nodes</th></tr><tr><td><a href=\"featurebydatatype.htm#{featureDataType}\">{featureDataType}</a></td><td><a href="featurebytype.htm#{featureType}">{featureType}</a></td><td>{nodeListHTML}</td></tr></table>'
    infoBlockMD = f'Data type|Feature type|Available for nodes\n---|---|---\n[`{featureDataType}`](featurebydatatype.md#{featureDataType.lower()})|[`{featureType}`](featurebytype.md#{featureType.lower()})|{nodeListMD}'

    # Outputting in Markdown format
    if typeOutput in ('md','both'):
        pageMD = f'{pageTitleMD}\n# Feature: {featureName}\n{infoBlockMD}\n## Description\n{featureDescription}\n## Feature Values\n{tableListMD} {footerMD} '
        fileNameMD = os.path.join(resultLocation, f"{featureName}.md")
        try:
            with open(fileNameMD, "w", encoding="utf-8") as file:
                file.write(pageMD)
                filesCreated += 1
                # Log if verbose mode is on
                if verbose: print(f"Markdown content written to {pathFull + fileNameMD}")
        except Exception as e:
            print(f"Exception: {e}")
            break  # Stops execution on encountering an exception

    # Outputting in HTML format
    if typeOutput in ('html','both'):
        pageHTML = f'<html><head>{htmlStyle}</head><body><p>{pageTitleHTML}</p>\n<h1 id=\"start\">Feature: {featureName}</h1>\n{infoBlockHTML}\n<h2>Description</h2>\n<p>{featureDescription}</p>\n<h2>Feature Values</h2>\n{tableListHTML} {footerHTML}'
        fileNameHTML = os.path.join(resultLocation, f"{featureName}.htm")
        try:
            with open(fileNameHTML, "w", encoding="utf-8") as file:
                file.write(pageHTML)
                filesCreated += 1
                # Log if verbose mode is on
                if verbose: print(f"HTML content written to {pathFull + fileNameHTML}")
        except Exception as e:
            print(f"Exception: {e}")
            break  # Stops execution on encountering an exception

# Reporting the number of files created
if filesCreated != 0:
    print(f'Finished (written {filesCreated} {"html and md" if typeOutput == "both" else typeOutput} files to directory {pathFull + resultLocation})')
else:
    print('No files written')


# ## 5.2 - Create the index pages<a class="anchor" id="bullet5x2"></a>
# ##### [Back to TOC](#TOC)

# In[7]:


# Initialize a counter for the number of files created
filesCreated = 0


# Example data function to create a list of examples for a given feature
def exampleData(feature):
    # Check if the feature exists in featureDict and has non-empty freqlist.
    if feature in featureDict and featureDict[feature]['freqlist']:
        # Get the first value from the freqlist
        freq_list = next(iter(featureDict[feature]['freqlist'].values()))['freq']
        # Use list comprehension to create the example list. 
        example_list = ' '.join(f'`{item[0]}`' for item in freq_list[:4])
        return example_list
    else:
        return "No values"

    
def writeToFile(fileName, content, fileType, verbose):
    """
    Writes content to a file.
    :param fileName: The name of the file to write to.
    :param content: The content to write.
    :param fileType: The type of file (e.g., 'md' for Markdown, 'html' for HTML).
    :param verbose: If True, prints a message upon successful writing.
    """
    global filesCreated
    try:
        with open(fileName, "w", encoding="utf-8") as file:
            file.write(content)
            filesCreated+=1
            if verbose: 
                print(f"{fileType.upper()} content written to {fileName}")
    except Exception as e:
        print(f"Exception while writing {fileType.upper()} file: {e}")

# Set up some lists
nodeFeatureList = []
typeFeatureList = []
dataTypeFeatureList = []

for featureName, featureData in featureDict.items():
    typeFeatureList.append((featureName,featureData.get('type')))
    dataTypeFeatureList.append((featureName,featureData.get('datatype')))
    for node in featureData.get('freqlist'):
        nodeFeatureList.append((node, featureName))
        
########################################################### 
# Create the page with overview per node type (e.g. word) #
###########################################################
        
pageMD=f'{pageTitleMD}\n# Overview features per nodetype\n'
pageHTML=f'<html><head>{htmlStyle}</head><body><p>{pageTitleHTML}</p>\n<h1>Overview features per nodetype</h1>'

# Sort the list alphabetically based on the second item of each tuple (featureName)
nodeFeatureList = sorted(nodeFeatureList, key=lambda x: x[1])
# Iterate over node types
for NodeType in F.otype.all:
    NodeItemTextMD=f'## {NodeType}\n\nFeature|Featuretype|Datatype|Description|Examples\n---|---|---|---|---\n' 
    NodeItemTextHTML=f'<h2 id=\"{NodeType}\">{NodeType}</h2>\n<table><tr><th>Feature</th><th>Featuretype</th><th>Datatype</th><th>Description</th><th>Examples</th></tr>\n' 
    for node, feature in nodeFeatureList:
        if node == NodeType: 
            featureData=featureDict[feature]
            featureDescription=featureData.get('descr')    
            featureType=featureData.get('type')  
            featureDataType=featureData.get('datatype')
            NodeItemTextMD+=f"[`{feature}`]({feature}.md#readme)|[`{featureType}`](featurebytype.md#{featureType})|[`{featureDataType}`](featurebydatatype.md#{featureDataType})|{featureDescription}|{exampleData(feature)}\n"
            NodeItemTextHTML+=f"<tr><td><a href=\"{feature}.htm#start\">{feature}</a></td><td><a href=\"featurebytype.htm#{featureType}\">{featureType}</td><td><a href=\"featurebydatatype.htm#{featureDataType}\">{featureDataType}</a></td><td>{featureDescription}</td><td>{exampleData(feature)}</td></tr>\n"
    NodeItemTextHTML+=f"</table>\n"
    pageHTML+=NodeItemTextHTML
    pageMD+=NodeItemTextMD
    
pageHTML+=f'{footerHTML}'
pageMD+=f'{footerMD}'
    
# Write to file by calling common function
if typeOutput in ('md','both'):
    fileNameMD = os.path.join(resultLocation, "featurebynodetype.md")
    writeToFile(fileNameMD, pageMD, 'md', verbose)

if typeOutput in ('html','both'):
    fileNameHTML = os.path.join(resultLocation, "featurebynodetype.htm")
    writeToFile(fileNameHTML, pageHTML, 'html', verbose)

####################################################################
# Create the page with overview per data type  (string or integer) #
####################################################################

pageMD=f'{pageTitleMD}\n# Overview features per datatype\n'
pageHTML=f'<html><head>{htmlStyle}</head><body><p>{pageTitleHTML}</p>\n<h1>Overview features per datatype</hl>'

# Sort the list alphabetically based on the second item of each tuple (featureName)
dataTypeFeatureList = sorted(dataTypeFeatureList, key=lambda x: x[1])

DataItemTextMD=DataItemTextHTML=''
for DataType in ('Integer','String'):
    DataItemTextMD=f'## {DataType}\n\nFeature|Featuretype|Available on nodes|Description|Examples\n---|---|---|---|---\n' 
    DataItemTextHTML=f'<h2 id=\"{DataType}\">{DataType}</h2>\n<table><tr><th>Feature</th><th>Featuretype</th><th>Available on nodes</th><th>Description</th><th>Examples</th></tr>\n' 
    for feature, featureDataType in dataTypeFeatureList:  
        if featureDataType == DataType: 
            featureDescription=featureDict[feature].get('descr')    
            featureType=featureDict[feature].get('type')  
            nodeListMD=nodeListHTML=''
            for thisNode in featureDict[feature]['freqlist']:
                nodeListMD+=f'[`{thisNode}`](featurebynodetype.md#{thisNode}) '
                nodeListHTML+=f'<a href=\"featurebynodetype.htm#{thisNode}\">{thisNode}</a> '
            DataItemTextMD+=f"[`{feature}`]({feature}.md#readme)|[`{featureType}`](featurebytype.md#{featureType.lower()})|{nodeListMD}|{featureDescription}|{exampleData(feature)}\n"
            DataItemTextHTML+=f"<tr><td><a href=\"{feature}.htm#start\">{feature}</a></td><td><a href=\"featurebytype.htm#{featureType}\">{featureType}</a></td><td>{nodeListHTML}</td><td>{featureDescription}</td><td>{exampleData(feature)}</td></tr>\n"
    DataItemTextHTML+=f"</table>\n"
    pageMD+=DataItemTextMD
    pageHTML+=DataItemTextHTML

pageHTML+=f'{footerHTML}'
pageMD+=f'{footerMD}'
    
   
# Write to file by calling common function
if typeOutput in ('md','both'):
    fileNameMD = os.path.join(resultLocation, "featurebydatatype.md")
    writeToFile(fileNameMD, pageMD, 'md', verbose)

if typeOutput in ('html','both'):
    fileNameHTML = os.path.join(resultLocation, "featurebydatatype.htm")
    writeToFile(fileNameHTML, pageHTML, 'html', verbose)
    
##################################################################
# Create the page with overview per feature type  (edge or node) #
##################################################################

pageMD=f'{pageTitleMD}\n# Overview features per type\n'
pageHTML=f'<html><head>{htmlStyle}</head><body><p>{pageTitleHTML}</p>\n<h1 id=\"start\">Overview features per type</hl>'

# Sort the list alphabetically based on the second item of each tuple (nodetype)
typeFeatureList = sorted(typeFeatureList, key=lambda x: x[1])
for featureType in ('Node','Edge'):
    ItemTextMD=f'## {featureType}\n\nFeature|Datatype|Available on nodes|Description|Examples\n---|---|---|---|---\n' 
    ItemTextHTML=f'<h2 id=\"{featureType}\">{featureType}</h2>\n<table><tr><th>Feature</th><th>Datatype</th><th>Available on nodes</th><th>Description</th><th>Examples</th></tr>\n' 
    for thisFeature, thisFeatureType in typeFeatureList: 
        if featureType == thisFeatureType:
            featureDescription=featureDict[thisFeature].get('descr')
            featureDataType=featureDict[thisFeature].get('datatype')
            nodeListMD=nodeListHTML=''
            for thisNode in featureDict[thisFeature]['freqlist']:
                nodeListMD+=f'[`{thisNode}`](featurebynodetype.md#{thisNode}) '
                nodeListHTML+=f'<a href=\"featurebynodetype.htm#{thisNode}\">{thisNode}</a> '
            ItemTextMD+=f"[`{thisFeature}`]({thisFeature}.md#readme)|[`{featureDataType}`](featurebydatatype.md#{featureDataType.lower()})|{nodeListMD}|{featureDescription}|{exampleData(thisFeature)}\n"
            ItemTextHTML+=f"<tr><td><a href=\"{thisFeature}.htm\">{thisFeature}</a></td><td><a href=\"featurebydatatype.htm#{featureDataType}\">{featureDataType}</a></td><td>{nodeListHTML}</td><td>{featureDescription}</td><td>{exampleData(thisFeature)}</td></tr>\n"
    ItemTextHTML+=f"</table>\n"
    pageMD+=ItemTextMD
    pageHTML+=ItemTextHTML

pageHTML+=f'{footerHTML}'
pageMD+=f'{footerMD}'

# Write to file by calling common function
if typeOutput in ('md','both'):
    fileNameMD = os.path.join(resultLocation, "featurebytype.md")
    writeToFile(fileNameMD, pageMD, 'md', verbose)

if typeOutput in ('html','both'):
    fileNameHTML = os.path.join(resultLocation, "featurebytype.htm")
    writeToFile(fileNameHTML, pageHTML, 'html', verbose)
    

# Reporting the number of files created
if filesCreated != 0:
    print(f'Finished (written {filesCreated} {"html and md" if typeOutput == "both" else typeOutput} files to directory {pathFull + resultLocation})')
else:
    print('No files written')


# # 6 - License<a class="anchor" id="bullet6"></a>
# ##### [Back to TOC](#TOC)

# Licenced under [Creative Commons Attribution 4.0 International (CC BY 4.0)](https://github.com/tonyjurg/Doc4TF/blob/main/LICENCE.md)