#!/usr/bin/env python
# coding: utf-8

# # Logging

# ## TL;DR
#  * There is a QCoDeS logging module: `qcodes.logger`
#  * Call `logger.start_all_logging` at the start of every script/session to make sure all log messages get stored to the `.qcodes/logs` folder in your home directory, or set `logger.start_logging_on_import` to `True` in your `qcodesrc.json` configuration.
#  * For debugging purposes you can log messages of an individual instrument (and see the VISA dialog).
#  * You can obtain all logging messages in a `pandas.DataFrame` for further analysis.

# ## Contents
# - [Introduction](#Introduction)
# - [Set up logging in QCoDeS](#Set-up-logging-in-QCoDeS)
#   - [QCoDeS default handlers](#QCoDeS-default-handlers)
#   - [IPython command history](#IPython-command-history)
# - [Temporarily elevating the logging level](#Temporarily-elevating-the-logging-level)
# - [Filtering log messages by instrument](#Filtering-log-messages-by-instrument)
# - [Capturing Pandas.DataFrame](#Capturing-Pandas.DataFrame)
# 

# ## Introduction
# Python offers an excellent logging framework that is widely used in QCoDeS. As a user, you might be interested in reading the log messages that are generated by QCoDeS for debugging and also profiling purposes.
# 
# All log records(i.e., messages and metadata) are created using a python `logging.logger` object. In QCoDeS there is a logger for every module with the same name as the module itself, i.e., you can use `logging.getLogger('qcodes.instrument.base')` for getting the logger of the QCoDeS `Instrument`. From the dots in the names of the loggers a hierarchy is defined, at the top of which is the so called *root logger* that catches all messages.
# The log records created in a logger are processed by the *handlers* that have been attached to this specific *logger*.
# The handlers can, for example, be used to output the log messages into different locations, like the console or a log file.
# 
# Every logger, handler and log record has an associated level. The default levels are: `CRITICAL`, `ERROR`, `WARNING`, `INFO` and `DEBUG`. The levels determine how messages are passed: a logger passes only records that exceeds its minimal passing level to its handlers and the handlers in turn only output those messages that exceed their minimal passing level.
# 
# 
# For a more detailed description of Python logging refer to the [official manual](https://docs.python.org/3.6/library/logging.html#module-logging) (This is all well summed up [in this chart](https://docs.python.org/3.6/howto/logging.html#logging-flow))

# ## Instrument communication loggers
# Not all the used loggers in QCoDeS follow the previously presented rule of naming the logger after the module. For instruments there are the following loggers:
#  - **qcodes.instrument.base.com**:
#    This logger logs all the communication of the `Instrument` and its descendants with the actual physical instruments, e.g.,  the messages generated by the `write` and `query` calls.
#  - **qcodes.instrument.base.com.visa**:
#   This is a sublogger that logs all the communication related messages of the `VisaInstrument` and its descendants.

# ## Set up logging in QCoDeS
# At the beginning of every experiment script file you should include the following lines:

# In[1]:


from qcodes.logger import start_all_logging

start_all_logging()


# The call to `start_all_logging` does two things:
# 1. Create handlers to output logging messages from the root logger to the console and to a log file.
# 2. Enable logging of the IPython commands into a separate log file.

# Alternatively we can configure qcodes to automatically start logging on every import of qcodes, by running the following code once. (This will persist the current configuration in `~\qcodesrc.json`)

# In[2]:


# Code is commented out to not set this by accident. Uncomment to activate
# from qcodes import config

# config.logger.start_logging_on_import = "always"
# config.save_to_home()


# ### QCoDeS default handlers
# The effect of the console handler can be seen in the two printed lines with a red background below the previous code cell. Every line corresponds to a log record. The fields of the records are separated by `¦` and have the following meaning:
#  0. time stamp
#  1. logger name, here the module name
#  2. record log level
#  3. name of the function that created the log message
#  4. line at which the message has been issued
#  5. actual log message
# 
# The same format is used for the file handler. By default QCoDeS logs to the file `<user directory>/.qcodes/logs/qcodes.log`. To avoid endlessly long files, the files roll over at midnight and get the date appended to their name where `qcodes.log` without date always gives you the log messages of the same day.
# 
# To configure the levels you want to log/handle, you need to edit the `qcodesrc.json` file in you home directory. The default values are:
# ```
#     "logger":{
#         "console_level": "WARNING",
#         "file_level": "INFO",
#         "logger_levels":{
#             "pyvisa": "INFO"
#         }
# ```
# While `console_level` and `file_level` describe the levels of the two default handlers described previously, `logger_levels` is a dictionary that can be used to limit the messages passed on by module level loggers. This means with these defaults, the `pyvisa` module will not log messages on a `DEBUG` level. Therefore setting the `console_level` to `DEBUG` will not show the `pyvisa` debug messages (you certainly don't want to see them as they log every single character that is passed to an instrument with an individual message). On the other hand setting the console level to warning will still suppress `pyvisa` info messages. 

# ### IPython command history
# The output above that follows these logging messages comes from IPython. It tells us that all issued IPython commands will be logged to a file. The file lives in the same log directory as the python logs and is called `command_history.log`. It will not roll over.
# 
# To change the command history logging directory you will need to call `start_command_history_logger` with the new path as an argument.

# ## Temporarily elevating the logging level
# Sometimes you might wish to catch more than that meets to the eye. To this end, one can temporarily elevate the logging level to catch all debug messages by using a context manager. This can simply be done via:

# In[3]:


import logging

from qcodes import logger

log = logging.getLogger("example_logger")

log.debug("This message will not be visible as the logging level is set to `DEBUG`")

with logger.console_level(logging.DEBUG):
    log.debug(
        "This message is visible as the logging level is temporarily elevated to `DEBUG`"
    )

log.debug("The level is back to what it used to be")


# For other handlers than the console handler, you can do the same thing using the `handler_level` context manager.

# ## Filtering log messages by instrument
# A very common use case for logging in QCoDeS is to analyze the communication with a VISA instrument. For this purpose the logger module provides a context manager to filter the logging messages by instrument.
# 
# Generally one could of course simply attach a handler to the logger of `qcodes.instrument_drivers.Vendor.Type` but this would not show the messages that are generated by the visa communication in `qcodes.instrument.visa`. Additionally with the approach implemented here one can not only filter messages for a given type of instrument but also of a specific instance.
# 
# To demonstrate this consider the following mock up of the AMI430 Magnet controller that consists of individual VISA instruments for each axis, here with the names `x`, `y` and `z`:

# In[4]:


import numpy as np

from qcodes.instrument_drivers.american_magnetics import AMIModel430, AMIModel4303D

mag_x = AMIModel430(
    "x", address="GPIB::1::INSTR", pyvisa_sim_file="AMI430.yaml", terminator="\n"
)
mag_y = AMIModel430(
    "y", address="GPIB::2::INSTR", pyvisa_sim_file="AMI430.yaml", terminator="\n"
)
mag_z = AMIModel430(
    "z", address="GPIB::3::INSTR", pyvisa_sim_file="AMI430.yaml", terminator="\n"
)

field_limit = [
    lambda x, y, z: x == 0 and y == 0 and z < 3,
    lambda x, y, z: np.linalg.norm([x, y, z]) < 2,
]

driver = AMIModel4303D("AMI430_3D", mag_x, mag_y, mag_z, field_limit)


# In the above messages you can see the prefix `[<instrument_name>(<instrument_type>)]`. To analyze the visa communication one can use `filter_instrument`:

# In[5]:


driver.cartesian((0, 0, 0))
with logger.console_level("DEBUG"):
    with logger.filter_instrument(mag_x):
        driver.cartesian((0, 0, 1))


# The output on the console shows as expected only messages from the `x` instrument. For multiple instruments pass a sequence of instruments to `filter_instrument` we write:

# In[6]:


driver.cartesian((0, 0, 0))
with logger.console_level("DEBUG"):
    with logger.filter_instrument((mag_x, mag_y)):
        driver.cartesian((0, 0, 1))


# ## Capturing Pandas.DataFrame
# To process the logs, especially with the timestamps, the [pandas module](https://pandas.pydata.org/) is highly suited. With `logger.log_to_dataframe` or `logger.logfile_to_dataframe` you can convert a log(file) to a pandas `DataFrame`. See [this](logfile_parsing.ipynb) notebook for an example.
# 
# You can also use a context manager to capture the logs directly into a `DataFrame`.

# In[7]:


from qcodes.logger import capture_dataframe


# In[8]:


with logger.console_level(logging.WARN):
    driver.cartesian((0, 0, 0))
    with capture_dataframe(level="DEBUG") as (handler, get_dataframe):
        driver.cartesian((0, 0, 1))
        df = get_dataframe()
        driver.cartesian((0, 0, 2))
        df2 = get_dataframe()  # this is the cummulative log


# In[9]:


df


# In[10]:


df2


# You can of course combine the context managers:

# In[11]:


with logger.console_level(logging.WARN):
    driver.cartesian((0, 0, 0))
    with capture_dataframe(level="DEBUG") as (handler, get_dataframe):
        with logger.filter_instrument(mag_x, handler=handler):
            driver.cartesian((0, 0, 1))
            df = get_dataframe()
df


# For an analysis of the timestamps please also refer to the log analysis [example notebook](logfile_parsing.ipynb).

# ## Telemetry

# The QCoDeS Measurement Context manager, DataSaverBuilder and DoND functions are instrumented as OpenTelemetry traces. This enables users to correlate log messages with the the measurement that they were performed within. See the (OpenTelemetry)[https://opentelemetry.io/] documentation for examples of how this can be used. 

# In[ ]:





#