#!/usr/bin/env python
# coding: utf-8

# # Exporting QCoDes Datasets

# This notebook demonstrates how we can export QCoDeS datasets to other file formats. 

# ## Setup

# First, we borrow an example from the measurement notebook to have some data to work with.

# In[1]:


get_ipython().run_line_magic('matplotlib', 'inline')
import shutil
from pathlib import Path

import numpy as np

import qcodes as qc
from qcodes.dataset import (
    Measurement,
    initialise_or_create_database_at,
    load_by_run_spec,
    load_from_netcdf,
    load_or_create_experiment,
    plot_dataset,
)
from qcodes.instrument_drivers.mock_instruments import (
    DummyInstrument,
    DummyInstrumentWithMeasurement,
)


# In[2]:


# preparatory mocking of physical setup
dac = DummyInstrument("dac", gates=["ch1", "ch2"])
dmm = DummyInstrumentWithMeasurement("dmm", setter_instr=dac)
station = qc.Station(dmm, dac)


# In[3]:


initialise_or_create_database_at(
    Path.cwd().parent / "example_output" / "export_example.db"
)
exp = load_or_create_experiment(
    experiment_name="exporting_data", sample_name="no sample"
)


# In[4]:


meas = Measurement(exp)
meas.register_parameter(dac.ch1)  # register the first independent parameter
meas.register_parameter(dac.ch2)  # register the second independent parameter
meas.register_parameter(
    dmm.v2, setpoints=(dac.ch1, dac.ch2)
)  # register the dependent one


# We then perform two very basic measurements using dummy instruments.

# In[5]:


# run a 2D sweep

with meas.run() as datasaver:
    for v1 in np.linspace(-1, 0, 200, endpoint=False):
        for v2 in np.linspace(-1, 1, 201):
            dac.ch1(v1)
            dac.ch2(v2)
            val = dmm.v2.get()
            datasaver.add_result((dac.ch1, v1), (dac.ch2, v2), (dmm.v2, val))

dataset1 = datasaver.dataset


# In[6]:


# run a 2D sweep

with meas.run() as datasaver:
    for v1 in np.linspace(0, 1, 200, endpoint=False):
        for v2 in np.linspace(1, 2, 201):
            dac.ch1(v1)
            dac.ch2(v2)
            val = dmm.v2.get()
            datasaver.add_result((dac.ch1, v1), (dac.ch2, v2), (dmm.v2, val))

dataset2 = datasaver.dataset


# ## Exporting data manually

# The dataset can be exported using the `export` method. Currently exporting to netcdf and csv is supported.

# In[7]:


dataset2.export("netcdf", path=Path.cwd().parent / "example_output")


# The `export_info` attribute contains information about where the dataset has been exported to:

# In[8]:


dataset2.export_info


# Looking at the signature of export we can see that in addition to the file format we can set the `prefix` and `path` to export to.

# In[9]:


get_ipython().run_line_magic('pinfo', 'dataset2.export')


# ## Export data automatically

# Datasets may also be exported automatically using the configuration options given in dataset config section. 
# Here you can toggle if a dataset should be exported automatically using the `export_automatic` option as well as set the default type, prefix, elements in the name, and path. See [the table here](https://microsoft.github.io/Qcodes/api/configuration/index.html#qcodes-default-configuration) for the relevant configuration options.
# 
# For more information about how to configure QCoDeS datasets see [the page about configuration](https://microsoft.github.io/Qcodes/api/configuration/index.html)  in the QCoDeS docs.
# 
# By default datasets are exported into a folder next to the database with the same name but ``.`` replaced by ``_`` e.g. if you store data to ``~/experiments.db`` the exported files will be storred in ``~/experiments_db``
# This folder is automatically created if it does not exist. 

# ## Automatically post process exported datasets.

# QCoDeD will attempt to call any [EntryPoint](https://setuptools.pypa.io/en/latest/userguide/entry_point.html#entry-points-for-plugins) registered for the group "qcodes.dataset.on_export". This will allow a user to setup a function that can trigger post processing such as backup to cloud, external drive, plotting or post process analysis. Functions registered for this entry point group are expected to take a Path to the file as input and return None. Please consult the [Setuptools docs](https://setuptools.pypa.io/en/latest/userguide/entry_point.html#entry-points-for-plugins) for more information on the use of EntryPoints. The entry point function must take the path to the exported file as a positional argument and take ``**kwargs`` for future compatibility. At the moment a single keyword argument ``automatic_export`` is passed to the function which indicates if the dataset was automatically or manually exported.

# ## Importing exported datasets into a new database

# The above dataset has been created in the following database

# In[10]:


qc.config.core.db_location


# Now lets imagine that we move the exported dataset to a different computer. To emulate this we will create a new database file and set it as the active database. 

# In[11]:


initialise_or_create_database_at(
    Path.cwd().parent / "example_output" / "reimport_example.db"
)


# In[12]:


qc.config.core.db_location


# We can then reload the dataset from the netcdf file as a DataSetInMem. This is a class that closely matches the regular DataSet class however its metadata may or may not be written to a database file and its data is not written to a database file. See more in [
# In memory dataset](./InMemoryDataSet.ipynb) . Concretely this means that the data captured in the dataset can be acceced via `dataset.cache.data` etc. and not via the methods directly on the dataset (`dataset.get_parameter_data` ...) 
# 
# Note that it is currently only possible to reload a dataset from a netcdf export and not from a csv export. This is due to the fact that a csv export only contains the raw data and not the metadata needed to recreate a dataset.

# In[13]:


loaded_ds = load_from_netcdf(dataset2.export_info.export_paths["nc"])


# In[14]:


type(loaded_ds)


# However, we can still export the data to Pandasa and xarray.

# In[15]:


loaded_ds.cache.to_xarray_dataset()


# And plot it using `plot_dataset`.

# In[16]:


plot_dataset(loaded_ds)


# Note that the dataset will have the same `captured_run_id` and `captured_counter` as before:

# In[17]:


captured_run_id = loaded_ds.captured_run_id
captured_run_id


# But do note that the `run_id` and `counter` are in general not preserved since they represent the datasets number in a given db file. 

# In[18]:


loaded_ds.run_id


# A loaded datasets metadata can be written to the current db file and subsequently the dataset including metadata and raw data reloaded from the database and netcdf file.

# In[19]:


loaded_ds.write_metadata_to_db()


# Now that the metadata has been written to a database the dataset can be plotted with [plottr](https://github.com/toolsforexperiments/plottr/) like a regular dataset.

# In[20]:


del loaded_ds


# In[21]:


reloaded_ds = load_by_run_spec(captured_run_id=captured_run_id)


# In[22]:


plot_dataset(reloaded_ds)


# Note that loading a dataset from the database will also load the raw data into `dataset.cache` provided that the `netcdf` file is still in the location where file was when the metadata was written to the database. Load_by_runspec and related functions will load data into a regular `DataSet` provided that the data can be found in the database otherwise it will be loaded into a `DataSetInMem`
# 
# 
# If the netcdf file cannot be found the dataset will load with a warning and the raw data will not be accessible from the dataset. 
# 
# If this happens because you have moved the location of a netcdf file you can use the method ``set_netcdf_location`` to set a new location for the the netcdf file in the dataset and database file. 
# Here we demonstrate this by copying the netcdf file and changing the location using this method.

# In[23]:


filepath = dataset2.export_info.export_paths["nc"]
new_file_path = str(Path.cwd().parent / "example_output" / "somefile.nc")
new_file_path


# In[24]:


shutil.copyfile(dataset2.export_info.export_paths["nc"], new_file_path)


# In[25]:


reloaded_ds.set_netcdf_location(new_file_path)
reloaded_ds.export_info