#!/usr/bin/env python
# coding: utf-8

# # OpenML - Machine Learning as a community
# > A description of how OpenML fits into traditional ML practices
# 
# - toc: true 
# - badges: true
# - comments: true
# - categories: [OpenML]
# - image: images/fastpages_posts/openml.png
# - author: Neeratyoy Mallik

# [OpenML](https://www.openml.org/) is an online Machine Learning (ML) experiments database accessible to everyone for free. The core idea is to have a single repository of datasets and results of ML experiments on them. Despite having gained a lot of popularity in recent years, with a plethora of tools now available, the numerous ML experimentations continue to happen in silos and not necessarily as one whole shared community.
# In this post, we shall try to get a brief glimpse of what OpenML offers and how it can fit our current Machine Learning practices.
# 
# Let us jump straight at getting our hands dirty by building a simple machine learning model. If it is simplicity we are looking for, it has to be the Iris dataset that we shall work with. In the example script below, we are going to load the Iris dataset available with scikit-learn, use 10-fold cross-validation to evaluate a Random Forest of 10 trees. Sounds trivial enough and is indeed less than 10 lines of code.

# In[1]:


from sklearn import datasets
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score


# In[2]:


# Loading Iris dataset
X, y = datasets.load_iris(return_X_y=True)
print(X.shape, y.shape)


# In[3]:


# Initializing a Random Forest with 
# arbitrary hyperparameters
# max_depth kept as 2 since Iris has
# only 4 features
clf = RandomForestClassifier(n_estimators=10, max_depth=2)


# In[4]:


scores = cross_val_score(clf, X, y, cv=5, scoring='accuracy')
print("Mean score : {:.5f}".format(scores.mean()))


# A simple script and we achieve a mean accuracy of **95.33%**. That was easy. It is really amazing how far we have come with ML tools that make it easy to get started. As a result, we have hundreds of thousands of people working with these tools every day. That inevitably leads to the reinvention of the wheel. The tasks that each individual ML practitioner performs often have significant overlaps and can be omitted by reusing what someone from the community has done already. At the end of the day, we didn't build a Random Forest model all the way from scratch. We gladly reused code written by generous folks from the community. The special attribute of our species is the ability to work as a collective wherein our combined intellect becomes larger than the individual sum of parts. Why not do the same for ML? I mean, can I see what other ML practitioners have done to get better scores on the Iris dataset?
# 
# Answering this is one of the targets of this post. We shall subsequently explore if this can be done, with the help of [OpenML](https://www.openml.org/). However, first, we shall briefly familiarize ourselves with few terminologies and see how we can split the earlier example we saw into modular components.

# ### OpenML Components

# <figure>
#   <img src="../images/fastpages-posts/openml.png" alt="Image source">
#   <figcaption></figcaption>
# </figure>

# Image source: <a href="https://medium.com/open-machine-learning/openml-1e0d43f0ae13">https://medium.com/open-machine-learning/openml-1e0d43f0ae13</a>

# **Dataset**: OpenML houses over 2k+ active datasets for various regression, classification, clustering, survival analysis, stream processing tasks and more. Any user can upload a dataset. Once uploaded, the server computes certain meta-features on the dataset - *Number of classes*, *Number of missing values*, *Number of features*, etc. With respect to our earlier example, the following line is the equivalent of fetching a dataset from OpenML.

# In[5]:


X, y = datasets.load_iris(return_X_y=True)


# **Task**: A task is linked to a specific dataset, defining what the target/dependent variable is. Also specifies evaluation measures such as - accuracy, precision, area under curve, etc. or the kind of estimation procedure to be used such as - 10-fold *cross-validation*, n% holdout set, etc. With respect to our earlier example, the *parameters* to the following function call capture the idea of a task.

# In[6]:


scores = cross_val_score(clf, X, y, cv=5, scoring='accuracy')


# **Flow**: Describes the kind of modelling to be performed. It could be a flow or a series of steps, i.e., a scikit-learn pipeline. For now, we have used a simple Random Forest model which is the *flow* component here.

# In[7]:


clf = RandomForestClassifier(n_estimators=10, max_depth=2)


# **Run**: Pairs a *flow* and task together which results in a *run*. The *run* has the predictions which are turned into *evaluations* by the server. This is effectively captured by the *execution* of the line:

# In[8]:


scores = cross_val_score(clf, X, y, cv=5, scoring='accuracy')


# Now, this may appear a little obfuscating given that we are trying to compartmentalize a simple 10-line code which works just fine. However, if we take a few seconds to go through the 4 components explained above, we can see that it makes our *training of a Random Forest* on Iris a series of modular tasks. Modules are such a fundamental concept in Computer Science. They are like Lego blocks. Once we have modules, it means we can plug and play at ease. The code snippet below attempts to rewrite the earlier example using the ideas of the OpenML components described, to give a glimpse of what we can potentially gain during experimentations.

# In[9]:


from sklearn import datasets
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score


# #### DATASET component

# In[10]:


# To load IRIS dataset as a dataset module/component
def dataset():
    X, y = datasets.load_iris(return_X_y=True)
    return X, y


# #### TASK component

# In[11]:


# Tasks here define the number of cross-validation folds
# and the scoring metric to be used for evaluation
def task_1(f):
    X, y = dataset()  # loads IRIS
    return cross_val_score(f, X, y, cv=5, 
                           scoring='accuracy')

def task_2(f):
    X, y = dataset()  # loads IRIS
    return cross_val_score(f, X, y, cv=15, 
                           scoring='balanced_accuracy')


# #### FLOW component

# In[12]:


# Flows determine the modelling technique to be applied
# Helps define a model irrespective of dataset or tasks
def flow_1():
    clf = RandomForestClassifier(n_estimators=10, max_depth=2)
    return clf

def flow_2():
    clf = SVC(gamma='auto', kernel='linear')
    return clf 


# #### RUN component

# In[13]:


# Runs essentially evaluates a task-flow pairing 
# and therefore in effect executs the modelling 
# of a dataset as per the task task definition
def run(task, flow):
    return task(flow)


# In[14]:


# Results for Random Forest
rf_task_1 = run(task_1, flow_1())
rf_task_2 = run(task_2, flow_1())
print("RF using task 1: {:<.5}; task 2: {:<.5}".format(rf_task_1.mean(), rf_task_2.mean()))

# Results for SVM
svm_task_1 = run(task_1, flow_2())
svm_task_2 = run(task_2, flow_2())
print("SVM using task 1: {:<.5}; task 2: {:<.5}".format(svm_task_1.mean(), svm_task_2.mean()))


# We can, therefore, compose various different tasks, flows, which are independent operations. Runs can then pair any such task and flow to construct an ML *workflow* and return the evaluated scores. This approach can help us define such components one-time, and we can extend this for any combination of a dataset, model, and for any number of evaluations in the future. Imagine if the entire ML *community* defines such tasks and various simple to complicated flows that they use in their daily practice. We can build custom working ML pipeline and even get to compare performances of our techniques on the same *task* with others! OpenML aims exactly for that. In the next section of this post, we shall scratch the surface of OpenML to see if we can actually do with OpenML what it promises.

# ### Using OpenML

# OpenML-Python can be installed using *pip* or by [cloning the git repo](https://openml.github.io/openml-python/develop/contributing.html#installation) and installing the current development version. So shall we then install OpenML? ;) It will be beneficial if the code snippets are tried out as this post is read. A consolidated Jupyter notebook with all the code can be found [here](https://nbviewer.jupyter.org/github/Neeratyoy/openml-python/blob/blog/OpenML%20-%20Machine%20Learning%20as%20a%20community.ipynb).
# 
# Now that we have OpenML, let us jump straight into figuring out how we can get the Iris dataset from there. We can always browse the[OpenML website](https://www.openml.org/) and search for Iris. That is the easy route. Let us get familiar with the programmatic approach and learn how to fish instead. The OpenML-Python API can be found [here](https://openml.github.io/openml-python/develop/api.html).

# #### Retrieving Iris from OpenML

# In the example below, we will list out all possible datasets available in OpenML. We can choose the output format. I'll go with *dataframe* so that we obtain a pandas DataFrame and can get a neat tabular representation to search and sort specific entries.

# In[15]:


import openml

import numpy as np
import pandas as pd


# In[16]:


# Fetching the list of all available datasets on OpenML
d = openml.datasets.list_datasets(output_format='dataframe')
print(d.shape)

# Listing column names or attributes that OpenML offers
for name in d.columns:
    print(name)


# In[17]:


print(d.head())


# The column names indicate that they contain the meta-information about each of the datasets, and at this instance, we have access to **2958** datasets as indicated by the shape of the dataframe. We shall try searching for 'iris' in the column *name* and also use the *version* column to sort the results.

# In[18]:


# Filtering dataset list to have 'iris' in the 'name' column
# then sorting the list based on the 'version'
d[d['name'].str.contains('iris')].sort_values(by='version').head()


# Okay, so the iris dataset with the version as 1 has an ID of **61**. For verification, we can check the [website for dataset ID 61](https://www.openml.org/d/61). We can see that it is the original Iris dataset which is of interest to us - 3 classes of 50 instances, with 4 numeric features. However, we shall retrieve the same information, as promised, programmatically.

# In[19]:


iris = openml.datasets.get_dataset(61)
iris


# In[20]:


iris.features


# In[21]:


print(iris.description)


# With the appropriate dataset available, let us briefly go back to the terminologies we discussed earlier. We have only used the *dataset* component so far. The *dataset* component is closely tied with the task component. To reiterate, the task would describe *how* the dataset will be used.

# #### Retrieving relevant tasks from OpenML

# We shall firstly list all available tasks that work with the Iris dataset. However, we are only treating Iris as a supervised classification problem and hence will filter accordingly. Following which, we will collect only the task IDs of the tasks relevant to us.

# In[22]:


df = openml.tasks.list_tasks(data_id=61, output_format='dataframe')
df.head()


# In[23]:


# Filtering only the Supervised Classification tasks on Iris
df.query("task_type=='Supervised Classification'").head()


# In[24]:


# Collecting all relevant task_ids
tasks = df.query("task_type=='Supervised Classification'")['tid'].to_numpy()
print(len(tasks))


# That settles the *task* component too. Notice how for one *dataset* (61), we obtain 11 task IDs which are of interest to us. This should illustrate the *one-to-many* relationship that *dataset-task components* can have. We have 2 more components to explore - *flows*, *runs*. We could list out all possible flows and filter out the ones we want, i.e., Random Forest. However, let us instead fetch all the evaluations made on the Iris dataset using the 11 tasks we collected above.
# 
# We shall subsequently work with the scikit-learn based task which has been uploaded/used the most. We shall then further filter out the list of evaluations from the selected task (task_id=59 in this case), depending on if Random Forest was used.

# In[25]:


# Listing all evaluations made on the 11 tasks collected above
# with evaluation metric as 'predictive_accuracy'
task_df = openml.evaluations.list_evaluations(function='predictive_accuracy', task=tasks, output_format='dataframe')
task_df.head()


# In[26]:


# Filtering based on sklearn (scikit-learn)
task_df = task_df[task_df['flow_name'].str.contains("sklearn")]
task_df.head()


# In[27]:


# Counting frequency of the different tasks used to
# solve Iris as a supervised classification using scikit-learn
task_df['task_id'].value_counts()


# In[28]:


# Retrieving the most used task
t = openml.tasks.get_task(59)
t


# In[29]:


# Filtering for only task_id=59
task_df = task_df.query("task_id==59")


# In[30]:


# Filtering based on Random Forest
task_rf =  task_df[task_df['flow_name'].str.contains("RandomForest")]
task_rf.head()


# #### Retrieving top-performing models from OpenML

# Since we are an ambitious bunch of ML practitioners who settle for nothing but the best, and also since most results will not be considered worth the effort if not matching or beating *state-of-the-art*, we shall aim for the best scores. We'll sort the filtered results we obtained based on the score or '*value*' and then extract the components from that run - *task* and *flow*.

# In[31]:


task_rf.sort_values(by='value', ascending=False).head()


# In[32]:


# Fetching the Random Forest flow with the best score
f = openml.flows.get_flow(2629)
f


# In[33]:


# Fetching the run with the best score for
# Random Forest on Iris
r = openml.runs.get_run(523926)
r


# Okay, let's take a pause and re-assess. From multiple users across the globe, who had uploaded runs to OpenML, for a Random Forest run on the Iris, the best score seen till now is **96.67%**. That is certainly better than the naive model we built at the beginning to achieve **95.33%**. We had used a basic 10-fold cross-validation to evaluate a Random Forest of 10 trees with a max depth of 2. Let us see, what the best run uses and if it differs from our approach.

# In[34]:


# The scoring metric used
t.evaluation_measure


# In[35]:


# The methodology used for estimations
t.estimation_procedure


# In[36]:


# The model used
f.name


# In[37]:


# The model parameters
for param in r.parameter_settings:
    name, value = param['oml:name'], param['oml:value']
    print("{:<25} : {:<10}".format(name, value))


# As evident, our initial approach is different on two fronts. We didn't explicitly use stratified sampling for our cross-validation. While the Random Forest hyperparameters are slightly different too (*max_depth=None*). That definitely sounds like a *to-do*, however, there is no reason why we should restrict ourselves to Random Forests. Remember, we are aiming *big* here. Given the [number of OpenML users](https://www.openml.org/search?type=user), there must be somebody who got a better score on Iris with some other model. Let us then retrieve that information. Programmatically, of course.
# 
# In summary, we are now going to sort the performance of all scikit-learn based models on Iris dataset as per the task definition with *task_id=59*.

# In[38]:


# Fetching top performances
task_df.sort_values(by='value', ascending=False).head()


# In[39]:


# Fetching best performing flow
f = openml.flows.get_flow(6048)
f


# In[40]:


# Fetching best performing run
r = openml.runs.get_run(2012943)

# The model parameters
for param in r.parameter_settings:
    name, value = param['oml:name'], param['oml:value']
    print("{:<25} : {:<10}".format(name, value))


# The highest score obtained among the uploaded results is **98.67%** using a [variant of SVM](https://scikit-learn.org/stable/modules/generated/sklearn.svm.NuSVC.html#sklearn.svm.NuSVC). However, if we check the corresponding flow description, we see that it is using an old scikit-learn version (0.18.1) and therefore may not be possible to replicate the exact results. However, in order to improve from our score of 95.33%, we should try running a *nu-SVC* on the same problem and see where we stand. Let's go for it. Via OpenML, of course.

# #### Running best performing flow on the required task

# In[41]:


import openml
import numpy as np
from sklearn.svm import NuSVC


# In[42]:


# Building the NuSVC model object with parameters found
clf = NuSVC(cache_size=200, class_weight=None, coef0=0.0,
   decision_function_shape=None, degree=3, gamma='auto', kernel='linear',
   max_iter=-1, nu=0.3, probability=True, random_state=3, shrinking=True,
   tol=3.2419092644286417e-05, verbose=False)


# In[43]:


# Obtaining task used earlier
t = openml.tasks.get_task(59)
t


# In[44]:


# Running the model on the task
# Internally, the model will be made into 
# an OpenML flow and we can choose to retrieve it
r, f = openml.runs.run_model_on_task(model=clf, task=t, upload_flow=False, return_flow=True)
f


# In[45]:


# To obtain the score (without uploading)
## r.publish() can be used to upload these results
## need to sign-in to https://www.openml.org/
score = []
evaluations = r.fold_evaluations['predictive_accuracy'][0]
for key in evaluations:
    score.append(evaluations[key])
print(np.mean(score))


# Lo and behold! We hit the magic number. I personally would have never tried out NuSVC and would have stuck around tweaking hyperparameters of the Random Forest. This is a new discovery of sorts for sure. I wonder though if anybody has tried XGBoost on Iris?
# 
# In any case, we can now upload the results of this run to OpenML using:

# In[46]:


r.publish()


# One would need to sign-in to https://www.openml.org/ and generate their respective *apikey*. The results would then be available for everyone to view and who knows, you can have your name against the *best-ever* performance measured on the Iris dataset!

# ---

# This post was in no ways intended to be a be-all-end-all guide to OpenML. The primary goal was to help form an acquaintance with the OpenML terminologies, introduce the API, establish connections with the general ML practices, and give a sneak-peek into the potential benefits of working together as a *community*. For a better understanding of OpenML, please explore the [documentation](https://openml.github.io/openml-python/develop/usage.html#usage). If one desires to continue from the examples given in this post and explore further, kindly refer to the [API](https://openml.github.io/openml-python/develop/api.html).
# 
# OpenML-Python is an open-source project and contributions from everyone in the form of Issues and Pull Requests are most welcome. Contribution to the OpenML community is in fact not limited to code contribution. Every single user can make the community richer by sharing data, experiments, results, using OpenML.
# 
# As ML practitioners, we may be dependent on tools for our tasks. However, as a collective, we can juice out its potential to a larger extent. Let us together, make ML more transparent, more democratic!

# ---

# Special thanks to Heidi, Bilge, Sahithya, Matthias, Ashwin for the ideas, feedback, and support.

# ---

# Related readings:
# * [To get started with OpenML-Python](https://openml.github.io/openml-python/develop/)
# * [OpenML-Python Github](https://github.com/openml/openml-python)
# * [The OpenML website](https://www.openml.org/)
# * [Miscellaneous reading on OpenML](https://openml.github.io/blog/)
# * [To get in touch!](https://www.openml.org/contact)