Comet.ml has an extensive interface to all of your data using a REST API through Comet.ml endpoints. Now, you can access this information easily through the Comet.ml Python SDK. Requires version comet_ml version 3.0.0 or greater.
To install comet_ml:
%pip install comet_ml --quiet
To run the following experiments, you'll need to set your COMET_API_KEY. The easiest way to to this is to use the comet_ml.init()
function:
import comet_ml
comet_ml.init()
To access the Python API through the Comet.ml SDK, you will need to make an API() instance.
Note: this is a new interface.
The new API:
APIExperiment
now works similarly to Experiment
, ExistingExperiment
, and OfflineExperiment
Let's try it out!
First, we import the API class and other libraries we will need. Note that this is a new interface and comes from comet_ml.api
:
from comet_ml import API
import comet_ml
import matplotlib.pyplot as plt
%matplotlib inline
and create the API instance:
comet_api = API()
Using the comet_api
instance, you can get the name of your workspaces:
comet_api.get()
['sampleworkspace', 'cometpublic']
If you reference your workspace by name using comet_api.get(WORKSPACE_NAME)
, you'll see your projects:
print(comet_api.get("cometpublic"))
['caffe2', 'ludwig', 'comet-sagemaker', 'fasttext', 'comet-notebooks', 'shap', 'parameter-space-exploration', 'comet-examples', 'fastai', '3d-histograms', 'pyspark-example', 'general', 'keras-example', 'home-credit']
Or, get the projects from another user or shared workspace:
comet_api.get("testuser")
['mnist']
Using the same method, you can refer to a project by name and get all of the experiments in a project:
comet_api.get("cometpublic", "fasttext")
[<APIExperiment 'cometpublic/fasttext/18e88395f6c44ffc9930c5323935f5d9'>, <APIExperiment 'cometpublic/fasttext/23db3466c7134a34b591de011c7595ba'>, <APIExperiment 'cometpublic/fasttext/52a4939f477c41e9936a3e16b87be324'>, <APIExperiment 'cometpublic/fasttext/990f58b9447b453ab18da1ea75cde85a'>, <APIExperiment 'cometpublic/fasttext/44ea2d68794d4c5e9e18b86c30562bf8'>, <APIExperiment 'cometpublic/fasttext/034d73233bdd45b7861c2acca66f90a1'>, <APIExperiment 'cometpublic/fasttext/86de6a38bd41489895bfdb44f74ab928'>, <APIExperiment 'cometpublic/fasttext/f0f23c9825834fd6a8d6ebfd2baeea26'>, <APIExperiment 'cometpublic/fasttext/328cf1303126404b96a32e8146c0bc89'>, <APIExperiment 'cometpublic/fasttext/dab224910ff44bb8aa0718612578584f'>, <APIExperiment 'cometpublic/fasttext/7005afd6a42446aab657e3d8255d01a3'>, <APIExperiment 'cometpublic/fasttext/17db4a9d68124b0bb6f78c805f78ab49'>, <APIExperiment 'cometpublic/fasttext/62d92ee769ca4ae19165deaf6653095c'>, <APIExperiment 'cometpublic/fasttext/be99897ab2a049868666f482f5cf1900'>, <APIExperiment 'cometpublic/fasttext/e34b7e2868324ed18a72e29e237b4608'>, <APIExperiment 'cometpublic/fasttext/dfb91dce76aa4abeaac61793b53ec8ab'>, <APIExperiment 'cometpublic/fasttext/e774ec41b7a64af8a423ed394a660705'>, <APIExperiment 'cometpublic/fasttext/b5812f495aea43729b2922c7d6102392'>, <APIExperiment 'cometpublic/fasttext/5e9f8097519f4b2f86e4daa8b3265300'>, <APIExperiment 'cometpublic/fasttext/43cd0aa2fec14a9a818e1b0e73a70a34'>, <APIExperiment 'cometpublic/fasttext/e64c5915920f481bab8f4cb4dbd615be'>]
Or, using the slash delimiter:
comet_api.get("cometpublic/fasttext")
[<APIExperiment 'cometpublic/fasttext/18e88395f6c44ffc9930c5323935f5d9'>, <APIExperiment 'cometpublic/fasttext/23db3466c7134a34b591de011c7595ba'>, <APIExperiment 'cometpublic/fasttext/52a4939f477c41e9936a3e16b87be324'>, <APIExperiment 'cometpublic/fasttext/990f58b9447b453ab18da1ea75cde85a'>, <APIExperiment 'cometpublic/fasttext/44ea2d68794d4c5e9e18b86c30562bf8'>, <APIExperiment 'cometpublic/fasttext/034d73233bdd45b7861c2acca66f90a1'>, <APIExperiment 'cometpublic/fasttext/86de6a38bd41489895bfdb44f74ab928'>, <APIExperiment 'cometpublic/fasttext/f0f23c9825834fd6a8d6ebfd2baeea26'>, <APIExperiment 'cometpublic/fasttext/328cf1303126404b96a32e8146c0bc89'>, <APIExperiment 'cometpublic/fasttext/dab224910ff44bb8aa0718612578584f'>, <APIExperiment 'cometpublic/fasttext/7005afd6a42446aab657e3d8255d01a3'>, <APIExperiment 'cometpublic/fasttext/17db4a9d68124b0bb6f78c805f78ab49'>, <APIExperiment 'cometpublic/fasttext/62d92ee769ca4ae19165deaf6653095c'>, <APIExperiment 'cometpublic/fasttext/be99897ab2a049868666f482f5cf1900'>, <APIExperiment 'cometpublic/fasttext/e34b7e2868324ed18a72e29e237b4608'>, <APIExperiment 'cometpublic/fasttext/dfb91dce76aa4abeaac61793b53ec8ab'>, <APIExperiment 'cometpublic/fasttext/e774ec41b7a64af8a423ed394a660705'>, <APIExperiment 'cometpublic/fasttext/b5812f495aea43729b2922c7d6102392'>, <APIExperiment 'cometpublic/fasttext/5e9f8097519f4b2f86e4daa8b3265300'>, <APIExperiment 'cometpublic/fasttext/43cd0aa2fec14a9a818e1b0e73a70a34'>, <APIExperiment 'cometpublic/fasttext/e64c5915920f481bab8f4cb4dbd615be'>]
And one more level, get an APIExperiment
object using the Experiment's ID:
comet_api.get("cometpublic", "fasttext", 'e64c5915920f481bab8f4cb4dbd615be')
<APIExperiment 'cometpublic/fasttext/e64c5915920f481bab8f4cb4dbd615be'>
Or, again using the slash shorthand:
comet_api.get("cometpublic/fasttext/e64c5915920f481bab8f4cb4dbd615be")
<APIExperiment 'cometpublic/fasttext/e64c5915920f481bab8f4cb4dbd615be'>
Let's get an experiment and save it to a variable named exp
:
exp = comet_api.get("cometpublic/fasttext/e64c5915920f481bab8f4cb4dbd615be")
exp
<APIExperiment 'cometpublic/fasttext/e64c5915920f481bab8f4cb4dbd615be'>
There are a number of items you get and set from the APIExperiment instance. For a complete reference, see: https://www.comet.ml/docs/python-sdk/API/
For example, we can explore the other
property, which shows items saved with Experiment.log_other(NAME, VALUE):
exp.get_others_summary()
[{'name': 'Name', 'valueMax': 'last', 'valueMin': 'last', 'valueCurrent': 'last'}, {'name': 'storage_size_bytes', 'valueMax': '0', 'valueMin': '0', 'valueCurrent': '0', 'timestampMax': 1574708415277, 'timestampMin': 1574708415277, 'timestampCurrent': 1574708415277}, {'name': 'trainable_params', 'valueMax': '376560', 'valueMin': '376560', 'valueCurrent': '376560', 'timestampMax': 1529089066169, 'timestampMin': 1529089066169, 'timestampCurrent': 1529089066169}]
In this example, we see that the experiment has the Name
"last". We can use Name
to also look up experiments:
exp = comet_api.get("cometpublic/fasttext/last")
exp.id, exp.name
('e64c5915920f481bab8f4cb4dbd615be', 'last')
Perhaps one of the most useful abilities for the Python API is to access your experiment's data in order to create a variation of a plot. To access the raw metric data, use the .get_metrics()
method of the APIExperiment:
len(exp.get_metrics())
2804
We see here were over 2800 metrics logged during the training of this experiment. We can get the first using indexing with an integer:
exp.get_metrics()[0]
{'metricName': 'val_acc', 'metricValue': '0.9832226913061168', 'timestamp': 1529089066713, 'step': 69, 'epoch': None, 'runContext': None, 'offset': 314}
That shows that the "acc" (accuracy) metric had a value of about 0.5 at step 1 of the experiment.
We can also filter on a single metric name, like so:
acc_metrics = exp.get_metrics("acc")
len(acc_metrics)
700
acc_metrics[0]
{'metricName': 'acc', 'metricValue': '0.5003255009651184', 'timestamp': 1529089066219, 'step': 1, 'epoch': None, 'runContext': None, 'offset': 29}
Therefore, exp.get_metrics("acc")
gives us the dictionary for all "acc" items. We can then easily use Python's built in zip and matplotlib to plot these values:
steps_acc = [(m["step"], float(m["metricValue"])) for m in acc_metrics]
This breaks up the data into (step, value) pairs:
steps_acc[0]
(1, 0.5003255009651184)
A little Python trick to separate the steps from the accuracies so we can easily use matplotlib:
steps, acc = zip(*steps_acc[:100]) # just the first 100 for now
plt.plot(steps, acc);
That's it for a quick overview. Now let's look in detail at each component.
comet_api.get()
reports your workspace names:
comet_api.get()
['sampleworkspace', 'cometpublic']
You can also interate over those names:
for workspace in comet_api.get():
print(workspace)
sampleworkspace cometpublic
As we saw above, you can also access other public workspaces as well:
comet_api.get("testuser")
['mnist']
Under get(WORKSPACE_NAME), you'll find the projects:
comet_api.get("cometpublic")
['caffe2', 'ludwig', 'comet-sagemaker', 'fasttext', 'comet-notebooks', 'shap', 'parameter-space-exploration', 'comet-examples', 'fastai', '3d-histograms', 'pyspark-example', 'general', 'keras-example', 'home-credit']
project = comet_api.get("cometpublic", "comet-notebooks")
## OR:
#project = comet_api.get("cometpublic/comet-notebooks")
If you just print out, or iterate over a project, you get access to the experiments:
project
[<APIExperiment 'cometpublic/comet-notebooks/7092a5e4c362453fb0b3f06785a1d30c'>, <APIExperiment 'cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20'>]
project[0].id, project[0].name
('7092a5e4c362453fb0b3f06785a1d30c', None)
project[1].id, project[1].name
('d21f94a1c71841d2961da1e6ddb5ab20', 'example 001')
Continuing with the dictionary-like access, you can see and iterate over the experiment ids:
comet_api.get("cometpublic", "comet-notebooks")
[<APIExperiment 'cometpublic/comet-notebooks/7092a5e4c362453fb0b3f06785a1d30c'>, <APIExperiment 'cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20'>]
exp = comet_api.get("cometpublic", "comet-notebooks", 'd21f94a1c71841d2961da1e6ddb5ab20')
## OR
# exp = comet_api.get("cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20")
exp
<APIExperiment 'cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20'>
exp = comet_api.get("cometpublic", "comet-notebooks", 'example 001')
## OR
## exp = comet_api.get("cometpublic/comet-notebooks/example 001")
exp
<APIExperiment 'cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20'>
You can also use regular expressions as the name for the experiment:
comet_api.get_experiments("cometpublic", "comet-notebooks", "example.*")
[<APIExperiment 'cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20'>]
The Python API provides programmatic access to the same query system in the web UI. There are 5 types of items that you can query:
log_metric()
log_other()
log_parameter()
add_tags()
To use these, you first need to import them:
from comet_ml.api import Metric, Metadata, Other, Parameter, Tag
You can then use these to build a query expression, like so:
comet_api.query("cometpublic", "general", Parameter("max_iter") == 100)
[<APIExperiment 'cometpublic/general/4d4fba0f6bac4420bb5948e41015c114'>, <APIExperiment 'cometpublic/general/be44937e416b4b62be79ee2e7c0c3c4e'>]
The API.query() method takes the following args:
A query is of the form:
((QUERY-VARIABLE OPERATOR VALUE) & ...)
# or:
(QUERY-VARIABLE.METHOD(VALUE) & ...)
where:
QUERY-VARIABLE
is Metric(NAME)
, Parameter(NAME)
, Other(NAME)
, Metadata(NAME)
, or Tag(VALUE)
.OPERATOR
is any of the standard mathematical operators ==
, <=
, >=
, !=
, <
, >
.METHOD
is between()
, contains()
, startswith()
, or endswith()
.You may also place the bitwise ~
not operator in front of an expression
which means to invert the expression. Use &
to combine additional
criteria. Currently, |
(bitwise or) is not supported.
VALUE
can be any query type, includeing string
, boolean
, double
,
datetime
, or timenumber
(number of seconds). None
and ""
are special
values that mean NULL
and EMPTY
, respectively. Use
API.get_project_variables(WORKSPACE, PROJECT_NAME)
to see query variables
and types for a project.
When using datetime
, be aware that the backend is using UTC datetimes. If you
do not receive the correct experiments via a datetime query, please check with
the web UI query builder to verify timezone of the server.
query()
returns a list of matching APIExperiments()
.
Examples:
# Find all experiments that have an acc metric value > .98:
>>> api.query("workspace", "project", Metric("acc") > .98)
[APIExperiment(), ...]
# Find all experiments that have a loss metric < .1 and
# a learning_rate parameter value >= 0.3:
>>> loss = Metric("loss")
>>> lr = Parameter("learning_rate")
>>> query = ((loss < .1) & (lr >= 0.3))
>>> api.query("workspace", "project", query)
[APIExperiment(), ...]
# Find all of the experiments tagged "My simple tag":
>>> tagged = Metric("My simple tag")
>>> api.query("workspace", "project", tagged)
[APIExperiment(), ...]
# Find all experiments started before Sept 24, 2019 at 5:00am:
>>> q = Metadata("start_server_timestamp") < datetime(2019, 9, 24, 5)
>>> api.query("workspace", "project", q)
[APIExperiment(), ...]
# Find all experiments lasting more that 2 minutes (in seconds):
>>> q = Metadata("duration") > (2 * 60)
>>> api.query("workspace", "project", q)
[APIExperiment(), ...]
Notes:
~
for not
on any expression~QUERY-VARIABLE.between(2,3)
for values not between 2 and 3(QUERY-VARIABLE == True)
for truth(QUERY-VARIABLE == False)
for not true(QUERY-VARIABLE == None)
for testing null(QUERY-VARIABLE != None)
or ~(QUERY-VARIABLE == None)
for testing not null(QUERY-VARIABLE == "")
for testing empty(QUERY-VARIABLE != "")
or ~(QUERY-VARIABLE == "")
for testing not emptyMetadata("start_server_timestamp")
or Metadata("end_server_timestamp")
Metadata("duration")
API.get_project_variables(WORKSPACE, PROJECT_NAME)
to see query variables
and types.Do not use 'and', 'or', 'not', 'is', or 'in'. These are logical operators and you must use mathematical operators for queries. For example, always use '==' where you might usually use 'is'.
How can you know what query variables are available for a project? Use API.get_query_variables(WORKSPACE, PROJECT_NAME):
vars = comet_api.get_query_variables("cometpublic", "general")
vars
[Metadata('hasNotes'), Metadata('duration'), Metric('sys.cpu.percent.04'), Parameter('warm_start'), Metric('sys.cpu.percent.avg'), Parameter('learning_rate'), Metric('train_reg_mean_loss'), Metadata('file_name'), Metric('sys.ram.total'), Parameter('beta2'), Metadata('experimentKey'), Parameter('run_arg_2'), Metric('sys.ram.used'), Parameter('intercept_scaling'), Metadata('end_server_timestamp'), Metric('sys.cpu.percent.08'), Metadata('file_path'), Metadata('runActive'), Parameter('C'), Metric('class/train_batch_mean_hits_at_k_class'), Parameter('run_arg_4'), Parameter('max_iter'), Metadata('runId'), Metric('sys.cpu.percent.03'), Parameter('curr_step'), Parameter('class_weight'), Parameter('multi_class'), Metric('sys.cpu.percent.01'), Parameter('fit_intercept'), Metadata('optimizationId'), Parameter('run_arg_3'), Metric('sys.cpu.percent.02'), Parameter('penalty'), Parameter('beta1'), Metadata('codeSHA'), Parameter('epsilon'), Parameter('solver'), Metric('class/train_mean_loss_class'), Metadata('installed_packages'), Metadata('last_message_time'), Metadata('hasImages'), Metadata('user_name'), Metadata('start_server_timestamp'), Parameter('tol'), Parameter('Optimizer'), Other('throttled'), Parameter('verbose'), Metric('sys.cpu.percent.05'), Metric('class/train_batch_accuracy_class'), Parameter('run_arg_1'), Metadata('hasHtml'), Parameter('dual'), Tag('ludwig'), Metric('sys.cpu.percent.07'), Parameter('n_jobs'), Parameter('run_arg_0'), Other('storage_size_bytes'), Metric('sys.cpu.percent.06'), Parameter('random_state'), Metric('sys.load.avg')]
In this brief dictionary representation, you will see that get_others_summary()
, get_metrics_summary()
and get_parameters_summary()
give summary data for each item:
exp.get_parameters_summary()
[{'name': 'f', 'valueMax': '/run/user/1000/jupyter/kernel-b1c4403c-c673-4e7f-90dc-853616d231e2.json', 'valueMin': '/run/user/1000/jupyter/kernel-b1c4403c-c673-4e7f-90dc-853616d231e2.json', 'valueCurrent': '/run/user/1000/jupyter/kernel-b1c4403c-c673-4e7f-90dc-853616d231e2.json', 'timestampMax': 1542823692798, 'timestampMin': 1542823692798, 'timestampCurrent': 1542823692798}]
exp.get_others_summary()[0]["name"], exp.get_others_summary()[0]["valueCurrent"]
('Name', 'example 001')
exp.get_metrics_summary("train_loss")
{'name': 'train_loss', 'valueMax': '2.3872790336608887', 'valueMin': '2.3872790336608887', 'valueCurrent': '2.3872790336608887', 'timestampMax': 1542823695820, 'timestampMin': 1542823695820, 'timestampCurrent': 1542823695820, 'runContextMax': 'train', 'runContextMin': 'train', 'runContextCurrent': 'train', 'stepMax': 0, 'stepMin': 0, 'stepCurrent': 0}
You can see more information on the methods and propeties on an APIExperiment instance here: https://www.comet.ml/docs/python-sdk/APIExperiment/
Just like when creating and logging data, you can also use the .display()
method to show the Comet.ml page for that experiment right in the notebook:
exp.display()
With this version of the API, you can now easily log, set, and add data to an experiment, either one that you just created, or one that existed previously.
To create a new APIExperiment, you can (assuming that your API key, workspace, and project name have been set in a Comet configure file):
from comet_ml.api import APIExperiment
api_experiment = APIExperiment()
If you have not set your API key, workspace, and project name, you can pass any and all to the constructor:
from comet_ml.api import APIExperiment
api_experiment = APIExperiment(api_key="MY-KEY",
workspace="MY-WORKSPACE",
project_name="MY-PROJECT")
To create an APIExperiment from a previously-made experiment, you can (assuming that your keys have been set):
from comet_ml.api import APIExperiment
api_experiment = APIExperiment(previous_experiment="7364746746743746") # previous experiment key or name
Once you have an APIExperiment, you can log, set, and add items to the experiment. You can use any of the following methods:
For many aspects of an experiment, you can log them with the Python API. However, to get the full power of the streaming Experiment
interface, you can create an ExistingExperiment
.
First, we can look up an experiment using any of the methods outlined here:
api_experiment = comet_api.get("cometpublic", "comet-notebooks", 'example 001')
Then, we can use the APIExperiment.id property to make an ExistingExperiment
:
existing_experiment = comet_ml.ExistingExperiment(previous_experiment=api_experiment.id)
COMET INFO: Experiment is live on comet.ml https://www.comet.ml/cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20
You can make changes to the saved data using the existing experiment:
existing_experiment.end()
COMET INFO: ---------------------------- COMET INFO: Comet.ml Experiment Summary: COMET INFO: Data: COMET INFO: url: https://www.comet.ml/cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20 COMET INFO: ---------------------------- COMET INFO: Uploading stats to Comet before program termination (may take several seconds)
As seen above, you can use the Query API that allows highly efficient queries of your data. However, you can also write your own query of sorts if you need to go beyond what the Query API provides.
Here is some code that prints out the names of experiments that have associated HTML (this can take a long time if you have many experiments):
%%time
workspace = "dsblank"
found = False
for project in comet_api.get(workspace):
if found:
break
print(" processing project", project, "...")
print(" processing experiment", exp.id, end="")
for exp in comet_api.get(workspace, project):
print(".", end="")
if exp.get_html() != None:
print("\nFound html in %s!" % exp.url)
found = True
break
print()
processing project release-2-0-12c ... processing experiment d21f94a1c71841d2961da1e6ddb5ab20. Found html in https://www.comet.ml/dsblank/release-2-0-12c/0f22eef674174082b8bcb2b78b116343! CPU times: user 45.1 ms, sys: 1.01 ms, total: 46.1 ms Wall time: 1 s
Here is a function that will find the first experiment that has associated images:
def find_image():
for workspace in comet_api.get():
for project in comet_api.get(workspace):
for exp in comet_api.get(workspace, project):
if exp.get_asset_list(asset_type="image") != []:
return exp
find_image()
<APIExperiment 'cometpublic/ludwig/02a0ed902ce2481fb6e2fc9009ee593c'>
Now, we get the experiment API and explore the APIExperiment.get_asset_list()
method:
comet_api.get('cometpublic/ludwig/02a0ed902ce2481fb6e2fc9009ee593c').get_asset_list(asset_type="image")
[{'fileName': 'Figure(640x480).svg', 'fileSize': 35241, 'runContext': None, 'step': None, 'link': 'https://s3.amazonaws.com/comet.ml/yLB5azBCky96p5Q5IJJJA4est.svg', 'compressedAssetLink': None, 'createdAt': 1555703549582, 'dir': 'media', 'canView': True, 'audio': False, 'video': False, 'histogram': False, 'image': True, 'type': 'image', 'metadata': None, 'assetId': '5b6a596de48041c08c3395d80cc436f6'}, {'fileName': 'Figure(640x480) (1).svg', 'fileSize': 34144, 'runContext': None, 'step': None, 'link': 'https://s3.amazonaws.com/comet.ml/kkn1Ca1jRnYCFa8SRCy2RMGUT.svg', 'compressedAssetLink': None, 'createdAt': 1555703600815, 'dir': 'media', 'canView': True, 'audio': False, 'video': False, 'histogram': False, 'image': True, 'type': 'image', 'metadata': None, 'assetId': '2d45468babff4bb08befcbc03f3af2fb'}, {'fileName': 'Figure(640x480) (2).svg', 'fileSize': 34700, 'runContext': None, 'step': None, 'link': 'https://s3.amazonaws.com/comet.ml/7eisOw4COHuLwtEYMNELdVURi.svg', 'compressedAssetLink': None, 'createdAt': 1555703605506, 'dir': 'media', 'canView': True, 'audio': False, 'video': False, 'histogram': False, 'image': True, 'type': 'image', 'metadata': None, 'assetId': '0bb15be9522644b88fe36528ae15fa6f'}, {'fileName': 'Figure(640x480) (3).svg', 'fileSize': 35032, 'runContext': None, 'step': None, 'link': 'https://s3.amazonaws.com/comet.ml/zCWecfRiaIw3EydI9L0BfN3y8.svg', 'compressedAssetLink': None, 'createdAt': 1555703613885, 'dir': 'media', 'canView': True, 'audio': False, 'video': False, 'histogram': False, 'image': True, 'type': 'image', 'metadata': None, 'assetId': '7e7e685298f848aea0d16f9dca76435e'}, {'fileName': 'Figure(640x480) (4).svg', 'fileSize': 35956, 'runContext': None, 'step': None, 'link': 'https://s3.amazonaws.com/comet.ml/TQ8EKEUBjYtY1XNhSmp0LBFa0.svg', 'compressedAssetLink': None, 'createdAt': 1555703619030, 'dir': 'media', 'canView': True, 'audio': False, 'video': False, 'histogram': False, 'image': True, 'type': 'image', 'metadata': None, 'assetId': '95ad3b25b2c145b8be69035a7819495f'}]
We can get a URL for the image, and display it in the notebook:
asset_list = comet_api.get('cometpublic/ludwig/02a0ed902ce2481fb6e2fc9009ee593c').get_asset_list(asset_type="image")
url = asset_list[0]["link"]
url
'https://s3.amazonaws.com/comet.ml/yLB5azBCky96p5Q5IJJJA4est.svg'
from IPython.display import Image
Image(url=url)
Now, let's write a short program that will find the run with the best accuracy given a workspace/project string:
Can we get all of the hidden_size
parameter values for the experiments in dsblank/pytorch?
[[p["valueCurrent"] for p in exp.get_parameters_summary()] for exp in comet_api.get("dsblank/pytorch")]
[['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', 'true', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28'], ['100', '128', '28', '0.01', '10', '3', '2', '28'], ['100', '128', '28', '0.01', '10', '3', '2', '28'], ['100', '128', '28', '0.01', '10', '3', '2', '28'], ['100', '128', '28', '0.01', '10', '3', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28', '0'], ['100', '128', '28', '0.01', '10', '2', '2', '28', '599'], ['100', '128', '28', '0.01', '10', '2', '2', '28', '599'], ['1199'], ['100', '128', '28', '0.01', '10', '2', '2', '28', '599'], ['100', '361fadf1c712', '/run/user/1000/jupyter/kernel-db3ca856-5af7-46ea-85f6-6fe462cae76b.json', '128', '28', '0.01', '10', '3', '2', '28'], ['100', '128', '28', '0.01', '10', '2', '2', '28', '599']]
experiments = [[(exp, "hidden_size", int(param["valueCurrent"]))
for param in exp.get_parameters_summary()
if param["name"] == "hidden_size"]
for exp in comet_api.get("dsblank/pytorch")]
experiments = [e[0] for e in experiments if len(e) > 0]
experiments[0]
(<APIExperiment 'dsblank/pytorch/5f540fc7894146f4bbfd21ebe8549fcf'>, 'hidden_size', 128)
To get an asset, you need to get the asset_id. You can see all of the assets related to a project using the APIExperiment.get_asset_list()
:
def find_asset(workspaces):
for ws in workspaces or comet_api.get():
for pj in comet_api.get(ws):
for exp in comet_api.get(ws, pj):
if exp.get_asset_list() != []:
return (exp, exp.get_asset_list())
exp, elist = find_asset(["cometpublic"])
exp
<APIExperiment 'cometpublic/ludwig/02a0ed902ce2481fb6e2fc9009ee593c'>
len(elist)
15
From there, you can use the APIExperiment.get_asset(asset_id)
method to get the asset.
description = exp.get_asset("0b256dc8858f4fbeb09228bc96074341", return_type="json")
print(description)
{'command': '/Users/ceceliashao/.local/bin/ludwig experiment --data_csv reuters-allcats.csv --model_definition_file model_definition.yaml', 'commit_hash': '5fbec5d311db', 'dataset_type': 'generic', 'input_data': 'reuters-allcats.csv', 'ludwig_version': '0.1.1', 'model_definition': {'combiner': {'type': 'concat'}, 'input_features': [{'encoder': 'parallel_cnn', 'level': 'word', 'name': 'text', 'tied_weights': None, 'type': 'text'}], 'output_features': [{'dependencies': [], 'loss': {'class_similarities_temperature': 0, 'class_weights': 1, 'confidence_penalty': 0, 'distortion': 1, 'labels_smoothing': 0, 'negative_samples': 0, 'robust_lambda': 0, 'sampler': None, 'type': 'softmax_cross_entropy', 'unique': False, 'weight': 1}, 'name': 'class', 'reduce_dependencies': 'sum', 'reduce_input': 'sum', 'top_k': 3, 'type': 'category'}], 'preprocessing': {'bag': {'fill_value': '', 'format': 'space', 'lowercase': False, 'missing_value_strategy': 'fill_with_const', 'most_common': 10000}, 'binary': {'fill_value': 0, 'missing_value_strategy': 'fill_with_const'}, 'category': {'fill_value': '<UNK>', 'lowercase': False, 'missing_value_strategy': 'fill_with_const', 'most_common': 10000}, 'force_split': False, 'image': {'in_memory': True, 'missing_value_strategy': 'backfill', 'resize_method': 'crop_or_pad'}, 'numerical': {'fill_value': 0, 'missing_value_strategy': 'fill_with_const'}, 'sequence': {'fill_value': '', 'format': 'space', 'lowercase': False, 'missing_value_strategy': 'fill_with_const', 'most_common': 20000, 'padding': 'right', 'padding_symbol': '<PAD>', 'sequence_length_limit': 256, 'unknown_symbol': '<UNK>'}, 'set': {'fill_value': '', 'format': 'space', 'lowercase': False, 'missing_value_strategy': 'fill_with_const', 'most_common': 10000}, 'split_probabilities': [0.7, 0.1, 0.2], 'stratify': None, 'text': {'char_format': 'characters', 'char_most_common': 70, 'char_sequence_length_limit': 1024, 'fill_value': '', 'lowercase': True, 'missing_value_strategy': 'fill_with_const', 'padding': 'right', 'padding_symbol': '<PAD>', 'unknown_symbol': '<UNK>', 'word_format': 'space_punct', 'word_most_common': 20000, 'word_sequence_length_limit': 256}, 'timeseries': {'fill_value': '', 'format': 'space', 'missing_value_strategy': 'fill_with_const', 'padding': 'right', 'padding_value': 0, 'timeseries_length_limit': 256}}, 'training': {'batch_size': 128, 'bucketing_field': None, 'decay': False, 'decay_rate': 0.96, 'decay_steps': 10000, 'dropout_rate': 0.0, 'early_stop': 5, 'epochs': 1, 'eval_batch_size': 0, 'gradient_clipping': None, 'increase_batch_size_on_plateau': 0, 'increase_batch_size_on_plateau_max': 512, 'increase_batch_size_on_plateau_patience': 5, 'increase_batch_size_on_plateau_rate': 2, 'learning_rate': 0.001, 'learning_rate_warmup_epochs': 5, 'optimizer': {'beta1': 0.9, 'beta2': 0.999, 'epsilon': 1e-08, 'type': 'adam'}, 'reduce_learning_rate_on_plateau': 0, 'reduce_learning_rate_on_plateau_patience': 5, 'reduce_learning_rate_on_plateau_rate': 0.5, 'regularization_lambda': 0, 'regularizer': 'l2', 'staircase': False, 'validation_field': 'combined', 'validation_measure': 'loss'}}, 'random_seed': 42}
We hope that this gives you some ideas of how you can use the Comet Python API!