Notebook

Running Tune experiments with BOHB¶

In this tutorial we introduce BOHB, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with BOHB and, as a result, allow you to seamlessly scale up a BOHB optimization process - without sacrificing performance.

Bayesian Optimization HyperBand (BOHB) combines the benefits of Bayesian optimization together with Bandit-based methods (e.g. HyperBand). BOHB does not rely on the gradient of the objective function, but instead, learns from samples of the search space. It is suitable for optimizing functions that are non-differentiable, with many local minima, or even unknown but only testable. Therefore, this approach belongs to the domain of "derivative-free optimization" and "black-box optimization".

In this example we minimize a simple objective to briefly demonstrate the usage of BOHB with Ray Tune via BOHBSearch. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume ConfigSpace==0.4.18 and hpbandster==0.7.4 libraries are installed. To learn more, please refer to the BOHB website.

In [1]:

# !pip install ray[tune]
!pip install ConfigSpace==0.4.18
!pip install hpbandster==0.7.4

Collecting ConfigSpace==0.4.18
  Using cached ConfigSpace-0.4.18-cp37-cp37m-macosx_10_15_x86_64.whl
Requirement already satisfied: pyparsing in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ConfigSpace==0.4.18) (2.4.7)
Requirement already satisfied: numpy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ConfigSpace==0.4.18) (1.21.6)
Requirement already satisfied: cython in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ConfigSpace==0.4.18) (0.29.30)
Installing collected packages: ConfigSpace
  Attempting uninstall: ConfigSpace
    Found existing installation: ConfigSpace 0.4.21
    Uninstalling ConfigSpace-0.4.21:
      Successfully uninstalled ConfigSpace-0.4.21
Successfully installed ConfigSpace-0.4.18
WARNING: There was an error checking the latest version of pip.
Requirement already satisfied: hpbandster==0.7.4 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (0.7.4)
Requirement already satisfied: netifaces in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hpbandster==0.7.4) (0.11.0)
Requirement already satisfied: scipy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hpbandster==0.7.4) (1.4.1)
Requirement already satisfied: Pyro4 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hpbandster==0.7.4) (4.82)
Requirement already satisfied: serpent in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hpbandster==0.7.4) (1.40)
Requirement already satisfied: statsmodels in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hpbandster==0.7.4) (0.13.2)
Requirement already satisfied: ConfigSpace in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hpbandster==0.7.4) (0.4.18)
Requirement already satisfied: numpy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hpbandster==0.7.4) (1.21.6)
Requirement already satisfied: cython in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ConfigSpace->hpbandster==0.7.4) (0.29.30)
Requirement already satisfied: pyparsing in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ConfigSpace->hpbandster==0.7.4) (2.4.7)
Requirement already satisfied: pandas>=0.25 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from statsmodels->hpbandster==0.7.4) (1.3.5)
Requirement already satisfied: packaging>=21.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from statsmodels->hpbandster==0.7.4) (21.3)
Requirement already satisfied: patsy>=0.5.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from statsmodels->hpbandster==0.7.4) (0.5.2)
Requirement already satisfied: python-dateutil>=2.7.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas>=0.25->statsmodels->hpbandster==0.7.4) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas>=0.25->statsmodels->hpbandster==0.7.4) (2022.1)
Requirement already satisfied: six in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from patsy>=0.5.2->statsmodels->hpbandster==0.7.4) (1.16.0)
WARNING: There was an error checking the latest version of pip.

Click below to see all the imports we need for this example. You can also launch directly into a Binder instance to run this notebook yourself. Just click on the rocket symbol at the top of the navigation.

In [2]:

import time

import ray
from ray import train, tune
from ray.tune.schedulers.hb_bohb import HyperBandForBOHB
from ray.tune.search.bohb import TuneBOHB
import ConfigSpace as CS

Let's start by defining a simple evaluation function. We artificially sleep for a bit (0.1 seconds) to simulate a long-running ML experiment. This setup assumes that we're running multiple steps of an experiment and try to tune two hyperparameters, namely width and height, and activation.

In [3]:

def evaluate(step, width, height, activation):
    time.sleep(0.1)
    activation_boost = 10 if activation=="relu" else 1
    return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost

Next, our objective function takes a Tune config, evaluates the score of your experiment in a training loop, and uses train.report to report the score back to Tune.

In [4]:

def objective(config):
    for step in range(config["steps"]):
        score = evaluate(step, config["width"], config["height"], config["activation"])
        train.report({"iterations": step, "mean_loss": score})

In [5]:

ray.init(configure_logging=False)

Out[5]:

Ray

Python version:	3.7.7
Ray version:	3.0.0.dev0
Dashboard:	http://127.0.0.1:8265

Next we define a search space. The critical assumption is that the optimal hyperparameters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time.

In [6]:

search_space = {
    "steps": 100,
    "width": tune.uniform(0, 20),
    "height": tune.uniform(-100, 100),
    "activation": tune.choice(["relu", "tanh"]),
}

Next we define the search algorithm built from TuneBOHB, constrained to a maximum of 4 concurrent trials with a ConcurrencyLimiter. Below algo will take care of the BO (Bayesian optimization) part of BOHB, while scheduler will take care the HB (HyperBand) part.

In [7]:

algo = TuneBOHB()
algo = tune.search.ConcurrencyLimiter(algo, max_concurrent=4)
scheduler = HyperBandForBOHB(
    time_attr="training_iteration",
    max_t=100,
    reduction_factor=4,
    stop_last_trials=False,
)

The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to 1000 samples. (you can decrease this if it takes too long on your machine).

In [8]:

num_samples = 1000

In [9]:

num_samples = 10

Finally, we run the experiment to minimize the "mean_loss" of the objective by searching within "steps": 100 via algo, num_samples times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute tuner.fit().

In [10]:

tuner = tune.Tuner(
    objective,
    tune_config=tune.TuneConfig(
        metric="mean_loss",
        mode="min",
        search_alg=algo,
        scheduler=scheduler,
        num_samples=num_samples,
    ),
    run_config=train.RunConfig(
        name="bohb_exp",
        stop={"training_iteration": 100},
    ),
    param_space=search_space,
)
results = tuner.fit()

Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.

== Status ==
Current time: 2022-07-22 15:07:54 (running for 00:00:26.41)
Memory usage on this node: 9.9/16.0 GiB
Using HyperBand: num_stopped=9 total_brackets=1 Round #0: Bracket(Max Size (n)=1, Milestone (r)=64, completed=66.8%): {TERMINATED: 10}
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.95 GiB heap, 0.0/2.0 GiB objects
Current best trial: a0c11456 with mean_loss=-4.53376204004117 and parameters={'steps': 100, 'width': 3.7250202606878258, 'height': -57.97769618290691, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/bohb_exp
Number of trials: 10/10 (10 TERMINATED)

Trial name	status	loc	activation	height	width	loss	iter	total time (s)	iterations	neg_mean_loss
objective_9e8d8b06	TERMINATED	127.0.0.1:45117	tanh	37.6516	12.2188	5.28254	16	2.23943	15	-5.28254
objective_a052a214	TERMINATED	127.0.0.1:45150	relu	-4.10627	17.9931	11.1524	4	0.531915	3	-11.1524
objective_a06180d6	TERMINATED	127.0.0.1:45151	tanh	89.5711	8.05512	12.884	4	0.534212	3	-12.884
objective_a077899e	TERMINATED	127.0.0.1:45152	relu	67.3538	13.6388	18.6994	4	0.538702	3	-18.6994
objective_a0865c76	TERMINATED	127.0.0.1:45153	relu	25.9876	9.57103	15.1819	4	0.559531	3	-15.1819
objective_a0a42d1e	TERMINATED	127.0.0.1:45154	relu	80.9133	13.0972	20.1201	4	0.538588	3	-20.1201
objective_a0c11456	TERMINATED	127.0.0.1:45117	tanh	-57.9777	3.72502	-4.53376	100	13.0563	99	4.53376
objective_a0dce442	TERMINATED	127.0.0.1:45200	tanh	-1.68715	1.87185	3.4575	16	2.27704	15	-3.4575
objective_a0f84156	TERMINATED	127.0.0.1:45157	tanh	65.9879	5.41575	11.4087	4	0.548028	3	-11.4087
objective_a1142416	TERMINATED	127.0.0.1:45201	tanh	5.35569	4.85644	2.74262	16	2.27749	15	-2.74262

Result for objective_9e8d8b06:
  date: 2022-07-22_15-07-31
  done: false
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.765164520733162
  neg_mean_loss: -14.765164520733162
  node_ip: 127.0.0.1
  pid: 45117
  time_since_restore: 0.10084700584411621
  time_this_iter_s: 0.10084700584411621
  time_total_s: 0.10084700584411621
  timestamp: 1658498851
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 9e8d8b06
  warmup_time: 0.005752086639404297
  
Result for objective_a06180d6:
  date: 2022-07-22_15-07-31
  done: false
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 19.957107852020386
  neg_mean_loss: -19.957107852020386
  node_ip: 127.0.0.1
  pid: 45117
  time_since_restore: 0.10333395004272461
  time_this_iter_s: 0.10333395004272461
  time_total_s: 0.10333395004272461
  timestamp: 1658498851
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a06180d6
  warmup_time: 0.005752086639404297
  
Result for objective_a0a42d1e:
  date: 2022-07-22_15-07-31
  done: false
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 28.091327935768636
  neg_mean_loss: -28.091327935768636
  node_ip: 127.0.0.1
  pid: 45117
  time_since_restore: 0.10419368743896484
  time_this_iter_s: 0.10419368743896484
  time_total_s: 0.10419368743896484
  timestamp: 1658498851
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a0a42d1e
  warmup_time: 0.005752086639404297
  
Result for objective_a0c11456:
  date: 2022-07-22_15-07-31
  done: false
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 5.202230381709309
  neg_mean_loss: -5.202230381709309
  node_ip: 127.0.0.1
  pid: 45117
  time_since_restore: 0.10060286521911621
  time_this_iter_s: 0.10060286521911621
  time_total_s: 0.10060286521911621
  timestamp: 1658498851
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a0c11456
  warmup_time: 0.005752086639404297
  
Result for objective_a0dce442:
  date: 2022-07-22_15-07-32
  done: false
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 10.831285273000042
  neg_mean_loss: -10.831285273000042
  node_ip: 127.0.0.1
  pid: 45117
  time_since_restore: 0.10183191299438477
  time_this_iter_s: 0.10183191299438477
  time_total_s: 0.10183191299438477
  timestamp: 1658498852
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a0dce442
  warmup_time: 0.005752086639404297
  
Result for objective_a0f84156:
  date: 2022-07-22_15-07-32
  done: false
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 17.598785421495204
  neg_mean_loss: -17.598785421495204
  node_ip: 127.0.0.1
  pid: 45117
  time_since_restore: 0.10328507423400879
  time_this_iter_s: 0.10328507423400879
  time_total_s: 0.10328507423400879
  timestamp: 1658498852
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a0f84156
  warmup_time: 0.005752086639404297
  
Result for objective_a1142416:
  date: 2022-07-22_15-07-32
  done: false
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 11.53556906389516
  neg_mean_loss: -11.53556906389516
  node_ip: 127.0.0.1
  pid: 45117
  time_since_restore: 0.10475611686706543
  time_this_iter_s: 0.10475611686706543
  time_total_s: 0.10475611686706543
  timestamp: 1658498852
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a1142416
  warmup_time: 0.005752086639404297
  
Result for objective_a052a214:
  date: 2022-07-22_15-07-34
  done: false
  experiment_id: 0fb96edfd1b34561b337e7146a3c64aa
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 19.5893732640325
  neg_mean_loss: -19.5893732640325
  node_ip: 127.0.0.1
  pid: 45126
  time_since_restore: 0.10437989234924316
  time_this_iter_s: 0.10437989234924316
  time_total_s: 0.10437989234924316
  timestamp: 1658498854
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a052a214
  warmup_time: 0.003122091293334961
  
Result for objective_a077899e:
  date: 2022-07-22_15-07-34
  done: false
  experiment_id: 008f8a44fb0c4475a3e9dad32f1bd2c9
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 26.735381559909975
  neg_mean_loss: -26.735381559909975
  node_ip: 127.0.0.1
  pid: 45129
  time_since_restore: 0.10449695587158203
  time_this_iter_s: 0.10449695587158203
  time_total_s: 0.10449695587158203
  timestamp: 1658498854
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a077899e
  warmup_time: 0.0031490325927734375
  
Result for objective_a0865c76:
  date: 2022-07-22_15-07-34
  done: false
  experiment_id: 06a194a4b7784ca5932a66e44e3cd815
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 22.598763488408313
  neg_mean_loss: -22.598763488408313
  node_ip: 127.0.0.1
  pid: 45132
  time_since_restore: 0.10512685775756836
  time_this_iter_s: 0.10512685775756836
  time_total_s: 0.10512685775756836
  timestamp: 1658498854
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a0865c76
  warmup_time: 0.00397801399230957

(objective pid=45117) 2022-07-22 15:07:34,468	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_9e8d8b06_1_activation=tanh,height=37.6516,steps=100,width=12.2188_2022-07-22_15-07-28/checkpoint_tmpf674c6
(objective pid=45117) 2022-07-22 15:07:34,469	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10084700584411621, '_episodes_total': 0}
(objective pid=45154) 2022-07-22 15:07:37,958	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_a0a42d1e_6_activation=relu,height=80.9133,steps=100,width=13.0972_2022-07-22_15-07-31/checkpoint_tmp30fafa
(objective pid=45154) 2022-07-22 15:07:37,958	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10419368743896484, '_episodes_total': 0}
(objective pid=45155) 2022-07-22 15:07:37,952	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_a0c11456_7_activation=tanh,height=-57.9777,steps=100,width=3.7250_2022-07-22_15-07-31/checkpoint_tmp7c225a
(objective pid=45155) 2022-07-22 15:07:37,952	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10060286521911621, '_episodes_total': 0}
(objective pid=45151) 2022-07-22 15:07:37,933	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_a06180d6_3_activation=tanh,height=89.5711,steps=100,width=8.0551_2022-07-22_15-07-31/checkpoint_tmpcde58c
(objective pid=45151) 2022-07-22 15:07:37,933	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10333395004272461, '_episodes_total': 0}
(objective pid=45150) 2022-07-22 15:07:37,929	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_a052a214_2_activation=relu,height=-4.1063,steps=100,width=17.9931_2022-07-22_15-07-31/checkpoint_tmp24072a
(objective pid=45150) 2022-07-22 15:07:37,929	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10437989234924316, '_episodes_total': 0}
(objective pid=45153) 2022-07-22 15:07:38,010	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_a0865c76_5_activation=relu,height=25.9876,steps=100,width=9.5710_2022-07-22_15-07-31/checkpoint_tmpad3367
(objective pid=45153) 2022-07-22 15:07:38,010	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10512685775756836, '_episodes_total': 0}
(objective pid=45158) 2022-07-22 15:07:38,010	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_a1142416_10_activation=tanh,height=5.3557,steps=100,width=4.8564_2022-07-22_15-07-32/checkpoint_tmpe18b15
(objective pid=45158) 2022-07-22 15:07:38,010	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10475611686706543, '_episodes_total': 0}
(objective pid=45156) 2022-07-22 15:07:38,008	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_a0dce442_8_activation=tanh,height=-1.6871,steps=100,width=1.8718_2022-07-22_15-07-32/checkpoint_tmp01ad4f
(objective pid=45156) 2022-07-22 15:07:38,008	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10183191299438477, '_episodes_total': 0}
(objective pid=45157) 2022-07-22 15:07:38,025	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_a0f84156_9_activation=tanh,height=65.9879,steps=100,width=5.4158_2022-07-22_15-07-32/checkpoint_tmp57fedd
(objective pid=45157) 2022-07-22 15:07:38,025	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10328507423400879, '_episodes_total': 0}
(objective pid=45152) 2022-07-22 15:07:38,022	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_a077899e_4_activation=relu,height=67.3538,steps=100,width=13.6388_2022-07-22_15-07-31/checkpoint_tmpfdb6b4
(objective pid=45152) 2022-07-22 15:07:38,022	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10449695587158203, '_episodes_total': 0}

Result for objective_a06180d6:
  date: 2022-07-22_15-07-38
  done: false
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 19.957107852020386
  neg_mean_loss: -19.957107852020386
  node_ip: 127.0.0.1
  pid: 45151
  time_since_restore: 0.10456109046936035
  time_this_iter_s: 0.10456109046936035
  time_total_s: 0.20789504051208496
  timestamp: 1658498858
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: a06180d6
  warmup_time: 0.010712862014770508
  
Result for objective_a0c11456:
  date: 2022-07-22_15-07-38
  done: false
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 5.202230381709309
  neg_mean_loss: -5.202230381709309
  node_ip: 127.0.0.1
  pid: 45155
  time_since_restore: 0.10466408729553223
  time_this_iter_s: 0.10466408729553223
  time_total_s: 0.20526695251464844
  timestamp: 1658498858
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: a0c11456
  warmup_time: 0.007361888885498047
  
Result for objective_a0a42d1e:
  date: 2022-07-22_15-07-38
  done: false
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 28.091327935768636
  neg_mean_loss: -28.091327935768636
  node_ip: 127.0.0.1
  pid: 45154
  time_since_restore: 0.10032892227172852
  time_this_iter_s: 0.10032892227172852
  time_total_s: 0.20452260971069336
  timestamp: 1658498858
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: a0a42d1e
  warmup_time: 0.008682966232299805
  
Result for objective_a0dce442:
  date: 2022-07-22_15-07-38
  done: false
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 10.831285273000042
  neg_mean_loss: -10.831285273000042
  node_ip: 127.0.0.1
  pid: 45156
  time_since_restore: 0.10120797157287598
  time_this_iter_s: 0.10120797157287598
  time_total_s: 0.20303988456726074
  timestamp: 1658498858
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: a0dce442
  warmup_time: 0.0077381134033203125
  
Result for objective_a1142416:
  date: 2022-07-22_15-07-38
  done: false
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 11.53556906389516
  neg_mean_loss: -11.53556906389516
  node_ip: 127.0.0.1
  pid: 45158
  time_since_restore: 0.10349392890930176
  time_this_iter_s: 0.10349392890930176
  time_total_s: 0.2082500457763672
  timestamp: 1658498858
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: a1142416
  warmup_time: 0.007157087326049805
  
Result for objective_a0f84156:
  date: 2022-07-22_15-07-38
  done: false
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 17.598785421495204
  neg_mean_loss: -17.598785421495204
  node_ip: 127.0.0.1
  pid: 45157
  time_since_restore: 0.10368680953979492
  time_this_iter_s: 0.10368680953979492
  time_total_s: 0.2069718837738037
  timestamp: 1658498858
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: a0f84156
  warmup_time: 0.006359100341796875
  
Result for objective_a052a214:
  date: 2022-07-22_15-07-38
  done: false
  episodes_total: 0
  experiment_id: 0fb96edfd1b34561b337e7146a3c64aa
  experiment_tag: 2_activation=relu,height=-4.1063,steps=100,width=17.9931
  hostname: Kais-MacBook-Pro.local
  iterations: 3
  iterations_since_restore: 4
  mean_loss: 11.152378612292932
  neg_mean_loss: -11.152378612292932
  node_ip: 127.0.0.1
  pid: 45150
  time_since_restore: 0.4275352954864502
  time_this_iter_s: 0.1124732494354248
  time_total_s: 0.5319151878356934
  timestamp: 1658498858
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 4
  trial_id: a052a214
  warmup_time: 0.010692834854125977
  
Result for objective_9e8d8b06:
  date: 2022-07-22_15-07-39
  done: false
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.765164520733162
  neg_mean_loss: -14.765164520733162
  node_ip: 127.0.0.1
  pid: 45117
  time_since_restore: 0.10145044326782227
  time_this_iter_s: 0.10145044326782227
  time_total_s: 0.6308252811431885
  timestamp: 1658498859
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 9e8d8b06
  warmup_time: 0.005752086639404297

(objective pid=45117) 2022-07-22 15:07:39,222	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_9e8d8b06_1_activation=tanh,height=37.6516,steps=100,width=12.2188_2022-07-22_15-07-28/checkpoint_tmpacb5ff
(objective pid=45117) 2022-07-22 15:07:39,223	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.5293748378753662, '_episodes_total': 0}
(objective pid=45199) 2022-07-22 15:07:41,833	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_a0c11456_7_activation=tanh,height=-57.9777,steps=100,width=3.7250_2022-07-22_15-07-31/checkpoint_tmpdf2071
(objective pid=45199) 2022-07-22 15:07:41,833	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.53450608253479, '_episodes_total': 0}
(objective pid=45200) 2022-07-22 15:07:41,833	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_a0dce442_8_activation=tanh,height=-1.6871,steps=100,width=1.8718_2022-07-22_15-07-32/checkpoint_tmp9cd0fc
(objective pid=45200) 2022-07-22 15:07:41,833	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.5349018573760986, '_episodes_total': 0}
(objective pid=45201) 2022-07-22 15:07:41,882	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_a1142416_10_activation=tanh,height=5.3557,steps=100,width=4.8564_2022-07-22_15-07-32/checkpoint_tmp7bd297
(objective pid=45201) 2022-07-22 15:07:41,882	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.5445291996002197, '_episodes_total': 0}

Result for objective_a0c11456:
  date: 2022-07-22_15-07-43
  done: false
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 11
  iterations_since_restore: 12
  mean_loss: -2.8360322412948014
  neg_mean_loss: 2.8360322412948014
  node_ip: 127.0.0.1
  pid: 45199
  time_since_restore: 1.3028512001037598
  time_this_iter_s: 0.10884499549865723
  time_total_s: 1.8373572826385498
  timestamp: 1658498863
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 12
  trial_id: a0c11456
  warmup_time: 0.0074689388275146484
  
Result for objective_a0dce442:
  date: 2022-07-22_15-07-43
  done: false
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 11
  iterations_since_restore: 12
  mean_loss: 4.100295137851358
  neg_mean_loss: -4.100295137851358
  node_ip: 127.0.0.1
  pid: 45200
  time_since_restore: 1.3082969188690186
  time_this_iter_s: 0.10707712173461914
  time_total_s: 1.8431987762451172
  timestamp: 1658498863
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 12
  trial_id: a0dce442
  warmup_time: 0.007519960403442383
  
Result for objective_a1142416:
  date: 2022-07-22_15-07-43
  done: false
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 11
  iterations_since_restore: 12
  mean_loss: 3.112339173810433
  neg_mean_loss: -3.112339173810433
  node_ip: 127.0.0.1
  pid: 45201
  time_since_restore: 1.2954580783843994
  time_this_iter_s: 0.1081690788269043
  time_total_s: 1.8399872779846191
  timestamp: 1658498863
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 12
  trial_id: a1142416
  warmup_time: 0.006086826324462891
  
Result for objective_a1142416:
  date: 2022-07-22_15-07-43
  done: true
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 15
  iterations_since_restore: 16
  mean_loss: 2.7426202715773025
  neg_mean_loss: -2.7426202715773025
  node_ip: 127.0.0.1
  pid: 45201
  time_since_restore: 1.7329609394073486
  time_this_iter_s: 0.1072549819946289
  time_total_s: 2.2774901390075684
  timestamp: 1658498863
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 16
  trial_id: a1142416
  warmup_time: 0.006086826324462891

(objective pid=45117) 2022-07-22 15:07:43,765	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp/objective_a0c11456_7_activation=tanh,height=-57.9777,steps=100,width=3.7250_2022-07-22_15-07-31/checkpoint_tmp3f09eb
(objective pid=45117) 2022-07-22 15:07:43,765	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 2.273958206176758, '_episodes_total': 0}

Result for objective_a0c11456:
  date: 2022-07-22_15-07-48
  done: false
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 40
  iterations_since_restore: 41
  mean_loss: -4.168842006342518
  neg_mean_loss: 4.168842006342518
  node_ip: 127.0.0.1
  pid: 45117
  time_since_restore: 4.4259912967681885
  time_this_iter_s: 0.10874629020690918
  time_total_s: 6.699949502944946
  timestamp: 1658498868
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 41
  trial_id: a0c11456
  warmup_time: 0.005752086639404297
  
Result for objective_a0c11456:
  date: 2022-07-22_15-07-53
  done: false
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 87
  iterations_since_restore: 88
  mean_loss: -4.498437215767879
  neg_mean_loss: 4.498437215767879
  node_ip: 127.0.0.1
  pid: 45117
  time_since_restore: 9.488422155380249
  time_this_iter_s: 0.10667800903320312
  time_total_s: 11.762380361557007
  timestamp: 1658498873
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 88
  trial_id: a0c11456
  warmup_time: 0.005752086639404297
  
Result for objective_a0c11456:
  date: 2022-07-22_15-07-54
  done: true
  episodes_total: 0
  experiment_id: c1cb9895c3f04e73b7cce9435cd92c68
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -4.53376204004117
  neg_mean_loss: 4.53376204004117
  node_ip: 127.0.0.1
  pid: 45117
  time_since_restore: 10.782363176345825
  time_this_iter_s: 0.1065070629119873
  time_total_s: 13.056321382522583
  timestamp: 1658498874
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 100
  trial_id: a0c11456
  warmup_time: 0.005752086639404297

Here are the hyperparameters found to minimize the mean loss of the defined objective.

In [11]:

print("Best hyperparameters found were: ", results.get_best_result().config)

Best hyperparameters found were:  {'steps': 100, 'width': 3.7250202606878258, 'height': -57.97769618290691, 'activation': 'tanh'}

Optional: Passing the search space via the TuneBOHB algorithm¶

We can define the hyperparameter search space using ConfigSpace, which is the format accepted by BOHB.

In [12]:

config_space = CS.ConfigurationSpace()
config_space.add_hyperparameter(
    CS.Constant("steps", 100)
)
config_space.add_hyperparameter(
    CS.UniformFloatHyperparameter("width", lower=0, upper=20)
)
config_space.add_hyperparameter(
    CS.UniformFloatHyperparameter("height", lower=-100, upper=100)
)
config_space.add_hyperparameter(
    CS.CategoricalHyperparameter(
        "activation", choices=["relu", "tanh"]
    )
)

/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/ipykernel_launcher.py:3: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
This is separate from the ipykernel package so we can avoid doing imports until
/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/ipykernel_launcher.py:6: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/ipykernel_launcher.py:9: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
if __name__ == "__main__":
/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/ipykernel_launcher.py:13: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
del sys.path[0]

Out[12]:

activation, Type: Categorical, Choices: {relu, tanh}, Default: relu

In [13]:

# As we are passing config space directly to the searcher,
# we need to define metric and mode in it as well, in addition
# to Tuner()
algo = TuneBOHB(
    space=config_space,
    metric="mean_loss",
    mode="max",
)
algo = tune.search.ConcurrencyLimiter(algo, max_concurrent=4)
scheduler = HyperBandForBOHB(
    time_attr="training_iteration",
    max_t=100,
    reduction_factor=4,
    stop_last_trials=False,
)

In [17]:

tuner = tune.Tuner(
    objective,
    tune_config=tune.TuneConfig(
        metric="mean_loss",
        mode="min",
        search_alg=algo,
        scheduler=scheduler,
        num_samples=num_samples,
    ),
    run_config=train.RunConfig(
        name="bohb_exp_2",
        stop={"training_iteration": 100},
    ),
)
results = tuner.fit()

== Status ==
Current time: 2022-07-22 15:11:40 (running for 00:00:29.52)
Memory usage on this node: 10.3/16.0 GiB
Using HyperBand: num_stopped=9 total_brackets=1 Round #0: Bracket(Max Size (n)=1, Milestone (r)=64, completed=66.8%): {TERMINATED: 10}
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.95 GiB heap, 0.0/2.0 GiB objects
Current best trial: 25b64488 with mean_loss=-3.74634537130406 and parameters={'activation': 'tanh', 'height': -48.451797714080236, 'steps': 100, 'width': 10.119125894538891}
Result logdir: /Users/kai/ray_results/bohb_exp_2
Number of trials: 10/10 (10 TERMINATED)

Trial name	status	loc	activation	height	steps	width	loss	iter	total time (s)	iterations	neg_mean_loss
objective_2397442c	TERMINATED	127.0.0.1:45401	tanh	32.8422	100	12.1847	4.80297	16	2.25951	15	-4.80297
objective_25b4a998	TERMINATED	127.0.0.1:45401	relu	20.2852	100	2.08202	18.1839	4	0.535426	3	-18.1839
objective_25b64488	TERMINATED	127.0.0.1:45453	tanh	-48.4518	100	10.1191	-3.74635	100	11.5319	99	3.74635
objective_25b7dfe6	TERMINATED	127.0.0.1:45403	relu	-18.8439	100	19.1277	9.59966	4	0.581903	3	-9.59966
objective_25cfab4e	TERMINATED	127.0.0.1:45404	relu	17.2057	100	0.317083	20.8519	4	0.59468	3	-20.8519
objective_278eba4c	TERMINATED	127.0.0.1:45454	relu	-27.0179	100	13.577	7.76626	16	2.31198	15	-7.76626
objective_279d01a6	TERMINATED	127.0.0.1:45407	relu	59.1103	100	2.4466	21.6781	4	0.575556	3	-21.6781
objective_27aa31e6	TERMINATED	127.0.0.1:45409	relu	50.058	100	17.3776	16.6153	4	0.537561	3	-16.6153
objective_27b7e2be	TERMINATED	127.0.0.1:45455	relu	-51.2093	100	8.94948	5.57235	16	2.64238	15	-5.57235
objective_27c59a80	TERMINATED	127.0.0.1:45446	relu	29.165	100	4.26995	17.3006	4	0.539177	3	-17.3006

Result for objective_2397442c:
  date: 2022-07-22_15-11-15
  done: false
  experiment_id: 1a4ebf62df50443492dc6df792fcb67a
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.284216630043918
  neg_mean_loss: -14.284216630043918
  node_ip: 127.0.0.1
  pid: 45353
  time_since_restore: 0.10108494758605957
  time_this_iter_s: 0.10108494758605957
  time_total_s: 0.10108494758605957
  timestamp: 1658499075
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 2397442c
  warmup_time: 0.002635955810546875
  
Result for objective_25b7dfe6:
  date: 2022-07-22_15-11-17
  done: false
  experiment_id: 0fd6607aa9674cb5a05b6ce63e474fd3
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 18.115606120430186
  neg_mean_loss: -18.115606120430186
  node_ip: 127.0.0.1
  pid: 45371
  time_since_restore: 0.10365676879882812
  time_this_iter_s: 0.10365676879882812
  time_total_s: 0.10365676879882812
  timestamp: 1658499077
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 25b7dfe6
  warmup_time: 0.005063056945800781
  
Result for objective_25b4a998:
  date: 2022-07-22_15-11-17
  done: false
  experiment_id: 20a5d76dc18749e4b1c9f15c5d8b43cf
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 22.028519616352035
  neg_mean_loss: -22.028519616352035
  node_ip: 127.0.0.1
  pid: 45369
  time_since_restore: 0.10431909561157227
  time_this_iter_s: 0.10431909561157227
  time_total_s: 0.10431909561157227
  timestamp: 1658499077
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 25b4a998
  warmup_time: 0.004379987716674805
  
Result for objective_25b64488:
  date: 2022-07-22_15-11-17
  done: false
  experiment_id: 75e7c1ad20a2495cac29630df6c3c782
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 6.154820228591976
  neg_mean_loss: -6.154820228591976
  node_ip: 127.0.0.1
  pid: 45370
  time_since_restore: 0.10407018661499023
  time_this_iter_s: 0.10407018661499023
  time_total_s: 0.10407018661499023
  timestamp: 1658499077
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 25b64488
  warmup_time: 0.003113985061645508
  
Result for objective_25cfab4e:
  date: 2022-07-22_15-11-18
  done: false
  experiment_id: fdc43ca37ed44cde857ca150a8f1e84f
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 21.72056884033249
  neg_mean_loss: -21.72056884033249
  node_ip: 127.0.0.1
  pid: 45378
  time_since_restore: 0.10431408882141113
  time_this_iter_s: 0.10431408882141113
  time_total_s: 0.10431408882141113
  timestamp: 1658499078
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 25cfab4e
  warmup_time: 0.0029649734497070312
  
Result for objective_279d01a6:
  date: 2022-07-22_15-11-18
  done: false
  experiment_id: 20a5d76dc18749e4b1c9f15c5d8b43cf
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 25.911032294938884
  neg_mean_loss: -25.911032294938884
  node_ip: 127.0.0.1
  pid: 45369
  time_since_restore: 0.10361599922180176
  time_this_iter_s: 0.10361599922180176
  time_total_s: 0.10361599922180176
  timestamp: 1658499078
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 279d01a6
  warmup_time: 0.004379987716674805
  
Result for objective_27aa31e6:
  date: 2022-07-22_15-11-18
  done: false
  experiment_id: 75e7c1ad20a2495cac29630df6c3c782
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 25.005798678308594
  neg_mean_loss: -25.005798678308594
  node_ip: 127.0.0.1
  pid: 45370
  time_since_restore: 0.10430693626403809
  time_this_iter_s: 0.10430693626403809
  time_total_s: 0.10430693626403809
  timestamp: 1658499078
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 27aa31e6
  warmup_time: 0.003113985061645508
  
Result for objective_27b7e2be:
  date: 2022-07-22_15-11-18
  done: false
  experiment_id: fdc43ca37ed44cde857ca150a8f1e84f
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.879072389639937
  neg_mean_loss: -14.879072389639937
  node_ip: 127.0.0.1
  pid: 45378
  time_since_restore: 0.10371994972229004
  time_this_iter_s: 0.10371994972229004
  time_total_s: 0.10371994972229004
  timestamp: 1658499078
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 27b7e2be
  warmup_time: 0.0029649734497070312
  
Result for objective_27c59a80:
  date: 2022-07-22_15-11-18
  done: false
  experiment_id: 20a5d76dc18749e4b1c9f15c5d8b43cf
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 22.916501000037474
  neg_mean_loss: -22.916501000037474
  node_ip: 127.0.0.1
  pid: 45369
  time_since_restore: 0.10353732109069824
  time_this_iter_s: 0.10353732109069824
  time_total_s: 0.10353732109069824
  timestamp: 1658499078
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 27c59a80
  warmup_time: 0.004379987716674805
  
Result for objective_278eba4c:
  date: 2022-07-22_15-11-20
  done: false
  experiment_id: 90186993d7ff42698c4615640d47d896
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 17.29821256734647
  neg_mean_loss: -17.29821256734647
  node_ip: 127.0.0.1
  pid: 45393
  time_since_restore: 0.10304784774780273
  time_this_iter_s: 0.10304784774780273
  time_total_s: 0.10304784774780273
  timestamp: 1658499080
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 278eba4c
  warmup_time: 0.0027189254760742188
  
Result for objective_2397442c:
  date: 2022-07-22_15-11-20
  done: false
  episodes_total: 0
  experiment_id: 1a4ebf62df50443492dc6df792fcb67a
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.284216630043918
  neg_mean_loss: -14.284216630043918
  node_ip: 127.0.0.1
  pid: 45370
  time_since_restore: 0.10411787033081055
  time_this_iter_s: 0.10411787033081055
  time_total_s: 0.20520281791687012
  timestamp: 1658499080
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 2397442c
  warmup_time: 0.003113985061645508

(objective pid=45370) 2022-07-22 15:11:20,826	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_2397442c_1_activation=tanh,height=32.8422,steps=100,width=12.1847_2022-07-22_15-11-11/checkpoint_tmpf4b290
(objective pid=45370) 2022-07-22 15:11:20,826	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10108494758605957, '_episodes_total': 0}

Result for objective_25b4a998:
  date: 2022-07-22_15-11-24
  done: false
  episodes_total: 0
  experiment_id: 20a5d76dc18749e4b1c9f15c5d8b43cf
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 22.028519616352035
  neg_mean_loss: -22.028519616352035
  node_ip: 127.0.0.1
  pid: 45401
  time_since_restore: 0.10445284843444824
  time_this_iter_s: 0.10445284843444824
  time_total_s: 0.2087719440460205
  timestamp: 1658499084
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 25b4a998
  warmup_time: 0.010488033294677734

(objective pid=45405) 2022-07-22 15:11:23,963	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_278eba4c_6_activation=relu,height=-27.0179,steps=100,width=13.5770_2022-07-22_15-11-18/checkpoint_tmpd366c5
(objective pid=45405) 2022-07-22 15:11:23,964	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10304784774780273, '_episodes_total': 0}
(objective pid=45401) 2022-07-22 15:11:23,959	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_25b4a998_2_activation=relu,height=20.2852,steps=100,width=2.0820_2022-07-22_15-11-14/checkpoint_tmpc96cdb
(objective pid=45401) 2022-07-22 15:11:23,959	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10431909561157227, '_episodes_total': 0}
(objective pid=45402) 2022-07-22 15:11:23,966	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_25b64488_3_activation=tanh,height=-48.4518,steps=100,width=10.1191_2022-07-22_15-11-14/checkpoint_tmp5ce175
(objective pid=45402) 2022-07-22 15:11:23,966	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10407018661499023, '_episodes_total': 0}
(objective pid=45407) 2022-07-22 15:11:23,966	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_279d01a6_7_activation=relu,height=59.1103,steps=100,width=2.4466_2022-07-22_15-11-18/checkpoint_tmp942eac
(objective pid=45407) 2022-07-22 15:11:23,966	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10361599922180176, '_episodes_total': 0}
(objective pid=45404) 2022-07-22 15:11:23,960	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_25cfab4e_5_activation=relu,height=17.2057,steps=100,width=0.3171_2022-07-22_15-11-15/checkpoint_tmpc8ceee
(objective pid=45404) 2022-07-22 15:11:23,960	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10431408882141113, '_episodes_total': 0}
(objective pid=45403) 2022-07-22 15:11:23,966	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_25b7dfe6_4_activation=relu,height=-18.8439,steps=100,width=19.1277_2022-07-22_15-11-14/checkpoint_tmp24108b
(objective pid=45403) 2022-07-22 15:11:23,966	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10365676879882812, '_episodes_total': 0}
(objective pid=45409) 2022-07-22 15:11:23,997	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_27aa31e6_8_activation=relu,height=50.0580,steps=100,width=17.3776_2022-07-22_15-11-18/checkpoint_tmp4218e5
(objective pid=45409) 2022-07-22 15:11:23,997	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10430693626403809, '_episodes_total': 0}

Result for objective_27aa31e6:
  date: 2022-07-22_15-11-24
  done: false
  episodes_total: 0
  experiment_id: 75e7c1ad20a2495cac29630df6c3c782
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 25.005798678308594
  neg_mean_loss: -25.005798678308594
  node_ip: 127.0.0.1
  pid: 45409
  time_since_restore: 0.10203695297241211
  time_this_iter_s: 0.10203695297241211
  time_total_s: 0.2063438892364502
  timestamp: 1658499084
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 27aa31e6
  warmup_time: 0.0062160491943359375
  
Result for objective_279d01a6:
  date: 2022-07-22_15-11-24
  done: false
  episodes_total: 0
  experiment_id: 20a5d76dc18749e4b1c9f15c5d8b43cf
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 25.911032294938884
  neg_mean_loss: -25.911032294938884
  node_ip: 127.0.0.1
  pid: 45407
  time_since_restore: 0.10443115234375
  time_this_iter_s: 0.10443115234375
  time_total_s: 0.20804715156555176
  timestamp: 1658499084
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 279d01a6
  warmup_time: 0.010073184967041016
  
Result for objective_25b7dfe6:
  date: 2022-07-22_15-11-24
  done: false
  episodes_total: 0
  experiment_id: 0fd6607aa9674cb5a05b6ce63e474fd3
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 18.115606120430186
  neg_mean_loss: -18.115606120430186
  node_ip: 127.0.0.1
  pid: 45403
  time_since_restore: 0.1044917106628418
  time_this_iter_s: 0.1044917106628418
  time_total_s: 0.20814847946166992
  timestamp: 1658499084
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 25b7dfe6
  warmup_time: 0.011971235275268555
  
Result for objective_25cfab4e:
  date: 2022-07-22_15-11-24
  done: false
  episodes_total: 0
  experiment_id: fdc43ca37ed44cde857ca150a8f1e84f
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 21.72056884033249
  neg_mean_loss: -21.72056884033249
  node_ip: 127.0.0.1
  pid: 45404
  time_since_restore: 0.10379600524902344
  time_this_iter_s: 0.10379600524902344
  time_total_s: 0.20811009407043457
  timestamp: 1658499084
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 25cfab4e
  warmup_time: 0.009023904800415039
  
Result for objective_25b64488:
  date: 2022-07-22_15-11-24
  done: false
  episodes_total: 0
  experiment_id: 75e7c1ad20a2495cac29630df6c3c782
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 6.154820228591976
  neg_mean_loss: -6.154820228591976
  node_ip: 127.0.0.1
  pid: 45402
  time_since_restore: 0.10440897941589355
  time_this_iter_s: 0.10440897941589355
  time_total_s: 0.2084791660308838
  timestamp: 1658499084
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 25b64488
  warmup_time: 0.011686325073242188

(objective pid=45424) 2022-07-22 15:11:24,383	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_27b7e2be_9_activation=relu,height=-51.2093,steps=100,width=8.9495_2022-07-22_15-11-18/checkpoint_tmp996dec
(objective pid=45424) 2022-07-22 15:11:24,384	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10371994972229004, '_episodes_total': 0}

Result for objective_27b7e2be:
  date: 2022-07-22_15-11-24
  done: false
  episodes_total: 0
  experiment_id: fdc43ca37ed44cde857ca150a8f1e84f
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.879072389639937
  neg_mean_loss: -14.879072389639937
  node_ip: 127.0.0.1
  pid: 45424
  time_since_restore: 0.1031639575958252
  time_this_iter_s: 0.1031639575958252
  time_total_s: 0.20688390731811523
  timestamp: 1658499084
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 27b7e2be
  warmup_time: 0.0069200992584228516

(objective pid=45446) 2022-07-22 15:11:26,749	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_27c59a80_10_activation=relu,height=29.1650,steps=100,width=4.2700_2022-07-22_15-11-18/checkpoint_tmp49d063
(objective pid=45446) 2022-07-22 15:11:26,749	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.10353732109069824, '_episodes_total': 0}

Result for objective_27c59a80:
  date: 2022-07-22_15-11-26
  done: false
  episodes_total: 0
  experiment_id: 20a5d76dc18749e4b1c9f15c5d8b43cf
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 22.916501000037474
  neg_mean_loss: -22.916501000037474
  node_ip: 127.0.0.1
  pid: 45446
  time_since_restore: 0.10502910614013672
  time_this_iter_s: 0.10502910614013672
  time_total_s: 0.20856642723083496
  timestamp: 1658499086
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 27c59a80
  warmup_time: 0.007359027862548828

(objective pid=45401) 2022-07-22 15:11:27,287	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_2397442c_1_activation=tanh,height=32.8422,steps=100,width=12.1847_2022-07-22_15-11-11/checkpoint_tmp2e0d09
(objective pid=45401) 2022-07-22 15:11:27,287	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.535153865814209, '_episodes_total': 0}

Result for objective_2397442c:
  date: 2022-07-22_15-11-27
  done: false
  episodes_total: 0
  experiment_id: 1a4ebf62df50443492dc6df792fcb67a
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.284216630043918
  neg_mean_loss: -14.284216630043918
  node_ip: 127.0.0.1
  pid: 45401
  time_since_restore: 0.1044008731842041
  time_this_iter_s: 0.1044008731842041
  time_total_s: 0.6395547389984131
  timestamp: 1658499087
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 2397442c
  warmup_time: 0.010488033294677734
  
Result for objective_278eba4c:
  date: 2022-07-22_15-11-29
  done: false
  episodes_total: 0
  experiment_id: 90186993d7ff42698c4615640d47d896
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 17.29821256734647
  neg_mean_loss: -17.29821256734647
  node_ip: 127.0.0.1
  pid: 45454
  time_since_restore: 0.1037449836730957
  time_this_iter_s: 0.1037449836730957
  time_total_s: 0.6844677925109863
  timestamp: 1658499089
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 278eba4c
  warmup_time: 0.006754875183105469

(objective pid=45454) 2022-07-22 15:11:29,879	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_278eba4c_6_activation=relu,height=-27.0179,steps=100,width=13.5770_2022-07-22_15-11-18/checkpoint_tmp2835d4
(objective pid=45454) 2022-07-22 15:11:29,879	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.5807228088378906, '_episodes_total': 0}
(objective pid=45455) 2022-07-22 15:11:29,909	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_27b7e2be_9_activation=relu,height=-51.2093,steps=100,width=8.9495_2022-07-22_15-11-18/checkpoint_tmpd7ea63
(objective pid=45455) 2022-07-22 15:11:29,910	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.9150340557098389, '_episodes_total': 0}
(objective pid=45453) 2022-07-22 15:11:29,930	INFO trainable.py:655 -- Restored on 127.0.0.1 from checkpoint: /Users/kai/ray_results/bohb_exp_2/objective_25b64488_3_activation=tanh,height=-48.4518,steps=100,width=10.1191_2022-07-22_15-11-14/checkpoint_tmp11824e
(objective pid=45453) 2022-07-22 15:11:29,930	INFO trainable.py:663 -- Current state after restoring: {'_iteration': 0, '_timesteps_total': 0, '_time_total': 0.5960800647735596, '_episodes_total': 0}

Result for objective_27b7e2be:
  date: 2022-07-22_15-11-30
  done: false
  episodes_total: 0
  experiment_id: fdc43ca37ed44cde857ca150a8f1e84f
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.879072389639937
  neg_mean_loss: -14.879072389639937
  node_ip: 127.0.0.1
  pid: 45455
  time_since_restore: 0.10332393646240234
  time_this_iter_s: 0.10332393646240234
  time_total_s: 1.0183579921722412
  timestamp: 1658499090
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 27b7e2be
  warmup_time: 0.006285190582275391
  
Result for objective_25b64488:
  date: 2022-07-22_15-11-30
  done: false
  episodes_total: 0
  experiment_id: 75e7c1ad20a2495cac29630df6c3c782
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 6.154820228591976
  neg_mean_loss: -6.154820228591976
  node_ip: 127.0.0.1
  pid: 45453
  time_since_restore: 0.1026160717010498
  time_this_iter_s: 0.1026160717010498
  time_total_s: 0.6986961364746094
  timestamp: 1658499090
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 25b64488
  warmup_time: 0.006460905075073242
  
Result for objective_25b64488:
  date: 2022-07-22_15-11-35
  done: false
  episodes_total: 0
  experiment_id: 75e7c1ad20a2495cac29630df6c3c782
  hostname: Kais-MacBook-Pro.local
  iterations: 46
  iterations_since_restore: 47
  mean_loss: -3.634865890857194
  neg_mean_loss: 3.634865890857194
  node_ip: 127.0.0.1
  pid: 45453
  time_since_restore: 5.1935131549835205
  time_this_iter_s: 0.10786604881286621
  time_total_s: 5.78959321975708
  timestamp: 1658499095
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 47
  trial_id: 25b64488
  warmup_time: 0.006460905075073242
  
Result for objective_25b64488:
  date: 2022-07-22_15-11-40
  done: false
  episodes_total: 0
  experiment_id: 75e7c1ad20a2495cac29630df6c3c782
  hostname: Kais-MacBook-Pro.local
  iterations: 93
  iterations_since_restore: 94
  mean_loss: -3.740036002402735
  neg_mean_loss: 3.740036002402735
  node_ip: 127.0.0.1
  pid: 45453
  time_since_restore: 10.256795167922974
  time_this_iter_s: 0.10682511329650879
  time_total_s: 10.852875232696533
  timestamp: 1658499100
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 94
  trial_id: 25b64488
  warmup_time: 0.006460905075073242
  
Result for objective_25b64488:
  date: 2022-07-22_15-11-40
  done: true
  episodes_total: 0
  experiment_id: 75e7c1ad20a2495cac29630df6c3c782
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -3.74634537130406
  neg_mean_loss: 3.74634537130406
  node_ip: 127.0.0.1
  pid: 45453
  time_since_restore: 10.935801029205322
  time_this_iter_s: 0.10489487648010254
  time_total_s: 11.531881093978882
  timestamp: 1658499100
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 100
  trial_id: 25b64488
  warmup_time: 0.006460905075073242

Here again are the hyperparameters found to minimize the mean loss of the defined objective.

In [18]:

print("Best hyperparameters found were: ", results.get_best_result().config)

Best hyperparameters found were:  {'activation': 'tanh', 'height': -48.451797714080236, 'steps': 100, 'width': 10.119125894538891}

In [19]:

ray.shutdown()