In this tutorial we introduce Optuna, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Optuna and, as a result, allow you to seamlessly scale up a Optuna optimization process - without sacrificing performance.
Similar to Ray Tune, Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative ("how" over "what" emphasis), define-by-run style user API. With Optuna, a user has the ability to dynamically construct the search spaces for the hyperparameters. Optuna falls in the domain of "derivative-free optimization" and "black-box optimization".
In this example we minimize a simple objective to briefly demonstrate the usage of Optuna with Ray Tune via OptunaSearch, including examples of conditional search spaces (string together relationships between hyperparameters), and the multi-objective problem (measure trade-offs among all important metrics). It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume optuna==2.9.1 library is installed. To learn more, please refer to Optuna website.
Please note that sophisticated schedulers, such as AsyncHyperBandScheduler, may not work correctly with multi-objective optimization, since they typically expect a scalar score to compare fitness among trials.
# !pip install ray[tune]
!pip install optuna==2.9.1
Requirement already satisfied: optuna==2.9.1 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (2.9.1) Requirement already satisfied: alembic in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.7.6) Requirement already satisfied: colorlog in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (6.6.0) Requirement already satisfied: PyYAML in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (5.4.1) Requirement already satisfied: scipy!=1.4.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.4.1) Requirement already satisfied: tqdm in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (4.64.0) Requirement already satisfied: numpy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.21.6) Requirement already satisfied: cliff in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (3.10.1) Requirement already satisfied: sqlalchemy>=1.1.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.4.32) Requirement already satisfied: packaging>=20.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (21.3) Requirement already satisfied: cmaes>=0.8.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (0.8.2) Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from packaging>=20.0->optuna==2.9.1) (2.4.7) Requirement already satisfied: importlib-metadata in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from sqlalchemy>=1.1.0->optuna==2.9.1) (4.10.1) Requirement already satisfied: greenlet!=0.4.17 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from sqlalchemy>=1.1.0->optuna==2.9.1) (1.1.2) Requirement already satisfied: importlib-resources in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from alembic->optuna==2.9.1) (5.4.0) Requirement already satisfied: Mako in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from alembic->optuna==2.9.1) (1.1.6) Requirement already satisfied: autopage>=0.4.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (0.5.0) Requirement already satisfied: cmd2>=1.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (2.4.0) Requirement already satisfied: PrettyTable>=0.7.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (3.2.0) Requirement already satisfied: stevedore>=2.0.1 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (3.5.0) Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (5.8.1) Requirement already satisfied: typing-extensions in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (4.1.1) Requirement already satisfied: attrs>=16.3.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (21.2.0) Requirement already satisfied: pyperclip>=1.6 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (1.8.2) Requirement already satisfied: wcwidth>=0.1.7 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (0.2.5) Requirement already satisfied: zipp>=0.5 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from importlib-metadata->sqlalchemy>=1.1.0->optuna==2.9.1) (3.7.0) Requirement already satisfied: MarkupSafe>=0.9.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from Mako->alembic->optuna==2.9.1) (2.0.1) WARNING: There was an error checking the latest version of pip.
Click below to see all the imports we need for this example. You can also launch directly into a Binder instance to run this notebook yourself. Just click on the rocket symbol at the top of the navigation.
import time
from typing import Dict, Optional, Any
import ray
from ray import train, tune
from ray.tune.search import ConcurrencyLimiter
from ray.tune.search.optuna import OptunaSearch
ray.init(configure_logging=False)
| Python version: | 3.7.7 |
| Ray version: | 3.0.0.dev0 |
| Dashboard: | http://127.0.0.1:8265 |
Let's start by defining a simple evaluation function.
An explicit math formula is queried here for demonstration, yet in practice this is typically a black-box function-- e.g. the performance results after training an ML model.
We artificially sleep for a bit (0.1 seconds) to simulate a long-running ML experiment.
This setup assumes that we're running multiple steps of an experiment while tuning three hyperparameters,
namely width, height, and activation.
def evaluate(step, width, height, activation):
time.sleep(0.1)
activation_boost = 10 if activation=="relu" else 0
return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost
Next, our objective function to be optimized takes a Tune config, evaluates the score of your experiment in a training loop,
and uses train.report to report the score back to Tune.
def objective(config):
for step in range(config["steps"]):
score = evaluate(step, config["width"], config["height"], config["activation"])
train.report({"iterations": step, "mean_loss": score})
Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparamters may be difficult to find in a short amount of time.
The simplest case is a search space with independent dimensions. In this case, a config dictionary will suffice.
search_space = {
"steps": 100,
"width": tune.uniform(0, 20),
"height": tune.uniform(-100, 100),
"activation": tune.choice(["relu", "tanh"]),
}
Here we define the Optuna search algorithm:
algo = OptunaSearch()
We also constrain the number of concurrent trials to 4 with a ConcurrencyLimiter.
algo = ConcurrencyLimiter(algo, max_concurrent=4)
The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to 1000 samples.
(you can decrease this if it takes too long on your machine).
num_samples = 1000
# We override here for our smoke tests.
num_samples = 10
Finally, we run the experiment to "min"imize the "mean_loss" of the objective by searching search_space via algo, num_samples times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute tuner.fit().
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
param_space=search_space,
)
results = tuner.fit()
Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.
[I 2022-07-22 15:21:47,769] A new study created in memory with name: optuna
| Trial name | status | loc | activation | height | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|
| objective_9f689668 | TERMINATED | 127.0.0.1:46141 | tanh | 32.9806 | 16.0406 | 3.36064 | 100 | 11.0409 | 99 | -3.36064 |
| objective_a11d2104 | TERMINATED | 127.0.0.1:46149 | relu | 72.2627 | 0.763881 | 18.3942 | 100 | 11.7018 | 99 | -18.3942 |
| objective_a11eef8e | TERMINATED | 127.0.0.1:46150 | relu | -32.8474 | 13.1144 | 6.79169 | 100 | 11.8071 | 99 | -6.79169 |
| objective_a1209e56 | TERMINATED | 127.0.0.1:46151 | tanh | 75.6408 | 10.4415 | 7.65989 | 100 | 11.6987 | 99 | -7.65989 |
| objective_a7b383e6 | TERMINATED | 127.0.0.1:46174 | relu | 50.1501 | 3.4612 | 15.2986 | 100 | 10.7015 | 99 | -15.2986 |
| objective_a9c844c8 | TERMINATED | 127.0.0.1:46183 | relu | -40.3931 | 14.0525 | 6.03205 | 100 | 10.6687 | 99 | -6.03205 |
| objective_a9cb308e | TERMINATED | 127.0.0.1:46184 | tanh | 48.4802 | 12.7746 | 4.92647 | 100 | 10.7122 | 99 | -4.92647 |
| objective_a9d8332e | TERMINATED | 127.0.0.1:46189 | tanh | -1.26682 | 10.9788 | -0.0355157 | 100 | 10.7089 | 99 | 0.0355157 |
| objective_af948a1a | TERMINATED | 127.0.0.1:46204 | tanh | 61.6381 | 11.0105 | 6.25472 | 100 | 10.744 | 99 | -6.25472 |
| objective_b1bad2fe | TERMINATED | 127.0.0.1:46209 | tanh | -16.4629 | 7.68847 | -1.51661 | 100 | 10.7274 | 99 | 1.51661 |
Result for objective_9f689668: date: 2022-07-22_15-21-51 done: false experiment_id: 7d29e5c95aa44d77becc24b23a326b9b hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 13.298063134872326 neg_mean_loss: -13.298063134872326 node_ip: 127.0.0.1 pid: 46141 time_since_restore: 0.10468196868896484 time_this_iter_s: 0.10468196868896484 time_total_s: 0.10468196868896484 timestamp: 1658499711 timesteps_since_restore: 0 training_iteration: 1 trial_id: 9f689668 warmup_time: 0.003930091857910156 Result for objective_a1209e56: date: 2022-07-22_15-21-54 done: false experiment_id: 70d9788c327c43fb94450e5084416b81 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 17.564083106485064 neg_mean_loss: -17.564083106485064 node_ip: 127.0.0.1 pid: 46151 time_since_restore: 0.10416483879089355 time_this_iter_s: 0.10416483879089355 time_total_s: 0.10416483879089355 timestamp: 1658499714 timesteps_since_restore: 0 training_iteration: 1 trial_id: a1209e56 warmup_time: 0.0026938915252685547 Result for objective_a11d2104: date: 2022-07-22_15-21-54 done: false experiment_id: af2598364a004cd28f37ede19afd1ffb hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 27.2262748487915 neg_mean_loss: -27.2262748487915 node_ip: 127.0.0.1 pid: 46149 time_since_restore: 0.10507392883300781 time_this_iter_s: 0.10507392883300781 time_total_s: 0.10507392883300781 timestamp: 1658499714 timesteps_since_restore: 0 training_iteration: 1 trial_id: a11d2104 warmup_time: 0.0031232833862304688 Result for objective_a11eef8e: date: 2022-07-22_15-21-54 done: false experiment_id: e4a5922603d44b459c23607a42d5e574 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 16.71525618736072 neg_mean_loss: -16.71525618736072 node_ip: 127.0.0.1 pid: 46150 time_since_restore: 0.10363006591796875 time_this_iter_s: 0.10363006591796875 time_total_s: 0.10363006591796875 timestamp: 1658499714 timesteps_since_restore: 0 training_iteration: 1 trial_id: a11eef8e warmup_time: 0.0028891563415527344 Result for objective_9f689668: date: 2022-07-22_15-21-56 done: false experiment_id: 7d29e5c95aa44d77becc24b23a326b9b hostname: Kais-MacBook-Pro.local iterations: 45 iterations_since_restore: 46 mean_loss: 3.434707064844201 neg_mean_loss: -3.434707064844201 node_ip: 127.0.0.1 pid: 46141 time_since_restore: 5.202661752700806 time_this_iter_s: 0.10527586936950684 time_total_s: 5.202661752700806 timestamp: 1658499716 timesteps_since_restore: 0 training_iteration: 46 trial_id: 9f689668 warmup_time: 0.003930091857910156 Result for objective_a1209e56: date: 2022-07-22_15-21-59 done: false experiment_id: 70d9788c327c43fb94450e5084416b81 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 7.763782433167051 neg_mean_loss: -7.763782433167051 node_ip: 127.0.0.1 pid: 46151 time_since_restore: 5.146381139755249 time_this_iter_s: 0.10691022872924805 time_total_s: 5.146381139755249 timestamp: 1658499719 timesteps_since_restore: 0 training_iteration: 48 trial_id: a1209e56 warmup_time: 0.0026938915252685547 Result for objective_a11eef8e: date: 2022-07-22_15-21-59 done: false experiment_id: e4a5922603d44b459c23607a42d5e574 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 6.874905099790281 neg_mean_loss: -6.874905099790281 node_ip: 127.0.0.1 pid: 46150 time_since_restore: 5.1618921756744385 time_this_iter_s: 0.10776615142822266 time_total_s: 5.1618921756744385 timestamp: 1658499719 timesteps_since_restore: 0 training_iteration: 48 trial_id: a11eef8e warmup_time: 0.0028891563415527344 Result for objective_a11d2104: date: 2022-07-22_15-21-59 done: false experiment_id: af2598364a004cd28f37ede19afd1ffb hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 19.4048102011043 neg_mean_loss: -19.4048102011043 node_ip: 127.0.0.1 pid: 46149 time_since_restore: 5.170748233795166 time_this_iter_s: 0.11556220054626465 time_total_s: 5.170748233795166 timestamp: 1658499719 timesteps_since_restore: 0 training_iteration: 48 trial_id: a11d2104 warmup_time: 0.0031232833862304688 Result for objective_9f689668: date: 2022-07-22_15-22-01 done: false experiment_id: 7d29e5c95aa44d77becc24b23a326b9b hostname: Kais-MacBook-Pro.local iterations: 92 iterations_since_restore: 93 mean_loss: 3.3653696881211697 neg_mean_loss: -3.3653696881211697 node_ip: 127.0.0.1 pid: 46141 time_since_restore: 10.248206853866577 time_this_iter_s: 0.10745501518249512 time_total_s: 10.248206853866577 timestamp: 1658499721 timesteps_since_restore: 0 training_iteration: 93 trial_id: 9f689668 warmup_time: 0.003930091857910156 Result for objective_9f689668: date: 2022-07-22_15-22-02 done: true experiment_id: 7d29e5c95aa44d77becc24b23a326b9b experiment_tag: 1_activation=tanh,height=32.9806,steps=100,width=16.0406 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 3.3606404197394255 neg_mean_loss: -3.3606404197394255 node_ip: 127.0.0.1 pid: 46141 time_since_restore: 11.04089903831482 time_this_iter_s: 0.10824704170227051 time_total_s: 11.04089903831482 timestamp: 1658499722 timesteps_since_restore: 0 training_iteration: 100 trial_id: 9f689668 warmup_time: 0.003930091857910156 Result for objective_a1209e56: date: 2022-07-22_15-22-04 done: false experiment_id: 70d9788c327c43fb94450e5084416b81 hostname: Kais-MacBook-Pro.local iterations: 90 iterations_since_restore: 91 mean_loss: 7.669375143981996 neg_mean_loss: -7.669375143981996 node_ip: 127.0.0.1 pid: 46151 time_since_restore: 9.724387168884277 time_this_iter_s: 0.10670638084411621 time_total_s: 9.724387168884277 timestamp: 1658499724 timesteps_since_restore: 0 training_iteration: 91 trial_id: a1209e56 warmup_time: 0.0026938915252685547 Result for objective_a11d2104: date: 2022-07-22_15-22-04 done: false experiment_id: af2598364a004cd28f37ede19afd1ffb hostname: Kais-MacBook-Pro.local iterations: 90 iterations_since_restore: 91 mean_loss: 18.49612779999574 neg_mean_loss: -18.49612779999574 node_ip: 127.0.0.1 pid: 46149 time_since_restore: 9.818917274475098 time_this_iter_s: 0.10725140571594238 time_total_s: 9.818917274475098 timestamp: 1658499724 timesteps_since_restore: 0 training_iteration: 91 trial_id: a11d2104 warmup_time: 0.0031232833862304688 Result for objective_a11eef8e: date: 2022-07-22_15-22-03 done: false experiment_id: e4a5922603d44b459c23607a42d5e574 hostname: Kais-MacBook-Pro.local iterations: 89 iterations_since_restore: 90 mean_loss: 6.800205168670805 neg_mean_loss: -6.800205168670805 node_ip: 127.0.0.1 pid: 46150 time_since_restore: 9.714852094650269 time_this_iter_s: 0.10286402702331543 time_total_s: 9.714852094650269 timestamp: 1658499723 timesteps_since_restore: 0 training_iteration: 90 trial_id: a11eef8e warmup_time: 0.0028891563415527344 Result for objective_a7b383e6: date: 2022-07-22_15-22-05 done: false experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 25.015010793208496 neg_mean_loss: -25.015010793208496 node_ip: 127.0.0.1 pid: 46174 time_since_restore: 0.10384607315063477 time_this_iter_s: 0.10384607315063477 time_total_s: 0.10384607315063477 timestamp: 1658499725 timesteps_since_restore: 0 training_iteration: 1 trial_id: a7b383e6 warmup_time: 0.002817869186401367 Result for objective_a1209e56: date: 2022-07-22_15-22-05 done: true experiment_id: 70d9788c327c43fb94450e5084416b81 experiment_tag: 4_activation=tanh,height=75.6408,steps=100,width=10.4415 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 7.659894851608276 neg_mean_loss: -7.659894851608276 node_ip: 127.0.0.1 pid: 46151 time_since_restore: 11.698678970336914 time_this_iter_s: 0.10874581336975098 time_total_s: 11.698678970336914 timestamp: 1658499725 timesteps_since_restore: 0 training_iteration: 100 trial_id: a1209e56 warmup_time: 0.0026938915252685547 Result for objective_a11d2104: date: 2022-07-22_15-22-05 done: true experiment_id: af2598364a004cd28f37ede19afd1ffb experiment_tag: 2_activation=relu,height=72.2627,steps=100,width=0.7639 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 18.394168929385685 neg_mean_loss: -18.394168929385685 node_ip: 127.0.0.1 pid: 46149 time_since_restore: 11.701792001724243 time_this_iter_s: 0.10734391212463379 time_total_s: 11.701792001724243 timestamp: 1658499725 timesteps_since_restore: 0 training_iteration: 100 trial_id: a11d2104 warmup_time: 0.0031232833862304688 Result for objective_a11eef8e: date: 2022-07-22_15-22-06 done: true experiment_id: e4a5922603d44b459c23607a42d5e574 experiment_tag: 3_activation=relu,height=-32.8474,steps=100,width=13.1144 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 6.791690049131352 neg_mean_loss: -6.791690049131352 node_ip: 127.0.0.1 pid: 46150 time_since_restore: 11.807135105133057 time_this_iter_s: 0.10672497749328613 time_total_s: 11.807135105133057 timestamp: 1658499726 timesteps_since_restore: 0 training_iteration: 100 trial_id: a11eef8e warmup_time: 0.0028891563415527344 Result for objective_a9cb308e: date: 2022-07-22_15-22-08 done: false experiment_id: fcc8b8f4880c471c8e8545228d7e591a hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 14.84802392036395 neg_mean_loss: -14.84802392036395 node_ip: 127.0.0.1 pid: 46184 time_since_restore: 0.1039888858795166 time_this_iter_s: 0.1039888858795166 time_total_s: 0.1039888858795166 timestamp: 1658499728 timesteps_since_restore: 0 training_iteration: 1 trial_id: a9cb308e warmup_time: 0.0027310848236083984 Result for objective_a9c844c8: date: 2022-07-22_15-22-08 done: false experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 15.960686077942514 neg_mean_loss: -15.960686077942514 node_ip: 127.0.0.1 pid: 46183 time_since_restore: 0.1004641056060791 time_this_iter_s: 0.1004641056060791 time_total_s: 0.1004641056060791 timestamp: 1658499728 timesteps_since_restore: 0 training_iteration: 1 trial_id: a9c844c8 warmup_time: 0.002599954605102539 Result for objective_a9d8332e: date: 2022-07-22_15-22-08 done: false experiment_id: 9c2f8aa7e4ed45c8867025673b146195 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 9.873318211243287 neg_mean_loss: -9.873318211243287 node_ip: 127.0.0.1 pid: 46189 time_since_restore: 0.10463905334472656 time_this_iter_s: 0.10463905334472656 time_total_s: 0.10463905334472656 timestamp: 1658499728 timesteps_since_restore: 0 training_iteration: 1 trial_id: a9d8332e warmup_time: 0.002753019332885742 Result for objective_a7b383e6: date: 2022-07-22_15-22-10 done: false experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 15.594128340598196 neg_mean_loss: -15.594128340598196 node_ip: 127.0.0.1 pid: 46174 time_since_restore: 5.12034797668457 time_this_iter_s: 0.10577988624572754 time_total_s: 5.12034797668457 timestamp: 1658499730 timesteps_since_restore: 0 training_iteration: 48 trial_id: a7b383e6 warmup_time: 0.002817869186401367 Result for objective_a9cb308e: date: 2022-07-22_15-22-13 done: false experiment_id: fcc8b8f4880c471c8e8545228d7e591a hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 5.011849709111575 neg_mean_loss: -5.011849709111575 node_ip: 127.0.0.1 pid: 46184 time_since_restore: 5.150436162948608 time_this_iter_s: 0.10827016830444336 time_total_s: 5.150436162948608 timestamp: 1658499733 timesteps_since_restore: 0 training_iteration: 48 trial_id: a9cb308e warmup_time: 0.0027310848236083984 Result for objective_a9c844c8: date: 2022-07-22_15-22-13 done: false experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 6.109835222238134 neg_mean_loss: -6.109835222238134 node_ip: 127.0.0.1 pid: 46183 time_since_restore: 5.136260032653809 time_this_iter_s: 0.10730218887329102 time_total_s: 5.136260032653809 timestamp: 1658499733 timesteps_since_restore: 0 training_iteration: 48 trial_id: a9c844c8 warmup_time: 0.002599954605102539 Result for objective_a9d8332e: date: 2022-07-22_15-22-13 done: false experiment_id: 9c2f8aa7e4ed45c8867025673b146195 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 0.06343129971586822 neg_mean_loss: -0.06343129971586822 node_ip: 127.0.0.1 pid: 46189 time_since_restore: 5.141700983047485 time_this_iter_s: 0.1074991226196289 time_total_s: 5.141700983047485 timestamp: 1658499733 timesteps_since_restore: 0 training_iteration: 48 trial_id: a9d8332e warmup_time: 0.002753019332885742 Result for objective_a7b383e6: date: 2022-07-22_15-22-15 done: false experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 15.313204013214245 neg_mean_loss: -15.313204013214245 node_ip: 127.0.0.1 pid: 46174 time_since_restore: 10.163710832595825 time_this_iter_s: 0.10754895210266113 time_total_s: 10.163710832595825 timestamp: 1658499735 timesteps_since_restore: 0 training_iteration: 95 trial_id: a7b383e6 warmup_time: 0.002817869186401367 Result for objective_a7b383e6: date: 2022-07-22_15-22-15 done: true experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa experiment_tag: 5_activation=relu,height=50.1501,steps=100,width=3.4612 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 15.298570798421121 neg_mean_loss: -15.298570798421121 node_ip: 127.0.0.1 pid: 46174 time_since_restore: 10.701509952545166 time_this_iter_s: 0.10684585571289062 time_total_s: 10.701509952545166 timestamp: 1658499735 timesteps_since_restore: 0 training_iteration: 100 trial_id: a7b383e6 warmup_time: 0.002817869186401367 Result for objective_af948a1a: date: 2022-07-22_15-22-18 done: false experiment_id: d27f58d6527a4d47ae4c12496f5275cf hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 16.16381214127525 neg_mean_loss: -16.16381214127525 node_ip: 127.0.0.1 pid: 46204 time_since_restore: 0.10407209396362305 time_this_iter_s: 0.10407209396362305 time_total_s: 0.10407209396362305 timestamp: 1658499738 timesteps_since_restore: 0 training_iteration: 1 trial_id: af948a1a warmup_time: 0.002669095993041992 Result for objective_a9cb308e: date: 2022-07-22_15-22-18 done: false experiment_id: fcc8b8f4880c471c8e8545228d7e591a hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 4.930613328484136 neg_mean_loss: -4.930613328484136 node_ip: 127.0.0.1 pid: 46184 time_since_restore: 10.175731897354126 time_this_iter_s: 0.10618400573730469 time_total_s: 10.175731897354126 timestamp: 1658499738 timesteps_since_restore: 0 training_iteration: 95 trial_id: a9cb308e warmup_time: 0.0027310848236083984 Result for objective_a9d8332e: date: 2022-07-22_15-22-18 done: false experiment_id: 9c2f8aa7e4ed45c8867025673b146195 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: -0.030712998363490673 neg_mean_loss: 0.030712998363490673 node_ip: 127.0.0.1 pid: 46189 time_since_restore: 10.171820163726807 time_this_iter_s: 0.10807609558105469 time_total_s: 10.171820163726807 timestamp: 1658499738 timesteps_since_restore: 0 training_iteration: 95 trial_id: a9d8332e warmup_time: 0.002753019332885742 Result for objective_a9c844c8: date: 2022-07-22_15-22-18 done: false experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e hostname: Kais-MacBook-Pro.local iterations: 95 iterations_since_restore: 96 mean_loss: 6.035035952030598 neg_mean_loss: -6.035035952030598 node_ip: 127.0.0.1 pid: 46183 time_since_restore: 10.23954701423645 time_this_iter_s: 0.10863399505615234 time_total_s: 10.23954701423645 timestamp: 1658499738 timesteps_since_restore: 0 training_iteration: 96 trial_id: a9c844c8 warmup_time: 0.002599954605102539 Result for objective_a9cb308e: date: 2022-07-22_15-22-19 done: true experiment_id: fcc8b8f4880c471c8e8545228d7e591a experiment_tag: 7_activation=tanh,height=48.4802,steps=100,width=12.7746 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 4.926474869576359 neg_mean_loss: -4.926474869576359 node_ip: 127.0.0.1 pid: 46184 time_since_restore: 10.712197065353394 time_this_iter_s: 0.10726809501647949 time_total_s: 10.712197065353394 timestamp: 1658499739 timesteps_since_restore: 0 training_iteration: 100 trial_id: a9cb308e warmup_time: 0.0027310848236083984 Result for objective_a9c844c8: date: 2022-07-22_15-22-19 done: true experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e experiment_tag: 6_activation=relu,height=-40.3931,steps=100,width=14.0525 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 6.03205335569619 neg_mean_loss: -6.03205335569619 node_ip: 127.0.0.1 pid: 46183 time_since_restore: 10.668684959411621 time_this_iter_s: 0.1074681282043457 time_total_s: 10.668684959411621 timestamp: 1658499739 timesteps_since_restore: 0 training_iteration: 100 trial_id: a9c844c8 warmup_time: 0.002599954605102539 Result for objective_a9d8332e: date: 2022-07-22_15-22-19 done: true experiment_id: 9c2f8aa7e4ed45c8867025673b146195 experiment_tag: 8_activation=tanh,height=-1.2668,steps=100,width=10.9788 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -0.035515719608698634 neg_mean_loss: 0.035515719608698634 node_ip: 127.0.0.1 pid: 46189 time_since_restore: 10.708936929702759 time_this_iter_s: 0.10948991775512695 time_total_s: 10.708936929702759 timestamp: 1658499739 timesteps_since_restore: 0 training_iteration: 100 trial_id: a9d8332e warmup_time: 0.002753019332885742 Result for objective_b1bad2fe: date: 2022-07-22_15-22-21 done: false experiment_id: f58c8e48e2324dd69c28f526bf96497c hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 8.353710439145445 neg_mean_loss: -8.353710439145445 node_ip: 127.0.0.1 pid: 46209 time_since_restore: 0.10369992256164551 time_this_iter_s: 0.10369992256164551 time_total_s: 0.10369992256164551 timestamp: 1658499741 timesteps_since_restore: 0 training_iteration: 1 trial_id: b1bad2fe warmup_time: 0.0026597976684570312 Result for objective_af948a1a: date: 2022-07-22_15-22-23 done: false experiment_id: d27f58d6527a4d47ae4c12496f5275cf hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 6.353387644558537 neg_mean_loss: -6.353387644558537 node_ip: 127.0.0.1 pid: 46204 time_since_restore: 5.161527872085571 time_this_iter_s: 0.10784792900085449 time_total_s: 5.161527872085571 timestamp: 1658499743 timesteps_since_restore: 0 training_iteration: 48 trial_id: af948a1a warmup_time: 0.002669095993041992 Result for objective_b1bad2fe: date: 2022-07-22_15-22-26 done: false experiment_id: f58c8e48e2324dd69c28f526bf96497c hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -1.3770075531258914 neg_mean_loss: 1.3770075531258914 node_ip: 127.0.0.1 pid: 46209 time_since_restore: 5.158898115158081 time_this_iter_s: 0.10701417922973633 time_total_s: 5.158898115158081 timestamp: 1658499746 timesteps_since_restore: 0 training_iteration: 48 trial_id: b1bad2fe warmup_time: 0.0026597976684570312 Result for objective_af948a1a: date: 2022-07-22_15-22-28 done: false experiment_id: d27f58d6527a4d47ae4c12496f5275cf hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 6.259506962612728 neg_mean_loss: -6.259506962612728 node_ip: 127.0.0.1 pid: 46204 time_since_restore: 10.20361876487732 time_this_iter_s: 0.10824179649353027 time_total_s: 10.20361876487732 timestamp: 1658499748 timesteps_since_restore: 0 training_iteration: 95 trial_id: af948a1a warmup_time: 0.002669095993041992 Result for objective_af948a1a: date: 2022-07-22_15-22-28 done: true experiment_id: d27f58d6527a4d47ae4c12496f5275cf experiment_tag: 9_activation=tanh,height=61.6381,steps=100,width=11.0105 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 6.254717826198094 neg_mean_loss: -6.254717826198094 node_ip: 127.0.0.1 pid: 46204 time_since_restore: 10.744001865386963 time_this_iter_s: 0.10536718368530273 time_total_s: 10.744001865386963 timestamp: 1658499748 timesteps_since_restore: 0 training_iteration: 100 trial_id: af948a1a warmup_time: 0.002669095993041992 Result for objective_b1bad2fe: date: 2022-07-22_15-22-31 done: false experiment_id: f58c8e48e2324dd69c28f526bf96497c hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: -1.5098109958909571 neg_mean_loss: 1.5098109958909571 node_ip: 127.0.0.1 pid: 46209 time_since_restore: 10.190011024475098 time_this_iter_s: 0.10639405250549316 time_total_s: 10.190011024475098 timestamp: 1658499751 timesteps_since_restore: 0 training_iteration: 95 trial_id: b1bad2fe warmup_time: 0.0026597976684570312 Result for objective_b1bad2fe: date: 2022-07-22_15-22-32 done: true experiment_id: f58c8e48e2324dd69c28f526bf96497c experiment_tag: 10_activation=tanh,height=-16.4629,steps=100,width=7.6885 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -1.5166144695235813 neg_mean_loss: 1.5166144695235813 node_ip: 127.0.0.1 pid: 46209 time_since_restore: 10.727396011352539 time_this_iter_s: 0.10742807388305664 time_total_s: 10.727396011352539 timestamp: 1658499752 timesteps_since_restore: 0 training_iteration: 100 trial_id: b1bad2fe warmup_time: 0.0026597976684570312
And now we have the hyperparameters found to minimize the mean loss.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 7.688465886501621, 'height': -16.46289560854555, 'activation': 'tanh'}
While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and
pass this information as a helpful starting point for the OptunaSearch object.
initial_params = [
{"width": 1, "height": 2, "activation": "relu"},
{"width": 4, "height": 2, "activation": "relu"},
]
Now the search_alg built using OptunaSearch takes points_to_evaluate.
searcher = OptunaSearch(points_to_evaluate=initial_params)
algo = ConcurrencyLimiter(searcher, max_concurrent=4)
And run the experiment with initial hyperparameter evaluations:
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
param_space=search_space,
)
results = tuner.fit()
[I 2022-07-22 15:22:32,644] A new study created in memory with name: optuna
/Users/kai/coding/ray/python/ray/tune/search/optuna/optuna_search.py:389: ExperimentalWarning: enqueue_trial is experimental (supported from v1.2.0). The interface can change in the future.
self._ot_study.enqueue_trial(point)
/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/optuna/study/study.py:857: ExperimentalWarning: create_trial is experimental (supported from v2.0.0). The interface can change in the future.
create_trial(state=TrialState.WAITING, system_attrs={"fixed_params": params})
/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/optuna/study/study.py:857: ExperimentalWarning: add_trial is experimental (supported from v2.0.0). The interface can change in the future.
create_trial(state=TrialState.WAITING, system_attrs={"fixed_params": params})
| Trial name | status | loc | activation | height | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|
| objective_b9acff32 | TERMINATED | 127.0.0.1:46220 | relu | 2 | 1 | 11.1174 | 100 | 10.6837 | 99 | -11.1174 |
| objective_bb3b2702 | TERMINATED | 127.0.0.1:46227 | relu | 2 | 4 | 10.4463 | 100 | 11.2816 | 99 | -10.4463 |
| objective_bb3c8714 | TERMINATED | 127.0.0.1:46228 | relu | -90.7016 | 0.566597 | 2.44285 | 100 | 11.2766 | 99 | -2.44285 |
| objective_bb3e0210 | TERMINATED | 127.0.0.1:46229 | tanh | -81.4501 | 14.783 | -8.07715 | 100 | 11.2196 | 99 | 8.07715 |
| objective_c19b9dde | TERMINATED | 127.0.0.1:46248 | relu | 86.466 | 0.308659 | 21.1122 | 100 | 10.7226 | 99 | -21.1122 |
| objective_c3762426 | TERMINATED | 127.0.0.1:46253 | tanh | -98.4632 | 0.723671 | -8.62148 | 100 | 10.7271 | 99 | 8.62148 |
| objective_c385528e | TERMINATED | 127.0.0.1:46256 | relu | -54.8368 | 11.0765 | 4.60669 | 100 | 10.7498 | 99 | -4.60669 |
| objective_c387a7c8 | TERMINATED | 127.0.0.1:46257 | tanh | 42.9466 | 8.01708 | 4.41909 | 100 | 10.7214 | 99 | -4.41909 |
| objective_c97678f8 | TERMINATED | 127.0.0.1:46272 | tanh | 9.25214 | 15.5288 | 0.989841 | 100 | 12.4591 | 99 | -0.989841 |
| objective_cb6180fe | TERMINATED | 127.0.0.1:46277 | relu | 41.6647 | 8.0585 | 14.2903 | 100 | 10.752 | 99 | -14.2903 |
Result for objective_b9acff32: date: 2022-07-22_15-22-35 done: false experiment_id: ec3c260f0ad348fa9b188f9c93c01eba hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 20.2 neg_mean_loss: -20.2 node_ip: 127.0.0.1 pid: 46220 time_since_restore: 0.10049605369567871 time_this_iter_s: 0.10049605369567871 time_total_s: 0.10049605369567871 timestamp: 1658499755 timesteps_since_restore: 0 training_iteration: 1 trial_id: b9acff32 warmup_time: 0.0029239654541015625 Result for objective_bb3e0210: date: 2022-07-22_15-22-37 done: false experiment_id: 08a7bc47c140405296e58a049cd2f6b4 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 1.8549886153795079 neg_mean_loss: -1.8549886153795079 node_ip: 127.0.0.1 pid: 46229 time_since_restore: 0.10361814498901367 time_this_iter_s: 0.10361814498901367 time_total_s: 0.10361814498901367 timestamp: 1658499757 timesteps_since_restore: 0 training_iteration: 1 trial_id: bb3e0210 warmup_time: 0.002833843231201172 Result for objective_bb3b2702: date: 2022-07-22_15-22-37 done: false experiment_id: 2f984120624b4e9aa3392fcc1b16b811 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 20.2 neg_mean_loss: -20.2 node_ip: 127.0.0.1 pid: 46227 time_since_restore: 0.10420107841491699 time_this_iter_s: 0.10420107841491699 time_total_s: 0.10420107841491699 timestamp: 1658499757 timesteps_since_restore: 0 training_iteration: 1 trial_id: bb3b2702 warmup_time: 0.002672910690307617 Result for objective_bb3c8714: date: 2022-07-22_15-22-37 done: false experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 10.929835533284928 neg_mean_loss: -10.929835533284928 node_ip: 127.0.0.1 pid: 46228 time_since_restore: 0.1048588752746582 time_this_iter_s: 0.1048588752746582 time_total_s: 0.1048588752746582 timestamp: 1658499757 timesteps_since_restore: 0 training_iteration: 1 trial_id: bb3c8714 warmup_time: 0.0027511119842529297 Result for objective_b9acff32: date: 2022-07-22_15-22-40 done: false experiment_id: ec3c260f0ad348fa9b188f9c93c01eba hostname: Kais-MacBook-Pro.local iterations: 48 iterations_since_restore: 49 mean_loss: 11.924137931034483 neg_mean_loss: -11.924137931034483 node_ip: 127.0.0.1 pid: 46220 time_since_restore: 5.207761287689209 time_this_iter_s: 0.10753417015075684 time_total_s: 5.207761287689209 timestamp: 1658499760 timesteps_since_restore: 0 training_iteration: 49 trial_id: b9acff32 warmup_time: 0.0029239654541015625 Result for objective_bb3e0210: date: 2022-07-22_15-22-43 done: false experiment_id: 08a7bc47c140405296e58a049cd2f6b4 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -8.003127819283387 neg_mean_loss: 8.003127819283387 node_ip: 127.0.0.1 pid: 46229 time_since_restore: 5.188760042190552 time_this_iter_s: 0.10853385925292969 time_total_s: 5.188760042190552 timestamp: 1658499763 timesteps_since_restore: 0 training_iteration: 48 trial_id: bb3e0210 warmup_time: 0.002833843231201172 Result for objective_bb3b2702: date: 2022-07-22_15-22-43 done: false experiment_id: 2f984120624b4e9aa3392fcc1b16b811 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 10.705050505050504 neg_mean_loss: -10.705050505050504 node_ip: 127.0.0.1 pid: 46227 time_since_restore: 5.1574530601501465 time_this_iter_s: 0.10719585418701172 time_total_s: 5.1574530601501465 timestamp: 1658499763 timesteps_since_restore: 0 training_iteration: 48 trial_id: bb3b2702 warmup_time: 0.002672910690307617 Result for objective_bb3c8714: date: 2022-07-22_15-22-43 done: false experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 3.6598337799021596 neg_mean_loss: -3.6598337799021596 node_ip: 127.0.0.1 pid: 46228 time_since_restore: 5.163677930831909 time_this_iter_s: 0.10699272155761719 time_total_s: 5.163677930831909 timestamp: 1658499763 timesteps_since_restore: 0 training_iteration: 48 trial_id: bb3c8714 warmup_time: 0.0027511119842529297 Result for objective_b9acff32: date: 2022-07-22_15-22-45 done: false experiment_id: ec3c260f0ad348fa9b188f9c93c01eba hostname: Kais-MacBook-Pro.local iterations: 95 iterations_since_restore: 96 mean_loss: 11.152380952380952 neg_mean_loss: -11.152380952380952 node_ip: 127.0.0.1 pid: 46220 time_since_restore: 10.250982284545898 time_this_iter_s: 0.1073312759399414 time_total_s: 10.250982284545898 timestamp: 1658499765 timesteps_since_restore: 0 training_iteration: 96 trial_id: b9acff32 warmup_time: 0.0029239654541015625 Result for objective_b9acff32: date: 2022-07-22_15-22-45 done: true experiment_id: ec3c260f0ad348fa9b188f9c93c01eba experiment_tag: 1_activation=relu,height=2,steps=100,width=1 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 11.11743119266055 neg_mean_loss: -11.11743119266055 node_ip: 127.0.0.1 pid: 46220 time_since_restore: 10.68371295928955 time_this_iter_s: 0.10785365104675293 time_total_s: 10.68371295928955 timestamp: 1658499765 timesteps_since_restore: 0 training_iteration: 100 trial_id: b9acff32 warmup_time: 0.0029239654541015625 Result for objective_bb3e0210: date: 2022-07-22_15-22-47 done: false experiment_id: 08a7bc47c140405296e58a049cd2f6b4 hostname: Kais-MacBook-Pro.local iterations: 93 iterations_since_restore: 94 mean_loss: -8.072800015147221 neg_mean_loss: 8.072800015147221 node_ip: 127.0.0.1 pid: 46229 time_since_restore: 10.099772214889526 time_this_iter_s: 0.10250687599182129 time_total_s: 10.099772214889526 timestamp: 1658499767 timesteps_since_restore: 0 training_iteration: 94 trial_id: bb3e0210 warmup_time: 0.002833843231201172 Result for objective_bb3c8714: date: 2022-07-22_15-22-47 done: false experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf hostname: Kais-MacBook-Pro.local iterations: 92 iterations_since_restore: 93 mean_loss: 2.53944357306219 neg_mean_loss: -2.53944357306219 node_ip: 127.0.0.1 pid: 46228 time_since_restore: 9.980368852615356 time_this_iter_s: 0.10658979415893555 time_total_s: 9.980368852615356 timestamp: 1658499767 timesteps_since_restore: 0 training_iteration: 93 trial_id: bb3c8714 warmup_time: 0.0027511119842529297 Result for objective_bb3b2702: date: 2022-07-22_15-22-47 done: false experiment_id: 2f984120624b4e9aa3392fcc1b16b811 hostname: Kais-MacBook-Pro.local iterations: 92 iterations_since_restore: 93 mean_loss: 10.464550264550265 neg_mean_loss: -10.464550264550265 node_ip: 127.0.0.1 pid: 46227 time_since_restore: 9.964121103286743 time_this_iter_s: 0.10547995567321777 time_total_s: 9.964121103286743 timestamp: 1658499767 timesteps_since_restore: 0 training_iteration: 93 trial_id: bb3b2702 warmup_time: 0.002672910690307617 Result for objective_c19b9dde: date: 2022-07-22_15-22-48 done: false experiment_id: fab35a2f68c347c4bb815c268e9500a6 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 28.646596578422436 neg_mean_loss: -28.646596578422436 node_ip: 127.0.0.1 pid: 46248 time_since_restore: 0.10230708122253418 time_this_iter_s: 0.10230708122253418 time_total_s: 0.10230708122253418 timestamp: 1658499768 timesteps_since_restore: 0 training_iteration: 1 trial_id: c19b9dde warmup_time: 0.0026540756225585938 Result for objective_bb3e0210: date: 2022-07-22_15-22-49 done: true experiment_id: 08a7bc47c140405296e58a049cd2f6b4 experiment_tag: 4_activation=tanh,height=-81.4501,steps=100,width=14.7830 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -8.077146761191683 neg_mean_loss: 8.077146761191683 node_ip: 127.0.0.1 pid: 46229 time_since_restore: 11.219618082046509 time_this_iter_s: 0.1108238697052002 time_total_s: 11.219618082046509 timestamp: 1658499769 timesteps_since_restore: 0 training_iteration: 100 trial_id: bb3e0210 warmup_time: 0.002833843231201172 Result for objective_bb3c8714: date: 2022-07-22_15-22-49 done: true experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf experiment_tag: 3_activation=relu,height=-90.7016,steps=100,width=0.5666 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 2.442852652712544 neg_mean_loss: -2.442852652712544 node_ip: 127.0.0.1 pid: 46228 time_since_restore: 11.276630163192749 time_this_iter_s: 0.10824012756347656 time_total_s: 11.276630163192749 timestamp: 1658499769 timesteps_since_restore: 0 training_iteration: 100 trial_id: bb3c8714 warmup_time: 0.0027511119842529297 Result for objective_bb3b2702: date: 2022-07-22_15-22-49 done: true experiment_id: 2f984120624b4e9aa3392fcc1b16b811 experiment_tag: 2_activation=relu,height=2,steps=100,width=4 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 10.446305418719211 neg_mean_loss: -10.446305418719211 node_ip: 127.0.0.1 pid: 46227 time_since_restore: 11.281576871871948 time_this_iter_s: 0.1068716049194336 time_total_s: 11.281576871871948 timestamp: 1658499769 timesteps_since_restore: 0 training_iteration: 100 trial_id: bb3b2702 warmup_time: 0.002672910690307617 Result for objective_c3762426: date: 2022-07-22_15-22-51 done: false experiment_id: 034596724b2a4c9f86726e9a15fb386b hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 0.15367937759969408 neg_mean_loss: -0.15367937759969408 node_ip: 127.0.0.1 pid: 46253 time_since_restore: 0.1050560474395752 time_this_iter_s: 0.1050560474395752 time_total_s: 0.1050560474395752 timestamp: 1658499771 timesteps_since_restore: 0 training_iteration: 1 trial_id: c3762426 warmup_time: 0.0028579235076904297 Result for objective_c385528e: date: 2022-07-22_15-22-51 done: false experiment_id: 3044e8f1500f4b16aba04b72105c67be hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 14.516317633001044 neg_mean_loss: -14.516317633001044 node_ip: 127.0.0.1 pid: 46256 time_since_restore: 0.10402488708496094 time_this_iter_s: 0.10402488708496094 time_total_s: 0.10402488708496094 timestamp: 1658499771 timesteps_since_restore: 0 training_iteration: 1 trial_id: c385528e warmup_time: 0.0027010440826416016 Result for objective_c387a7c8: date: 2022-07-22_15-22-51 done: false experiment_id: fc7f3d46b136437cab526c2fa6ea2944 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 14.294664350689382 neg_mean_loss: -14.294664350689382 node_ip: 127.0.0.1 pid: 46257 time_since_restore: 0.10461091995239258 time_this_iter_s: 0.10461091995239258 time_total_s: 0.10461091995239258 timestamp: 1658499771 timesteps_since_restore: 0 training_iteration: 1 trial_id: c387a7c8 warmup_time: 0.0025899410247802734 Result for objective_c19b9dde: date: 2022-07-22_15-22-53 done: false experiment_id: fab35a2f68c347c4bb815c268e9500a6 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 22.72706638821166 neg_mean_loss: -22.72706638821166 node_ip: 127.0.0.1 pid: 46248 time_since_restore: 5.118642330169678 time_this_iter_s: 0.10826611518859863 time_total_s: 5.118642330169678 timestamp: 1658499773 timesteps_since_restore: 0 training_iteration: 48 trial_id: c19b9dde warmup_time: 0.0026540756225585938 Result for objective_c3762426: date: 2022-07-22_15-22-56 done: false experiment_id: 034596724b2a4c9f86726e9a15fb386b hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -7.5742395372701825 neg_mean_loss: 7.5742395372701825 node_ip: 127.0.0.1 pid: 46253 time_since_restore: 5.14322304725647 time_this_iter_s: 0.10359907150268555 time_total_s: 5.14322304725647 timestamp: 1658499776 timesteps_since_restore: 0 training_iteration: 48 trial_id: c3762426 warmup_time: 0.0028579235076904297 Result for objective_c385528e: date: 2022-07-22_15-22-56 done: false experiment_id: 3044e8f1500f4b16aba04b72105c67be hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 4.704784733826337 neg_mean_loss: -4.704784733826337 node_ip: 127.0.0.1 pid: 46256 time_since_restore: 5.16935396194458 time_this_iter_s: 0.10719513893127441 time_total_s: 5.16935396194458 timestamp: 1658499776 timesteps_since_restore: 0 training_iteration: 48 trial_id: c385528e warmup_time: 0.0027010440826416016 Result for objective_c387a7c8: date: 2022-07-22_15-22-56 done: false experiment_id: fc7f3d46b136437cab526c2fa6ea2944 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 4.553194134463865 neg_mean_loss: -4.553194134463865 node_ip: 127.0.0.1 pid: 46257 time_since_restore: 5.1458728313446045 time_this_iter_s: 0.10374999046325684 time_total_s: 5.1458728313446045 timestamp: 1658499776 timesteps_since_restore: 0 training_iteration: 48 trial_id: c387a7c8 warmup_time: 0.0025899410247802734 Result for objective_c19b9dde: date: 2022-07-22_15-22-58 done: false experiment_id: fab35a2f68c347c4bb815c268e9500a6 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 21.20978139671461 neg_mean_loss: -21.20978139671461 node_ip: 127.0.0.1 pid: 46248 time_since_restore: 10.183105230331421 time_this_iter_s: 0.10718917846679688 time_total_s: 10.183105230331421 timestamp: 1658499778 timesteps_since_restore: 0 training_iteration: 95 trial_id: c19b9dde warmup_time: 0.0026540756225585938 Result for objective_c19b9dde: date: 2022-07-22_15-22-59 done: true experiment_id: fab35a2f68c347c4bb815c268e9500a6 experiment_tag: 5_activation=relu,height=86.4660,steps=100,width=0.3087 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 21.112246386473707 neg_mean_loss: -21.112246386473707 node_ip: 127.0.0.1 pid: 46248 time_since_restore: 10.722583055496216 time_this_iter_s: 0.10788774490356445 time_total_s: 10.722583055496216 timestamp: 1658499779 timesteps_since_restore: 0 training_iteration: 100 trial_id: c19b9dde warmup_time: 0.0026540756225585938 Result for objective_c97678f8: date: 2022-07-22_15-23-01 done: false experiment_id: 7c4187f99b9246bdbae534ce20f33bfc hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 10.925214276212655 neg_mean_loss: -10.925214276212655 node_ip: 127.0.0.1 pid: 46272 time_since_restore: 0.1039581298828125 time_this_iter_s: 0.1039581298828125 time_total_s: 0.1039581298828125 timestamp: 1658499781 timesteps_since_restore: 0 training_iteration: 1 trial_id: c97678f8 warmup_time: 0.002711057662963867 Result for objective_c3762426: date: 2022-07-22_15-23-01 done: false experiment_id: 034596724b2a4c9f86726e9a15fb386b hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: -8.564680574270156 neg_mean_loss: 8.564680574270156 node_ip: 127.0.0.1 pid: 46253 time_since_restore: 10.190189123153687 time_this_iter_s: 0.10848307609558105 time_total_s: 10.190189123153687 timestamp: 1658499781 timesteps_since_restore: 0 training_iteration: 95 trial_id: c3762426 warmup_time: 0.0028579235076904297 Result for objective_c387a7c8: date: 2022-07-22_15-23-01 done: false experiment_id: fc7f3d46b136437cab526c2fa6ea2944 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 4.425622066068654 neg_mean_loss: -4.425622066068654 node_ip: 127.0.0.1 pid: 46257 time_since_restore: 10.181554794311523 time_this_iter_s: 0.10594582557678223 time_total_s: 10.181554794311523 timestamp: 1658499781 timesteps_since_restore: 0 training_iteration: 95 trial_id: c387a7c8 warmup_time: 0.0025899410247802734 Result for objective_c385528e: date: 2022-07-22_15-23-01 done: false experiment_id: 3044e8f1500f4b16aba04b72105c67be hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 4.611447627123378 neg_mean_loss: -4.611447627123378 node_ip: 127.0.0.1 pid: 46256 time_since_restore: 10.214791059494019 time_this_iter_s: 0.10866808891296387 time_total_s: 10.214791059494019 timestamp: 1658499781 timesteps_since_restore: 0 training_iteration: 95 trial_id: c385528e warmup_time: 0.0027010440826416016 Result for objective_c3762426: date: 2022-07-22_15-23-02 done: true experiment_id: 034596724b2a4c9f86726e9a15fb386b experiment_tag: 6_activation=tanh,height=-98.4632,steps=100,width=0.7237 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -8.621481572769046 neg_mean_loss: 8.621481572769046 node_ip: 127.0.0.1 pid: 46253 time_since_restore: 10.72708010673523 time_this_iter_s: 0.10719490051269531 time_total_s: 10.72708010673523 timestamp: 1658499782 timesteps_since_restore: 0 training_iteration: 100 trial_id: c3762426 warmup_time: 0.0028579235076904297 Result for objective_c387a7c8: date: 2022-07-22_15-23-02 done: true experiment_id: fc7f3d46b136437cab526c2fa6ea2944 experiment_tag: 8_activation=tanh,height=42.9466,steps=100,width=8.0171 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 4.419090335709762 neg_mean_loss: -4.419090335709762 node_ip: 127.0.0.1 pid: 46257 time_since_restore: 10.721372842788696 time_this_iter_s: 0.10873699188232422 time_total_s: 10.721372842788696 timestamp: 1658499782 timesteps_since_restore: 0 training_iteration: 100 trial_id: c387a7c8 warmup_time: 0.0025899410247802734 Result for objective_c385528e: date: 2022-07-22_15-23-02 done: true experiment_id: 3044e8f1500f4b16aba04b72105c67be experiment_tag: 7_activation=relu,height=-54.8368,steps=100,width=11.0765 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 4.606686500095965 neg_mean_loss: -4.606686500095965 node_ip: 127.0.0.1 pid: 46256 time_since_restore: 10.749844074249268 time_this_iter_s: 0.10710501670837402 time_total_s: 10.749844074249268 timestamp: 1658499782 timesteps_since_restore: 0 training_iteration: 100 trial_id: c385528e warmup_time: 0.0027010440826416016 Result for objective_cb6180fe: date: 2022-07-22_15-23-04 done: false experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 24.166473702046588 neg_mean_loss: -24.166473702046588 node_ip: 127.0.0.1 pid: 46277 time_since_restore: 0.10389995574951172 time_this_iter_s: 0.10389995574951172 time_total_s: 0.10389995574951172 timestamp: 1658499784 timesteps_since_restore: 0 training_iteration: 1 trial_id: cb6180fe warmup_time: 0.0027260780334472656 Result for objective_c97678f8: date: 2022-07-22_15-23-06 done: false experiment_id: 7c4187f99b9246bdbae534ce20f33bfc hostname: Kais-MacBook-Pro.local iterations: 31 iterations_since_restore: 32 mean_loss: 1.1287174007116527 neg_mean_loss: -1.1287174007116527 node_ip: 127.0.0.1 pid: 46272 time_since_restore: 5.140314340591431 time_this_iter_s: 0.10663819313049316 time_total_s: 5.140314340591431 timestamp: 1658499786 timesteps_since_restore: 0 training_iteration: 32 trial_id: c97678f8 warmup_time: 0.002711057662963867 Result for objective_cb6180fe: date: 2022-07-22_15-23-09 done: false experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 14.42370878193022 neg_mean_loss: -14.42370878193022 node_ip: 127.0.0.1 pid: 46277 time_since_restore: 5.139087915420532 time_this_iter_s: 0.10686802864074707 time_total_s: 5.139087915420532 timestamp: 1658499789 timesteps_since_restore: 0 training_iteration: 48 trial_id: cb6180fe warmup_time: 0.0027260780334472656 Result for objective_c97678f8: date: 2022-07-22_15-23-11 done: false experiment_id: 7c4187f99b9246bdbae534ce20f33bfc hostname: Kais-MacBook-Pro.local iterations: 78 iterations_since_restore: 79 mean_loss: 1.0070978091228147 neg_mean_loss: -1.0070978091228147 node_ip: 127.0.0.1 pid: 46272 time_since_restore: 10.206932067871094 time_this_iter_s: 0.10774993896484375 time_total_s: 10.206932067871094 timestamp: 1658499791 timesteps_since_restore: 0 training_iteration: 79 trial_id: c97678f8 warmup_time: 0.002711057662963867 Result for objective_c97678f8: date: 2022-07-22_15-23-14 done: true experiment_id: 7c4187f99b9246bdbae534ce20f33bfc experiment_tag: 9_activation=tanh,height=9.2521,steps=100,width=15.5288 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 0.9898408262377837 neg_mean_loss: -0.9898408262377837 node_ip: 127.0.0.1 pid: 46272 time_since_restore: 12.459095239639282 time_this_iter_s: 0.10785508155822754 time_total_s: 12.459095239639282 timestamp: 1658499794 timesteps_since_restore: 0 training_iteration: 100 trial_id: c97678f8 warmup_time: 0.002711057662963867 Result for objective_cb6180fe: date: 2022-07-22_15-23-15 done: false experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 14.296767042885353 neg_mean_loss: -14.296767042885353 node_ip: 127.0.0.1 pid: 46277 time_since_restore: 10.220713138580322 time_this_iter_s: 0.10750126838684082 time_total_s: 10.220713138580322 timestamp: 1658499795 timesteps_since_restore: 0 training_iteration: 95 trial_id: cb6180fe warmup_time: 0.0027260780334472656 Result for objective_cb6180fe: date: 2022-07-22_15-23-15 done: true experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea experiment_tag: 10_activation=relu,height=41.6647,steps=100,width=8.0585 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 14.29026803363612 neg_mean_loss: -14.29026803363612 node_ip: 127.0.0.1 pid: 46277 time_since_restore: 10.752035856246948 time_this_iter_s: 0.1066899299621582 time_total_s: 10.752035856246948 timestamp: 1658499795 timesteps_since_restore: 0 training_iteration: 100 trial_id: cb6180fe warmup_time: 0.0027260780334472656
We take another look at the optimal hyperparamters.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 0.7236705053153614, 'height': -98.46320622400306, 'activation': 'tanh'}
Sometimes we may want to build a more complicated search space that has conditional dependencies on other hyperparameters. In this case, we pass a define-by-run function to the search_alg argument in ray.tune().
def define_by_run_func(trial) -> Optional[Dict[str, Any]]:
"""Define-by-run function to create the search space.
Ensure no actual computation takes place here. That should go into
the trainable passed to ``Tuner()`` (in this example, that's
``objective``).
For more information, see https://optuna.readthedocs.io/en/stable\
/tutorial/10_key_features/002_configurations.html
This function should either return None or a dict with constant values.
"""
activation = trial.suggest_categorical("activation", ["relu", "tanh"])
# Define-by-run allows for conditional search spaces.
if activation == "relu":
trial.suggest_float("width", 0, 20)
trial.suggest_float("height", -100, 100)
else:
trial.suggest_float("width", -1, 21)
trial.suggest_float("height", -101, 101)
# Return all constants in a dictionary.
return {"steps": 100}
As before, we create the search_alg from OptunaSearch and ConcurrencyLimiter, this time we define the scope of search via the space argument and provide no initialization. We also must specific metric and mode when using space.
searcher = OptunaSearch(space=define_by_run_func, metric="mean_loss", mode="min")
algo = ConcurrencyLimiter(searcher, max_concurrent=4)
[I 2022-07-22 15:23:15,784] A new study created in memory with name: optuna
Running the experiment with a define-by-run search space:
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
search_alg=algo,
num_samples=num_samples,
),
)
results = tuner.fit()
| Trial name | status | loc | activation | height | steps | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|---|
| objective_d363eed6 | TERMINATED | 127.0.0.1:46290 | relu | 96.9804 | 100 | 6.47293 | 19.8517 | 100 | 10.735 | 99 | -19.8517 |
| objective_d4f3a700 | TERMINATED | 127.0.0.1:46298 | tanh | -76.4387 | 100 | 17.7358 | -7.58724 | 100 | 11.396 | 99 | 7.58724 |
| objective_d4f50ce4 | TERMINATED | 127.0.0.1:46299 | relu | -89.8609 | 100 | 6.28321 | 1.17212 | 100 | 11.4182 | 99 | -1.17212 |
| objective_d4f67908 | TERMINATED | 127.0.0.1:46300 | relu | 67.9266 | 100 | 14.0599 | 16.864 | 100 | 11.341 | 99 | -16.864 |
| objective_db5c0402 | TERMINATED | 127.0.0.1:46317 | tanh | -54.13 | 100 | 11.4712 | -5.32571 | 100 | 10.705 | 99 | 5.32571 |
| objective_dd487b56 | TERMINATED | 127.0.0.1:46322 | relu | -25.9818 | 100 | 0.758497 | 8.57703 | 100 | 10.6916 | 99 | -8.57703 |
| objective_dd4b4e94 | TERMINATED | 127.0.0.1:46323 | relu | 54.6085 | 100 | 1.2361 | 16.2163 | 100 | 10.6711 | 99 | -16.2163 |
| objective_dd5a8bca | TERMINATED | 127.0.0.1:46328 | tanh | 5.22131 | 100 | 17.0952 | 0.580871 | 100 | 10.7017 | 99 | -0.580871 |
| objective_e3482178 | TERMINATED | 127.0.0.1:46341 | tanh | 78.844 | 100 | 15.5079 | 7.94912 | 100 | 12.6417 | 99 | -7.94912 |
| objective_e532d6e0 | TERMINATED | 127.0.0.1:46353 | tanh | -66.9988 | 100 | 7.15087 | -6.56059 | 100 | 10.7486 | 99 | 6.56059 |
Result for objective_d363eed6: date: 2022-07-22_15-23-18 done: false experiment_id: af8f846e60254bc794827e76909df4f0 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 29.698044266420816 neg_mean_loss: -29.698044266420816 node_ip: 127.0.0.1 pid: 46290 time_since_restore: 0.10497498512268066 time_this_iter_s: 0.10497498512268066 time_total_s: 0.10497498512268066 timestamp: 1658499798 timesteps_since_restore: 0 training_iteration: 1 trial_id: d363eed6 warmup_time: 0.0027971267700195312 Result for objective_d4f3a700: date: 2022-07-22_15-23-21 done: false experiment_id: 52e5f2a557cf4466939b6c80bb9ff905 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 2.3561338711040003 neg_mean_loss: -2.3561338711040003 node_ip: 127.0.0.1 pid: 46298 time_since_restore: 0.10145115852355957 time_this_iter_s: 0.10145115852355957 time_total_s: 0.10145115852355957 timestamp: 1658499801 timesteps_since_restore: 0 training_iteration: 1 trial_id: d4f3a700 warmup_time: 0.0033049583435058594 Result for objective_d4f67908: date: 2022-07-22_15-23-21 done: false experiment_id: 230c6da41a834386be865bb19780b3a5 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 26.792658178391687 neg_mean_loss: -26.792658178391687 node_ip: 127.0.0.1 pid: 46300 time_since_restore: 0.10428690910339355 time_this_iter_s: 0.10428690910339355 time_total_s: 0.10428690910339355 timestamp: 1658499801 timesteps_since_restore: 0 training_iteration: 1 trial_id: d4f67908 warmup_time: 0.0030798912048339844 Result for objective_d4f50ce4: date: 2022-07-22_15-23-21 done: false experiment_id: f2396d75f50b417f9fae16d1c9ddcda4 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 11.013906266700227 neg_mean_loss: -11.013906266700227 node_ip: 127.0.0.1 pid: 46299 time_since_restore: 0.1020050048828125 time_this_iter_s: 0.1020050048828125 time_total_s: 0.1020050048828125 timestamp: 1658499801 timesteps_since_restore: 0 training_iteration: 1 trial_id: d4f50ce4 warmup_time: 0.0027348995208740234 Result for objective_d363eed6: date: 2022-07-22_15-23-23 done: false experiment_id: af8f846e60254bc794827e76909df4f0 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 20.016284953435374 neg_mean_loss: -20.016284953435374 node_ip: 127.0.0.1 pid: 46290 time_since_restore: 5.138504981994629 time_this_iter_s: 0.10720086097717285 time_total_s: 5.138504981994629 timestamp: 1658499803 timesteps_since_restore: 0 training_iteration: 48 trial_id: d363eed6 warmup_time: 0.0027971267700195312 Result for objective_d4f3a700: date: 2022-07-22_15-23-26 done: false experiment_id: 52e5f2a557cf4466939b6c80bb9ff905 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -7.525324371061028 neg_mean_loss: 7.525324371061028 node_ip: 127.0.0.1 pid: 46298 time_since_restore: 5.153075218200684 time_this_iter_s: 0.10515189170837402 time_total_s: 5.153075218200684 timestamp: 1658499806 timesteps_since_restore: 0 training_iteration: 48 trial_id: d4f3a700 warmup_time: 0.0033049583435058594 Result for objective_d4f67908: date: 2022-07-22_15-23-26 done: false experiment_id: 230c6da41a834386be865bb19780b3a5 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 16.94173032408916 neg_mean_loss: -16.94173032408916 node_ip: 127.0.0.1 pid: 46300 time_since_restore: 5.153704881668091 time_this_iter_s: 0.10537075996398926 time_total_s: 5.153704881668091 timestamp: 1658499806 timesteps_since_restore: 0 training_iteration: 48 trial_id: d4f67908 warmup_time: 0.0030798912048339844 Result for objective_d4f50ce4: date: 2022-07-22_15-23-26 done: false experiment_id: f2396d75f50b417f9fae16d1c9ddcda4 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 1.3414410369095506 neg_mean_loss: -1.3414410369095506 node_ip: 127.0.0.1 pid: 46299 time_since_restore: 5.1622138023376465 time_this_iter_s: 0.10337996482849121 time_total_s: 5.1622138023376465 timestamp: 1658499806 timesteps_since_restore: 0 training_iteration: 48 trial_id: d4f50ce4 warmup_time: 0.0027348995208740234 Result for objective_d363eed6: date: 2022-07-22_15-23-28 done: false experiment_id: af8f846e60254bc794827e76909df4f0 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 19.85973747786284 neg_mean_loss: -19.85973747786284 node_ip: 127.0.0.1 pid: 46290 time_since_restore: 10.1943039894104 time_this_iter_s: 0.12109899520874023 time_total_s: 10.1943039894104 timestamp: 1658499808 timesteps_since_restore: 0 training_iteration: 95 trial_id: d363eed6 warmup_time: 0.0027971267700195312 Result for objective_d363eed6: date: 2022-07-22_15-23-29 done: true experiment_id: af8f846e60254bc794827e76909df4f0 experiment_tag: 1_activation=relu,height=96.9804,steps=100,width=6.4729 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 19.851696631549423 neg_mean_loss: -19.851696631549423 node_ip: 127.0.0.1 pid: 46290 time_since_restore: 10.734967947006226 time_this_iter_s: 0.10534882545471191 time_total_s: 10.734967947006226 timestamp: 1658499809 timesteps_since_restore: 0 training_iteration: 100 trial_id: d363eed6 warmup_time: 0.0027971267700195312 Result for objective_d4f67908: date: 2022-07-22_15-23-31 done: false experiment_id: 230c6da41a834386be865bb19780b3a5 hostname: Kais-MacBook-Pro.local iterations: 93 iterations_since_restore: 94 mean_loss: 16.86855533914186 neg_mean_loss: -16.86855533914186 node_ip: 127.0.0.1 pid: 46300 time_since_restore: 10.096334218978882 time_this_iter_s: 0.10534310340881348 time_total_s: 10.096334218978882 timestamp: 1658499811 timesteps_since_restore: 0 training_iteration: 94 trial_id: d4f67908 warmup_time: 0.0030798912048339844 Result for objective_d4f50ce4: date: 2022-07-22_15-23-31 done: false experiment_id: f2396d75f50b417f9fae16d1c9ddcda4 hostname: Kais-MacBook-Pro.local iterations: 92 iterations_since_restore: 93 mean_loss: 1.183958166642718 neg_mean_loss: -1.183958166642718 node_ip: 127.0.0.1 pid: 46299 time_since_restore: 9.976859092712402 time_this_iter_s: 0.1057121753692627 time_total_s: 9.976859092712402 timestamp: 1658499811 timesteps_since_restore: 0 training_iteration: 93 trial_id: d4f50ce4 warmup_time: 0.0027348995208740234 Result for objective_d4f3a700: date: 2022-07-22_15-23-31 done: false experiment_id: 52e5f2a557cf4466939b6c80bb9ff905 hostname: Kais-MacBook-Pro.local iterations: 93 iterations_since_restore: 94 mean_loss: -7.583604593881161 neg_mean_loss: 7.583604593881161 node_ip: 127.0.0.1 pid: 46298 time_since_restore: 10.091833114624023 time_this_iter_s: 0.10643887519836426 time_total_s: 10.091833114624023 timestamp: 1658499811 timesteps_since_restore: 0 training_iteration: 94 trial_id: d4f3a700 warmup_time: 0.0033049583435058594 Result for objective_db5c0402: date: 2022-07-22_15-23-31 done: false experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 4.587004175501654 neg_mean_loss: -4.587004175501654 node_ip: 127.0.0.1 pid: 46317 time_since_restore: 0.10449004173278809 time_this_iter_s: 0.10449004173278809 time_total_s: 0.10449004173278809 timestamp: 1658499811 timesteps_since_restore: 0 training_iteration: 1 trial_id: db5c0402 warmup_time: 0.003406047821044922 Result for objective_d4f67908: date: 2022-07-22_15-23-32 done: true experiment_id: 230c6da41a834386be865bb19780b3a5 experiment_tag: 4_activation=relu,height=67.9266,steps=100,width=14.0599 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 16.86398832185618 neg_mean_loss: -16.86398832185618 node_ip: 127.0.0.1 pid: 46300 time_since_restore: 11.341041088104248 time_this_iter_s: 0.10513019561767578 time_total_s: 11.341041088104248 timestamp: 1658499812 timesteps_since_restore: 0 training_iteration: 100 trial_id: d4f67908 warmup_time: 0.0030798912048339844 Result for objective_d4f3a700: date: 2022-07-22_15-23-32 done: true experiment_id: 52e5f2a557cf4466939b6c80bb9ff905 experiment_tag: 2_activation=tanh,height=-76.4387,steps=100,width=17.7358 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -7.587236125634034 neg_mean_loss: 7.587236125634034 node_ip: 127.0.0.1 pid: 46298 time_since_restore: 11.396040916442871 time_this_iter_s: 0.10666394233703613 time_total_s: 11.396040916442871 timestamp: 1658499812 timesteps_since_restore: 0 training_iteration: 100 trial_id: d4f3a700 warmup_time: 0.0033049583435058594 Result for objective_d4f50ce4: date: 2022-07-22_15-23-32 done: true experiment_id: f2396d75f50b417f9fae16d1c9ddcda4 experiment_tag: 3_activation=relu,height=-89.8609,steps=100,width=6.2832 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 1.1721245345451354 neg_mean_loss: -1.1721245345451354 node_ip: 127.0.0.1 pid: 46299 time_since_restore: 11.418162822723389 time_this_iter_s: 0.11828303337097168 time_total_s: 11.418162822723389 timestamp: 1658499812 timesteps_since_restore: 0 training_iteration: 100 trial_id: d4f50ce4 warmup_time: 0.0027348995208740234 Result for objective_dd4b4e94: date: 2022-07-22_15-23-35 done: false experiment_id: c6036d4a26674f86946fd7c8c895e85c hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 25.460850093028082 neg_mean_loss: -25.460850093028082 node_ip: 127.0.0.1 pid: 46323 time_since_restore: 0.10411691665649414 time_this_iter_s: 0.10411691665649414 time_total_s: 0.10411691665649414 timestamp: 1658499815 timesteps_since_restore: 0 training_iteration: 1 trial_id: dd4b4e94 warmup_time: 0.003142118453979492 Result for objective_dd487b56: date: 2022-07-22_15-23-35 done: false experiment_id: 75e118a5733e454fab54f4a20c036852 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 17.401819651158622 neg_mean_loss: -17.401819651158622 node_ip: 127.0.0.1 pid: 46322 time_since_restore: 0.1048891544342041 time_this_iter_s: 0.1048891544342041 time_total_s: 0.1048891544342041 timestamp: 1658499815 timesteps_since_restore: 0 training_iteration: 1 trial_id: dd487b56 warmup_time: 0.0028679370880126953 Result for objective_dd5a8bca: date: 2022-07-22_15-23-35 done: false experiment_id: 775683e6d95a4705846a37e809b474d7 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 10.522131438315931 neg_mean_loss: -10.522131438315931 node_ip: 127.0.0.1 pid: 46328 time_since_restore: 0.10494732856750488 time_this_iter_s: 0.10494732856750488 time_total_s: 0.10494732856750488 timestamp: 1658499815 timesteps_since_restore: 0 training_iteration: 1 trial_id: dd5a8bca warmup_time: 0.002931833267211914 Result for objective_db5c0402: date: 2022-07-22_15-23-36 done: false experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -5.230895562358142 neg_mean_loss: 5.230895562358142 node_ip: 127.0.0.1 pid: 46317 time_since_restore: 5.120880842208862 time_this_iter_s: 0.1076059341430664 time_total_s: 5.120880842208862 timestamp: 1658499816 timesteps_since_restore: 0 training_iteration: 48 trial_id: db5c0402 warmup_time: 0.003406047821044922 Result for objective_dd4b4e94: date: 2022-07-22_15-23-40 done: false experiment_id: c6036d4a26674f86946fd7c8c895e85c hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 16.929346028466988 neg_mean_loss: -16.929346028466988 node_ip: 127.0.0.1 pid: 46323 time_since_restore: 5.137623071670532 time_this_iter_s: 0.10670709609985352 time_total_s: 5.137623071670532 timestamp: 1658499820 timesteps_since_restore: 0 training_iteration: 48 trial_id: dd4b4e94 warmup_time: 0.003142118453979492 Result for objective_dd487b56: date: 2022-07-22_15-23-40 done: false experiment_id: 75e118a5733e454fab54f4a20c036852 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 9.592430453048937 neg_mean_loss: -9.592430453048937 node_ip: 127.0.0.1 pid: 46322 time_since_restore: 5.140692949295044 time_this_iter_s: 0.1086587905883789 time_total_s: 5.140692949295044 timestamp: 1658499820 timesteps_since_restore: 0 training_iteration: 48 trial_id: dd487b56 warmup_time: 0.0028679370880126953 Result for objective_dd5a8bca: date: 2022-07-22_15-23-40 done: false experiment_id: 775683e6d95a4705846a37e809b474d7 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 0.6450610360030851 neg_mean_loss: -0.6450610360030851 node_ip: 127.0.0.1 pid: 46328 time_since_restore: 5.138496160507202 time_this_iter_s: 0.10809993743896484 time_total_s: 5.138496160507202 timestamp: 1658499820 timesteps_since_restore: 0 training_iteration: 48 trial_id: dd5a8bca warmup_time: 0.002931833267211914 Result for objective_db5c0402: date: 2022-07-22_15-23-41 done: false experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: -5.321109063262834 neg_mean_loss: 5.321109063262834 node_ip: 127.0.0.1 pid: 46317 time_since_restore: 10.162945985794067 time_this_iter_s: 0.10671615600585938 time_total_s: 10.162945985794067 timestamp: 1658499821 timesteps_since_restore: 0 training_iteration: 95 trial_id: db5c0402 warmup_time: 0.003406047821044922 Result for objective_db5c0402: date: 2022-07-22_15-23-42 done: true experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8 experiment_tag: 5_activation=tanh,height=-54.1300,steps=100,width=11.4712 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -5.325709301325668 neg_mean_loss: 5.325709301325668 node_ip: 127.0.0.1 pid: 46317 time_since_restore: 10.704953908920288 time_this_iter_s: 0.1082158088684082 time_total_s: 10.704953908920288 timestamp: 1658499822 timesteps_since_restore: 0 training_iteration: 100 trial_id: db5c0402 warmup_time: 0.003406047821044922 Result for objective_dd4b4e94: date: 2022-07-22_15-23-45 done: false experiment_id: c6036d4a26674f86946fd7c8c895e85c hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 16.253282229309562 neg_mean_loss: -16.253282229309562 node_ip: 127.0.0.1 pid: 46323 time_since_restore: 10.140441179275513 time_this_iter_s: 0.10501408576965332 time_total_s: 10.140441179275513 timestamp: 1658499825 timesteps_since_restore: 0 training_iteration: 95 trial_id: dd4b4e94 warmup_time: 0.003142118453979492 Result for objective_dd487b56: date: 2022-07-22_15-23-45 done: false experiment_id: 75e118a5733e454fab54f4a20c036852 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 8.631851056235014 neg_mean_loss: -8.631851056235014 node_ip: 127.0.0.1 pid: 46322 time_since_restore: 10.156053066253662 time_this_iter_s: 0.10682201385498047 time_total_s: 10.156053066253662 timestamp: 1658499825 timesteps_since_restore: 0 training_iteration: 95 trial_id: dd487b56 warmup_time: 0.0028679370880126953 Result for objective_e3482178: date: 2022-07-22_15-23-45 done: false experiment_id: 6709b686be234eb996457210f3efbe08 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 17.884403838898738 neg_mean_loss: -17.884403838898738 node_ip: 127.0.0.1 pid: 46341 time_since_restore: 0.10499191284179688 time_this_iter_s: 0.10499191284179688 time_total_s: 0.10499191284179688 timestamp: 1658499825 timesteps_since_restore: 0 training_iteration: 1 trial_id: e3482178 warmup_time: 0.002877950668334961 Result for objective_dd5a8bca: date: 2022-07-22_15-23-45 done: false experiment_id: 775683e6d95a4705846a37e809b474d7 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 0.583976365762005 neg_mean_loss: -0.583976365762005 node_ip: 127.0.0.1 pid: 46328 time_since_restore: 10.165286302566528 time_this_iter_s: 0.10989904403686523 time_total_s: 10.165286302566528 timestamp: 1658499825 timesteps_since_restore: 0 training_iteration: 95 trial_id: dd5a8bca warmup_time: 0.002931833267211914 Result for objective_dd4b4e94: date: 2022-07-22_15-23-45 done: true experiment_id: c6036d4a26674f86946fd7c8c895e85c experiment_tag: 7_activation=relu,height=54.6085,steps=100,width=1.2361 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 16.216283787762706 neg_mean_loss: -16.216283787762706 node_ip: 127.0.0.1 pid: 46323 time_since_restore: 10.67109203338623 time_this_iter_s: 0.1058969497680664 time_total_s: 10.67109203338623 timestamp: 1658499825 timesteps_since_restore: 0 training_iteration: 100 trial_id: dd4b4e94 warmup_time: 0.003142118453979492 Result for objective_dd487b56: date: 2022-07-22_15-23-45 done: true experiment_id: 75e118a5733e454fab54f4a20c036852 experiment_tag: 6_activation=relu,height=-25.9818,steps=100,width=0.7585 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 8.577028987245935 neg_mean_loss: -8.577028987245935 node_ip: 127.0.0.1 pid: 46322 time_since_restore: 10.691627979278564 time_this_iter_s: 0.10821914672851562 time_total_s: 10.691627979278564 timestamp: 1658499825 timesteps_since_restore: 0 training_iteration: 100 trial_id: dd487b56 warmup_time: 0.0028679370880126953 Result for objective_dd5a8bca: date: 2022-07-22_15-23-45 done: true experiment_id: 775683e6d95a4705846a37e809b474d7 experiment_tag: 8_activation=tanh,height=5.2213,steps=100,width=17.0952 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 0.5808712318418436 neg_mean_loss: -0.5808712318418436 node_ip: 127.0.0.1 pid: 46328 time_since_restore: 10.701741218566895 time_this_iter_s: 0.11321616172790527 time_total_s: 10.701741218566895 timestamp: 1658499825 timesteps_since_restore: 0 training_iteration: 100 trial_id: dd5a8bca warmup_time: 0.002931833267211914 Result for objective_e532d6e0: date: 2022-07-22_15-23-48 done: false experiment_id: 34d05fd425474b82a450c22c2a063843 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 3.30012197692201 neg_mean_loss: -3.30012197692201 node_ip: 127.0.0.1 pid: 46353 time_since_restore: 0.10400509834289551 time_this_iter_s: 0.10400509834289551 time_total_s: 0.10400509834289551 timestamp: 1658499828 timesteps_since_restore: 0 training_iteration: 1 trial_id: e532d6e0 warmup_time: 0.0028901100158691406 Result for objective_e3482178: date: 2022-07-22_15-23-50 done: false experiment_id: 6709b686be234eb996457210f3efbe08 hostname: Kais-MacBook-Pro.local iterations: 29 iterations_since_restore: 30 mean_loss: 8.10192352359083 neg_mean_loss: -8.10192352359083 node_ip: 127.0.0.1 pid: 46341 time_since_restore: 5.183844089508057 time_this_iter_s: 0.10574007034301758 time_total_s: 5.183844089508057 timestamp: 1658499830 timesteps_since_restore: 0 training_iteration: 30 trial_id: e3482178 warmup_time: 0.002877950668334961 Result for objective_e532d6e0: date: 2022-07-22_15-23-53 done: false experiment_id: 34d05fd425474b82a450c22c2a063843 hostname: Kais-MacBook-Pro.local iterations: 48 iterations_since_restore: 49 mean_loss: -6.416785967274913 neg_mean_loss: 6.416785967274913 node_ip: 127.0.0.1 pid: 46353 time_since_restore: 5.212116003036499 time_this_iter_s: 0.10714387893676758 time_total_s: 5.212116003036499 timestamp: 1658499833 timesteps_since_restore: 0 training_iteration: 49 trial_id: e532d6e0 warmup_time: 0.0028901100158691406 Result for objective_e3482178: date: 2022-07-22_15-23-55 done: false experiment_id: 6709b686be234eb996457210f3efbe08 hostname: Kais-MacBook-Pro.local iterations: 76 iterations_since_restore: 77 mean_loss: 7.968536513457016 neg_mean_loss: -7.968536513457016 node_ip: 127.0.0.1 pid: 46341 time_since_restore: 10.190052032470703 time_this_iter_s: 0.10843491554260254 time_total_s: 10.190052032470703 timestamp: 1658499835 timesteps_since_restore: 0 training_iteration: 77 trial_id: e3482178 warmup_time: 0.002877950668334961 Result for objective_e3482178: date: 2022-07-22_15-23-57 done: true experiment_id: 6709b686be234eb996457210f3efbe08 experiment_tag: 9_activation=tanh,height=78.8440,steps=100,width=15.5079 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 7.9491170267942515 neg_mean_loss: -7.9491170267942515 node_ip: 127.0.0.1 pid: 46341 time_since_restore: 12.64172887802124 time_this_iter_s: 0.10761594772338867 time_total_s: 12.64172887802124 timestamp: 1658499837 timesteps_since_restore: 0 training_iteration: 100 trial_id: e3482178 warmup_time: 0.002877950668334961 Result for objective_e532d6e0: date: 2022-07-22_15-23-58 done: false experiment_id: 34d05fd425474b82a450c22c2a063843 hostname: Kais-MacBook-Pro.local iterations: 95 iterations_since_restore: 96 mean_loss: -6.554810275125985 neg_mean_loss: 6.554810275125985 node_ip: 127.0.0.1 pid: 46353 time_since_restore: 10.305040121078491 time_this_iter_s: 0.10702300071716309 time_total_s: 10.305040121078491 timestamp: 1658499838 timesteps_since_restore: 0 training_iteration: 96 trial_id: e532d6e0 warmup_time: 0.0028901100158691406 Result for objective_e532d6e0: date: 2022-07-22_15-23-58 done: true experiment_id: 34d05fd425474b82a450c22c2a063843 experiment_tag: 10_activation=tanh,height=-66.9988,steps=100,width=7.1509 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -6.560589957038967 neg_mean_loss: 6.560589957038967 node_ip: 127.0.0.1 pid: 46353 time_since_restore: 10.74857211112976 time_this_iter_s: 0.1075899600982666 time_total_s: 10.74857211112976 timestamp: 1658499838 timesteps_since_restore: 0 training_iteration: 100 trial_id: e532d6e0 warmup_time: 0.0028901100158691406
We take a look again at the optimal hyperparameters.
print("Best hyperparameters for loss found were: ", results.get_best_result("mean_loss", "min").config)
Best hyperparameters for loss found were: {'activation': 'tanh', 'width': 17.73584230792165, 'height': -76.43866128895999, 'steps': 100}
Finally, let's take a look at the multi-objective case.
def multi_objective(config):
# Hyperparameters
width, height = config["width"], config["height"]
for step in range(config["steps"]):
# Iterative training function - can be any arbitrary training procedure
intermediate_score = evaluate(step, config["width"], config["height"], config["activation"])
# Feed the score back back to Tune.
train.report({
"iterations": step, "loss": intermediate_score, "gain": intermediate_score * width
})
We define the OptunaSearch object this time with metric and mode as list arguments.
searcher = OptunaSearch(metric=["loss", "gain"], mode=["min", "max"])
algo = ConcurrencyLimiter(searcher, max_concurrent=4)
tuner = tune.Tuner(
multi_objective,
tune_config=tune.TuneConfig(
search_alg=algo,
num_samples=num_samples,
),
param_space=search_space
)
results = tuner.fit()
[I 2022-07-22 15:26:50,680] A new study created in memory with name: optuna
| Trial name | status | loc | activation | height | width | iter | total time (s) | iterations | loss | gain |
|---|---|---|---|---|---|---|---|---|---|---|
| multi_objective_5378f1e8 | TERMINATED | 127.0.0.1:46621 | tanh | -47.5236 | 10.9256 | 100 | 10.7064 | 99 | -4.66076 | -50.9216 |
| multi_objective_550962f4 | TERMINATED | 127.0.0.1:46631 | relu | -40.6828 | 4.34953 | 100 | 11.3948 | 99 | 6.15869 | 26.7874 |
| multi_objective_550ad9a4 | TERMINATED | 127.0.0.1:46632 | tanh | -42.0563 | 2.02124 | 100 | 11.2755 | 99 | -3.72967 | -7.53857 |
| multi_objective_550c4faa | TERMINATED | 127.0.0.1:46633 | relu | -56.2838 | 5.65237 | 100 | 11.407 | 99 | 4.54719 | 25.7024 |
| multi_objective_5b6c4ba2 | TERMINATED | 127.0.0.1:46679 | relu | 54.9133 | 16.9039 | 100 | 10.6698 | 99 | 15.5507 | 262.867 |
| multi_objective_5d51f2d2 | TERMINATED | 127.0.0.1:46686 | tanh | 16.5793 | 5.16709 | 100 | 10.7451 | 99 | 1.84967 | 9.55744 |
| multi_objective_5d636760 | TERMINATED | 127.0.0.1:46689 | tanh | -13.8895 | 13.6043 | 100 | 10.6897 | 99 | -1.31525 | -17.8931 |
| multi_objective_5d66093e | TERMINATED | 127.0.0.1:46690 | relu | -51.6714 | 19.8211 | 100 | 10.7246 | 99 | 4.88357 | 96.7974 |
| multi_objective_634c9444 | TERMINATED | 127.0.0.1:46710 | relu | -12.3384 | 15.1097 | 100 | 13.1986 | 99 | 8.83256 | 133.457 |
| multi_objective_65469a2e | TERMINATED | 127.0.0.1:46719 | tanh | 19.6774 | 13.6172 | 100 | 10.7279 | 99 | 2.04137 | 27.7978 |
Result for multi_objective_5378f1e8: date: 2022-07-22_15-26-53 done: false experiment_id: 4ce03cab420a449a9f4827c3b6b76c38 gain: 57.33359096413655 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 loss: 5.247638036988612 node_ip: 127.0.0.1 pid: 46621 time_since_restore: 0.10418081283569336 time_this_iter_s: 0.10418081283569336 time_total_s: 0.10418081283569336 timestamp: 1658500013 timesteps_since_restore: 0 training_iteration: 1 trial_id: 5378f1e8 warmup_time: 0.002785921096801758 Result for multi_objective_550ad9a4: date: 2022-07-22_15-26-56 done: false experiment_id: 102a4b06449b428b9870a75d02812be6 gain: 11.711822825618256 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 loss: 5.794371166191576 node_ip: 127.0.0.1 pid: 46632 time_since_restore: 0.10288000106811523 time_this_iter_s: 0.10288000106811523 time_total_s: 0.10288000106811523 timestamp: 1658500016 timesteps_since_restore: 0 training_iteration: 1 trial_id: 550ad9a4 warmup_time: 0.0030989646911621094 Result for multi_objective_550c4faa: date: 2022-07-22_15-26-56 done: false experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d gain: 81.23378320141829 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 loss: 14.371624912716872 node_ip: 127.0.0.1 pid: 46633 time_since_restore: 0.10215306282043457 time_this_iter_s: 0.10215306282043457 time_total_s: 0.10215306282043457 timestamp: 1658500016 timesteps_since_restore: 0 training_iteration: 1 trial_id: 550c4faa warmup_time: 0.0036542415618896484 Result for multi_objective_550962f4: date: 2022-07-22_15-26-56 done: false experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64 gain: 69.29543163212045 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 loss: 15.93172358423837 node_ip: 127.0.0.1 pid: 46631 time_since_restore: 0.10244321823120117 time_this_iter_s: 0.10244321823120117 time_total_s: 0.10244321823120117 timestamp: 1658500016 timesteps_since_restore: 0 training_iteration: 1 trial_id: 550962f4 warmup_time: 0.003384828567504883 Result for multi_objective_5378f1e8: date: 2022-07-22_15-26-58 done: false experiment_id: 4ce03cab420a449a9f4827c3b6b76c38 gain: -49.83538614085625 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 loss: -4.561341152769549 node_ip: 127.0.0.1 pid: 46621 time_since_restore: 5.1368408203125 time_this_iter_s: 0.1180417537689209 time_total_s: 5.1368408203125 timestamp: 1658500018 timesteps_since_restore: 0 training_iteration: 48 trial_id: 5378f1e8 warmup_time: 0.002785921096801758 Result for multi_objective_550ad9a4: date: 2022-07-22_15-27-01 done: false experiment_id: 102a4b06449b428b9870a75d02812be6 gain: -6.575568930702184 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 loss: -3.2532328725144097 node_ip: 127.0.0.1 pid: 46632 time_since_restore: 5.146106719970703 time_this_iter_s: 0.10509181022644043 time_total_s: 5.146106719970703 timestamp: 1658500021 timesteps_since_restore: 0 training_iteration: 48 trial_id: 550ad9a4 warmup_time: 0.0030989646911621094 Result for multi_objective_550c4faa: date: 2022-07-22_15-27-01 done: false experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d gain: 26.760529995257745 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 loss: 4.73438862995905 node_ip: 127.0.0.1 pid: 46633 time_since_restore: 5.1661481857299805 time_this_iter_s: 0.10656309127807617 time_total_s: 5.1661481857299805 timestamp: 1658500021 timesteps_since_restore: 0 training_iteration: 48 trial_id: 550c4faa warmup_time: 0.0036542415618896484 Result for multi_objective_550962f4: date: 2022-07-22_15-27-01 done: false experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64 gain: 27.828615153405433 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 loss: 6.398081286367129 node_ip: 127.0.0.1 pid: 46631 time_since_restore: 5.169018983840942 time_this_iter_s: 0.10616683959960938 time_total_s: 5.169018983840942 timestamp: 1658500021 timesteps_since_restore: 0 training_iteration: 48 trial_id: 550962f4 warmup_time: 0.003384828567504883 Result for multi_objective_5378f1e8: date: 2022-07-22_15-27-03 done: false experiment_id: 4ce03cab420a449a9f4827c3b6b76c38 gain: -50.86883186480195 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 loss: -4.65593053743818 node_ip: 127.0.0.1 pid: 46621 time_since_restore: 10.17048692703247 time_this_iter_s: 0.10630917549133301 time_total_s: 10.17048692703247 timestamp: 1658500023 timesteps_since_restore: 0 training_iteration: 95 trial_id: 5378f1e8 warmup_time: 0.002785921096801758 Result for multi_objective_5378f1e8: date: 2022-07-22_15-27-04 done: true experiment_id: 4ce03cab420a449a9f4827c3b6b76c38 experiment_tag: 1_activation=tanh,height=-47.5236,steps=100,width=10.9256 gain: -50.92155508732405 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 loss: -4.66075619693878 node_ip: 127.0.0.1 pid: 46621 time_since_restore: 10.706397771835327 time_this_iter_s: 0.10694766044616699 time_total_s: 10.706397771835327 timestamp: 1658500024 timesteps_since_restore: 0 training_iteration: 100 trial_id: 5378f1e8 warmup_time: 0.002785921096801758 Result for multi_objective_550ad9a4: date: 2022-07-22_15-27-05 done: false experiment_id: 102a4b06449b428b9870a75d02812be6 gain: -7.479635489825483 hostname: Kais-MacBook-Pro.local iterations: 93 iterations_since_restore: 94 loss: -3.700515697784228 node_ip: 127.0.0.1 pid: 46632 time_since_restore: 10.056212902069092 time_this_iter_s: 0.10589313507080078 time_total_s: 10.056212902069092 timestamp: 1658500025 timesteps_since_restore: 0 training_iteration: 94 trial_id: 550ad9a4 warmup_time: 0.0030989646911621094 Result for multi_objective_550c4faa: date: 2022-07-22_15-27-05 done: false experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d gain: 25.776502806907246 hostname: Kais-MacBook-Pro.local iterations: 92 iterations_since_restore: 93 loss: 4.5602976409942295 node_ip: 127.0.0.1 pid: 46633 time_since_restore: 9.99274206161499 time_this_iter_s: 0.10583209991455078 time_total_s: 9.99274206161499 timestamp: 1658500025 timesteps_since_restore: 0 training_iteration: 93 trial_id: 550c4faa warmup_time: 0.0036542415618896484 Result for multi_objective_550962f4: date: 2022-07-22_15-27-05 done: false experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64 gain: 26.860636112316993 hostname: Kais-MacBook-Pro.local iterations: 92 iterations_since_restore: 93 loss: 6.175533072801938 node_ip: 127.0.0.1 pid: 46631 time_since_restore: 9.971143007278442 time_this_iter_s: 0.1054527759552002 time_total_s: 9.971143007278442 timestamp: 1658500025 timesteps_since_restore: 0 training_iteration: 93 trial_id: 550962f4 warmup_time: 0.003384828567504883 Result for multi_objective_5b6c4ba2: date: 2022-07-22_15-27-06 done: false experiment_id: f143c125a0734166ac832270ac08dbb6 gain: 430.90195437776714 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 loss: 25.491329340230966 node_ip: 127.0.0.1 pid: 46679 time_since_restore: 0.10215902328491211 time_this_iter_s: 0.10215902328491211 time_total_s: 0.10215902328491211 timestamp: 1658500026 timesteps_since_restore: 0 training_iteration: 1 trial_id: 5b6c4ba2 warmup_time: 0.002663135528564453 Result for multi_objective_550ad9a4: date: 2022-07-22_15-27-07 done: true experiment_id: 102a4b06449b428b9870a75d02812be6 experiment_tag: 3_activation=tanh,height=-42.0563,steps=100,width=2.0212 gain: -7.538566547680232 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 loss: -3.729671571085163 node_ip: 127.0.0.1 pid: 46632 time_since_restore: 11.27550482749939 time_this_iter_s: 0.10642290115356445 time_total_s: 11.27550482749939 timestamp: 1658500027 timesteps_since_restore: 0 training_iteration: 100 trial_id: 550ad9a4 warmup_time: 0.0030989646911621094 Result for multi_objective_550962f4: date: 2022-07-22_15-27-07 done: true experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64 experiment_tag: 2_activation=relu,height=-40.6828,steps=100,width=4.3495 gain: 26.787356208375382 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 loss: 6.15868527856241 node_ip: 127.0.0.1 pid: 46631 time_since_restore: 11.394789934158325 time_this_iter_s: 0.10491585731506348 time_total_s: 11.394789934158325 timestamp: 1658500027 timesteps_since_restore: 0 training_iteration: 100 trial_id: 550962f4 warmup_time: 0.003384828567504883 Result for multi_objective_550c4faa: date: 2022-07-22_15-27-07 done: true experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d experiment_tag: 4_activation=relu,height=-56.2838,steps=100,width=5.6524 gain: 25.702421216002982 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 loss: 4.547191359402326 node_ip: 127.0.0.1 pid: 46633 time_since_restore: 11.406961917877197 time_this_iter_s: 0.12177181243896484 time_total_s: 11.406961917877197 timestamp: 1658500027 timesteps_since_restore: 0 training_iteration: 100 trial_id: 550c4faa warmup_time: 0.0036542415618896484 Result for multi_objective_5d51f2d2: date: 2022-07-22_15-27-09 done: false experiment_id: 1433194530b14db4b77ce40a69d65407 gain: 60.23765295445784 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 loss: 11.657934184433229 node_ip: 127.0.0.1 pid: 46686 time_since_restore: 0.10274410247802734 time_this_iter_s: 0.10274410247802734 time_total_s: 0.10274410247802734 timestamp: 1658500029 timesteps_since_restore: 0 training_iteration: 1 trial_id: 5d51f2d2 warmup_time: 0.0028769969940185547 Result for multi_objective_5d636760: date: 2022-07-22_15-27-10 done: false experiment_id: b8dccac5d8914569b1c2ca061ad47a56 gain: 117.14725547845826 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 loss: 8.611046741256526 node_ip: 127.0.0.1 pid: 46689 time_since_restore: 0.10460686683654785 time_this_iter_s: 0.10460686683654785 time_total_s: 0.10460686683654785 timestamp: 1658500030 timesteps_since_restore: 0 training_iteration: 1 trial_id: 5d636760 warmup_time: 0.0028090476989746094 Result for multi_objective_5d66093e: date: 2022-07-22_15-27-10 done: false experiment_id: e7259e986278458899b46b485e069ec9 gain: 294.00300634729234 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 loss: 14.832864636325777 node_ip: 127.0.0.1 pid: 46690 time_since_restore: 0.10294318199157715 time_this_iter_s: 0.10294318199157715 time_total_s: 0.10294318199157715 timestamp: 1658500030 timesteps_since_restore: 0 training_iteration: 1 trial_id: 5d66093e warmup_time: 0.0027511119842529297 Result for multi_objective_5b6c4ba2: date: 2022-07-22_15-27-11 done: false experiment_id: f143c125a0734166ac832270ac08dbb6 gain: 263.92128797150474 hostname: Kais-MacBook-Pro.local iterations: 48 iterations_since_restore: 49 loss: 15.6130748612977 node_ip: 127.0.0.1 pid: 46679 time_since_restore: 5.2078468799591064 time_this_iter_s: 0.10803008079528809 time_total_s: 5.2078468799591064 timestamp: 1658500031 timesteps_since_restore: 0 training_iteration: 49 trial_id: 5b6c4ba2 warmup_time: 0.002663135528564453 Result for multi_objective_5d51f2d2: date: 2022-07-22_15-27-14 done: false experiment_id: 1433194530b14db4b77ce40a69d65407 gain: 10.610216987384797 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 loss: 2.053420165872076 node_ip: 127.0.0.1 pid: 46686 time_since_restore: 5.178762912750244 time_this_iter_s: 0.10710978507995605 time_total_s: 5.178762912750244 timestamp: 1658500034 timesteps_since_restore: 0 training_iteration: 48 trial_id: 5d51f2d2 warmup_time: 0.0028769969940185547 Result for multi_objective_5d636760: date: 2022-07-22_15-27-15 done: false experiment_id: b8dccac5d8914569b1c2ca061ad47a56 gain: -16.800839364465826 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 loss: -1.234965449842409 node_ip: 127.0.0.1 pid: 46689 time_since_restore: 5.146665811538696 time_this_iter_s: 0.10526585578918457 time_total_s: 5.146665811538696 timestamp: 1658500035 timesteps_since_restore: 0 training_iteration: 48 trial_id: 5d636760 warmup_time: 0.0028090476989746094 Result for multi_objective_5d66093e: date: 2022-07-22_15-27-15 done: false experiment_id: e7259e986278458899b46b485e069ec9 gain: 97.89753254610552 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 loss: 4.939068026982622 node_ip: 127.0.0.1 pid: 46690 time_since_restore: 5.1532862186431885 time_this_iter_s: 0.10752320289611816 time_total_s: 5.1532862186431885 timestamp: 1658500035 timesteps_since_restore: 0 training_iteration: 48 trial_id: 5d66093e warmup_time: 0.0027511119842529297 Result for multi_objective_5b6c4ba2: date: 2022-07-22_15-27-16 done: false experiment_id: f143c125a0734166ac832270ac08dbb6 gain: 262.90943552209455 hostname: Kais-MacBook-Pro.local iterations: 95 iterations_since_restore: 96 loss: 15.553215620072208 node_ip: 127.0.0.1 pid: 46679 time_since_restore: 10.240519762039185 time_this_iter_s: 0.10443782806396484 time_total_s: 10.240519762039185 timestamp: 1658500036 timesteps_since_restore: 0 training_iteration: 96 trial_id: 5b6c4ba2 warmup_time: 0.002663135528564453 Result for multi_objective_5b6c4ba2: date: 2022-07-22_15-27-17 done: true experiment_id: f143c125a0734166ac832270ac08dbb6 experiment_tag: 5_activation=relu,height=54.9133,steps=100,width=16.9039 gain: 262.86741923011425 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 loss: 15.550730017197484 node_ip: 127.0.0.1 pid: 46679 time_since_restore: 10.669770956039429 time_this_iter_s: 0.10598993301391602 time_total_s: 10.669770956039429 timestamp: 1658500037 timesteps_since_restore: 0 training_iteration: 100 trial_id: 5b6c4ba2 warmup_time: 0.002663135528564453 Result for multi_objective_634c9444: date: 2022-07-22_15-27-19 done: false experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee gain: 283.55076351521427 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 loss: 18.766157678691147 node_ip: 127.0.0.1 pid: 46710 time_since_restore: 0.10465264320373535 time_this_iter_s: 0.10465264320373535 time_total_s: 0.10465264320373535 timestamp: 1658500039 timesteps_since_restore: 0 training_iteration: 1 trial_id: 634c9444 warmup_time: 0.004335880279541016 Result for multi_objective_5d51f2d2: date: 2022-07-22_15-27-20 done: false experiment_id: 1433194530b14db4b77ce40a69d65407 gain: 9.60907229177291 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 loss: 1.8596662860626851 node_ip: 127.0.0.1 pid: 46686 time_since_restore: 10.218497037887573 time_this_iter_s: 0.10553288459777832 time_total_s: 10.218497037887573 timestamp: 1658500040 timesteps_since_restore: 0 training_iteration: 95 trial_id: 5d51f2d2 warmup_time: 0.0028769969940185547 Result for multi_objective_5d636760: date: 2022-07-22_15-27-20 done: false experiment_id: b8dccac5d8914569b1c2ca061ad47a56 gain: -17.840160182810624 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 loss: -1.311361948500235 node_ip: 127.0.0.1 pid: 46689 time_since_restore: 10.16199278831482 time_this_iter_s: 0.10700488090515137 time_total_s: 10.16199278831482 timestamp: 1658500040 timesteps_since_restore: 0 training_iteration: 95 trial_id: 5d636760 warmup_time: 0.0028090476989746094 Result for multi_objective_5d66093e: date: 2022-07-22_15-27-20 done: false experiment_id: e7259e986278458899b46b485e069ec9 gain: 96.85061995053432 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 loss: 4.886249816008858 node_ip: 127.0.0.1 pid: 46690 time_since_restore: 10.197196006774902 time_this_iter_s: 0.10463690757751465 time_total_s: 10.197196006774902 timestamp: 1658500040 timesteps_since_restore: 0 training_iteration: 95 trial_id: 5d66093e warmup_time: 0.0027511119842529297 Result for multi_objective_5d51f2d2: date: 2022-07-22_15-27-20 done: true experiment_id: 1433194530b14db4b77ce40a69d65407 experiment_tag: 6_activation=tanh,height=16.5793,steps=100,width=5.1671 gain: 9.557436804323405 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 loss: 1.8496731491334866 node_ip: 127.0.0.1 pid: 46686 time_since_restore: 10.745142936706543 time_this_iter_s: 0.10509181022644043 time_total_s: 10.745142936706543 timestamp: 1658500040 timesteps_since_restore: 0 training_iteration: 100 trial_id: 5d51f2d2 warmup_time: 0.0028769969940185547 Result for multi_objective_5d636760: date: 2022-07-22_15-27-20 done: true experiment_id: b8dccac5d8914569b1c2ca061ad47a56 experiment_tag: 7_activation=tanh,height=-13.8895,steps=100,width=13.6043 gain: -17.893079155032332 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 loss: -1.3152518197690004 node_ip: 127.0.0.1 pid: 46689 time_since_restore: 10.689707040786743 time_this_iter_s: 0.10322403907775879 time_total_s: 10.689707040786743 timestamp: 1658500040 timesteps_since_restore: 0 training_iteration: 100 trial_id: 5d636760 warmup_time: 0.0028090476989746094 Result for multi_objective_5d66093e: date: 2022-07-22_15-27-20 done: true experiment_id: e7259e986278458899b46b485e069ec9 experiment_tag: 8_activation=relu,height=-51.6714,steps=100,width=19.8211 gain: 96.79744897015512 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 loss: 4.8835672653630535 node_ip: 127.0.0.1 pid: 46690 time_since_restore: 10.72464108467102 time_this_iter_s: 0.10328292846679688 time_total_s: 10.72464108467102 timestamp: 1658500040 timesteps_since_restore: 0 training_iteration: 100 trial_id: 5d66093e warmup_time: 0.0027511119842529297 Result for multi_objective_65469a2e: date: 2022-07-22_15-27-23 done: false experiment_id: 015ee29ddb8b4c239d85a022a23d4093 gain: 162.96737941207124 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 loss: 11.96774070054979 node_ip: 127.0.0.1 pid: 46719 time_since_restore: 0.10403180122375488 time_this_iter_s: 0.10403180122375488 time_total_s: 0.10403180122375488 timestamp: 1658500043 timesteps_since_restore: 0 training_iteration: 1 trial_id: 65469a2e warmup_time: 0.0033850669860839844 Result for multi_objective_634c9444: date: 2022-07-22_15-27-24 done: false experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee gain: 136.50874589912112 hostname: Kais-MacBook-Pro.local iterations: 24 iterations_since_restore: 25 loss: 9.034518610724309 node_ip: 127.0.0.1 pid: 46710 time_since_restore: 5.125189781188965 time_this_iter_s: 0.11187601089477539 time_total_s: 5.125189781188965 timestamp: 1658500044 timesteps_since_restore: 0 training_iteration: 25 trial_id: 634c9444 warmup_time: 0.004335880279541016 Result for multi_objective_65469a2e: date: 2022-07-22_15-27-28 done: false experiment_id: 015ee29ddb8b4c239d85a022a23d4093 gain: 28.890088364068774 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 loss: 2.1215846238952016 node_ip: 127.0.0.1 pid: 46719 time_since_restore: 5.174077749252319 time_this_iter_s: 0.10734677314758301 time_total_s: 5.174077749252319 timestamp: 1658500048 timesteps_since_restore: 0 training_iteration: 48 trial_id: 65469a2e warmup_time: 0.0033850669860839844 Result for multi_objective_634c9444: date: 2022-07-22_15-27-30 done: false experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee gain: 133.84933935751317 hostname: Kais-MacBook-Pro.local iterations: 71 iterations_since_restore: 72 loss: 8.85851188137237 node_ip: 127.0.0.1 pid: 46710 time_since_restore: 10.204095840454102 time_this_iter_s: 0.10942888259887695 time_total_s: 10.204095840454102 timestamp: 1658500050 timesteps_since_restore: 0 training_iteration: 72 trial_id: 634c9444 warmup_time: 0.004335880279541016 Result for multi_objective_634c9444: date: 2022-07-22_15-27-33 done: true experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee experiment_tag: 9_activation=relu,height=-12.3384,steps=100,width=15.1097 gain: 133.45728949146132 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 loss: 8.83256495915982 node_ip: 127.0.0.1 pid: 46710 time_since_restore: 13.198630809783936 time_this_iter_s: 0.1067969799041748 time_total_s: 13.198630809783936 timestamp: 1658500053 timesteps_since_restore: 0 training_iteration: 100 trial_id: 634c9444 warmup_time: 0.004335880279541016 Result for multi_objective_65469a2e: date: 2022-07-22_15-27-33 done: false experiment_id: 015ee29ddb8b4c239d85a022a23d4093 gain: 27.850744704436202 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 loss: 2.0452589477867855 node_ip: 127.0.0.1 pid: 46719 time_since_restore: 10.19201397895813 time_this_iter_s: 0.10735702514648438 time_total_s: 10.19201397895813 timestamp: 1658500053 timesteps_since_restore: 0 training_iteration: 95 trial_id: 65469a2e warmup_time: 0.0033850669860839844 Result for multi_objective_65469a2e: date: 2022-07-22_15-27-34 done: true experiment_id: 015ee29ddb8b4c239d85a022a23d4093 experiment_tag: 10_activation=tanh,height=19.6774,steps=100,width=13.6172 gain: 27.797824972416166 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 loss: 2.041372712191397 node_ip: 127.0.0.1 pid: 46719 time_since_restore: 10.727944135665894 time_this_iter_s: 0.10885024070739746 time_total_s: 10.727944135665894 timestamp: 1658500054 timesteps_since_restore: 0 training_iteration: 100 trial_id: 65469a2e warmup_time: 0.0033850669860839844
Now there are two hyperparameter sets for the two objectives.
print("Best hyperparameters for loss found were: ", results.get_best_result("loss", "min").config)
print("Best hyperparameters for gain found were: ", results.get_best_result("gain", "max").config)
Best hyperparameters for loss found were: {'steps': 100, 'width': 10.925599395387751, 'height': -47.52361963011387, 'activation': 'tanh'}
Best hyperparameters for gain found were: {'steps': 100, 'width': 16.90386360893735, 'height': 54.91329340230965, 'activation': 'relu'}
We can mix-and-match the use of initial hyperparameter evaluations, conditional search spaces via define-by-run functions, and multi-objective tasks. This is also true of scheduler usage, with the exception of multi-objective optimization-- schedulers typically rely on a single scalar score, rather than the two scores we use here: loss, gain.
ray.shutdown()