In this tutorial we introduce BlendSearch and CFO, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with FLAML and, as a result, allow you to seamlessly scale up a BlendSearch and CFO optimization process - without sacrificing performance.
Fast Library for Automated Machine Learning & Tuning (FLAML) does not rely on the gradient of the objective function, but instead, learns from samples of the search space. It is suitable for optimizing functions that are non-differentiable, with many local minima, or even unknown but only testable. Therefore, it is necessarily belongs to the domain of "derivative-free optimization" and "black-box optimization".
FLAML has two primary algorithms: (1) Frugal Optimization for Cost-related Hyperparameters (CFO) begins with a low-cost initial point and gradually moves to a high-cost region as needed. It is a local search method that leverages randomized direct search method with an adaptive step-size and random restarts. As a local search method, it has an appealing provable convergence rate and bounded cost but may get trapped in suboptimal local minima. (2) Economical Hyperparameter Optimization With Blended Search Strategy (BlendSearch) combines CFO's local search with global search, making it less suspectable to local minima traps. It leverages the frugality of CFO and the space exploration ability of global search methods such as Bayesian optimization.
In this example we minimize a simple objective to briefly demonstrate the usage of
FLAML with Ray Tune via BlendSearch and CFO. It's useful to keep in mind that
despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit
or explicit objective. Here we assume flaml==1.1.1 and optuna==2.8.0 libraries
are installed. To learn more, please refer to
the FLAML website.
Click below to see all the imports we need for this example. You can also launch directly into a Binder instance to run this notebook yourself. Just click on the rocket symbol at the top of the navigation.
import time
import ray
from ray import tune
from ray.air import session
from ray.tune.search import ConcurrencyLimiter
from ray.tune.search.flaml import BlendSearch, CFO
/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/flaml/searcher/blendsearch.py:14: DeprecationWarning: The module `ray.tune.suggest` has been moved to `ray.tune.search` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.suggest` with `ray.tune.search`. from ray.tune.suggest import Searcher /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/flaml/searcher/blendsearch.py:15: DeprecationWarning: The module `ray.tune.suggest.optuna` has been moved to `ray.tune.search.optuna` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.suggest.optuna` with `ray.tune.search.optuna`. from ray.tune.suggest.optuna import OptunaSearch as GlobalSearch /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/flaml/tune/sample.py:39: DeprecationWarning: The module `ray.tune.sample` has been moved to `ray.tune.search.sample` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.sample` with `ray.tune.search.sample`. from ray.tune.sample import _BackwardsCompatibleNumpyRng /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/flaml/tune/space.py:6: DeprecationWarning: The module `ray.tune.suggest.variant_generator` has been moved to `ray.tune.search.variant_generator` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.suggest.variant_generator` with `ray.tune.search.variant_generator`. from ray.tune.suggest.variant_generator import generate_variants
Let's start by defining a simple evaluation function.
We artificially sleep for a bit (0.1 seconds) to simulate a long-running ML experiment.
This setup assumes that we're running multiple steps of an experiment and try to
tune three hyperparameters, namely width and height, and activation.
def evaluate(step, width, height, activation):
time.sleep(0.1)
activation_boost = 10 if activation=="relu" else 1
return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost
Next, our objective function takes a Tune config, evaluates the score of your
experiment in a training loop, and uses session.report to report the score back to Tune.
def objective(config):
for step in range(config["steps"]):
score = evaluate(step, config["width"], config["height"], config["activation"])
session.report({"iterations": step, "mean_loss": score})
ray.init(configure_logging=False)
| Python version: | 3.7.7 |
| Ray version: | 3.0.0.dev0 |
| Dashboard: | http://127.0.0.1:8265 |
This example demonstrates the usage of Economical Hyperparameter Optimization With Blended Search Strategy (BlendSearch) with Ray Tune.
Now we define the search algorithm built from BlendSearch, constrained to a
maximum of 4 concurrent trials with a ConcurrencyLimiter.
algo = BlendSearch()
algo = ConcurrencyLimiter(algo, max_concurrent=4)
The number of samples this Tune run is set to 1000.
(you can decrease this if it takes too long on your machine).
num_samples = 1000
# If 1000 samples take too long, you can reduce this number.
# We override this number here for our smoke tests.
num_samples = 10
Next we define a search space. The critical assumption is that the optimal hyperparameters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time.
search_config = {
"steps": 100,
"width": tune.uniform(0, 20),
"height": tune.uniform(-100, 100),
"activation": tune.choice(["relu, tanh"])
}
Finally, we run the experiment to "min"imize the "mean_loss" of the objective by
searching search_config via algo, num_samples times. This previous sentence is
fully characterizes the search problem we aim to solve. With this in mind, observe
how efficient it is to execute tuner.fit().
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
param_space=search_config,
)
results = tuner.fit()
Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.
[I 2022-07-22 15:18:10,659] A new study created in memory with name: optuna
| Trial name | status | loc | activation | height | steps | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|---|
| objective_1e0f814e | TERMINATED | 127.0.0.1:45801 | relu, tanh | 29.4532 | 100 | 1.94864 | 4.43814 | 100 | 11.061 | 99 | -4.43814 |
| objective_1fc231f8 | TERMINATED | 127.0.0.1:45809 | relu, tanh | -95.8496 | 100 | 15.4264 | -8.51991 | 100 | 11.6807 | 99 | 8.51991 |
| objective_1fc3b668 | TERMINATED | 127.0.0.1:45810 | relu, tanh | 49.7608 | 100 | 12.673 | 6.05515 | 100 | 11.7372 | 99 | -6.05515 |
| objective_1fc58b1e | TERMINATED | 127.0.0.1:45811 | relu, tanh | -55.0407 | 100 | 9.97014 | -4.40377 | 100 | 11.6335 | 99 | 4.40377 |
| objective_265c09ee | TERMINATED | 127.0.0.1:45837 | relu, tanh | 52.1061 | 100 | 3.96126 | 6.45927 | 100 | 10.7539 | 99 | -6.45927 |
| objective_28587494 | TERMINATED | 127.0.0.1:45842 | relu, tanh | -82.332 | 100 | 3.38222 | -6.94321 | 100 | 10.7116 | 99 | 6.94321 |
| objective_28682bfa | TERMINATED | 127.0.0.1:45845 | relu, tanh | -94.9771 | 100 | 17.681 | -8.4409 | 100 | 10.7471 | 99 | 8.4409 |
| objective_28788388 | TERMINATED | 127.0.0.1:45848 | relu, tanh | 90.6787 | 100 | 13.7072 | 10.141 | 100 | 10.7525 | 99 | -10.141 |
| objective_2e3cd63e | TERMINATED | 127.0.0.1:45864 | relu, tanh | 2.43845 | 100 | 0.0789653 | 6.85628 | 100 | 10.7006 | 99 | -6.85628 |
| objective_3046455a | TERMINATED | 127.0.0.1:45869 | relu, tanh | 22.5052 | 100 | 16.2524 | 3.31229 | 100 | 10.7176 | 99 | -3.31229 |
Result for objective_1e0f814e: date: 2022-07-22_15-18-14 done: false experiment_id: 623da6c1aa28400a9d85da19bdc5721d hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 13.945321626643842 neg_mean_loss: -13.945321626643842 node_ip: 127.0.0.1 pid: 45801 time_since_restore: 0.10402607917785645 time_this_iter_s: 0.10402607917785645 time_total_s: 0.10402607917785645 timestamp: 1658499494 timesteps_since_restore: 0 training_iteration: 1 trial_id: 1e0f814e warmup_time: 0.003920078277587891 Result for objective_1fc58b1e: date: 2022-07-22_15-18-17 done: false experiment_id: da508e209a1946e89ec15a3f1a5ce5ef hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 5.495932910616953 neg_mean_loss: -5.495932910616953 node_ip: 127.0.0.1 pid: 45811 time_since_restore: 0.10269379615783691 time_this_iter_s: 0.10269379615783691 time_total_s: 0.10269379615783691 timestamp: 1658499497 timesteps_since_restore: 0 training_iteration: 1 trial_id: 1fc58b1e warmup_time: 0.004099130630493164 Result for objective_1fc231f8: date: 2022-07-22_15-18-17 done: false experiment_id: 24c1fa443be246229e2c2ed9581452de hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 1.4150389871880282 neg_mean_loss: -1.4150389871880282 node_ip: 127.0.0.1 pid: 45809 time_since_restore: 0.10376811027526855 time_this_iter_s: 0.10376811027526855 time_total_s: 0.10376811027526855 timestamp: 1658499497 timesteps_since_restore: 0 training_iteration: 1 trial_id: 1fc231f8 warmup_time: 0.004148244857788086 Result for objective_1fc3b668: date: 2022-07-22_15-18-17 done: false experiment_id: 5cdd489afb0e43b7a44aa4484765dc92 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 15.976077650772236 neg_mean_loss: -15.976077650772236 node_ip: 127.0.0.1 pid: 45810 time_since_restore: 0.1012420654296875 time_this_iter_s: 0.1012420654296875 time_total_s: 0.1012420654296875 timestamp: 1658499497 timesteps_since_restore: 0 training_iteration: 1 trial_id: 1fc3b668 warmup_time: 0.002666950225830078 Result for objective_1e0f814e: date: 2022-07-22_15-18-19 done: false experiment_id: 623da6c1aa28400a9d85da19bdc5721d hostname: Kais-MacBook-Pro.local iterations: 44 iterations_since_restore: 45 mean_loss: 4.989814690087961 neg_mean_loss: -4.989814690087961 node_ip: 127.0.0.1 pid: 45801 time_since_restore: 5.1339499950408936 time_this_iter_s: 0.10842394828796387 time_total_s: 5.1339499950408936 timestamp: 1658499499 timesteps_since_restore: 0 training_iteration: 45 trial_id: 1e0f814e warmup_time: 0.003920078277587891 Result for objective_1fc58b1e: date: 2022-07-22_15-18-22 done: false experiment_id: da508e209a1946e89ec15a3f1a5ce5ef hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -4.295122851662673 neg_mean_loss: 4.295122851662673 node_ip: 127.0.0.1 pid: 45811 time_since_restore: 5.167006015777588 time_this_iter_s: 0.10817217826843262 time_total_s: 5.167006015777588 timestamp: 1658499502 timesteps_since_restore: 0 training_iteration: 48 trial_id: 1fc58b1e warmup_time: 0.004099130630493164 Result for objective_1fc231f8: date: 2022-07-22_15-18-22 done: false experiment_id: 24c1fa443be246229e2c2ed9581452de hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -8.44891425494968 neg_mean_loss: 8.44891425494968 node_ip: 127.0.0.1 pid: 45809 time_since_restore: 5.138737916946411 time_this_iter_s: 0.10868191719055176 time_total_s: 5.138737916946411 timestamp: 1658499502 timesteps_since_restore: 0 training_iteration: 48 trial_id: 1fc231f8 warmup_time: 0.004148244857788086 Result for objective_1fc3b668: date: 2022-07-22_15-18-22 done: false experiment_id: 5cdd489afb0e43b7a44aa4484765dc92 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 6.141195146339472 neg_mean_loss: -6.141195146339472 node_ip: 127.0.0.1 pid: 45810 time_since_restore: 5.181504011154175 time_this_iter_s: 0.10758399963378906 time_total_s: 5.181504011154175 timestamp: 1658499502 timesteps_since_restore: 0 training_iteration: 48 trial_id: 1fc3b668 warmup_time: 0.002666950225830078 Result for objective_1e0f814e: date: 2022-07-22_15-18-24 done: false experiment_id: 623da6c1aa28400a9d85da19bdc5721d hostname: Kais-MacBook-Pro.local iterations: 91 iterations_since_restore: 92 mean_loss: 4.479149291114477 neg_mean_loss: -4.479149291114477 node_ip: 127.0.0.1 pid: 45801 time_since_restore: 10.196602821350098 time_this_iter_s: 0.10757303237915039 time_total_s: 10.196602821350098 timestamp: 1658499504 timesteps_since_restore: 0 training_iteration: 92 trial_id: 1e0f814e warmup_time: 0.003920078277587891 Result for objective_1e0f814e: date: 2022-07-22_15-18-25 done: true experiment_id: 623da6c1aa28400a9d85da19bdc5721d experiment_tag: 1_activation=relu_tanh,height=29.4532,steps=100,width=1.9486 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 4.438137591322411 neg_mean_loss: -4.438137591322411 node_ip: 127.0.0.1 pid: 45801 time_since_restore: 11.060971975326538 time_this_iter_s: 0.10748600959777832 time_total_s: 11.060971975326538 timestamp: 1658499505 timesteps_since_restore: 0 training_iteration: 100 trial_id: 1e0f814e warmup_time: 0.003920078277587891 Result for objective_1fc231f8: date: 2022-07-22_15-18-26 done: false experiment_id: 24c1fa443be246229e2c2ed9581452de hostname: Kais-MacBook-Pro.local iterations: 90 iterations_since_restore: 91 mean_loss: -8.513449547227397 neg_mean_loss: 8.513449547227397 node_ip: 127.0.0.1 pid: 45809 time_since_restore: 9.72878909111023 time_this_iter_s: 0.10644912719726562 time_total_s: 9.72878909111023 timestamp: 1658499506 timesteps_since_restore: 0 training_iteration: 91 trial_id: 1fc231f8 warmup_time: 0.004148244857788086 Result for objective_1fc3b668: date: 2022-07-22_15-18-26 done: false experiment_id: 5cdd489afb0e43b7a44aa4484765dc92 hostname: Kais-MacBook-Pro.local iterations: 89 iterations_since_restore: 90 mean_loss: 6.063959309753095 neg_mean_loss: -6.063959309753095 node_ip: 127.0.0.1 pid: 45810 time_since_restore: 9.6911301612854 time_this_iter_s: 0.10597920417785645 time_total_s: 9.6911301612854 timestamp: 1658499506 timesteps_since_restore: 0 training_iteration: 90 trial_id: 1fc3b668 warmup_time: 0.002666950225830078 Result for objective_1fc58b1e: date: 2022-07-22_15-18-27 done: false experiment_id: da508e209a1946e89ec15a3f1a5ce5ef hostname: Kais-MacBook-Pro.local iterations: 91 iterations_since_restore: 92 mean_loss: -4.395049451529077 neg_mean_loss: 4.395049451529077 node_ip: 127.0.0.1 pid: 45811 time_since_restore: 9.873695850372314 time_this_iter_s: 0.10703802108764648 time_total_s: 9.873695850372314 timestamp: 1658499507 timesteps_since_restore: 0 training_iteration: 92 trial_id: 1fc58b1e warmup_time: 0.004099130630493164 Result for objective_265c09ee: date: 2022-07-22_15-18-28 done: false experiment_id: a7543e8d696745edbfe36a6f4ebe4071 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 16.210614243979172 neg_mean_loss: -16.210614243979172 node_ip: 127.0.0.1 pid: 45837 time_since_restore: 0.1049339771270752 time_this_iter_s: 0.1049339771270752 time_total_s: 0.1049339771270752 timestamp: 1658499508 timesteps_since_restore: 0 training_iteration: 1 trial_id: 265c09ee warmup_time: 0.0026078224182128906 Result for objective_1fc58b1e: date: 2022-07-22_15-18-28 done: true experiment_id: da508e209a1946e89ec15a3f1a5ce5ef experiment_tag: 4_activation=relu_tanh,height=-55.0407,steps=100,width=9.9701 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -4.403770601366095 neg_mean_loss: 4.403770601366095 node_ip: 127.0.0.1 pid: 45811 time_since_restore: 11.633522987365723 time_this_iter_s: 0.10734701156616211 time_total_s: 11.633522987365723 timestamp: 1658499508 timesteps_since_restore: 0 training_iteration: 100 trial_id: 1fc58b1e warmup_time: 0.004099130630493164 Result for objective_1fc231f8: date: 2022-07-22_15-18-28 done: true experiment_id: 24c1fa443be246229e2c2ed9581452de experiment_tag: 2_activation=relu_tanh,height=-95.8496,steps=100,width=15.4264 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -8.519908298508359 neg_mean_loss: 8.519908298508359 node_ip: 127.0.0.1 pid: 45809 time_since_restore: 11.680676937103271 time_this_iter_s: 0.10814285278320312 time_total_s: 11.680676937103271 timestamp: 1658499508 timesteps_since_restore: 0 training_iteration: 100 trial_id: 1fc231f8 warmup_time: 0.004148244857788086 Result for objective_1fc3b668: date: 2022-07-22_15-18-29 done: true experiment_id: 5cdd489afb0e43b7a44aa4484765dc92 experiment_tag: 3_activation=relu_tanh,height=49.7608,steps=100,width=12.6730 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 6.055152568795227 neg_mean_loss: -6.055152568795227 node_ip: 127.0.0.1 pid: 45810 time_since_restore: 11.73720097541809 time_this_iter_s: 0.10816287994384766 time_total_s: 11.73720097541809 timestamp: 1658499509 timesteps_since_restore: 0 training_iteration: 100 trial_id: 1fc3b668 warmup_time: 0.002666950225830078 Result for objective_28587494: date: 2022-07-22_15-18-31 done: false experiment_id: 750bd707fda147f0bd615c32243d64e5 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 2.766796283480206 neg_mean_loss: -2.766796283480206 node_ip: 127.0.0.1 pid: 45842 time_since_restore: 0.10265803337097168 time_this_iter_s: 0.10265803337097168 time_total_s: 0.10265803337097168 timestamp: 1658499511 timesteps_since_restore: 0 training_iteration: 1 trial_id: '28587494' warmup_time: 0.0028159618377685547 Result for objective_28682bfa: date: 2022-07-22_15-18-31 done: false experiment_id: 5487e6a1cf70423096cf6285197fb824 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 1.5022935169945555 neg_mean_loss: -1.5022935169945555 node_ip: 127.0.0.1 pid: 45845 time_since_restore: 0.10289216041564941 time_this_iter_s: 0.10289216041564941 time_total_s: 0.10289216041564941 timestamp: 1658499511 timesteps_since_restore: 0 training_iteration: 1 trial_id: 28682bfa warmup_time: 0.003072023391723633 Result for objective_28788388: date: 2022-07-22_15-18-31 done: false experiment_id: e959dae493a240c489a3193352c46f3a hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 20.067866923898734 neg_mean_loss: -20.067866923898734 node_ip: 127.0.0.1 pid: 45848 time_since_restore: 0.1038503646850586 time_this_iter_s: 0.1038503646850586 time_total_s: 0.1038503646850586 timestamp: 1658499511 timesteps_since_restore: 0 training_iteration: 1 trial_id: '28788388' warmup_time: 0.0027070045471191406 Result for objective_265c09ee: date: 2022-07-22_15-18-33 done: false experiment_id: a7543e8d696745edbfe36a6f4ebe4071 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 6.720352557756592 neg_mean_loss: -6.720352557756592 node_ip: 127.0.0.1 pid: 45837 time_since_restore: 5.148331165313721 time_this_iter_s: 0.10472607612609863 time_total_s: 5.148331165313721 timestamp: 1658499513 timesteps_since_restore: 0 training_iteration: 48 trial_id: 265c09ee warmup_time: 0.0026078224182128906 Result for objective_28587494: date: 2022-07-22_15-18-36 done: false experiment_id: 750bd707fda147f0bd615c32243d64e5 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -6.641362320132167 neg_mean_loss: 6.641362320132167 node_ip: 127.0.0.1 pid: 45842 time_since_restore: 5.175055980682373 time_this_iter_s: 0.10797286033630371 time_total_s: 5.175055980682373 timestamp: 1658499516 timesteps_since_restore: 0 training_iteration: 48 trial_id: '28587494' warmup_time: 0.0028159618377685547 Result for objective_28682bfa: date: 2022-07-22_15-18-36 done: false experiment_id: 5487e6a1cf70423096cf6285197fb824 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -8.378801379858526 neg_mean_loss: 8.378801379858526 node_ip: 127.0.0.1 pid: 45845 time_since_restore: 5.192234039306641 time_this_iter_s: 0.10880398750305176 time_total_s: 5.192234039306641 timestamp: 1658499516 timesteps_since_restore: 0 training_iteration: 48 trial_id: 28682bfa warmup_time: 0.003072023391723633 Result for objective_28788388: date: 2022-07-22_15-18-36 done: false experiment_id: e959dae493a240c489a3193352c46f3a hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 10.22071644495505 neg_mean_loss: -10.22071644495505 node_ip: 127.0.0.1 pid: 45848 time_since_restore: 5.175441265106201 time_this_iter_s: 0.10689902305603027 time_total_s: 5.175441265106201 timestamp: 1658499516 timesteps_since_restore: 0 training_iteration: 48 trial_id: '28788388' warmup_time: 0.0027070045471191406 Result for objective_265c09ee: date: 2022-07-22_15-18-38 done: false experiment_id: a7543e8d696745edbfe36a6f4ebe4071 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 6.472149118155721 neg_mean_loss: -6.472149118155721 node_ip: 127.0.0.1 pid: 45837 time_since_restore: 10.2187819480896 time_this_iter_s: 0.10822892189025879 time_total_s: 10.2187819480896 timestamp: 1658499518 timesteps_since_restore: 0 training_iteration: 95 trial_id: 265c09ee warmup_time: 0.0026078224182128906 Result for objective_265c09ee: date: 2022-07-22_15-18-38 done: true experiment_id: a7543e8d696745edbfe36a6f4ebe4071 experiment_tag: 5_activation=relu_tanh,height=52.1061,steps=100,width=3.9613 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 6.459268729660066 neg_mean_loss: -6.459268729660066 node_ip: 127.0.0.1 pid: 45837 time_since_restore: 10.753926277160645 time_this_iter_s: 0.10714435577392578 time_total_s: 10.753926277160645 timestamp: 1658499518 timesteps_since_restore: 0 training_iteration: 100 trial_id: 265c09ee warmup_time: 0.0026078224182128906 Result for objective_2e3cd63e: date: 2022-07-22_15-18-41 done: false experiment_id: a2f52d042c5d43bcbb59c6a16249fdb9 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 11.243845267715532 neg_mean_loss: -11.243845267715532 node_ip: 127.0.0.1 pid: 45864 time_since_restore: 0.10467910766601562 time_this_iter_s: 0.10467910766601562 time_total_s: 0.10467910766601562 timestamp: 1658499521 timesteps_since_restore: 0 training_iteration: 1 trial_id: 2e3cd63e warmup_time: 0.002730131149291992 Result for objective_28587494: date: 2022-07-22_15-18-41 done: false experiment_id: 750bd707fda147f0bd615c32243d64e5 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: -6.928259075209273 neg_mean_loss: 6.928259075209273 node_ip: 127.0.0.1 pid: 45842 time_since_restore: 10.179664850234985 time_this_iter_s: 0.10827279090881348 time_total_s: 10.179664850234985 timestamp: 1658499521 timesteps_since_restore: 0 training_iteration: 95 trial_id: '28587494' warmup_time: 0.0028159618377685547 Result for objective_28682bfa: date: 2022-07-22_15-18-41 done: false experiment_id: 5487e6a1cf70423096cf6285197fb824 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: -8.437898356861577 neg_mean_loss: 8.437898356861577 node_ip: 127.0.0.1 pid: 45845 time_since_restore: 10.208577156066895 time_this_iter_s: 0.10761713981628418 time_total_s: 10.208577156066895 timestamp: 1658499521 timesteps_since_restore: 0 training_iteration: 95 trial_id: 28682bfa warmup_time: 0.003072023391723633 Result for objective_28788388: date: 2022-07-22_15-18-41 done: false experiment_id: e959dae493a240c489a3193352c46f3a hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 10.144880256980718 neg_mean_loss: -10.144880256980718 node_ip: 127.0.0.1 pid: 45848 time_since_restore: 10.215306997299194 time_this_iter_s: 0.10723686218261719 time_total_s: 10.215306997299194 timestamp: 1658499521 timesteps_since_restore: 0 training_iteration: 95 trial_id: '28788388' warmup_time: 0.0027070045471191406 Result for objective_28587494: date: 2022-07-22_15-18-42 done: true experiment_id: 750bd707fda147f0bd615c32243d64e5 experiment_tag: 6_activation=relu_tanh,height=-82.3320,steps=100,width=3.3822 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -6.943213699003365 neg_mean_loss: 6.943213699003365 node_ip: 127.0.0.1 pid: 45842 time_since_restore: 10.711625099182129 time_this_iter_s: 0.10703706741333008 time_total_s: 10.711625099182129 timestamp: 1658499522 timesteps_since_restore: 0 training_iteration: 100 trial_id: '28587494' warmup_time: 0.0028159618377685547 Result for objective_28682bfa: date: 2022-07-22_15-18-42 done: true experiment_id: 5487e6a1cf70423096cf6285197fb824 experiment_tag: 7_activation=relu_tanh,height=-94.9771,steps=100,width=17.6810 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -8.440901810803183 neg_mean_loss: 8.440901810803183 node_ip: 127.0.0.1 pid: 45845 time_since_restore: 10.747072219848633 time_this_iter_s: 0.1084749698638916 time_total_s: 10.747072219848633 timestamp: 1658499522 timesteps_since_restore: 0 training_iteration: 100 trial_id: 28682bfa warmup_time: 0.003072023391723633 Result for objective_28788388: date: 2022-07-22_15-18-42 done: true experiment_id: e959dae493a240c489a3193352c46f3a experiment_tag: 8_activation=relu_tanh,height=90.6787,steps=100,width=13.7072 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 10.141019147716873 neg_mean_loss: -10.141019147716873 node_ip: 127.0.0.1 pid: 45848 time_since_restore: 10.752525091171265 time_this_iter_s: 0.10643196105957031 time_total_s: 10.752525091171265 timestamp: 1658499522 timesteps_since_restore: 0 training_iteration: 100 trial_id: '28788388' warmup_time: 0.0027070045471191406 Result for objective_3046455a: date: 2022-07-22_15-18-44 done: false experiment_id: f282080699d848f69c4b8626bd4c2d91 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 13.250521336587763 neg_mean_loss: -13.250521336587763 node_ip: 127.0.0.1 pid: 45869 time_since_restore: 0.10329103469848633 time_this_iter_s: 0.10329103469848633 time_total_s: 0.10329103469848633 timestamp: 1658499524 timesteps_since_restore: 0 training_iteration: 1 trial_id: 3046455a warmup_time: 0.0027010440826416016 Result for objective_2e3cd63e: date: 2022-07-22_15-18-46 done: false experiment_id: a2f52d042c5d43bcbb59c6a16249fdb9 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 8.5370623175226 neg_mean_loss: -8.5370623175226 node_ip: 127.0.0.1 pid: 45864 time_since_restore: 5.129793882369995 time_this_iter_s: 0.10869097709655762 time_total_s: 5.129793882369995 timestamp: 1658499526 timesteps_since_restore: 0 training_iteration: 48 trial_id: 2e3cd63e warmup_time: 0.002730131149291992 Result for objective_3046455a: date: 2022-07-22_15-18-49 done: false experiment_id: f282080699d848f69c4b8626bd4c2d91 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 3.3797430580474837 neg_mean_loss: -3.3797430580474837 node_ip: 127.0.0.1 pid: 45869 time_since_restore: 5.139970064163208 time_this_iter_s: 0.10870695114135742 time_total_s: 5.139970064163208 timestamp: 1658499529 timesteps_since_restore: 0 training_iteration: 48 trial_id: 3046455a warmup_time: 0.0027010440826416016 Result for objective_2e3cd63e: date: 2022-07-22_15-18-51 done: false experiment_id: a2f52d042c5d43bcbb59c6a16249fdb9 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 6.983470378488619 neg_mean_loss: -6.983470378488619 node_ip: 127.0.0.1 pid: 45864 time_since_restore: 10.16285490989685 time_this_iter_s: 0.1078939437866211 time_total_s: 10.16285490989685 timestamp: 1658499531 timesteps_since_restore: 0 training_iteration: 95 trial_id: 2e3cd63e warmup_time: 0.002730131149291992 Result for objective_2e3cd63e: date: 2022-07-22_15-18-51 done: true experiment_id: a2f52d042c5d43bcbb59c6a16249fdb9 experiment_tag: 9_activation=relu_tanh,height=2.4385,steps=100,width=0.0790 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 6.8562837197212945 neg_mean_loss: -6.8562837197212945 node_ip: 127.0.0.1 pid: 45864 time_since_restore: 10.700610876083374 time_this_iter_s: 0.10786271095275879 time_total_s: 10.700610876083374 timestamp: 1658499531 timesteps_since_restore: 0 training_iteration: 100 trial_id: 2e3cd63e warmup_time: 0.002730131149291992 Result for objective_3046455a: date: 2022-07-22_15-18-54 done: false experiment_id: f282080699d848f69c4b8626bd4c2d91 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 3.315552368411653 neg_mean_loss: -3.315552368411653 node_ip: 127.0.0.1 pid: 45869 time_since_restore: 10.18196415901184 time_this_iter_s: 0.10727214813232422 time_total_s: 10.18196415901184 timestamp: 1658499534 timesteps_since_restore: 0 training_iteration: 95 trial_id: 3046455a warmup_time: 0.0027010440826416016 Result for objective_3046455a: date: 2022-07-22_15-18-55 done: true experiment_id: f282080699d848f69c4b8626bd4c2d91 experiment_tag: 10_activation=relu_tanh,height=22.5052,steps=100,width=16.2524 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 3.3122882595656677 neg_mean_loss: -3.3122882595656677 node_ip: 127.0.0.1 pid: 45869 time_since_restore: 10.717580080032349 time_this_iter_s: 0.1054232120513916 time_total_s: 10.717580080032349 timestamp: 1658499535 timesteps_since_restore: 0 training_iteration: 100 trial_id: 3046455a warmup_time: 0.0027010440826416016
Here are the hyperparamters found to minimize the mean loss of the defined objective.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 15.42641286533492, 'height': -95.8496101281197, 'activation': 'relu, tanh'}
Define the time budget in seconds:
time_budget_s = 30
Similarly we define a search space, but this time we feed it as an argument to
BlendSearch rather than Tuner()'s param_space argument.
We next define the time budget via set_search_properties.
And once again include the ConcurrencyLimiter.
algo = BlendSearch(
metric="mean_loss",
mode="min",
space={
"width": tune.uniform(0, 20),
"height": tune.uniform(-100, 100),
"activation": tune.choice(["relu", "tanh"]),
},
)
algo.set_search_properties(config={"time_budget_s": time_budget_s})
algo = ConcurrencyLimiter(algo, max_concurrent=4)
You passed a `space` parameter to OptunaSearch that contained unresolved search space definitions. OptunaSearch should however be instantiated with fully configured search spaces only. To use Ray Tune's automatic search space conversion, pass the space definition as part of the `param_space` argument to `Tuner()` instead.
[I 2022-07-22 15:18:55,531] A new study created in memory with name: optuna
Now we run the experiment, this time with the time_budget included as an argument.
Note: We allow for virtually infinite num_samples by passing -1, so that the
experiment is stopped according to the time budget rather than a sample limit.
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=-1,
time_budget_s=time_budget_s,
),
param_space={"steps": 100},
)
results = tuner.fit()
| Trial name | status | loc | activation | height | steps | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|---|
| objective_38449df6 | TERMINATED | 127.0.0.1:45883 | relu | 29.4532 | 100 | 1.94864 | 13.4381 | 100 | 10.6555 | 99 | -13.4381 |
| objective_39d44fea | TERMINATED | 127.0.0.1:45889 | tanh | -95.8496 | 100 | 15.4264 | -8.51991 | 100 | 11.2945 | 99 | 8.51991 |
| objective_39d5c046 | TERMINATED | 127.0.0.1:45890 | tanh | -55.0407 | 100 | 9.97014 | -4.40377 | 100 | 11.169 | 99 | 4.40377 |
| objective_39d71608 | TERMINATED | 127.0.0.1:45891 | tanh | -82.332 | 100 | 3.38222 | -6.94321 | 100 | 11.3021 | 99 | 6.94321 |
| objective_402f9534 | TERMINATED | 127.0.0.1:45909 | relu | 2.43845 | 100 | 0.0789653 | 15.8563 | 100 | 10.7275 | 99 | -15.8563 |
| objective_4208c6aa | TERMINATED | 127.0.0.1:45914 | relu | -41.6248 | 100 | 14.4351 | 5.90701 | 100 | 10.811 | 99 | -5.90701 |
| objective_421aeb64 | TERMINATED | 127.0.0.1:45917 | relu | -94.9771 | 100 | 17.681 | 0.559098 | 100 | 10.8502 | 99 | -0.559098 |
| objective_421d81ee | TERMINATED | 127.0.0.1:45918 | tanh | -96.7222 | 100 | 13.1718 | -8.59611 | 100 | 10.8456 | 99 | 8.59611 |
| objective_481b0de6 | TERMINATED | 127.0.0.1:45933 | relu | -81.3401 | 100 | 15.2514 | 2.48132 | 11 | 1.17882 | 10 | -2.48132 |
| objective_4a00fc10 | TERMINATED | 127.0.0.1:45940 | tanh | -100 | 100 | 11.0922 | |||||
| objective_4a167946 | TERMINATED | 127.0.0.1:45945 | tanh | -71.566 | 100 | 10.8509 | |||||
| objective_4a1908a0 | TERMINATED | 127.0.0.1:45946 | relu | -100 | 100 | 12.4575 |
Result for objective_38449df6: date: 2022-07-22_15-18-58 done: false experiment_id: 22ed7d2e0f154e3d89fddc8f7f0b743f hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 22.945321626643842 neg_mean_loss: -22.945321626643842 node_ip: 127.0.0.1 pid: 45883 time_since_restore: 0.1004023551940918 time_this_iter_s: 0.1004023551940918 time_total_s: 0.1004023551940918 timestamp: 1658499538 timesteps_since_restore: 0 training_iteration: 1 trial_id: 38449df6 warmup_time: 0.0025548934936523438 Result for objective_39d5c046: date: 2022-07-22_15-19-00 done: false experiment_id: faa00d9b1a2b4014bf01bab455bf9dbc hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 5.495932910616953 neg_mean_loss: -5.495932910616953 node_ip: 127.0.0.1 pid: 45890 time_since_restore: 0.10018515586853027 time_this_iter_s: 0.10018515586853027 time_total_s: 0.10018515586853027 timestamp: 1658499540 timesteps_since_restore: 0 training_iteration: 1 trial_id: 39d5c046 warmup_time: 0.0029740333557128906 Result for objective_39d71608: date: 2022-07-22_15-19-00 done: false experiment_id: 43ea2d3839fb42cabf83009ddd82acbd hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 2.766796283480206 neg_mean_loss: -2.766796283480206 node_ip: 127.0.0.1 pid: 45891 time_since_restore: 0.10022497177124023 time_this_iter_s: 0.10022497177124023 time_total_s: 0.10022497177124023 timestamp: 1658499540 timesteps_since_restore: 0 training_iteration: 1 trial_id: 39d71608 warmup_time: 0.003571033477783203 Result for objective_39d44fea: date: 2022-07-22_15-19-00 done: false experiment_id: d45c2138630140c3915a785b0e544a10 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 1.4150389871880282 neg_mean_loss: -1.4150389871880282 node_ip: 127.0.0.1 pid: 45889 time_since_restore: 0.10021686553955078 time_this_iter_s: 0.10021686553955078 time_total_s: 0.10021686553955078 timestamp: 1658499540 timesteps_since_restore: 0 training_iteration: 1 trial_id: 39d44fea warmup_time: 0.003281831741333008 Result for objective_38449df6: date: 2022-07-22_15-19-03 done: false experiment_id: 22ed7d2e0f154e3d89fddc8f7f0b743f hostname: Kais-MacBook-Pro.local iterations: 48 iterations_since_restore: 49 mean_loss: 13.91118054261762 neg_mean_loss: -13.91118054261762 node_ip: 127.0.0.1 pid: 45883 time_since_restore: 5.199498176574707 time_this_iter_s: 0.10724401473999023 time_total_s: 5.199498176574707 timestamp: 1658499543 timesteps_since_restore: 0 training_iteration: 49 trial_id: 38449df6 warmup_time: 0.0025548934936523438 Result for objective_39d5c046: date: 2022-07-22_15-19-05 done: false experiment_id: faa00d9b1a2b4014bf01bab455bf9dbc hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -4.295122851662673 neg_mean_loss: 4.295122851662673 node_ip: 127.0.0.1 pid: 45890 time_since_restore: 5.137373924255371 time_this_iter_s: 0.1063838005065918 time_total_s: 5.137373924255371 timestamp: 1658499545 timesteps_since_restore: 0 training_iteration: 48 trial_id: 39d5c046 warmup_time: 0.0029740333557128906 Result for objective_39d44fea: date: 2022-07-22_15-19-05 done: false experiment_id: d45c2138630140c3915a785b0e544a10 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -8.44891425494968 neg_mean_loss: 8.44891425494968 node_ip: 127.0.0.1 pid: 45889 time_since_restore: 5.140375852584839 time_this_iter_s: 0.10918521881103516 time_total_s: 5.140375852584839 timestamp: 1658499545 timesteps_since_restore: 0 training_iteration: 48 trial_id: 39d44fea warmup_time: 0.003281831741333008 Result for objective_39d71608: date: 2022-07-22_15-19-05 done: false experiment_id: 43ea2d3839fb42cabf83009ddd82acbd hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -6.641362320132167 neg_mean_loss: 6.641362320132167 node_ip: 127.0.0.1 pid: 45891 time_since_restore: 5.144212007522583 time_this_iter_s: 0.1064291000366211 time_total_s: 5.144212007522583 timestamp: 1658499545 timesteps_since_restore: 0 training_iteration: 48 trial_id: 39d71608 warmup_time: 0.003571033477783203 Result for objective_38449df6: date: 2022-07-22_15-19-08 done: false experiment_id: 22ed7d2e0f154e3d89fddc8f7f0b743f hostname: Kais-MacBook-Pro.local iterations: 95 iterations_since_restore: 96 mean_loss: 13.457824286707531 neg_mean_loss: -13.457824286707531 node_ip: 127.0.0.1 pid: 45883 time_since_restore: 10.220572233200073 time_this_iter_s: 0.10792016983032227 time_total_s: 10.220572233200073 timestamp: 1658499548 timesteps_since_restore: 0 training_iteration: 96 trial_id: 38449df6 warmup_time: 0.0025548934936523438 Result for objective_38449df6: date: 2022-07-22_15-19-08 done: true experiment_id: 22ed7d2e0f154e3d89fddc8f7f0b743f experiment_tag: 1_activation=relu,height=29.4532,steps=100,width=1.9486 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 13.43813759132241 neg_mean_loss: -13.43813759132241 node_ip: 127.0.0.1 pid: 45883 time_since_restore: 10.655535221099854 time_this_iter_s: 0.10872602462768555 time_total_s: 10.655535221099854 timestamp: 1658499548 timesteps_since_restore: 0 training_iteration: 100 trial_id: 38449df6 warmup_time: 0.0025548934936523438 Result for objective_39d5c046: date: 2022-07-22_15-19-10 done: false experiment_id: faa00d9b1a2b4014bf01bab455bf9dbc hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: -4.398492005249027 neg_mean_loss: 4.398492005249027 node_ip: 127.0.0.1 pid: 45890 time_since_restore: 10.1424720287323 time_this_iter_s: 0.10685110092163086 time_total_s: 10.1424720287323 timestamp: 1658499550 timesteps_since_restore: 0 training_iteration: 95 trial_id: 39d5c046 warmup_time: 0.0029740333557128906 Result for objective_39d44fea: date: 2022-07-22_15-19-10 done: false experiment_id: d45c2138630140c3915a785b0e544a10 hostname: Kais-MacBook-Pro.local iterations: 93 iterations_since_restore: 94 mean_loss: -8.515740400738455 neg_mean_loss: 8.515740400738455 node_ip: 127.0.0.1 pid: 45889 time_since_restore: 10.051257848739624 time_this_iter_s: 0.10633492469787598 time_total_s: 10.051257848739624 timestamp: 1658499550 timesteps_since_restore: 0 training_iteration: 94 trial_id: 39d44fea warmup_time: 0.003281831741333008 Result for objective_39d71608: date: 2022-07-22_15-19-10 done: false experiment_id: 43ea2d3839fb42cabf83009ddd82acbd hostname: Kais-MacBook-Pro.local iterations: 93 iterations_since_restore: 94 mean_loss: -6.925081133185133 neg_mean_loss: 6.925081133185133 node_ip: 127.0.0.1 pid: 45891 time_since_restore: 10.069098949432373 time_this_iter_s: 0.10451602935791016 time_total_s: 10.069098949432373 timestamp: 1658499550 timesteps_since_restore: 0 training_iteration: 94 trial_id: 39d71608 warmup_time: 0.003571033477783203 Result for objective_402f9534: date: 2022-07-22_15-19-11 done: false experiment_id: 66af4092dc134e049ed47a56532133ce hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 20.243845267715532 neg_mean_loss: -20.243845267715532 node_ip: 127.0.0.1 pid: 45909 time_since_restore: 0.10260891914367676 time_this_iter_s: 0.10260891914367676 time_total_s: 0.10260891914367676 timestamp: 1658499551 timesteps_since_restore: 0 training_iteration: 1 trial_id: 402f9534 warmup_time: 0.0026721954345703125 Result for objective_39d5c046: date: 2022-07-22_15-19-11 done: true experiment_id: faa00d9b1a2b4014bf01bab455bf9dbc experiment_tag: 3_activation=tanh,height=-55.0407,steps=100,width=9.9701 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -4.403770601366095 neg_mean_loss: 4.403770601366095 node_ip: 127.0.0.1 pid: 45890 time_since_restore: 11.169048070907593 time_this_iter_s: 0.1081700325012207 time_total_s: 11.169048070907593 timestamp: 1658499551 timesteps_since_restore: 0 training_iteration: 100 trial_id: 39d5c046 warmup_time: 0.0029740333557128906 Result for objective_39d44fea: date: 2022-07-22_15-19-12 done: true experiment_id: d45c2138630140c3915a785b0e544a10 experiment_tag: 2_activation=tanh,height=-95.8496,steps=100,width=15.4264 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -8.519908298508359 neg_mean_loss: 8.519908298508359 node_ip: 127.0.0.1 pid: 45889 time_since_restore: 11.29450273513794 time_this_iter_s: 0.1167898178100586 time_total_s: 11.29450273513794 timestamp: 1658499552 timesteps_since_restore: 0 training_iteration: 100 trial_id: 39d44fea warmup_time: 0.003281831741333008 Result for objective_39d71608: date: 2022-07-22_15-19-12 done: true experiment_id: 43ea2d3839fb42cabf83009ddd82acbd experiment_tag: 4_activation=tanh,height=-82.3320,steps=100,width=3.3822 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -6.943213699003365 neg_mean_loss: 6.943213699003365 node_ip: 127.0.0.1 pid: 45891 time_since_restore: 11.302148818969727 time_this_iter_s: 0.12157797813415527 time_total_s: 11.302148818969727 timestamp: 1658499552 timesteps_since_restore: 0 training_iteration: 100 trial_id: 39d71608 warmup_time: 0.003571033477783203 Result for objective_4208c6aa: date: 2022-07-22_15-19-14 done: false experiment_id: ce292c59f7c74e1ab42de6004eb3d846 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 15.837521363412662 neg_mean_loss: -15.837521363412662 node_ip: 127.0.0.1 pid: 45914 time_since_restore: 0.10442900657653809 time_this_iter_s: 0.10442900657653809 time_total_s: 0.10442900657653809 timestamp: 1658499554 timesteps_since_restore: 0 training_iteration: 1 trial_id: 4208c6aa warmup_time: 0.002707958221435547 Result for objective_421aeb64: date: 2022-07-22_15-19-14 done: false experiment_id: a4b264c05660423ea645cef7a9cd38bb hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 10.502293516994555 neg_mean_loss: -10.502293516994555 node_ip: 127.0.0.1 pid: 45917 time_since_restore: 0.1030268669128418 time_this_iter_s: 0.1030268669128418 time_total_s: 0.1030268669128418 timestamp: 1658499554 timesteps_since_restore: 0 training_iteration: 1 trial_id: 421aeb64 warmup_time: 0.0031020641326904297 Result for objective_421d81ee: date: 2022-07-22_15-19-14 done: false experiment_id: 836a62c82d6c467fb232414586872fd0 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 1.3277844573815027 neg_mean_loss: -1.3277844573815027 node_ip: 127.0.0.1 pid: 45918 time_since_restore: 0.10497617721557617 time_this_iter_s: 0.10497617721557617 time_total_s: 0.10497617721557617 timestamp: 1658499554 timesteps_since_restore: 0 training_iteration: 1 trial_id: 421d81ee warmup_time: 0.002989053726196289 Result for objective_402f9534: date: 2022-07-22_15-19-16 done: false experiment_id: 66af4092dc134e049ed47a56532133ce hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 17.5370623175226 neg_mean_loss: -17.5370623175226 node_ip: 127.0.0.1 pid: 45909 time_since_restore: 5.13571572303772 time_this_iter_s: 0.1079568862915039 time_total_s: 5.13571572303772 timestamp: 1658499556 timesteps_since_restore: 0 training_iteration: 48 trial_id: 402f9534 warmup_time: 0.0026721954345703125 Result for objective_4208c6aa: date: 2022-07-22_15-19-19 done: false experiment_id: ce292c59f7c74e1ab42de6004eb3d846 hostname: Kais-MacBook-Pro.local iterations: 46 iterations_since_restore: 47 mean_loss: 5.98588603781584 neg_mean_loss: -5.98588603781584 node_ip: 127.0.0.1 pid: 45914 time_since_restore: 5.11784291267395 time_this_iter_s: 0.106842041015625 time_total_s: 5.11784291267395 timestamp: 1658499559 timesteps_since_restore: 0 training_iteration: 47 trial_id: 4208c6aa warmup_time: 0.002707958221435547 Result for objective_421aeb64: date: 2022-07-22_15-19-19 done: false experiment_id: a4b264c05660423ea645cef7a9cd38bb hostname: Kais-MacBook-Pro.local iterations: 46 iterations_since_restore: 47 mean_loss: 0.6237521179327707 neg_mean_loss: -0.6237521179327707 node_ip: 127.0.0.1 pid: 45917 time_since_restore: 5.119860887527466 time_this_iter_s: 0.10904288291931152 time_total_s: 5.119860887527466 timestamp: 1658499559 timesteps_since_restore: 0 training_iteration: 47 trial_id: 421aeb64 warmup_time: 0.0031020641326904297 Result for objective_421d81ee: date: 2022-07-22_15-19-19 done: false experiment_id: 836a62c82d6c467fb232414586872fd0 hostname: Kais-MacBook-Pro.local iterations: 46 iterations_since_restore: 47 mean_loss: -8.509852624896407 neg_mean_loss: 8.509852624896407 node_ip: 127.0.0.1 pid: 45918 time_since_restore: 5.103626012802124 time_this_iter_s: 0.10568881034851074 time_total_s: 5.103626012802124 timestamp: 1658499559 timesteps_since_restore: 0 training_iteration: 47 trial_id: 421d81ee warmup_time: 0.002989053726196289 Result for objective_402f9534: date: 2022-07-22_15-19-21 done: false experiment_id: 66af4092dc134e049ed47a56532133ce hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 15.983470378488619 neg_mean_loss: -15.983470378488619 node_ip: 127.0.0.1 pid: 45909 time_since_restore: 10.194732666015625 time_this_iter_s: 0.11167168617248535 time_total_s: 10.194732666015625 timestamp: 1658499561 timesteps_since_restore: 0 training_iteration: 95 trial_id: 402f9534 warmup_time: 0.0026721954345703125 Result for objective_402f9534: date: 2022-07-22_15-19-22 done: true experiment_id: 66af4092dc134e049ed47a56532133ce experiment_tag: 5_activation=relu,height=2.4385,steps=100,width=0.0790 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 15.856283719721294 neg_mean_loss: -15.856283719721294 node_ip: 127.0.0.1 pid: 45909 time_since_restore: 10.727468013763428 time_this_iter_s: 0.10604405403137207 time_total_s: 10.727468013763428 timestamp: 1658499562 timesteps_since_restore: 0 training_iteration: 100 trial_id: 402f9534 warmup_time: 0.0026721954345703125 Result for objective_4208c6aa: date: 2022-07-22_15-19-24 done: false experiment_id: ce292c59f7c74e1ab42de6004eb3d846 hostname: Kais-MacBook-Pro.local iterations: 93 iterations_since_restore: 94 mean_loss: 5.911460436218253 neg_mean_loss: -5.911460436218253 node_ip: 127.0.0.1 pid: 45914 time_since_restore: 10.168487071990967 time_this_iter_s: 0.10619020462036133 time_total_s: 10.168487071990967 timestamp: 1658499564 timesteps_since_restore: 0 training_iteration: 94 trial_id: 4208c6aa warmup_time: 0.002707958221435547 Result for objective_481b0de6: date: 2022-07-22_15-19-24 done: false experiment_id: ceafc8f530054af6bc676fd081f84c46 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 11.865986796947288 neg_mean_loss: -11.865986796947288 node_ip: 127.0.0.1 pid: 45933 time_since_restore: 0.10329604148864746 time_this_iter_s: 0.10329604148864746 time_total_s: 0.10329604148864746 timestamp: 1658499564 timesteps_since_restore: 0 training_iteration: 1 trial_id: 481b0de6 warmup_time: 0.0028421878814697266 Result for objective_421aeb64: date: 2022-07-22_15-19-24 done: false experiment_id: a4b264c05660423ea645cef7a9cd38bb hostname: Kais-MacBook-Pro.local iterations: 93 iterations_since_restore: 94 mean_loss: 0.5627408539120644 neg_mean_loss: -0.5627408539120644 node_ip: 127.0.0.1 pid: 45917 time_since_restore: 10.207238912582397 time_this_iter_s: 0.1081080436706543 time_total_s: 10.207238912582397 timestamp: 1658499564 timesteps_since_restore: 0 training_iteration: 94 trial_id: 421aeb64 warmup_time: 0.0031020641326904297 Result for objective_421d81ee: date: 2022-07-22_15-19-24 done: false experiment_id: 836a62c82d6c467fb232414586872fd0 hostname: Kais-MacBook-Pro.local iterations: 93 iterations_since_restore: 94 mean_loss: -8.591242584097142 neg_mean_loss: 8.591242584097142 node_ip: 127.0.0.1 pid: 45918 time_since_restore: 10.2047119140625 time_this_iter_s: 0.1085958480834961 time_total_s: 10.2047119140625 timestamp: 1658499564 timesteps_since_restore: 0 training_iteration: 94 trial_id: 421d81ee warmup_time: 0.002989053726196289 Result for objective_4208c6aa: date: 2022-07-22_15-19-25 done: true experiment_id: ce292c59f7c74e1ab42de6004eb3d846 experiment_tag: 6_activation=relu,height=-41.6248,steps=100,width=14.4351 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 5.907010419420164 neg_mean_loss: -5.907010419420164 node_ip: 127.0.0.1 pid: 45914 time_since_restore: 10.811041831970215 time_this_iter_s: 0.10398101806640625 time_total_s: 10.811041831970215 timestamp: 1658499565 timesteps_since_restore: 0 training_iteration: 100 trial_id: 4208c6aa warmup_time: 0.002707958221435547 Result for objective_421aeb64: date: 2022-07-22_15-19-25 done: true experiment_id: a4b264c05660423ea645cef7a9cd38bb experiment_tag: 7_activation=relu,height=-94.9771,steps=100,width=17.6810 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 0.559098189196817 neg_mean_loss: -0.559098189196817 node_ip: 127.0.0.1 pid: 45917 time_since_restore: 10.850210905075073 time_this_iter_s: 0.10677385330200195 time_total_s: 10.850210905075073 timestamp: 1658499565 timesteps_since_restore: 0 training_iteration: 100 trial_id: 421aeb64 warmup_time: 0.0031020641326904297 Result for objective_421d81ee: date: 2022-07-22_15-19-25 done: true experiment_id: 836a62c82d6c467fb232414586872fd0 experiment_tag: 8_activation=tanh,height=-96.7222,steps=100,width=13.1718 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -8.596112689018389 neg_mean_loss: 8.596112689018389 node_ip: 127.0.0.1 pid: 45918 time_since_restore: 10.845621109008789 time_this_iter_s: 0.10427713394165039 time_total_s: 10.845621109008789 timestamp: 1658499565 timesteps_since_restore: 0 training_iteration: 100 trial_id: 421d81ee warmup_time: 0.002989053726196289
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 13.171830039895717, 'height': -96.72215542618497, 'activation': 'tanh'}
This example demonstrates the usage of Frugal Optimization for Cost-related Hyperparameters (CFO) with Ray Tune.
We now define the search algorithm as built from CFO, constrained to a maximum of 4
concurrent trials with a ConcurrencyLimiter.
algo = CFO()
algo = ConcurrencyLimiter(algo, max_concurrent=4)
The number of samples is the number of hyperparameter combinations that will be
tried out. This Tune run is set to 1000 samples.
(you can decrease this if it takes too long on your machine).
num_samples = 1000
# If 1000 samples take too long, you can reduce this number.
# We override this number here for our smoke tests.
num_samples = 10
Next we define a search space. The critical assumption is that the optimal hyperparameters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time.
search_config = {
"steps": 100,
"width": tune.uniform(0, 20),
"height": tune.uniform(-100, 100),
"activation": tune.choice(["relu, tanh"])
}
Finally, we run the experiment to "min"imize the "mean_loss" of the objective
by searching search_config via algo, num_samples times. This previous sentence
is fully characterizes the search problem we aim to solve. With this in mind,
notice how efficient it is to execute tuner.fit().
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
param_space=search_config,
)
results = tuner.fit()
| Trial name | status | loc | activation | height | steps | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|---|
| objective_4bc281fe | TERMINATED | 127.0.0.1:45958 | relu, tanh | 29.4532 | 100 | 1.94864 | 4.43814 | 100 | 10.7193 | 99 | -4.43814 |
| objective_4d4ed536 | TERMINATED | 127.0.0.1:45963 | relu, tanh | -26.2681 | 100 | 7.25618 | -1.48952 | 100 | 11.1932 | 99 | 1.48952 |
| objective_4d5064aa | TERMINATED | 127.0.0.1:45964 | relu, tanh | -46.7215 | 100 | 15.1097 | -3.60574 | 100 | 11.1983 | 99 | 3.60574 |
| objective_4d51b38c | TERMINATED | 127.0.0.1:45965 | relu, tanh | -97.2777 | 100 | 15.1242 | -8.66142 | 100 | 11.1977 | 99 | 8.66142 |
| objective_53b39e84 | TERMINATED | 127.0.0.1:45983 | relu, tanh | 25.7812 | 100 | 1.77704 | 4.11597 | 100 | 10.7964 | 99 | -4.11597 |
| objective_558c6132 | TERMINATED | 127.0.0.1:45990 | relu, tanh | 33.1252 | 100 | 2.12024 | 4.76727 | 100 | 10.7659 | 99 | -4.76727 |
| objective_558f27aa | TERMINATED | 127.0.0.1:45991 | relu, tanh | 40.929 | 100 | 0.0138032 | 13.8907 | 100 | 10.785 | 99 | -13.8907 |
| objective_5592ca68 | TERMINATED | 127.0.0.1:45992 | relu, tanh | 17.9775 | 100 | 3.88348 | 3.05125 | 100 | 10.7802 | 99 | -3.05125 |
| objective_5b999414 | TERMINATED | 127.0.0.1:46011 | relu, tanh | 11.0354 | 100 | 3.27423 | 2.40281 | 100 | 12.6262 | 99 | -2.40281 |
| objective_5d8462c2 | TERMINATED | 127.0.0.1:46018 | relu, tanh | 24.9195 | 100 | 4.49273 | 3.71184 | 100 | 10.7339 | 99 | -3.71184 |
Result for objective_4bc281fe: date: 2022-07-22_15-19-30 done: false experiment_id: 129b24a058ee40fd991fb40844988057 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 13.945321626643842 neg_mean_loss: -13.945321626643842 node_ip: 127.0.0.1 pid: 45958 time_since_restore: 0.10440444946289062 time_this_iter_s: 0.10440444946289062 time_total_s: 0.10440444946289062 timestamp: 1658499570 timesteps_since_restore: 0 training_iteration: 1 trial_id: 4bc281fe warmup_time: 0.003987789154052734 Result for objective_4d4ed536: date: 2022-07-22_15-19-33 done: false experiment_id: 6c18c359d1574792ac99ed37265d5d87 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 8.373189005362395 neg_mean_loss: -8.373189005362395 node_ip: 127.0.0.1 pid: 45963 time_since_restore: 0.10463309288024902 time_this_iter_s: 0.10463309288024902 time_total_s: 0.10463309288024902 timestamp: 1658499573 timesteps_since_restore: 0 training_iteration: 1 trial_id: 4d4ed536 warmup_time: 0.0028281211853027344 Result for objective_4d51b38c: date: 2022-07-22_15-19-33 done: false experiment_id: 5c9cf6fc35c4411b8ec13e0b90b791a4 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 1.2722313329577961 neg_mean_loss: -1.2722313329577961 node_ip: 127.0.0.1 pid: 45965 time_since_restore: 0.10346817970275879 time_this_iter_s: 0.10346817970275879 time_total_s: 0.10346817970275879 timestamp: 1658499573 timesteps_since_restore: 0 training_iteration: 1 trial_id: 4d51b38c warmup_time: 0.0027761459350585938 Result for objective_4d5064aa: date: 2022-07-22_15-19-33 done: false experiment_id: 93ed3aa2df8c4f71b0b1e9437c062ccf hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 6.327848408616047 neg_mean_loss: -6.327848408616047 node_ip: 127.0.0.1 pid: 45964 time_since_restore: 0.10395598411560059 time_this_iter_s: 0.10395598411560059 time_total_s: 0.10395598411560059 timestamp: 1658499573 timesteps_since_restore: 0 training_iteration: 1 trial_id: 4d5064aa warmup_time: 0.002546072006225586 Result for objective_4bc281fe: date: 2022-07-22_15-19-35 done: false experiment_id: 129b24a058ee40fd991fb40844988057 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 4.9297078000588614 neg_mean_loss: -4.9297078000588614 node_ip: 127.0.0.1 pid: 45958 time_since_restore: 5.122281312942505 time_this_iter_s: 0.10893416404724121 time_total_s: 5.122281312942505 timestamp: 1658499575 timesteps_since_restore: 0 training_iteration: 48 trial_id: 4bc281fe warmup_time: 0.003987789154052734 Result for objective_4d4ed536: date: 2022-07-22_15-19-38 done: false experiment_id: 6c18c359d1574792ac99ed37265d5d87 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -1.3419437208788234 neg_mean_loss: 1.3419437208788234 node_ip: 127.0.0.1 pid: 45963 time_since_restore: 5.132510185241699 time_this_iter_s: 0.10718894004821777 time_total_s: 5.132510185241699 timestamp: 1658499578 timesteps_since_restore: 0 training_iteration: 48 trial_id: 4d4ed536 warmup_time: 0.0028281211853027344 Result for objective_4d51b38c: date: 2022-07-22_15-19-38 done: false experiment_id: 5c9cf6fc35c4411b8ec13e0b90b791a4 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -8.58904124947806 neg_mean_loss: 8.58904124947806 node_ip: 127.0.0.1 pid: 45965 time_since_restore: 5.1296000480651855 time_this_iter_s: 0.10642719268798828 time_total_s: 5.1296000480651855 timestamp: 1658499578 timesteps_since_restore: 0 training_iteration: 48 trial_id: 4d51b38c warmup_time: 0.0027761459350585938 Result for objective_4d5064aa: date: 2022-07-22_15-19-38 done: false experiment_id: 93ed3aa2df8c4f71b0b1e9437c062ccf hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: -3.5332931133029772 neg_mean_loss: 3.5332931133029772 node_ip: 127.0.0.1 pid: 45964 time_since_restore: 5.1527419090271 time_this_iter_s: 0.10794281959533691 time_total_s: 5.1527419090271 timestamp: 1658499578 timesteps_since_restore: 0 training_iteration: 48 trial_id: 4d5064aa warmup_time: 0.002546072006225586 Result for objective_4bc281fe: date: 2022-07-22_15-19-41 done: false experiment_id: 129b24a058ee40fd991fb40844988057 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 4.462994199505408 neg_mean_loss: -4.462994199505408 node_ip: 127.0.0.1 pid: 45958 time_since_restore: 10.180925369262695 time_this_iter_s: 0.10581827163696289 time_total_s: 10.180925369262695 timestamp: 1658499581 timesteps_since_restore: 0 training_iteration: 95 trial_id: 4bc281fe warmup_time: 0.003987789154052734 Result for objective_4bc281fe: date: 2022-07-22_15-19-41 done: true experiment_id: 129b24a058ee40fd991fb40844988057 experiment_tag: 1_activation=relu_tanh,height=29.4532,steps=100,width=1.9486 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 4.438137591322411 neg_mean_loss: -4.438137591322411 node_ip: 127.0.0.1 pid: 45958 time_since_restore: 10.71930718421936 time_this_iter_s: 0.10692596435546875 time_total_s: 10.71930718421936 timestamp: 1658499581 timesteps_since_restore: 0 training_iteration: 100 trial_id: 4bc281fe warmup_time: 0.003987789154052734 Result for objective_4d4ed536: date: 2022-07-22_15-19-43 done: false experiment_id: 6c18c359d1574792ac99ed37265d5d87 hostname: Kais-MacBook-Pro.local iterations: 93 iterations_since_restore: 94 mean_loss: -1.48078832328251 neg_mean_loss: 1.48078832328251 node_ip: 127.0.0.1 pid: 45963 time_since_restore: 10.033550262451172 time_this_iter_s: 0.10465312004089355 time_total_s: 10.033550262451172 timestamp: 1658499583 timesteps_since_restore: 0 training_iteration: 94 trial_id: 4d4ed536 warmup_time: 0.0028281211853027344 Result for objective_4d51b38c: date: 2022-07-22_15-19-43 done: false experiment_id: 5c9cf6fc35c4411b8ec13e0b90b791a4 hostname: Kais-MacBook-Pro.local iterations: 93 iterations_since_restore: 94 mean_loss: -8.657174711294605 neg_mean_loss: 8.657174711294605 node_ip: 127.0.0.1 pid: 45965 time_since_restore: 10.026285171508789 time_this_iter_s: 0.10726714134216309 time_total_s: 10.026285171508789 timestamp: 1658499583 timesteps_since_restore: 0 training_iteration: 94 trial_id: 4d51b38c warmup_time: 0.0027761459350585938 Result for objective_4d5064aa: date: 2022-07-22_15-19-43 done: false experiment_id: 93ed3aa2df8c4f71b0b1e9437c062ccf hostname: Kais-MacBook-Pro.local iterations: 93 iterations_since_restore: 94 mean_loss: -3.601490481891755 neg_mean_loss: 3.601490481891755 node_ip: 127.0.0.1 pid: 45964 time_since_restore: 10.049538135528564 time_this_iter_s: 0.10654830932617188 time_total_s: 10.049538135528564 timestamp: 1658499583 timesteps_since_restore: 0 training_iteration: 94 trial_id: 4d5064aa warmup_time: 0.002546072006225586 Result for objective_53b39e84: date: 2022-07-22_15-19-44 done: false experiment_id: abfeeaaa7eae4ad798c655bed2d1e223 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 13.57812082128205 neg_mean_loss: -13.57812082128205 node_ip: 127.0.0.1 pid: 45983 time_since_restore: 0.10458230972290039 time_this_iter_s: 0.10458230972290039 time_total_s: 0.10458230972290039 timestamp: 1658499584 timesteps_since_restore: 0 training_iteration: 1 trial_id: 53b39e84 warmup_time: 0.002878904342651367 Result for objective_4d4ed536: date: 2022-07-22_15-19-44 done: true experiment_id: 6c18c359d1574792ac99ed37265d5d87 experiment_tag: 2_activation=relu_tanh,height=-26.2681,steps=100,width=7.2562 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -1.489516678620475 neg_mean_loss: 1.489516678620475 node_ip: 127.0.0.1 pid: 45963 time_since_restore: 11.193175077438354 time_this_iter_s: 0.11231780052185059 time_total_s: 11.193175077438354 timestamp: 1658499584 timesteps_since_restore: 0 training_iteration: 100 trial_id: 4d4ed536 warmup_time: 0.0028281211853027344 Result for objective_4d51b38c: date: 2022-07-22_15-19-44 done: true experiment_id: 5c9cf6fc35c4411b8ec13e0b90b791a4 experiment_tag: 4_activation=relu_tanh,height=-97.2777,steps=100,width=15.1242 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -8.661424748129757 neg_mean_loss: 8.661424748129757 node_ip: 127.0.0.1 pid: 45965 time_since_restore: 11.19774603843689 time_this_iter_s: 0.10802578926086426 time_total_s: 11.19774603843689 timestamp: 1658499584 timesteps_since_restore: 0 training_iteration: 100 trial_id: 4d51b38c warmup_time: 0.0027761459350585938 Result for objective_4d5064aa: date: 2022-07-22_15-19-44 done: true experiment_id: 93ed3aa2df8c4f71b0b1e9437c062ccf experiment_tag: 3_activation=relu_tanh,height=-46.7215,steps=100,width=15.1097 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: -3.6057445346106167 neg_mean_loss: 3.6057445346106167 node_ip: 127.0.0.1 pid: 45964 time_since_restore: 11.198285102844238 time_this_iter_s: 0.10615801811218262 time_total_s: 11.198285102844238 timestamp: 1658499584 timesteps_since_restore: 0 training_iteration: 100 trial_id: 4d5064aa warmup_time: 0.002546072006225586 Result for objective_558c6132: date: 2022-07-22_15-19-47 done: false experiment_id: 98df2ca60c5f4a33a3da6870c3d8db4c hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 14.31252243200564 neg_mean_loss: -14.31252243200564 node_ip: 127.0.0.1 pid: 45990 time_since_restore: 0.10317206382751465 time_this_iter_s: 0.10317206382751465 time_total_s: 0.10317206382751465 timestamp: 1658499587 timesteps_since_restore: 0 training_iteration: 1 trial_id: 558c6132 warmup_time: 0.003049135208129883 Result for objective_558f27aa: date: 2022-07-22_15-19-47 done: false experiment_id: 73def1ffc7f443669328b88695453248 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 15.092898207127158 neg_mean_loss: -15.092898207127158 node_ip: 127.0.0.1 pid: 45991 time_since_restore: 0.10495114326477051 time_this_iter_s: 0.10495114326477051 time_total_s: 0.10495114326477051 timestamp: 1658499587 timesteps_since_restore: 0 training_iteration: 1 trial_id: 558f27aa warmup_time: 0.0029478073120117188 Result for objective_5592ca68: date: 2022-07-22_15-19-47 done: false experiment_id: 1de409fb58a84c3bba522620ec7e1fe4 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 12.79774504616053 neg_mean_loss: -12.79774504616053 node_ip: 127.0.0.1 pid: 45992 time_since_restore: 0.10448837280273438 time_this_iter_s: 0.10448837280273438 time_total_s: 0.10448837280273438 timestamp: 1658499587 timesteps_since_restore: 0 training_iteration: 1 trial_id: 5592ca68 warmup_time: 0.0027618408203125 Result for objective_53b39e84: date: 2022-07-22_15-19-49 done: false experiment_id: abfeeaaa7eae4ad798c655bed2d1e223 hostname: Kais-MacBook-Pro.local iterations: 46 iterations_since_restore: 47 mean_loss: 4.668111815725785 neg_mean_loss: -4.668111815725785 node_ip: 127.0.0.1 pid: 45983 time_since_restore: 5.113715171813965 time_this_iter_s: 0.10550904273986816 time_total_s: 5.113715171813965 timestamp: 1658499589 timesteps_since_restore: 0 training_iteration: 47 trial_id: 53b39e84 warmup_time: 0.002878904342651367 Result for objective_558c6132: date: 2022-07-22_15-19-52 done: false experiment_id: 98df2ca60c5f4a33a3da6870c3d8db4c hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 5.2245036670526375 neg_mean_loss: -5.2245036670526375 node_ip: 127.0.0.1 pid: 45990 time_since_restore: 5.1552488803863525 time_this_iter_s: 0.10927176475524902 time_total_s: 5.1552488803863525 timestamp: 1658499592 timesteps_since_restore: 0 training_iteration: 48 trial_id: 558c6132 warmup_time: 0.003049135208129883 Result for objective_558f27aa: date: 2022-07-22_15-19-52 done: false experiment_id: 73def1ffc7f443669328b88695453248 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 14.483670237907232 neg_mean_loss: -14.483670237907232 node_ip: 127.0.0.1 pid: 45991 time_since_restore: 5.174212217330933 time_this_iter_s: 0.10701107978820801 time_total_s: 5.174212217330933 timestamp: 1658499592 timesteps_since_restore: 0 training_iteration: 48 trial_id: 558f27aa warmup_time: 0.0029478073120117188 Result for objective_5592ca68: date: 2022-07-22_15-19-52 done: false experiment_id: 1de409fb58a84c3bba522620ec7e1fe4 hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 3.3171620341364583 neg_mean_loss: -3.3171620341364583 node_ip: 127.0.0.1 pid: 45992 time_since_restore: 5.171269178390503 time_this_iter_s: 0.10331320762634277 time_total_s: 5.171269178390503 timestamp: 1658499592 timesteps_since_restore: 0 training_iteration: 48 trial_id: 5592ca68 warmup_time: 0.0027618408203125 Result for objective_53b39e84: date: 2022-07-22_15-19-54 done: false experiment_id: abfeeaaa7eae4ad798c655bed2d1e223 hostname: Kais-MacBook-Pro.local iterations: 93 iterations_since_restore: 94 mean_loss: 4.148686061272061 neg_mean_loss: -4.148686061272061 node_ip: 127.0.0.1 pid: 45983 time_since_restore: 10.137071132659912 time_this_iter_s: 0.10901093482971191 time_total_s: 10.137071132659912 timestamp: 1658499594 timesteps_since_restore: 0 training_iteration: 94 trial_id: 53b39e84 warmup_time: 0.002878904342651367 Result for objective_53b39e84: date: 2022-07-22_15-19-54 done: true experiment_id: abfeeaaa7eae4ad798c655bed2d1e223 experiment_tag: 5_activation=relu_tanh,height=25.7812,steps=100,width=1.7770 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 4.1159662035734215 neg_mean_loss: -4.1159662035734215 node_ip: 127.0.0.1 pid: 45983 time_since_restore: 10.796404361724854 time_this_iter_s: 0.10543107986450195 time_total_s: 10.796404361724854 timestamp: 1658499594 timesteps_since_restore: 0 training_iteration: 100 trial_id: 53b39e84 warmup_time: 0.002878904342651367 Result for objective_5b999414: date: 2022-07-22_15-19-57 done: false experiment_id: 5548713c4f5046c0a44e1946db840d73 hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 12.103540718376786 neg_mean_loss: -12.103540718376786 node_ip: 127.0.0.1 pid: 46011 time_since_restore: 0.10435795783996582 time_this_iter_s: 0.10435795783996582 time_total_s: 0.10435795783996582 timestamp: 1658499597 timesteps_since_restore: 0 training_iteration: 1 trial_id: 5b999414 warmup_time: 0.002862691879272461 Result for objective_558c6132: date: 2022-07-22_15-19-57 done: false experiment_id: 98df2ca60c5f4a33a3da6870c3d8db4c hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 4.790299222921972 neg_mean_loss: -4.790299222921972 node_ip: 127.0.0.1 pid: 45990 time_since_restore: 10.228116989135742 time_this_iter_s: 0.146165132522583 time_total_s: 10.228116989135742 timestamp: 1658499597 timesteps_since_restore: 0 training_iteration: 95 trial_id: 558c6132 warmup_time: 0.003049135208129883 Result for objective_558f27aa: date: 2022-07-22_15-19-57 done: false experiment_id: 73def1ffc7f443669328b88695453248 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 13.944411303108408 neg_mean_loss: -13.944411303108408 node_ip: 127.0.0.1 pid: 45991 time_since_restore: 10.248456001281738 time_this_iter_s: 0.13899707794189453 time_total_s: 10.248456001281738 timestamp: 1658499597 timesteps_since_restore: 0 training_iteration: 95 trial_id: 558f27aa warmup_time: 0.0029478073120117188 Result for objective_5592ca68: date: 2022-07-22_15-19-57 done: false experiment_id: 1de409fb58a84c3bba522620ec7e1fe4 hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 3.064378230421662 neg_mean_loss: -3.064378230421662 node_ip: 127.0.0.1 pid: 45992 time_since_restore: 10.244185209274292 time_this_iter_s: 0.16223502159118652 time_total_s: 10.244185209274292 timestamp: 1658499597 timesteps_since_restore: 0 training_iteration: 95 trial_id: 5592ca68 warmup_time: 0.0027618408203125 Result for objective_558c6132: date: 2022-07-22_15-19-58 done: true experiment_id: 98df2ca60c5f4a33a3da6870c3d8db4c experiment_tag: 6_activation=relu_tanh,height=33.1252,steps=100,width=2.1202 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 4.767266385536331 neg_mean_loss: -4.767266385536331 node_ip: 127.0.0.1 pid: 45990 time_since_restore: 10.765881776809692 time_this_iter_s: 0.10750794410705566 time_total_s: 10.765881776809692 timestamp: 1658499598 timesteps_since_restore: 0 training_iteration: 100 trial_id: 558c6132 warmup_time: 0.003049135208129883 Result for objective_558f27aa: date: 2022-07-22_15-19-58 done: true experiment_id: 73def1ffc7f443669328b88695453248 experiment_tag: 7_activation=relu_tanh,height=40.9290,steps=100,width=0.0138 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 13.890665978269285 neg_mean_loss: -13.890665978269285 node_ip: 127.0.0.1 pid: 45991 time_since_restore: 10.784975051879883 time_this_iter_s: 0.10729503631591797 time_total_s: 10.784975051879883 timestamp: 1658499598 timesteps_since_restore: 0 training_iteration: 100 trial_id: 558f27aa warmup_time: 0.0029478073120117188 Result for objective_5592ca68: date: 2022-07-22_15-19-58 done: true experiment_id: 1de409fb58a84c3bba522620ec7e1fe4 experiment_tag: 8_activation=relu_tanh,height=17.9775,steps=100,width=3.8835 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 3.0512532903599245 neg_mean_loss: -3.0512532903599245 node_ip: 127.0.0.1 pid: 45992 time_since_restore: 10.780152082443237 time_this_iter_s: 0.10820770263671875 time_total_s: 10.780152082443237 timestamp: 1658499598 timesteps_since_restore: 0 training_iteration: 100 trial_id: 5592ca68 warmup_time: 0.0027618408203125 Result for objective_5d8462c2: date: 2022-07-22_15-20-00 done: false experiment_id: 37975742a6894daeb580d222effadf3c hostname: Kais-MacBook-Pro.local iterations: 0 iterations_since_restore: 1 mean_loss: 13.491949373944276 neg_mean_loss: -13.491949373944276 node_ip: 127.0.0.1 pid: 46018 time_since_restore: 0.1021728515625 time_this_iter_s: 0.1021728515625 time_total_s: 0.1021728515625 timestamp: 1658499600 timesteps_since_restore: 0 training_iteration: 1 trial_id: 5d8462c2 warmup_time: 0.0030527114868164062 Result for objective_5b999414: date: 2022-07-22_15-20-02 done: false experiment_id: 5548713c4f5046c0a44e1946db840d73 hostname: Kais-MacBook-Pro.local iterations: 29 iterations_since_restore: 30 mean_loss: 3.0563505542773735 neg_mean_loss: -3.0563505542773735 node_ip: 127.0.0.1 pid: 46011 time_since_restore: 5.116016149520874 time_this_iter_s: 0.1060340404510498 time_total_s: 5.116016149520874 timestamp: 1658499602 timesteps_since_restore: 0 training_iteration: 30 trial_id: 5b999414 warmup_time: 0.002862691879272461 Result for objective_5d8462c2: date: 2022-07-22_15-20-05 done: false experiment_id: 37975742a6894daeb580d222effadf3c hostname: Kais-MacBook-Pro.local iterations: 47 iterations_since_restore: 48 mean_loss: 3.9441144421537513 neg_mean_loss: -3.9441144421537513 node_ip: 127.0.0.1 pid: 46018 time_since_restore: 5.143904685974121 time_this_iter_s: 0.1046457290649414 time_total_s: 5.143904685974121 timestamp: 1658499605 timesteps_since_restore: 0 training_iteration: 48 trial_id: 5d8462c2 warmup_time: 0.0030527114868164062 Result for objective_5b999414: date: 2022-07-22_15-20-07 done: false experiment_id: 5548713c4f5046c0a44e1946db840d73 hostname: Kais-MacBook-Pro.local iterations: 76 iterations_since_restore: 77 mean_loss: 2.4898772989369 neg_mean_loss: -2.4898772989369 node_ip: 127.0.0.1 pid: 46011 time_since_restore: 10.162328004837036 time_this_iter_s: 0.10597825050354004 time_total_s: 10.162328004837036 timestamp: 1658499607 timesteps_since_restore: 0 training_iteration: 77 trial_id: 5b999414 warmup_time: 0.002862691879272461 Result for objective_5b999414: date: 2022-07-22_15-20-09 done: true experiment_id: 5548713c4f5046c0a44e1946db840d73 experiment_tag: 9_activation=relu_tanh,height=11.0354,steps=100,width=3.2742 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 2.4028084118568103 neg_mean_loss: -2.4028084118568103 node_ip: 127.0.0.1 pid: 46011 time_since_restore: 12.626188039779663 time_this_iter_s: 0.10697293281555176 time_total_s: 12.626188039779663 timestamp: 1658499609 timesteps_since_restore: 0 training_iteration: 100 trial_id: 5b999414 warmup_time: 0.002862691879272461 Result for objective_5d8462c2: date: 2022-07-22_15-20-10 done: false experiment_id: 37975742a6894daeb580d222effadf3c hostname: Kais-MacBook-Pro.local iterations: 94 iterations_since_restore: 95 mean_loss: 3.723261470545891 neg_mean_loss: -3.723261470545891 node_ip: 127.0.0.1 pid: 46018 time_since_restore: 10.198927879333496 time_this_iter_s: 0.10776281356811523 time_total_s: 10.198927879333496 timestamp: 1658499610 timesteps_since_restore: 0 training_iteration: 95 trial_id: 5d8462c2 warmup_time: 0.0030527114868164062 Result for objective_5d8462c2: date: 2022-07-22_15-20-11 done: true experiment_id: 37975742a6894daeb580d222effadf3c experiment_tag: 10_activation=relu_tanh,height=24.9195,steps=100,width=4.4927 hostname: Kais-MacBook-Pro.local iterations: 99 iterations_since_restore: 100 mean_loss: 3.711835922326217 neg_mean_loss: -3.711835922326217 node_ip: 127.0.0.1 pid: 46018 time_since_restore: 10.733852863311768 time_this_iter_s: 0.10374593734741211 time_total_s: 10.733852863311768 timestamp: 1658499611 timesteps_since_restore: 0 training_iteration: 100 trial_id: 5d8462c2 warmup_time: 0.0030527114868164062
Here are the hyperparameters found to minimize the mean loss of the defined objective.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 15.124213652112319, 'height': -97.27768667042203, 'activation': 'relu, tanh'}
ray.shutdown()