In this tutorial we introduce Ax, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Ax and, as a result, allow you to seamlessly scale up a Ax optimization process - without sacrificing performance.
Ax is a platform for optimizing any kind of experiment, including machine learning experiments, A/B tests, and simulations. Ax can optimize discrete configurations (e.g., variants of an A/B test) using multi-armed bandit optimization, and continuous/ordered configurations (e.g. float/int parameters) using Bayesian optimization. Results of A/B tests and simulations with reinforcement learning agents often exhibit high amounts of noise. Ax supports state-of-the-art algorithms which work better than traditional Bayesian optimization in high-noise settings. Ax also supports multi-objective and constrained optimization which are common to real-world problems (e.g. improving load time without increasing data use). Ax belongs to the domain of "derivative-free" and "black-box" optimization.
In this example we minimize a simple objective to briefly demonstrate the usage of AxSearch with Ray Tune via AxSearch. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume ax-platform==0.2.4 library is installed withe python version >= 3.7. To learn more, please refer to the Ax website.
# !pip install ray[tune]
!pip install ax-platform==0.2.4
Requirement already satisfied: ax-platform==0.2.4 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (0.2.4) Requirement already satisfied: botorch==0.6.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (0.6.2) Requirement already satisfied: jinja2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (3.0.3) Requirement already satisfied: pandas in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (1.3.5) Requirement already satisfied: scipy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (1.4.1) Requirement already satisfied: plotly in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (5.6.0) Requirement already satisfied: scikit-learn in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (0.24.2) Requirement already satisfied: typeguard in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (2.13.3) Requirement already satisfied: gpytorch>=1.6 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from botorch==0.6.2->ax-platform==0.2.4) (1.6.0) Requirement already satisfied: torch>=1.9 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from botorch==0.6.2->ax-platform==0.2.4) (1.9.0) Requirement already satisfied: multipledispatch in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from botorch==0.6.2->ax-platform==0.2.4) (0.6.0) Requirement already satisfied: MarkupSafe>=2.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from jinja2->ax-platform==0.2.4) (2.0.1) Requirement already satisfied: pytz>=2017.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas->ax-platform==0.2.4) (2022.1) Requirement already satisfied: numpy>=1.17.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas->ax-platform==0.2.4) (1.21.6) Requirement already satisfied: python-dateutil>=2.7.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas->ax-platform==0.2.4) (2.8.2) Requirement already satisfied: tenacity>=6.2.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from plotly->ax-platform==0.2.4) (8.0.1) Requirement already satisfied: six in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from plotly->ax-platform==0.2.4) (1.16.0) Requirement already satisfied: joblib>=0.11 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn->ax-platform==0.2.4) (1.1.0) Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn->ax-platform==0.2.4) (3.0.0) Requirement already satisfied: typing-extensions in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from torch>=1.9->botorch==0.6.2->ax-platform==0.2.4) (4.1.1) WARNING: There was an error checking the latest version of pip.
Click below to see all the imports we need for this example. You can also launch directly into a Binder instance to run this notebook yourself. Just click on the rocket symbol at the top of the navigation.
import numpy as np
import time
import ray
from ray import train, tune
from ray.tune.search.ax import AxSearch
Let's start by defining a classic benchmark for global optimization.
The form here is explicit for demonstration, yet it is typically a black-box.
We artificially sleep for a bit (0.02 seconds) to simulate a long-running ML experiment.
This setup assumes that we're running multiple steps of an experiment and try to tune 6-dimensions of the x hyperparameter.
def landscape(x):
"""
Hartmann 6D function containing 6 local minima.
It is a classic benchmark for developing global optimization algorithms.
"""
alpha = np.array([1.0, 1.2, 3.0, 3.2])
A = np.array(
[
[10, 3, 17, 3.5, 1.7, 8],
[0.05, 10, 17, 0.1, 8, 14],
[3, 3.5, 1.7, 10, 17, 8],
[17, 8, 0.05, 10, 0.1, 14],
]
)
P = 10 ** (-4) * np.array(
[
[1312, 1696, 5569, 124, 8283, 5886],
[2329, 4135, 8307, 3736, 1004, 9991],
[2348, 1451, 3522, 2883, 3047, 6650],
[4047, 8828, 8732, 5743, 1091, 381],
]
)
y = 0.0
for j, alpha_j in enumerate(alpha):
t = 0
for k in range(6):
t += A[j, k] * ((x[k] - P[j, k]) ** 2)
y -= alpha_j * np.exp(-t)
return y
Next, our objective function takes a Tune config, evaluates the landscape of our experiment in a training loop,
and uses train.report to report the landscape back to Tune.
def objective(config):
for i in range(config["iterations"]):
x = np.array([config.get("x{}".format(i + 1)) for i in range(6)])
train.report(
{"timesteps_total": i, "landscape": landscape(x), "l2norm": np.sqrt((x ** 2).sum())}
)
time.sleep(0.02)
Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparamters may be difficult to find in a short amount of time.
search_space = {
"iterations":100,
"x1": tune.uniform(0.0, 1.0),
"x2": tune.uniform(0.0, 1.0),
"x3": tune.uniform(0.0, 1.0),
"x4": tune.uniform(0.0, 1.0),
"x5": tune.uniform(0.0, 1.0),
"x6": tune.uniform(0.0, 1.0)
}
ray.init(configure_logging=False)
Now we define the search algorithm from AxSearch. If you want to constrain your parameters or even the space of outcomes, that can be easily done by passing the argumentsas below.
algo = AxSearch(
parameter_constraints=["x1 + x2 <= 2.0"],
outcome_constraints=["l2norm <= 1.25"],
)
We also use ConcurrencyLimiter to constrain to 4 concurrent trials.
algo = tune.search.ConcurrencyLimiter(algo, max_concurrent=4)
The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to 1000 samples.
You can decrease this if it takes too long on your machine, or you can set a time limit easily through stop argument in the train.RunConfig() as we will show here.
num_samples = 100
stop_timesteps = 200
# Reducing samples for smoke tests
num_samples = 10
Finally, we run the experiment to find the global minimum of the provided landscape (which contains 5 false minima). The argument to metric, "landscape", is provided via the objective function's session.report. The experiment "min"imizes the "mean_loss" of the landscape by searching within search_space via algo, num_samples times or when "timesteps_total": stop_timesteps. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute tuner.fit().
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="landscape",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
run_config=train.RunConfig(
name="ax",
stop={"timesteps_total": stop_timesteps}
),
param_space=search_space,
)
results = tuner.fit()
[INFO 07-22 15:04:18] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 6 decimal points. [INFO 07-22 15:04:18] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[FixedParameter(name='iterations', parameter_type=INT, value=100), RangeParameter(name='x1', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x3', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x4', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x5', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x6', parameter_type=FLOAT, range=[0.0, 1.0])], parameter_constraints=[ParameterConstraint(1.0*x1 + 1.0*x2 <= 2.0)]). [INFO 07-22 15:04:18] ax.modelbridge.dispatch_utils: Using Bayesian optimization since there are more ordered parameters than there are categories for the unordered categorical parameters. [INFO 07-22 15:04:18] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 12 trials, GPEI for subsequent trials]). Iterations after 12 will take longer to generate due to model-fitting. Detected sequential enforcement. Be sure to use a ConcurrencyLimiter.
| Trial name | status | loc | iterations | x1 | x2 | x3 | x4 | x5 | x6 | iter | total time (s) | ts | landscape | l2norm |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| objective_2dfbe86a | TERMINATED | 127.0.0.1:44721 | 100 | 0.0558336 | 0.0896192 | 0.958956 | 0.234474 | 0.174516 | 0.970311 | 100 | 2.57372 | 99 | -0.805233 | 1.39917 |
| objective_2fa776c0 | TERMINATED | 127.0.0.1:44726 | 100 | 0.744772 | 0.754537 | 0.0950125 | 0.273877 | 0.0966829 | 0.368943 | 100 | 2.6361 | 99 | -0.11286 | 1.16341 |
| objective_2fabaa1a | TERMINATED | 127.0.0.1:44727 | 100 | 0.405704 | 0.374626 | 0.935628 | 0.222185 | 0.787212 | 0.00812439 | 100 | 2.62393 | 99 | -0.11348 | 1.35995 |
| objective_2faee7c0 | TERMINATED | 127.0.0.1:44728 | 100 | 0.664728 | 0.207519 | 0.359514 | 0.704578 | 0.755882 | 0.812402 | 100 | 2.62069 | 99 | -0.0119837 | 1.53035 |
| objective_313d3d3a | TERMINATED | 127.0.0.1:44747 | 100 | 0.0418746 | 0.992783 | 0.906027 | 0.594429 | 0.825393 | 0.646362 | 100 | 3.16233 | 99 | -0.00677976 | 1.80573 |
| objective_32c9acd8 | TERMINATED | 127.0.0.1:44726 | 100 | 0.126064 | 0.703408 | 0.344681 | 0.337363 | 0.401396 | 0.679202 | 100 | 3.12119 | 99 | -0.904622 | 1.16864 |
| objective_32cf8ca2 | TERMINATED | 127.0.0.1:44756 | 100 | 0.0910936 | 0.304138 | 0.869848 | 0.405435 | 0.567922 | 0.228608 | 100 | 2.70791 | 99 | -0.146532 | 1.18178 |
| objective_32d8dd20 | TERMINATED | 127.0.0.1:44758 | 100 | 0.603178 | 0.409057 | 0.729056 | 0.0825984 | 0.572948 | 0.508304 | 100 | 2.64158 | 99 | -0.247223 | 1.28691 |
| objective_34adf04a | TERMINATED | 127.0.0.1:44768 | 100 | 0.454189 | 0.271772 | 0.530871 | 0.991841 | 0.691843 | 0.472366 | 100 | 2.70327 | 99 | -0.0132915 | 1.49917 |
| objective_34b7abda | TERMINATED | 127.0.0.1:44771 | 100 | 0.265264 | 0.924884 | 0.151716 | 0.436026 | 0.85731 | 0.0898102 | 100 | 2.68521 | 99 | -1.66244 | 1.37185 |
[INFO 07-22 15:04:19] ax.service.ax_client: Generated new trial 0 with parameters {'x1': 0.055834, 'x2': 0.089619, 'x3': 0.958956, 'x4': 0.234474, 'x5': 0.174516, 'x6': 0.970311, 'iterations': 100}.
[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 1 with parameters {'x1': 0.744772, 'x2': 0.754537, 'x3': 0.095012, 'x4': 0.273877, 'x5': 0.096683, 'x6': 0.368943, 'iterations': 100}.
[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 2 with parameters {'x1': 0.405704, 'x2': 0.374626, 'x3': 0.935628, 'x4': 0.222185, 'x5': 0.787212, 'x6': 0.008124, 'iterations': 100}.
[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 3 with parameters {'x1': 0.664728, 'x2': 0.207519, 'x3': 0.359514, 'x4': 0.704578, 'x5': 0.755882, 'x6': 0.812402, 'iterations': 100}.
Result for objective_2dfbe86a: date: 2022-07-22_15-04-22 done: false experiment_id: 4ef8a12ac94a4f4fa483ec18e347967f hostname: Kais-MacBook-Pro.local iterations_since_restore: 1 l2norm: 1.3991721132671366 landscape: -0.8052333562869153 node_ip: 127.0.0.1 pid: 44721 time_since_restore: 0.00022912025451660156 time_this_iter_s: 0.00022912025451660156 time_total_s: 0.00022912025451660156 timestamp: 1658498662 timesteps_since_restore: 0 timesteps_total: 0 training_iteration: 1 trial_id: 2dfbe86a warmup_time: 0.0035619735717773438
[INFO 07-22 15:04:24] ax.service.ax_client: Completed trial 0 with data: {'landscape': (-0.805233, None), 'l2norm': (1.399172, None)}.
[INFO 07-22 15:04:24] ax.service.ax_client: Generated new trial 4 with parameters {'x1': 0.041875, 'x2': 0.992783, 'x3': 0.906027, 'x4': 0.594429, 'x5': 0.825393, 'x6': 0.646362, 'iterations': 100}.
Result for objective_2faee7c0: date: 2022-07-22_15-04-24 done: false experiment_id: 3699644e85ac439cb7c1a36ed0976307 hostname: Kais-MacBook-Pro.local iterations_since_restore: 1 l2norm: 1.530347488145437 landscape: -0.011983676977099367 node_ip: 127.0.0.1 pid: 44728 time_since_restore: 0.00022292137145996094 time_this_iter_s: 0.00022292137145996094 time_total_s: 0.00022292137145996094 timestamp: 1658498664 timesteps_since_restore: 0 timesteps_total: 0 training_iteration: 1 trial_id: 2faee7c0 warmup_time: 0.0027179718017578125 Result for objective_2fa776c0: date: 2022-07-22_15-04-24 done: false experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7 hostname: Kais-MacBook-Pro.local iterations_since_restore: 1 l2norm: 1.1634068454629019 landscape: -0.11285961764770336 node_ip: 127.0.0.1 pid: 44726 time_since_restore: 0.000225067138671875 time_this_iter_s: 0.000225067138671875 time_total_s: 0.000225067138671875 timestamp: 1658498664 timesteps_since_restore: 0 timesteps_total: 0 training_iteration: 1 trial_id: 2fa776c0 warmup_time: 0.0026290416717529297 Result for objective_2dfbe86a: date: 2022-07-22_15-04-24 done: true experiment_id: 4ef8a12ac94a4f4fa483ec18e347967f experiment_tag: 1_iterations=100,x1=0.0558,x2=0.0896,x3=0.9590,x4=0.2345,x5=0.1745,x6=0.9703 hostname: Kais-MacBook-Pro.local iterations_since_restore: 100 l2norm: 1.3991721132671366 landscape: -0.8052333562869153 node_ip: 127.0.0.1 pid: 44721 time_since_restore: 2.573719024658203 time_this_iter_s: 0.0251619815826416 time_total_s: 2.573719024658203 timestamp: 1658498664 timesteps_since_restore: 0 timesteps_total: 99 training_iteration: 100 trial_id: 2dfbe86a warmup_time: 0.0035619735717773438 Result for objective_2fabaa1a: date: 2022-07-22_15-04-24 done: false experiment_id: eb9287e4fe5f44c7868dc943e2642312 hostname: Kais-MacBook-Pro.local iterations_since_restore: 1 l2norm: 1.3599537840291782 landscape: -0.11348012497414121 node_ip: 127.0.0.1 pid: 44727 time_since_restore: 0.00022077560424804688 time_this_iter_s: 0.00022077560424804688 time_total_s: 0.00022077560424804688 timestamp: 1658498664 timesteps_since_restore: 0 timesteps_total: 0 training_iteration: 1 trial_id: 2fabaa1a warmup_time: 0.0025510787963867188
[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 3 with data: {'landscape': (-0.011984, None), 'l2norm': (1.530347, None)}.
[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 5 with parameters {'x1': 0.126064, 'x2': 0.703408, 'x3': 0.344681, 'x4': 0.337363, 'x5': 0.401396, 'x6': 0.679202, 'iterations': 100}.
[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 1 with data: {'landscape': (-0.11286, None), 'l2norm': (1.163407, None)}.
[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 6 with parameters {'x1': 0.091094, 'x2': 0.304138, 'x3': 0.869848, 'x4': 0.405435, 'x5': 0.567922, 'x6': 0.228608, 'iterations': 100}.
[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 2 with data: {'landscape': (-0.11348, None), 'l2norm': (1.359954, None)}.
[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 7 with parameters {'x1': 0.603178, 'x2': 0.409057, 'x3': 0.729056, 'x4': 0.082598, 'x5': 0.572948, 'x6': 0.508304, 'iterations': 100}.
Result for objective_313d3d3a: date: 2022-07-22_15-04-27 done: false experiment_id: fa7afd557e154fbebe4f54d8eedb3573 hostname: Kais-MacBook-Pro.local iterations_since_restore: 1 l2norm: 1.805729990121368 landscape: -0.006779757704679272 node_ip: 127.0.0.1 pid: 44747 time_since_restore: 0.00021076202392578125 time_this_iter_s: 0.00021076202392578125 time_total_s: 0.00021076202392578125 timestamp: 1658498667 timesteps_since_restore: 0 timesteps_total: 0 training_iteration: 1 trial_id: 313d3d3a warmup_time: 0.0029790401458740234 Result for objective_2faee7c0: date: 2022-07-22_15-04-27 done: true experiment_id: 3699644e85ac439cb7c1a36ed0976307 experiment_tag: 4_iterations=100,x1=0.6647,x2=0.2075,x3=0.3595,x4=0.7046,x5=0.7559,x6=0.8124 hostname: Kais-MacBook-Pro.local iterations_since_restore: 100 l2norm: 1.530347488145437 landscape: -0.011983676977099367 node_ip: 127.0.0.1 pid: 44728 time_since_restore: 2.6206929683685303 time_this_iter_s: 0.027359962463378906 time_total_s: 2.6206929683685303 timestamp: 1658498667 timesteps_since_restore: 0 timesteps_total: 99 training_iteration: 100 trial_id: 2faee7c0 warmup_time: 0.0027179718017578125 Result for objective_2fa776c0: date: 2022-07-22_15-04-27 done: true experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7 experiment_tag: 2_iterations=100,x1=0.7448,x2=0.7545,x3=0.0950,x4=0.2739,x5=0.0967,x6=0.3689 hostname: Kais-MacBook-Pro.local iterations_since_restore: 100 l2norm: 1.1634068454629019 landscape: -0.11285961764770336 node_ip: 127.0.0.1 pid: 44726 time_since_restore: 2.6361019611358643 time_this_iter_s: 0.0264589786529541 time_total_s: 2.6361019611358643 timestamp: 1658498667 timesteps_since_restore: 0 timesteps_total: 99 training_iteration: 100 trial_id: 2fa776c0 warmup_time: 0.0026290416717529297 Result for objective_32c9acd8: date: 2022-07-22_15-04-27 done: false experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7 hostname: Kais-MacBook-Pro.local iterations_since_restore: 1 l2norm: 1.1686440476629836 landscape: -0.9046216637367911 node_ip: 127.0.0.1 pid: 44726 time_since_restore: 0.00020194053649902344 time_this_iter_s: 0.00020194053649902344 time_total_s: 0.00020194053649902344 timestamp: 1658498667 timesteps_since_restore: 0 timesteps_total: 0 training_iteration: 1 trial_id: 32c9acd8 warmup_time: 0.0026290416717529297 Result for objective_2fabaa1a: date: 2022-07-22_15-04-27 done: true experiment_id: eb9287e4fe5f44c7868dc943e2642312 experiment_tag: 3_iterations=100,x1=0.4057,x2=0.3746,x3=0.9356,x4=0.2222,x5=0.7872,x6=0.0081 hostname: Kais-MacBook-Pro.local iterations_since_restore: 100 l2norm: 1.3599537840291782 landscape: -0.11348012497414121 node_ip: 127.0.0.1 pid: 44727 time_since_restore: 2.623929977416992 time_this_iter_s: 0.032716989517211914 time_total_s: 2.623929977416992 timestamp: 1658498667 timesteps_since_restore: 0 timesteps_total: 99 training_iteration: 100 trial_id: 2fabaa1a warmup_time: 0.0025510787963867188 Result for objective_32d8dd20: date: 2022-07-22_15-04-30 done: false experiment_id: 171527593b0f4cbf941c0a03faaf0953 hostname: Kais-MacBook-Pro.local iterations_since_restore: 1 l2norm: 1.2869105702896437 landscape: -0.24722262157458608 node_ip: 127.0.0.1 pid: 44758 time_since_restore: 0.00021886825561523438 time_this_iter_s: 0.00021886825561523438 time_total_s: 0.00021886825561523438 timestamp: 1658498670 timesteps_since_restore: 0 timesteps_total: 0 training_iteration: 1 trial_id: 32d8dd20 warmup_time: 0.002732992172241211 Result for objective_32cf8ca2: date: 2022-07-22_15-04-29 done: false experiment_id: 37610500f6df493aae4e7e46bb21bf09 hostname: Kais-MacBook-Pro.local iterations_since_restore: 1 l2norm: 1.1817810425508524 landscape: -0.14653248187442922 node_ip: 127.0.0.1 pid: 44756 time_since_restore: 0.00025081634521484375 time_this_iter_s: 0.00025081634521484375 time_total_s: 0.00025081634521484375 timestamp: 1658498669 timesteps_since_restore: 0 timesteps_total: 0 training_iteration: 1 trial_id: 32cf8ca2 warmup_time: 0.0032138824462890625
[INFO 07-22 15:04:30] ax.service.ax_client: Completed trial 4 with data: {'landscape': (-0.00678, None), 'l2norm': (1.80573, None)}.
[INFO 07-22 15:04:30] ax.service.ax_client: Generated new trial 8 with parameters {'x1': 0.454189, 'x2': 0.271772, 'x3': 0.530871, 'x4': 0.991841, 'x5': 0.691843, 'x6': 0.472366, 'iterations': 100}.
[INFO 07-22 15:04:30] ax.service.ax_client: Completed trial 5 with data: {'landscape': (-0.904622, None), 'l2norm': (1.168644, None)}.
[INFO 07-22 15:04:30] ax.service.ax_client: Generated new trial 9 with parameters {'x1': 0.265264, 'x2': 0.924884, 'x3': 0.151716, 'x4': 0.436026, 'x5': 0.85731, 'x6': 0.08981, 'iterations': 100}.
Result for objective_313d3d3a: date: 2022-07-22_15-04-30 done: true experiment_id: fa7afd557e154fbebe4f54d8eedb3573 experiment_tag: 5_iterations=100,x1=0.0419,x2=0.9928,x3=0.9060,x4=0.5944,x5=0.8254,x6=0.6464 hostname: Kais-MacBook-Pro.local iterations_since_restore: 100 l2norm: 1.805729990121368 landscape: -0.006779757704679272 node_ip: 127.0.0.1 pid: 44747 time_since_restore: 3.1623308658599854 time_this_iter_s: 0.02911996841430664 time_total_s: 3.1623308658599854 timestamp: 1658498670 timesteps_since_restore: 0 timesteps_total: 99 training_iteration: 100 trial_id: 313d3d3a warmup_time: 0.0029790401458740234 Result for objective_32c9acd8: date: 2022-07-22_15-04-30 done: true experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7 experiment_tag: 6_iterations=100,x1=0.1261,x2=0.7034,x3=0.3447,x4=0.3374,x5=0.4014,x6=0.6792 hostname: Kais-MacBook-Pro.local iterations_since_restore: 100 l2norm: 1.1686440476629836 landscape: -0.9046216637367911 node_ip: 127.0.0.1 pid: 44726 time_since_restore: 3.1211891174316406 time_this_iter_s: 0.02954697608947754 time_total_s: 3.1211891174316406 timestamp: 1658498670 timesteps_since_restore: 0 timesteps_total: 99 training_iteration: 100 trial_id: 32c9acd8 warmup_time: 0.0026290416717529297
[INFO 07-22 15:04:32] ax.service.ax_client: Completed trial 7 with data: {'landscape': (-0.247223, None), 'l2norm': (1.286911, None)}.
[INFO 07-22 15:04:32] ax.service.ax_client: Completed trial 6 with data: {'landscape': (-0.146532, None), 'l2norm': (1.181781, None)}.
Result for objective_32d8dd20: date: 2022-07-22_15-04-32 done: true experiment_id: 171527593b0f4cbf941c0a03faaf0953 experiment_tag: 8_iterations=100,x1=0.6032,x2=0.4091,x3=0.7291,x4=0.0826,x5=0.5729,x6=0.5083 hostname: Kais-MacBook-Pro.local iterations_since_restore: 100 l2norm: 1.2869105702896437 landscape: -0.24722262157458608 node_ip: 127.0.0.1 pid: 44758 time_since_restore: 2.6415798664093018 time_this_iter_s: 0.026781082153320312 time_total_s: 2.6415798664093018 timestamp: 1658498672 timesteps_since_restore: 0 timesteps_total: 99 training_iteration: 100 trial_id: 32d8dd20 warmup_time: 0.002732992172241211 Result for objective_32cf8ca2: date: 2022-07-22_15-04-32 done: true experiment_id: 37610500f6df493aae4e7e46bb21bf09 experiment_tag: 7_iterations=100,x1=0.0911,x2=0.3041,x3=0.8698,x4=0.4054,x5=0.5679,x6=0.2286 hostname: Kais-MacBook-Pro.local iterations_since_restore: 100 l2norm: 1.1817810425508524 landscape: -0.14653248187442922 node_ip: 127.0.0.1 pid: 44756 time_since_restore: 2.707913875579834 time_this_iter_s: 0.027456998825073242 time_total_s: 2.707913875579834 timestamp: 1658498672 timesteps_since_restore: 0 timesteps_total: 99 training_iteration: 100 trial_id: 32cf8ca2 warmup_time: 0.0032138824462890625 Result for objective_34adf04a: date: 2022-07-22_15-04-33 done: false experiment_id: 4f65c5b68f5c49d98fda388e37c83deb hostname: Kais-MacBook-Pro.local iterations_since_restore: 1 l2norm: 1.4991655675380078 landscape: -0.01329150870283869 node_ip: 127.0.0.1 pid: 44768 time_since_restore: 0.00021600723266601562 time_this_iter_s: 0.00021600723266601562 time_total_s: 0.00021600723266601562 timestamp: 1658498673 timesteps_since_restore: 0 timesteps_total: 0 training_iteration: 1 trial_id: 34adf04a warmup_time: 0.0027239322662353516 Result for objective_34b7abda: date: 2022-07-22_15-04-33 done: false experiment_id: f135a2c40f5644ba9d2ae096a9dd10e0 hostname: Kais-MacBook-Pro.local iterations_since_restore: 1 l2norm: 1.3718451333547932 landscape: -1.6624439263544026 node_ip: 127.0.0.1 pid: 44771 time_since_restore: 0.0002338886260986328 time_this_iter_s: 0.0002338886260986328 time_total_s: 0.0002338886260986328 timestamp: 1658498673 timesteps_since_restore: 0 timesteps_total: 0 training_iteration: 1 trial_id: 34b7abda warmup_time: 0.002721071243286133
[INFO 07-22 15:04:35] ax.service.ax_client: Completed trial 8 with data: {'landscape': (-0.013292, None), 'l2norm': (1.499166, None)}.
[INFO 07-22 15:04:35] ax.service.ax_client: Completed trial 9 with data: {'landscape': (-1.662444, None), 'l2norm': (1.371845, None)}.
Result for objective_34adf04a: date: 2022-07-22_15-04-35 done: true experiment_id: 4f65c5b68f5c49d98fda388e37c83deb experiment_tag: 9_iterations=100,x1=0.4542,x2=0.2718,x3=0.5309,x4=0.9918,x5=0.6918,x6=0.4724 hostname: Kais-MacBook-Pro.local iterations_since_restore: 100 l2norm: 1.4991655675380078 landscape: -0.01329150870283869 node_ip: 127.0.0.1 pid: 44768 time_since_restore: 2.7032668590545654 time_this_iter_s: 0.029300928115844727 time_total_s: 2.7032668590545654 timestamp: 1658498675 timesteps_since_restore: 0 timesteps_total: 99 training_iteration: 100 trial_id: 34adf04a warmup_time: 0.0027239322662353516 Result for objective_34b7abda: date: 2022-07-22_15-04-35 done: true experiment_id: f135a2c40f5644ba9d2ae096a9dd10e0 experiment_tag: 10_iterations=100,x1=0.2653,x2=0.9249,x3=0.1517,x4=0.4360,x5=0.8573,x6=0.0898 hostname: Kais-MacBook-Pro.local iterations_since_restore: 100 l2norm: 1.3718451333547932 landscape: -1.6624439263544026 node_ip: 127.0.0.1 pid: 44771 time_since_restore: 2.6852078437805176 time_this_iter_s: 0.029579877853393555 time_total_s: 2.6852078437805176 timestamp: 1658498675 timesteps_since_restore: 0 timesteps_total: 99 training_iteration: 100 trial_id: 34b7abda warmup_time: 0.002721071243286133
And now we have the hyperparameters found to minimize the mean loss.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'iterations': 100, 'x1': 0.26526361983269453, 'x2': 0.9248840995132923, 'x3': 0.15171580761671066, 'x4': 0.43602637108415365, 'x5': 0.8573104059323668, 'x6': 0.08981018699705601}
ray.shutdown()