To run any of Eden's notebooks, please check the guides on our Wiki page.
There you will find instructions on how to deploy the notebooks on your local system, on Google Colab, or on MyBinder, as well as other useful links, troubleshooting tips, and more.
Note: If you find any issues while executing the notebook, don't hesitate to open an issue on Github. We will try to reply as soon as possible.
In this notebook, we are going to cover a technique called Automated Machine Learning (AutoML). These systems have arisen in the past years to allow computers to automatically find the most suitable Machine Learning (ML) pipeline matching a specific task and dataset. AutoML systems could provide insights to ML engineers resulting in better models deployed in a shorter period of time. They are considered meta-level ML algorithms which use different components as building blocks for finding the optimal ML pipeline structures (Feurer et al., 2015; Kotthoff et al., 2017). These systems automatically evaluate multiple pipeline configurations, trying to improve the global performance iteratively. Several open-source technologies have raised awareness of the strengths and limitations of the AutoML systems, e.g. AutoKeras, AutoSklearn, Auto-WEKA, H2O AutoML, TPOT, autoxgboost, and OBOE. Additionally, different AutoML cloud-solutions are now being offered by IT firms, such as Google Cloud AutoML Vision, Microsoft Azure Machine Learning, and Apple Create ML; they offer user-friendly interfaces and require little expertise in Machine Learning to train models. In this notebook, we will be using AutoKeras. If you want to check out AutoSklearn, please have a look in our previous notebooks:
In the agricultural domain, some recent research studies have made use of the AutoML technique in the past few years, using it to process time series as well as proximal and satellite images. In Hayashi et al., (2019), the authors tested whether AutoML was a useful tool for the identification of pest insect species by using three aphid species. They constructed models that were trained by photographs of those species under various conditions in Google Cloud AutoML Vision, and compared their accuracies of identification. Since the rates of correct identification were over 96% when the models were trained with 400 images per class, they considered AutoML to be useful for pest species identification. In Montellano (2019), the author used AutoML through the same platform to classify different types of butterflies, image fruits, and larval host plants; their average accuracy was around 97.1%. In Hsieh et al., (2019), AutoML was implemented along with neural network algorithms to classify whether the conditions of rice blast disease were exacerbated or relieved by using five years of climatic data. Although the experiments showed 72% accuracy on average, the model obtained an accuracy of 89% in the exacerbation case. Hence, the effectiveness of the proposed classification model, which combined multiple machine learning models, was confirmed. Finally, an AutoML approach has been applied in Kiala et al., (2020), in an attempt to map the Parthenium weed. The authors constructed models by using AutoML technology and 16 other classifiers that were trained by satellite pictures of Sentinel-2 and Landsat 8. The AutoML model achieved a higher overall classification accuracy of 88.15% using Sentinel-2 and 74% using Landsat 8, results that confirmed the significance of the AutoML in mapping Parthenium weed infestations using satellite imagery. In Koh et al., (2020), authors used wheat lodging assessment with UAV images for high-throughput plant phenotyping. They compared AutoKeras in image classification and regression tasks to transfer learning techniques. Finally, in Espejo-Garcia et. al, (2021), the authors integrated AutoSklearn and AutoKeras for addressing the problem of weed identification in 2 different datasets.
In agriculture, since weeds compete with crops in the domain of space, light, and nutrients, they are an important problem that can lead to a poorer harvest. To avoid this, weeds should be removed at every step of growth, but especially at the initial stages. For that reason, identifying weeds accurately through Deep Learning has arisen as an important objective.
from matplotlib import pyplot as plt
import numpy as np
from sklearn.model_selection import train_test_split
import torch
from torch import nn
from torch import optim
from torch.utils.data import random_split
from torch.utils.data import DataLoader, TensorDataset
from torch.nn import functional as F
import torchvision
from torchvision import datasets, transforms
import optuna
import os
import cv2
from tqdm import tqdm
from glob import glob
import random
from pathlib import Path
/home/beast/anaconda3/envs/eden_pytorch_transfer/lib/python3.8/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
def normalize_data(data):
return (data - np.min(data)) / (np.max(data) - np.min(data))
# Function for plotting images.
def plot_sample(X):
# Plotting 9 sample images
num_rows = 3
num_cols = 3
plt.figure(figsize=(8, 8))
for ix in range(num_rows*num_cols):
plt.subplot(num_cols, num_rows, ix+1)
plt.imshow(random.choice(X))
plt.axis("off")
plt.show()
def read_data(path_list, im_size=(128,128)):
X = []
y = []
# Exctract the file-names of the datasets we read and create a label dictionary.
tag2idx = {tag.split(os.path.sep)[-1]:i for i, tag in enumerate(path_list)}
for path in path_list:
for im_file in tqdm(glob(path + '*/*')): # Read all files in path
try:
# os.path.separator is OS agnostic (either '/' or '\'),[-2] to grab folder name.
label = im_file.split(os.path.sep)[-2]
im = cv2.imread(im_file)
# Resize to appropriate dimensions.You can try different interpolation methods.
im = cv2.resize(im, im_size,interpolation=cv2.INTER_LINEAR)
# By default OpenCV read with BGR format, return back to RGB.
im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)
X.append(im)
y.append(tag2idx[label])# Append the label name to y
except Exception as e:
# In case annotations or metadata are found
print("Not a picture")
X = np.array(X) #Convert list to numpy array.
y = np.eye(len(np.unique(y)))[y].astype(np.uint8)
return X, y #np.array(y)
def get_dataloaders(x, y):
"""
It returns the DataLoaders necessary for training and validating.
"""
data_transforms = transforms.Compose(
[transforms.RandomHorizontalFlip(),
transforms.ColorJitter(brightness=0.1, contrast=0.2, saturation=0, hue=0)]
)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33)
train_dataset = TensorDataset(torch.permute(torch.Tensor(x_train), (0, 3, 1, 2)),
torch.Tensor(y_train)) # create your datset
train_dataset.transform = data_transforms
train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE) # create your dataloader
valid_dataset = TensorDataset(torch.permute(torch.Tensor(x_test), (0, 3, 1, 2)), torch.Tensor(y_test)) # create your datset
valid_dataset.transform = data_transforms
valid_loader = DataLoader(valid_dataset, batch_size=BATCH_SIZE) # create your dataloader
return train_loader, valid_loader
DEVICE = torch.device("cuda:0") if torch.cuda.is_available() else torch.device("cpu")
NUM_CLASSES = 4
NUM_EPOCHS = 50
BATCH_SIZE = 16
INPUT_SIZE = 128
IM_SIZE = (128, 128)
NUM_TRIALS = 100
PATH_LIST = [
"Cotton-100619-Healthy-zz-V1-20210225102300",
"Black nightsade-220519-Weed-zz-V1-20210225102034",
"Tomato-240519-Healthy-zz-V1-202102251037402",
"Velvet leaf-220519-Weed-zz-V1-20210225104123",
]
i = 0
for path in PATH_LIST:
# Define paths in an OS agnostic way.
PATH_LIST[i] = str(
Path(Path.cwd()).parents[0].joinpath("eden_library_datasets").joinpath(path)
)
i += 1
x, y = read_data(PATH_LIST, IM_SIZE)
100%|██████████| 47/47 [00:07<00:00, 6.39it/s] 62%|██████▏ | 77/124 [00:11<00:05, 8.57it/s]
Not a picture
100%|██████████| 124/124 [00:18<00:00, 6.72it/s] 64%|██████▍ | 130/202 [00:29<00:18, 3.92it/s]
Not a picture
100%|██████████| 202/202 [00:45<00:00, 4.44it/s] 69%|██████▉ | 84/121 [00:12<00:04, 8.59it/s]
Not a picture
100%|██████████| 121/121 [00:18<00:00, 6.60it/s]
plot_sample(x)
train_loader, valid_loader = get_dataloaders(x, y)
class ConvNet(nn.Module):
def __init__(self, trial):
super(ConvNet, self).__init__()
self.conv1 = nn.Conv2d(
in_channels=3, out_channels=32, kernel_size=3, stride=1, padding=1
)
dropout_rate_1 = trial.suggest_float("dropout_rate_1", 0, 0.5, step=0.1)
self.drop_1 = nn.Dropout(p=dropout_rate_1)
self.conv2 = nn.Conv2d(
in_channels=32, out_channels=64, kernel_size=3, stride=1, padding=1
)
dropout_rate_2 = trial.suggest_float("dropout_rate_2", 0, 0.5, step=0.1)
self.drop_2 = nn.Dropout(p=dropout_rate_2)
self.conv3 = nn.Conv2d(
in_channels=64, out_channels=128, kernel_size=3, stride=1, padding=1
)
dropout_rate_3 = trial.suggest_float("dropout_rate_3", 0, 0.5, step=0.1)
self.drop_3 = nn.Dropout(p=dropout_rate_3)
fc2_input_dim = trial.suggest_int("fc2_input_dim", 32, 128, 32)
self.fc1 = nn.Linear(128 * 16 * 16, fc2_input_dim)
dropout_rate_4 = trial.suggest_float("dropout_rate2", 0, 0.5, step=0.1)
self.drop_4 = nn.Dropout(p=dropout_rate_4)
self.fc2 = nn.Linear(fc2_input_dim, NUM_CLASSES)
def forward(self, x):
x = F.relu(F.max_pool2d(self.conv1(x), kernel_size=2))
x = self.drop_1(x)
x = F.relu(F.max_pool2d(self.conv2(x), kernel_size=2))
x = self.drop_2(x)
x = F.relu(F.max_pool2d(self.conv3(x), kernel_size=2))
x = self.drop_3(x)
x = x.view(x.size(0), -1)
x = F.relu(self.fc1(x))
x = self.drop_4(x)
x = self.fc2(x)
return x
def objective(trial):
# Generate the model.
model = ConvNet(trial).to(DEVICE)
# Generate the optimizers.
# try Adam, AdaDelta and Adagrad
optimizer_name = trial.suggest_categorical(
"optimizer", ["Adam", "Adadelta", "Adagrad"]
)
lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True)
optimizer = getattr(optim, optimizer_name)(model.parameters(), lr=lr)
criterion = nn.CrossEntropyLoss()
# Training of the model.
for epoch in range(NUM_EPOCHS):
model.train()
for batch_idx, (images, labels) in enumerate(train_loader):
images, labels = images.to(DEVICE), labels.to(DEVICE)
optimizer.zero_grad()
output = model(images)
loss = criterion(output, labels)
loss.backward()
optimizer.step()
# Validation of the model.
model.eval()
correct = 0
with torch.no_grad():
for batch_idx, (images, labels) in enumerate(valid_loader):
images, labels = images.to(DEVICE), labels.to(DEVICE)
output = model(images)
# Get the index of the max log-probability.
pred = output.argmax(dim=1, keepdim=True)
labels_norm = labels.argmax(dim=1, keepdim=True)
correct += pred.eq(labels_norm.view_as(pred)).sum().item()
accuracy = correct / len(valid_loader.dataset)
trial.report(accuracy, epoch)
# Handle pruning based on the intermediate value.
if trial.should_prune():
raise optuna.exceptions.TrialPruned()
return accuracy
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=NUM_TRIALS)
[I 2022-06-16 13:20:56,416] A new study created in memory with name: no-name-1528260c-d726-4fd1-bbb3-e81a92d6d12f [I 2022-06-16 13:21:02,490] Trial 0 finished with value: 0.5337423312883436 and parameters: {'dropout_rate_1': 0.0, 'dropout_rate_2': 0.30000000000000004, 'dropout_rate_3': 0.2, 'fc2_input_dim': 32, 'dropout_rate2': 0.2, 'optimizer': 'Adam', 'lr': 0.0023547824182933727}. Best is trial 0 with value: 0.5337423312883436. [I 2022-06-16 13:21:08,769] Trial 1 finished with value: 0.32515337423312884 and parameters: {'dropout_rate_1': 0.5, 'dropout_rate_2': 0.2, 'dropout_rate_3': 0.0, 'fc2_input_dim': 32, 'dropout_rate2': 0.2, 'optimizer': 'Adadelta', 'lr': 1.642636552632549e-05}. Best is trial 0 with value: 0.5337423312883436. [I 2022-06-16 13:21:14,740] Trial 2 finished with value: 0.5398773006134969 and parameters: {'dropout_rate_1': 0.2, 'dropout_rate_2': 0.4, 'dropout_rate_3': 0.5, 'fc2_input_dim': 32, 'dropout_rate2': 0.0, 'optimizer': 'Adam', 'lr': 0.011375394397880201}. Best is trial 2 with value: 0.5398773006134969. [I 2022-06-16 13:21:20,561] Trial 3 finished with value: 0.44171779141104295 and parameters: {'dropout_rate_1': 0.0, 'dropout_rate_2': 0.1, 'dropout_rate_3': 0.0, 'fc2_input_dim': 32, 'dropout_rate2': 0.0, 'optimizer': 'Adam', 'lr': 0.0015958925310999462}. Best is trial 2 with value: 0.5398773006134969. [I 2022-06-16 13:21:26,634] Trial 4 finished with value: 0.5521472392638037 and parameters: {'dropout_rate_1': 0.0, 'dropout_rate_2': 0.1, 'dropout_rate_3': 0.2, 'fc2_input_dim': 32, 'dropout_rate2': 0.4, 'optimizer': 'Adam', 'lr': 0.010220873309105993}. Best is trial 4 with value: 0.5521472392638037. [I 2022-06-16 13:21:26,766] Trial 5 pruned. [I 2022-06-16 13:21:26,902] Trial 6 pruned. [I 2022-06-16 13:21:27,053] Trial 7 pruned. [I 2022-06-16 13:21:27,192] Trial 8 pruned. [I 2022-06-16 13:21:27,331] Trial 9 pruned. [I 2022-06-16 13:21:27,484] Trial 10 pruned. [I 2022-06-16 13:21:27,640] Trial 11 pruned. [I 2022-06-16 13:21:27,792] Trial 12 pruned. [I 2022-06-16 13:21:27,946] Trial 13 pruned. [I 2022-06-16 13:21:28,113] Trial 14 pruned. [I 2022-06-16 13:21:28,259] Trial 15 pruned. [I 2022-06-16 13:21:28,411] Trial 16 pruned. [I 2022-06-16 13:21:28,559] Trial 17 pruned. [I 2022-06-16 13:21:28,711] Trial 18 pruned. [I 2022-06-16 13:21:28,858] Trial 19 pruned. [I 2022-06-16 13:21:29,009] Trial 20 pruned. [I 2022-06-16 13:21:29,156] Trial 21 pruned. [I 2022-06-16 13:21:29,305] Trial 22 pruned. [I 2022-06-16 13:21:29,450] Trial 23 pruned. [I 2022-06-16 13:21:29,606] Trial 24 pruned. [I 2022-06-16 13:21:29,759] Trial 25 pruned. [I 2022-06-16 13:21:29,920] Trial 26 pruned. [I 2022-06-16 13:21:30,073] Trial 27 pruned. [I 2022-06-16 13:21:30,238] Trial 28 pruned. [I 2022-06-16 13:21:30,392] Trial 29 pruned. [I 2022-06-16 13:21:30,552] Trial 30 pruned. [I 2022-06-16 13:21:30,700] Trial 31 pruned. [I 2022-06-16 13:21:30,852] Trial 32 pruned. [I 2022-06-16 13:21:31,006] Trial 33 pruned. [I 2022-06-16 13:21:31,157] Trial 34 pruned. [I 2022-06-16 13:21:31,319] Trial 35 pruned. [I 2022-06-16 13:21:31,480] Trial 36 pruned. [I 2022-06-16 13:21:31,649] Trial 37 pruned. [I 2022-06-16 13:21:31,829] Trial 38 pruned. [I 2022-06-16 13:21:31,987] Trial 39 pruned. [I 2022-06-16 13:21:32,151] Trial 40 pruned. [I 2022-06-16 13:21:32,311] Trial 41 pruned. [I 2022-06-16 13:21:32,469] Trial 42 pruned. [I 2022-06-16 13:21:32,638] Trial 43 pruned. [I 2022-06-16 13:21:32,809] Trial 44 pruned. [I 2022-06-16 13:21:32,961] Trial 45 pruned. [I 2022-06-16 13:21:33,117] Trial 46 pruned. [I 2022-06-16 13:21:33,277] Trial 47 pruned. [I 2022-06-16 13:21:33,431] Trial 48 pruned. [I 2022-06-16 13:21:33,578] Trial 49 pruned. [I 2022-06-16 13:21:33,737] Trial 50 pruned. [I 2022-06-16 13:21:33,890] Trial 51 pruned. [I 2022-06-16 13:21:34,041] Trial 52 pruned. [I 2022-06-16 13:21:34,193] Trial 53 pruned. [I 2022-06-16 13:21:34,350] Trial 54 pruned. [I 2022-06-16 13:21:34,511] Trial 55 pruned. [I 2022-06-16 13:21:34,666] Trial 56 pruned. [I 2022-06-16 13:21:34,835] Trial 57 pruned. [I 2022-06-16 13:21:35,015] Trial 58 pruned. [I 2022-06-16 13:21:35,177] Trial 59 pruned. [I 2022-06-16 13:21:35,971] Trial 60 pruned. [I 2022-06-16 13:21:36,126] Trial 61 pruned. [I 2022-06-16 13:21:36,277] Trial 62 pruned. [I 2022-06-16 13:21:36,425] Trial 63 pruned. [I 2022-06-16 13:21:36,571] Trial 64 pruned. [I 2022-06-16 13:21:36,719] Trial 65 pruned. [I 2022-06-16 13:21:36,867] Trial 66 pruned. [I 2022-06-16 13:21:37,030] Trial 67 pruned. [I 2022-06-16 13:21:37,186] Trial 68 pruned. [I 2022-06-16 13:21:37,345] Trial 69 pruned. [I 2022-06-16 13:21:37,516] Trial 70 pruned. [I 2022-06-16 13:21:37,685] Trial 71 pruned. [I 2022-06-16 13:21:37,841] Trial 72 pruned. [I 2022-06-16 13:21:37,996] Trial 73 pruned. [I 2022-06-16 13:21:38,159] Trial 74 pruned. [I 2022-06-16 13:21:38,325] Trial 75 pruned. [I 2022-06-16 13:21:38,483] Trial 76 pruned. [I 2022-06-16 13:21:45,054] Trial 77 finished with value: 0.5337423312883436 and parameters: {'dropout_rate_1': 0.0, 'dropout_rate_2': 0.30000000000000004, 'dropout_rate_3': 0.5, 'fc2_input_dim': 32, 'dropout_rate2': 0.2, 'optimizer': 'Adam', 'lr': 0.006094994675772609}. Best is trial 4 with value: 0.5521472392638037. [I 2022-06-16 13:21:45,203] Trial 78 pruned. [I 2022-06-16 13:21:45,362] Trial 79 pruned. [I 2022-06-16 13:21:48,024] Trial 80 pruned. [I 2022-06-16 13:21:48,183] Trial 81 pruned. [I 2022-06-16 13:21:48,341] Trial 82 pruned. [I 2022-06-16 13:21:48,497] Trial 83 pruned. [I 2022-06-16 13:21:48,652] Trial 84 pruned. [I 2022-06-16 13:21:48,813] Trial 85 pruned. [I 2022-06-16 13:21:48,979] Trial 86 pruned. [I 2022-06-16 13:21:49,134] Trial 87 pruned. [I 2022-06-16 13:21:49,282] Trial 88 pruned. [I 2022-06-16 13:21:49,438] Trial 89 pruned. [I 2022-06-16 13:21:49,594] Trial 90 pruned. [I 2022-06-16 13:21:49,750] Trial 91 pruned. [I 2022-06-16 13:21:52,329] Trial 92 pruned. [I 2022-06-16 13:21:52,485] Trial 93 pruned. [I 2022-06-16 13:21:58,628] Trial 94 finished with value: 0.8834355828220859 and parameters: {'dropout_rate_1': 0.0, 'dropout_rate_2': 0.30000000000000004, 'dropout_rate_3': 0.5, 'fc2_input_dim': 32, 'dropout_rate2': 0.2, 'optimizer': 'Adam', 'lr': 0.0037981920114391983}. Best is trial 94 with value: 0.8834355828220859. [I 2022-06-16 13:21:58,788] Trial 95 pruned. [I 2022-06-16 13:21:58,946] Trial 96 pruned. [I 2022-06-16 13:21:59,105] Trial 97 pruned. [I 2022-06-16 13:21:59,263] Trial 98 pruned. [I 2022-06-16 13:21:59,426] Trial 99 pruned.
best_trial = study.best_trial
print(f"Accuracy: {best_trial.value}")
print(f"Best hyperparameters: {best_trial.params}")
Accuracy: 0.8834355828220859 Best hyperparameters: {'dropout_rate_1': 0.0, 'dropout_rate_2': 0.30000000000000004, 'dropout_rate_3': 0.5, 'fc2_input_dim': 32, 'dropout_rate2': 0.2, 'optimizer': 'Adam', 'lr': 0.0037981920114391983}
optuna.visualization.plot_contour(study, params=["optimizer", "lr"])
optuna.visualization.plot_param_importances(study)