DeepCube Use Case 3: Fire Hazard Forecasting in the Mediterranean¶
Notebook 2: Fire danger maps in Greece with Deep Learning models¶
This notebook shows how to load our pretrained models for Fire Danger prediction to perform inference on the published greece_wildfire_datacube or its cloud version.
For details about how our models were trained, have a look at our relevant paper "Deep Learning Methods for Daily Wildfire Danger Forecasting" available on arXiv, which is accepted to the workshop on Artificial Intelligence for Humanitarian Assistance and Disaster Response at the 35th Conference on Neural Information Processing Systems (NeurIPS 2021)
# uncomment the lines if you download the dataset locally# !wget -O dataset_greece.nc https://zenodo.org/record/4943354/files/dataset_greece.nc?download=1# ds = xr.open_dataset('./dataset_greece.nc')# comment out the following two lines if you want to access the dataset locallyurl='https://storage.de.cloud.ovh.net/v1/AUTH_84d6da8e37fe4bb5aea18902da8c1170/uc3/uc3cube.zarr'ds=xr.open_zarr(fsspec.get_mapper(url),consolidated=True)
In [3]:
ds
Out[3]:
<xarray.Dataset>
Dimensions: (time: 4314, x: 700, y: 562)
Coordinates:
band int64 ...
spatial_ref (time) int64 dask.array<chunksize=(288,), meta=np.ndarray>
* time (time) datetime64[ms] 2009-03-06 ... 2020-12-26
* x (x) float64 19.86 19.87 19.89 ... 28.16 28.17 28.18
* y (y) float64 41.62 41.61 41.59 ... 34.96 34.95 34.94
Data variables: (12/58)
1 km 16 days EVI (time, y, x) float64 dask.array<chunksize=(288, 50, 50), meta=np.ndarray>
1 km 16 days NDVI (time, y, x) float64 dask.array<chunksize=(288, 50, 50), meta=np.ndarray>
1 km 16 days VI Quality (time, y, x) float64 dask.array<chunksize=(288, 50, 50), meta=np.ndarray>
ET_500m (time, y, x) float64 dask.array<chunksize=(288, 50, 50), meta=np.ndarray>
ET_QC_500m (time, y, x) float64 dask.array<chunksize=(288, 50, 50), meta=np.ndarray>
FparExtra_QC (time, y, x) float64 dask.array<chunksize=(288, 50, 50), meta=np.ndarray>
... ...
population_density_2020 (y, x) float32 dask.array<chunksize=(50, 50), meta=np.ndarray>
roads_density_2020 (y, x) float64 dask.array<chunksize=(50, 50), meta=np.ndarray>
slope_max (y, x) float32 dask.array<chunksize=(50, 50), meta=np.ndarray>
slope_mean (y, x) float32 dask.array<chunksize=(50, 50), meta=np.ndarray>
slope_min (y, x) float32 dask.array<chunksize=(50, 50), meta=np.ndarray>
slope_std (y, x) float32 dask.array<chunksize=(50, 50), meta=np.ndarray>
We see that the dataset has x,y and time dimensions.
More specifically it contains 4314 days (from 06/03/2009 to 06/12/2020) of 700x562 rasters
Dynamic variables like the burned_areas have all three dimensions, while static variables like clc_2012 misses the temporal component and only has x, y dimensions.
From the datacube, we extract four different datasets that are used to train four different models.
For a given pixel (i.e., a cell representing a 1 km x 1 km square region) and a given day, we extract input-target pairs for 4 different modeling modalities:
First, the pixel dataset, where we extract the input attributes and their last 10-day average (only for the dynamic input attributes). This is used to train a Random Forest (RF) model (is not shown in this notebook).
Second, the temporal dataset, where we extract the last 10-day time-series of the input attributes. This is used to train an Long Short-Term Memory (LSTM) model.
Third, the spatial dataset, where we extract 25km x 25km patches spatially centered around the given pixel. This is used to train a Convolutional Neural Network (CNN) model.
Fourth, the spatio-temporal dataset, where we extract 25km x 25km x 10 days blocks centered spatially around the given pixel. This is used to train a convLSTM model.
# these are the features that change in timedynamic_features=['Fpar_500m','Lai_500m','LST_Day_1km','LST_Night_1km','1 km 16 days NDVI','1 km 16 days EVI','era5_max_u10','era5_max_v10','era5_max_t2m','era5_max_tp','era5_min_u10','era5_min_v10','era5_min_t2m']# these are the features that remain the same through timestatic_features=['dem_mean','aspect_mean','slope_mean','roads_density_2020','population_density']# the settings that were used to train our best modelsbest_settings={'lstm':{'dynamic_features':dynamic_features,'static_features':static_features,'hidden_size':64,'lstm_layers':1,'attention':False},'cnn':{'dynamic_features':dynamic_features,'static_features':static_features,'hidden_size':16},'convlstm':{'dynamic_features':dynamic_features,'static_features':static_features,'hidden_size':16,'lstm_layers':1},}
The models have been trained using data for the years 2009-2019. We will perform inference on two days of the test set (year 2020).
In [9]:
fromsrc.greece_fire_datasetimportFireDatasetWholeDaydefpredict_map(day,pl_module,access_mode,problem_class,patch_size,lag,dynamic_features,static_features,nan_fill,batch_size):""" This function returns the predictions of model (pl_module) for a given day. """dataset=FireDatasetWholeDay(ds,day,access_mode,problem_class,patch_size,lag,dynamic_features,static_features,nan_fill)len_x=dataset.len_xlen_y=dataset.len_ypl_module.eval()num_iterations=max(1,len(dataset)//batch_size)dataloader=DataLoader(dataset,batch_size=batch_size,shuffle=False,num_workers=num_workers,prefetch_factor=batch_size)outputs=[]fori,(dynamic,static)intqdm(enumerate(dataloader),total=num_iterations):ifpl_module.on_gpu:inputs=inputs.cuda()logits=pl_module(inputs)preds_proba=torch.exp(logits)[:,1]outputs.append(preds_proba.detach().cpu())outputs=torch.cat(outputs,dim=0)outputs=outputs.reshape(len_y,len_x)outputs=outputs.detach().cpu().numpy().squeeze()returnoutputs
In [10]:
# days for which to create fire danger mapsdays=[4150,4155]patch_size=25lag=10# value used to fill nansnan_fill=-1.0acc_mod_bs=[('spatial',cnn,4096,'CNN'),('temporal',lstm,10196,'LSTM'),('spatiotemporal',convlstm,512,'ConvLSTM'),]problem_class='classification'num_workers=12# these are used to mask out land cover classes that are not supposed to be dangerous for fire (sea, water bodies, urban areas)clc=np.isin(ds['clc_2012'].values,list(range(12,33)),invert=True)pop_den=np.isnan(ds['population_density_2012'].values)fordayindays:gc.collect()day=int(day)print(day)# get a validation batch from the validation dat loaderfig,ax=plt.subplots(1,4,figsize=(20,5))fig.suptitle('Fire danger for Greece on {}'.format(ds['time'][day].dt.strftime('%d/%m/%Y').values))subplot_number=141fori,(access_mode,pl_module,batch_size,name)inenumerate(acc_mod_bs):ifaccess_mode=='spatial':lag=0else:lag=10ifaccess_mode=='temporal':patch_size=0else:patch_size=25outputs=predict_map(day,pl_module,access_mode,problem_class,patch_size,lag,dynamic_features,static_features,nan_fill,batch_size)outputs[clc]=0outputs[pop_den]=0plt.subplot(subplot_number),plt.imshow(outputs,vmin=0,vmax=1,cmap='Spectral_r')plt.title(name)subplot_number+=1fwi=ds.fwi[day].valuesfwi[clc]=0fwi[pop_den]=0plt.subplot(subplot_number),plt.imshow(fwi,vmin=0,vmax=50,cmap='Spectral_r')plt.title('FWI')plt.show()
We see that the danger maps created by the Deep Learning models follow closely, but offer a better resolution than the empirical Fire Weather Index that is widely used as an operational tool. This is quite an interesting result, considering that our models have been trained using the historical burned areas as target and do not have a direct notion of what Fire Danger is. This is quite a step forward in showing that fire danger can be learned in a data driven way and it is possible to leverage what DL models can learn from the complex interactions of the Fire Drivers.
We see that the danger maps created by the Deep Learning models follow closely, but offer a better resolution than the empirical Fire Weather Index that is widely used as an operational tool. This is quite an interesting result, considering that our models have been trained using the historical burned areas as target and do not have a direct notion of what Fire Danger is. This is quite a step forward in showing that fire danger can be learned in a data driven way and it is possible to leverage what DL models can learn from the complex interactions of the Fire Drivers.
In this notebook, we saw how to load pytorch lighning pretrained Deep Learning modules and use them on inference to create fire danger maps. In a later notebook we'll use explainable AI methods to understand our models' predictions. Stay tuned!