Warning¶
This tutorial is a work in progress, it does not cover all the functionalities of the preprocessing package yet.
preprocessing tutorial¶
In this Notebook, we will demonstrate how to preprocess brain MR images with the BrainLes preprocessing package.
Requirements¶
This tutorial requires:
- Python 3.10+ environment
- Docker
- Linux (Windows and Mac OS will be supported in future versions)
optional (but recommended):
- CUDA
Setup Colab environment¶
If you installed the packages and requirements on your own machine, you can skip this section and start from the import section. Otherwise you can follow and execute the tutorial on your browser. In order to start working on the notebook, click on the following button, this will open this page in the Colab environment and you will be able to execute the code on your own.
Now that you are visualizing the notebook in Colab, run the next cell to install the packages we will use. There are few things you should follow in order to properly set the notebook up:
- Warning: This notebook was not authored by Google. Click on 'Run anyway'.
- When the installation commands are done, there might be "Restart runtime" button at the end of the output. Please, click it.
Installation of dependencies¶
In [1]:
%pip install brainles_preprocessing auxiliary
%pip install matplotlib
%load_ext autoreload
%autoreload 2
Requirement already satisfied: brainles_preprocessing in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (0.0.24) Requirement already satisfied: auxiliary in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (0.0.40) Requirement already satisfied: BrainLes-HD-BET>=0.0.5 in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (from brainles_preprocessing) (0.0.5) Requirement already satisfied: nibabel<4.0.0,>=3.2.1 in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (from brainles_preprocessing) (3.2.2) Requirement already satisfied: numpy<2.0.0,>=1.23.0 in /home/florian/.local/lib/python3.10/site-packages (from brainles_preprocessing) (1.26.2) Requirement already satisfied: path<17.0.0,>=16.2.0 in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (from brainles_preprocessing) (16.9.0) Requirement already satisfied: pathlib<2.0.0,>=1.0.1 in /home/florian/.local/lib/python3.10/site-packages (from brainles_preprocessing) (1.0.1) Requirement already satisfied: rich<14.0.0,>=13.6.0 in /home/florian/.local/lib/python3.10/site-packages (from brainles_preprocessing) (13.7.0) Requirement already satisfied: tqdm<5.0.0,>=4.64.1 in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (from brainles_preprocessing) (4.66.1) Requirement already satisfied: ttictoc<0.6.0,>=0.5.6 in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (from brainles_preprocessing) (0.5.6) Requirement already satisfied: pillow>=10.0.0 in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (from auxiliary) (10.2.0) Requirement already satisfied: tifffile>=2023.8.25 in /home/florian/.local/lib/python3.10/site-packages (from auxiliary) (2023.12.9) Requirement already satisfied: torch>=0.4.1 in /home/florian/.local/lib/python3.10/site-packages (from BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (2.1.2) Requirement already satisfied: scikit-image>=0.21.0 in /home/florian/.local/lib/python3.10/site-packages (from BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (0.22.0) Requirement already satisfied: SimpleITK>=2.2.1 in /home/florian/.local/lib/python3.10/site-packages (from BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (2.3.1) Requirement already satisfied: packaging>=14.3 in /home/florian/.local/lib/python3.10/site-packages (from nibabel<4.0.0,>=3.2.1->brainles_preprocessing) (23.2) Requirement already satisfied: setuptools in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (from nibabel<4.0.0,>=3.2.1->brainles_preprocessing) (68.2.2) Requirement already satisfied: markdown-it-py>=2.2.0 in /home/florian/.local/lib/python3.10/site-packages (from rich<14.0.0,>=13.6.0->brainles_preprocessing) (3.0.0) Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/florian/.local/lib/python3.10/site-packages (from rich<14.0.0,>=13.6.0->brainles_preprocessing) (2.17.2) Requirement already satisfied: mdurl~=0.1 in /home/florian/.local/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich<14.0.0,>=13.6.0->brainles_preprocessing) (0.1.2) Requirement already satisfied: scipy>=1.8 in /home/florian/.local/lib/python3.10/site-packages (from scikit-image>=0.21.0->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (1.11.4) Requirement already satisfied: networkx>=2.8 in /home/florian/.local/lib/python3.10/site-packages (from scikit-image>=0.21.0->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (3.2.1) Requirement already satisfied: imageio>=2.27 in /home/florian/.local/lib/python3.10/site-packages (from scikit-image>=0.21.0->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (2.33.1) Requirement already satisfied: lazy_loader>=0.3 in /home/florian/.local/lib/python3.10/site-packages (from scikit-image>=0.21.0->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (0.3) Requirement already satisfied: filelock in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (3.13.1) Requirement already satisfied: typing-extensions in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (4.8.0) Requirement already satisfied: sympy in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (1.12) Requirement already satisfied: jinja2 in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (3.1.2) Requirement already satisfied: fsspec in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (2023.10.0) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (12.1.105) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (12.1.105) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (12.1.105) Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (8.9.2.26) Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (12.1.3.1) Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (11.0.2.54) Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (10.3.2.106) Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (11.4.5.107) Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (12.1.0.106) Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (2.18.1) Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (12.1.105) Requirement already satisfied: triton==2.1.0 in /home/florian/.local/lib/python3.10/site-packages (from torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (2.1.0) Requirement already satisfied: nvidia-nvjitlink-cu12 in /home/florian/.local/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (12.3.52) Requirement already satisfied: MarkupSafe>=2.0 in /home/florian/.local/lib/python3.10/site-packages (from jinja2->torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (2.1.3) Requirement already satisfied: mpmath>=0.19 in /home/florian/.local/lib/python3.10/site-packages (from sympy->torch>=0.4.1->BrainLes-HD-BET>=0.0.5->brainles_preprocessing) (1.3.0) Note: you may need to restart the kernel to use updated packages. Requirement already satisfied: matplotlib in /home/florian/.local/lib/python3.10/site-packages (3.3.1) Requirement already satisfied: certifi>=2020.06.20 in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (from matplotlib) (2024.2.2) Requirement already satisfied: cycler>=0.10 in /home/florian/.local/lib/python3.10/site-packages (from matplotlib) (0.12.1) Requirement already satisfied: kiwisolver>=1.0.1 in /home/florian/.local/lib/python3.10/site-packages (from matplotlib) (1.4.5) Requirement already satisfied: numpy>=1.15 in /home/florian/.local/lib/python3.10/site-packages (from matplotlib) (1.26.2) Requirement already satisfied: pillow>=6.2.0 in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (from matplotlib) (10.2.0) Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (from matplotlib) (3.1.1) Requirement already satisfied: python-dateutil>=2.1 in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (from matplotlib) (2.8.2) Requirement already satisfied: six>=1.5 in /home/florian/miniconda3/envs/aurora_tutorial/lib/python3.10/site-packages (from python-dateutil>=2.1->matplotlib) (1.16.0) Note: you may need to restart the kernel to use updated packages.
By running the next cell you are going to create a folder in your Google Drive. All the files for this tutorial will be uploaded to this folder. After the first execution you might receive some warning and notification, please follow these instructions:
- Permit this notebook to access your Google Drive files? Click on 'Yes', and select your account.
Google Drive for desktop wants to access your Google Account. Click on 'Allow'.
- At this point, a folder has been created and you can navigate it through the lefthand panel in Colab, you might also have received an email that informs you about the access on your Google Drive
In [2]:
# Create a folder in your Google Drive
# from google.colab import drive
# drive.mount('/content/drive')
In [3]:
# Don't run this cell if you already cloned the repo
# !git clone https://github.com/BrainLesion/tutorials.git
In [4]:
# make files from the repo available in colab
import sys
COLAB_BASE_PATH = "/content/tutorials/BraTS-Toolkit/"
sys.path.insert(0, COLAB_BASE_PATH)
Imports¶
In [ ]:
Raw Input data¶
TODO describe raw input data. Two exams from TCIA/TCGA with T1, T1c, T2, T2-FLAIR with skull etc. TODO visualize it
Preprocessing¶
BraTS challenge algorithms expect co-registered, skullstripped files in SRI-24 space, to achieve this preprocessing is required. Instead of using the vanilla preprocessing pipeline from BraTS Toolkit, we recommend using the new BrainLes preprocessing package.
First, we define a function that procoesses an exam.
In [1]:
from auxiliary.normalization.percentile_normalizer import PercentileNormalizer
from auxiliary.turbopath import turbopath
from tqdm import tqdm
from brainles_preprocessing.brain_extraction import HDBetExtractor
from brainles_preprocessing.modality import Modality
from brainles_preprocessing.preprocessor import Preprocessor
from brainles_preprocessing.registration import NiftyRegRegistrator
def preprocess_exam_in_brats_style(inputDir: str) -> None:
"""
Perform BRATS (Brain Tumor Segmentation) style preprocessing on MRI exam data.
Args:
inputDir (str): Path to the directory containing raw MRI files for an exam.
Raises:
Exception: If any error occurs during the preprocessing.
Example:
brat_style_preprocess_exam("/path/to/exam_directory")
This function preprocesses MRI exam data following the BRATS style, which includes the following steps:
1. Normalization using a percentile normalizer.
2. Registration and correction using NiftyReg.
3. Brain extraction using HDBet.
The processed data is saved in a structured directory within the input directory.
Args:
inputDir (str): Path to the directory containing raw MRI files for an exam.
Returns:
None
"""
inputDir = turbopath(inputDir)
print("*** start ***")
brainles_dir = turbopath(inputDir) + "/" + inputDir.name + "_brainles"
norm_bet_dir = brainles_dir / "normalized_bet"
t1_file = inputDir.files("*t1.nii.gz")
t1c_file = inputDir.files("*t1c.nii.gz")
t2_file = inputDir.files("*t2.nii.gz")
flair_file = inputDir.files("*fla.nii.gz")
# we check that we have only one file of each type
if len(t1_file) == len(t1c_file) == len(t2_file) == len(flair_file) == 1:
t1File = t1_file[0]
t1cFile = t1c_file[0]
t2File = t2_file[0]
flaFile = flair_file[0]
# normalizer
percentile_normalizer = PercentileNormalizer(
lower_percentile=0.1,
upper_percentile=99.9,
lower_limit=0,
upper_limit=1,
)
# define modalities
center = Modality(
modality_name="t1c",
input_path=t1cFile,
normalized_bet_output_path=norm_bet_dir / inputDir.name
+ "_t1c_bet_normalized.nii.gz",
atlas_correction=True,
normalizer=percentile_normalizer,
)
moving_modalities = [
Modality(
modality_name="t1",
input_path=t1File,
normalized_bet_output_path=norm_bet_dir / inputDir.name
+ "_t1_bet_normalized.nii.gz",
atlas_correction=True,
normalizer=percentile_normalizer,
),
Modality(
modality_name="t2",
input_path=t2File,
normalized_bet_output_path=norm_bet_dir / inputDir.name
+ "_t2_bet_normalized.nii.gz",
atlas_correction=True,
normalizer=percentile_normalizer,
),
Modality(
modality_name="flair",
input_path=flaFile,
normalized_bet_output_path=norm_bet_dir / inputDir.name
+ "_fla_bet_normalized.nii.gz",
atlas_correction=True,
normalizer=percentile_normalizer,
),
]
preprocessor = Preprocessor(
center_modality=center,
moving_modalities=moving_modalities,
registrator=NiftyRegRegistrator(),
brain_extractor=HDBetExtractor(),
# optional: we provide a temporary directory as a sandbox for the preprocessin
temp_folder="temporary_directory",
limit_cuda_visible_devices="0",
)
preprocessor.run(
save_dir_coregistration=brainles_dir + "/co-registration",
save_dir_atlas_registration=brainles_dir + "/atlas-registration",
save_dir_atlas_correction=brainles_dir + "/atlas-correction",
save_dir_brain_extraction=brainles_dir + "/brain-extraction",
)
now we loop through the exams to preprocess the data:
In [2]:
EXAMPLE_DATA_DIR = turbopath("data")
exams = EXAMPLE_DATA_DIR.dirs()
for exam in tqdm(exams):
print("processing:", exam)
preprocess_exam_in_brats_style(exam)
0%| | 0/2 [00:00<?, ?it/s]
processing: /home/florian/flow/BrainLesion/tutorials/preprocessing/data/OtherEXampleFromTCIA *** start ***
50%|█████ | 1/2 [03:56<03:56, 236.47s/it]
processing: /home/florian/flow/BrainLesion/tutorials/preprocessing/data/TCGA-DU-7294 *** start ***
100%|██████████| 2/2 [10:59<00:00, 329.95s/it]