Model Overview

A pre-trained model for volumetric (3D) segmentation of brain tumor subregions from multimodal MRIs based on BraTS 2018 data. The whole pipeline is modified from clara_pt_brain_mri_segmentation .

The model is trained to segment 3 nested subregions of primary brain tumors (gliomas): the "enhancing tumor" (ET), the "tumor core" (TC), the "whole tumor" (WT) based on 4 aligned input MRI scans (T1c, T1, T2, FLAIR). - The ET is described by areas that show hyper intensity in T1c when compared to T1, but also when compared to "healthy" white matter in T1c. - The TC describes the bulk of the tumor, which is what is typically resected. The TC entails the ET, as well as the necrotic (fluid-filled) and the non-enhancing (solid) parts of the tumor. - The WT describes the complete extent of the disease, as it entails the TC and the peritumoral edema (ED), which is typically depicted by hyper-intense signal in FLAIR.

Data

The training data is from the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2018 .

• Target: 3 tumor subregions
• Task: Segmentation
• Modality: MRI
• Size: 285 3D volumes (4 channels each)

The provided labelled data was partitioned, based on our own split, into training (200 studies), validation (42 studies) and testing (43 studies) datasets.

Preprocessing

The data list/split can be created with the script scripts/prepare_datalist.py .

python scripts/prepare_datalist.py --path your-brats18-dataset-path

Training configuration

This model utilized a similar approach described in 3D MRI brain tumor segmentation using autoencoder regularization, which was a winning method in BraTS2018 [1]. The training was performed with the following:

• GPU: At least 16GB of GPU memory.
• Actual Model Input: 224 x 224 x 144
• AMP: True
• Optimizer: Adam
• Learning Rate: 1e-4
• Loss: DiceLoss

Input

4 channel aligned MRIs at 1 x 1 x 1 mm - T1c - T1 - T2 - FLAIR

Output

3 channels - Label 0: TC tumor subregion - Label 1: WT tumor subregion - Label 2: ET tumor subregion

Performance

Dice score was used for evaluating the performance of the model. This model achieved Dice scores on the validation data of: - Tumor core (TC): 0.8559 - Whole tumor (WT): 0.9026 - Enhancing tumor (ET): 0.7905 - Average: 0.8518

Training Loss and Dice

Validation Dice

MONAI Bundle Commands

In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle. The CLI supports flexible use cases, such as overriding configs at runtime and predefining arguments in a file.

For more details usage instructions, visit the MONAI Bundle Configuration Page .

Execute training:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf

Override the train config to execute multi-GPU training:

torchrun --standalone --nnodes=1 --nproc_per_node=8 -m monai.bundle run training --meta_file configs/metadata.json --config_file "['configs/train.json','configs/multi_gpu_train.json']" --logging_file configs/logging.conf

Please note that the distributed training-related options depend on the actual running environment; thus, users may need to remove --standalone , modify --nnodes , or do some other necessary changes according to the machine used. For more details, please refer to pytorch's official tutorial .

Override the train config to execute evaluation with the trained model:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf

Execute inference:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf

References

[1] Myronenko, Andriy. "3D MRI brain tumor segmentation using autoencoder regularization." International MICCAI Brainlesion Workshop. Springer, Cham, 2018. https://arxiv.org/abs/1810.11654.

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Description

A pre-trained model for breast-density classification.

Model Overview

This model is trained using transfer learning on InceptionV3. The model weights were fine tuned using the Mayo Clinic Data. The details of training and data is outlined in https://arxiv.org/abs/2202.08238.

The bundle does not support torchscript.

Input and Output Formats

The input image should have the size [3, 299, 299]. The output is an array with probabilities for each of the four class.

Sample Data

In the folder sample_data few example input images are stored for each category of images. These images are stored in jpeg format for sharing purpose.

Input and Output Formats

The input image should have the size [299, 299, 3]. For a dicom image which are single channel. The channel can be repeated 3 times. The output is an array with probabilities for each of the four class.

Commands Example

Create a json file with names of all the input files. Execute the following command

python scripts/create_dataset.py -base_dir <path to the bundle root dir>/sample_data -output_file configs/sample_image_data.json

Change the filename for the field data with the absolute path for sample_image_data.json

Add scripts folder to your python path as follows

export PYTHONPATH=$PYTHONPATH:<path to the bundle root dir>/scripts

Execute Inference

The inference can be executed as follows

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json configs/logging.conf

Execute training

It is a work in progress and will be shared in the next version soon.

Contributors

This model is made available from Center for Augmented Intelligence in Imaging, Mayo Clinic Florida. For questions email Vikash Gupta (gupta.vikash@mayo.edu).

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Model Overview

A pre-trained model for the endoscopic inbody classification task and trained using the SEResNet50 structure, whose details can be found in [1]. All datasets are from private samples of Activ Surgical . Samples in training and validation dataset are from the same 4 videos, while test samples are from different two videos.

The PyTorch model and torchscript model are shared in google drive. Modify the bundle_root parameter specified in configs/train.json and configs/inference.json to reflect where models are downloaded. Expected directory path to place downloaded models is models/ under bundle_root .

Data

The datasets used in this work were provided by Activ Surgical .

We've provided a link of 20 samples (10 in-body and 10 out-body) to show what this dataset looks like.

Preprocessing

After downloading this dataset, python script in scripts folder named data_process can be used to generate label json files by running the command below and modifying datapath to path of unziped downloaded data. Generated label json files will be stored in label folder under the bundle path.

python scripts/data_process.py --datapath /path/to/data/root

By default, label path parameter in train.json and inference.json of this bundle is point to the generated label folder under bundle path. If you move these generated label files to another place, please modify the train_json , val_json and test_json parameters specified in configs/train.json and configs/inference.json to where these label files are.

The input label json should be a list made up by dicts which includes image and label keys. An example format is shown below.

[
    {
        "image":"/path/to/image/image_name0.jpg",
        "label": 0
    },
    {
        "image":"/path/to/image/image_name1.jpg",
        "label": 0
    },
    {
        "image":"/path/to/image/image_name2.jpg",
        "label": 1
    },
    ....
    {
        "image":"/path/to/image/image_namek.jpg",
        "label": 0
    },
]

Training configuration

The training as performed with the following: - GPU: At least 12GB of GPU memory - Actual Model Input: 256 x 256 x 3 - Optimizer: Adam - Learning Rate: 1e-3

Input

A three channel video frame

Output

Two Channels - Label 0: in body - Label 1: out body

Performance

Accuracy was used for evaluating the performance of the model. This model achieves an accuracy score of 0.98

Training Loss

Validation Accuracy

MONAI Bundle Commands

In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle. The CLI supports flexible use cases, such as overriding configs at runtime and predefining arguments in a file.

For more details usage instructions, visit the MONAI Bundle Configuration Page .

Execute training:

python -m monai.bundle run training \
    --meta_file configs/metadata.json \
    --config_file configs/train.json \
    --logging_file configs/logging.conf

Override the train config to execute multi-GPU training:

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run training \
    --meta_file configs/metadata.json \
    --config_file "['configs/train.json','configs/multi_gpu_train.json']" \
    --logging_file configs/logging.conf

Please note that the distributed training-related options depend on the actual running environment; thus, users may need to remove --standalone , modify --nnodes , or do some other necessary changes according to the machine used. For more details, please refer to pytorch's official tutorial .

In addition, if using the 20 samples example dataset, the preprocessing script will divide the samples to 16 training samples, 2 validation samples and 2 test samples. However, pytorch multi-gpu training requires number of samples in dataloader larger than gpu numbers. Therefore, please use no more than 2 gpus to run this bundle if using the 20 samples example dataset.

Override the train config to execute evaluation with the trained model:

python -m monai.bundle run evaluating \
    --meta_file configs/metadata.json \
    --config_file "['configs/train.json','configs/evaluate.json']" \
    --logging_file configs/logging.conf

Execute inference:

python -m monai.bundle run evaluating \
    --meta_file configs/metadata.json \
    --config_file configs/inference.json \
    --logging_file configs/logging.conf

The classification result of every images in test.json will be printed to the screen.

Export checkpoint to TorchScript file:

python -m monai.bundle ckpt_export network_def \
    --filepath models/model.ts \
    --ckpt_file models/model.pt \
    --meta_file configs/metadata.json \
    --config_file configs/inference.json

References

[1] J. Hu, L. Shen and G. Sun, Squeeze-and-Excitation Networks, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132-7141. https://arxiv.org/pdf/1709.01507.pdf

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Model Overview

A pre-trained model for the endoscopic tool segmentation task and is trained using a flexible unet structure with an efficient-b2 [1] as the backbone and a UNet architecture [2] as the decoder. Datasets use private samples from Activ Surgical .

The PyTorch model and torchscript model are shared in google drive. Details can be found in large_files.yml file. Modify the "bundle_root" parameter specified in configs/train.json and configs/inference.json to reflect where models are downloaded. Expected directory path to place downloaded models is "models/" under "bundle_root".

Data

Datasets used in this work were provided by Activ Surgical .

Since datasets are private, existing public datasets like EndoVis 2017 can be used to train a similar model.

Preprocessing

When using EndoVis or any other dataset, it should be divided into "train", "valid" and "test" folders. Samples in each folder would better be images and converted to jpg format. Otherwise, "images", "labels", "val_images" and "val_labels" parameters in "configs/train.json" and "datalist" in "configs/inference.json" should be modified to fit given dataset. After that, "dataset_dir" parameter in "configs/train.json" and "configs/inference.json" should be changed to root folder which contains previous "train", "valid" and "test" folders.

Please notice that loading data operation in this bundle is adaptive. If images and labels are not in the same format, it may lead to a mismatching problem. For example, if images are in jpg format and labels are in npy format, PIL and Numpy readers will be used separately to load images and labels. Since these two readers have their own way to parse file's shape, loaded labels will be transpose of the correct ones and incur a missmatching problem.

Training configuration

The training as performed with the following: - GPU: At least 12GB of GPU memory - Actual Model Input: 736 x 480 x 3 - Optimizer: Adam - Learning Rate: 1e-4

Input

A three channel video frame

Output

Two channels: - Label 1: tools - Label 0: everything else

Performance

IoU was used for evaluating the performance of the model. This model achieves a mean IoU score of 0.87.

Training Loss

Validation IoU

MONAI Bundle Commands

In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle. The CLI supports flexible use cases, such as overriding configs at runtime and predefining arguments in a file.

For more details usage instructions, visit the MONAI Bundle Configuration Page .

Execute training:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf

Override the train config to execute multi-GPU training:

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run training --meta_file configs/metadata.json --config_file "['configs/train.json','configs/multi_gpu_train.json']" --logging_file configs/logging.conf

Please note that the distributed training-related options depend on the actual running environment; thus, users may need to remove --standalone , modify --nnodes , or do some other necessary changes according to the machine used. For more details, please refer to pytorch's official tutorial .

Override the train config to execute evaluation with the trained model:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf

Override the train config and evaluate config to execute multi-GPU evaluation:

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json','configs/multi_gpu_evaluate.json']" --logging_file configs/logging.conf

#### Execute inference:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf

#### Export checkpoint to TorchScript file:

python -m monai.bundle ckpt_export network_def --filepath models/model.ts --ckpt_file models/model.pt --meta_file configs/metadata.json --config_file configs/inference.json ```

References

[1] Tan, M. and Le, Q. V. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, 2019a. https://arxiv.org/pdf/1905.11946.pdf

[2] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015. https://arxiv.org/pdf/1505.04597.pdf

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Model Overview

A pre-trained model for volumetric (3D) detection of the lung nodule from CT image.

This model is trained on LUNA16 dataset (https://luna16.grand-challenge.org/Home/), using the RetinaNet (Lin, Tsung-Yi, et al. "Focal loss for dense object detection." ICCV 2017. https://arxiv.org/abs/1708.02002).

Data

The dataset we are experimenting in this example is LUNA16 (https://luna16.grand-challenge.org/Home/), which is based on LIDC-IDRI database [3,4,5].

LUNA16 is a public dataset of CT lung nodule detection. Using raw CT scans, the goal is to identify locations of possible nodules, and to assign a probability for being a nodule to each location.

Disclaimer: We are not the host of the data. Please make sure to read the requirements and usage policies of the data and give credit to the authors of the dataset! We acknowledge the National Cancer Institute and the Foundation for the National Institutes of Health, and their critical role in the creation of the free publicly available LIDC/IDRI Database used in this study.

10-fold data splitting

We follow the official 10-fold data splitting from LUNA16 challenge and generate data split json files using the script from nnDetection .

Please download the resulted json files from https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/LUNA16_datasplit-20220615T233840Z-001.zip.

In these files, the values of "box" are the ground truth boxes in world coordinate.

Data resampling

The raw CT images in LUNA16 have various of voxel sizes. The first step is to resample them to the same voxel size. In this model, we resampled them into 0.703125 x 0.703125 x 1.25 mm.

Please following the instruction in Section 3.1 of https://github.com/Project-MONAI/tutorials/tree/main/detection to do the resampling.

Data download

The mhd/raw original data can be downloaded from LUNA16 . The DICOM original data can be downloaded from LIDC-IDRI database [3,4,5]. You will need to resample the original data to start training.

Alternatively, we provide resampled nifti images and a copy of original mhd/raw images from LUNA16 for users to download.

Training configuration

The training was performed with the following:

• GPU: at least 16GB GPU memory
• Actual Model Input: 192 x 192 x 80
• AMP: True
• Optimizer: Adam
• Learning Rate: 1e-2
• Loss: BCE loss and L1 loss

Input

1 channel - List of 3D CT patches

Output

In Training Mode: A dictionary of classification and box regression loss.

In Evaluation Mode: A list of dictionaries of predicted box, classification label, and classification score.

Performance

Coco metric is used for evaluating the performance of the model. The pre-trained model was trained and validated on data fold 0. This model achieves a mAP=0.853, mAR=0.994, AP(IoU=0.1)=0.862, AR(IoU=0.1)=1.0.

Training Loss

Validation Accuracy

The validation accuracy in this curve is the mean of mAP, mAR, AP(IoU=0.1), and AR(IoU=0.1) in Coco metric.

MONAI Bundle Commands

In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle. The CLI supports flexible use cases, such as overriding configs at runtime and predefining arguments in a file.

For more details usage instructions, visit the MONAI Bundle Configuration Page .

Execute training:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf

Override the train config to execute evaluation with the trained model:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf

Execute inference on resampled LUNA16 images by setting "whether_raw_luna16": false in inference.json :

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf

With the same command, we can execute inference on original LUNA16 images by setting "whether_raw_luna16": true in inference.json . Remember to also set "data_list_file_path": "$@bundle_root + '/LUNA16_datasplit/mhd_original/dataset_fold0.json'" and change "dataset_dir" .

Note that in inference.json, the transform "LoadImaged" in "preprocessing" and "AffineBoxToWorldCoordinated" in "postprocessing" has "affine_lps_to_ras": true . This depends on the input images. LUNA16 needs "affine_lps_to_ras": true . It is possible that your inference dataset should set "affine_lps_to_ras": false .

References

[1] Lin, Tsung-Yi, et al. "Focal loss for dense object detection." ICCV 2017. https://arxiv.org/abs/1708.02002)

[2] Baumgartner and Jaeger et al. "nnDetection: A self-configuring method for medical object detection." MICCAI 2021. https://arxiv.org/pdf/2106.00817.pdf

[3] Armato III, S. G., McLennan, G., Bidaut, L., McNitt-Gray, M. F., Meyer, C. R., Reeves, A. P., Zhao, B., Aberle, D. R., Henschke, C. I., Hoffman, E. A., Kazerooni, E. A., MacMahon, H., Van Beek, E. J. R., Yankelevitz, D., Biancardi, A. M., Bland, P. H., Brown, M. S., Engelmann, R. M., Laderach, G. E., Max, D., Pais, R. C. , Qing, D. P. Y. , Roberts, R. Y., Smith, A. R., Starkey, A., Batra, P., Caligiuri, P., Farooqi, A., Gladish, G. W., Jude, C. M., Munden, R. F., Petkovska, I., Quint, L. E., Schwartz, L. H., Sundaram, B., Dodd, L. E., Fenimore, C., Gur, D., Petrick, N., Freymann, J., Kirby, J., Hughes, B., Casteele, A. V., Gupte, S., Sallam, M., Heath, M. D., Kuhn, M. H., Dharaiya, E., Burns, R., Fryd, D. S., Salganicoff, M., Anand, V., Shreter, U., Vastagh, S., Croft, B. Y., Clarke, L. P. (2015). Data From LIDC-IDRI [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/K9/TCIA.2015.LO9QL9SX

[4] Armato SG 3rd, McLennan G, Bidaut L, McNitt-Gray MF, Meyer CR, Reeves AP, Zhao B, Aberle DR, Henschke CI, Hoffman EA, Kazerooni EA, MacMahon H, Van Beeke EJ, Yankelevitz D, Biancardi AM, Bland PH, Brown MS, Engelmann RM, Laderach GE, Max D, Pais RC, Qing DP, Roberts RY, Smith AR, Starkey A, Batrah P, Caligiuri P, Farooqi A, Gladish GW, Jude CM, Munden RF, Petkovska I, Quint LE, Schwartz LH, Sundaram B, Dodd LE, Fenimore C, Gur D, Petrick N, Freymann J, Kirby J, Hughes B, Casteele AV, Gupte S, Sallamm M, Heath MD, Kuhn MH, Dharaiya E, Burns R, Fryd DS, Salganicoff M, Anand V, Shreter U, Vastagh S, Croft BY. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A completed reference database of lung nodules on CT scans. Medical Physics, 38: 915--931, 2011. DOI: https://doi.org/10.1118/1.3528204

[5] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., & Prior, F. (2013). The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. Journal of Digital Imaging, 26(6), 1045–1057. https://doi.org/10.1007/s10278-013-9622-7

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

MedNIST GAN Hand Model

This model is a generator for creating images like the Hand category in the MedNIST dataset. It was trained as a GAN and accepts random values as inputs to produce an image output. The train.json file describes the training process along with the definition of the discriminator network used, and is based on the MONAI GAN tutorials .

This is a demonstration network meant to just show the training process for this sort of network with MONAI, its outputs are not particularly good and are of the same tiny size as the images in MedNIST. The training process was very short so a network with a longer training time would produce better results.

Downloading the Dataset

Download the dataset from here and extract the contents to a convenient location.

The MedNIST dataset was gathered from several sets from TCIA , the RSNA Bone Age Challenge , and the NIH Chest X-ray dataset .

The dataset is kindly made available by Dr. Bradley J. Erickson M.D., Ph.D. (Department of Radiology, Mayo Clinic) under the Creative Commons CC BY-SA 4.0 license .

If you use the MedNIST dataset, please acknowledge the source.

Training

Assuming the current directory is the bundle directory, and the dataset was extracted to the directory ./MedNIST , the following command will train the network for 50 epochs:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf --bundle_root .

Not also the output from the training will be placed in the models directory but will not overwrite the model.pt file that may be there already. You will have to manually rename the most recent checkpoint file to model.pt to use the inference script mentioned below after checking the results are correct. This saved checkpoint contains a dictionary with the generator weights stored as model and omits the discriminator.

Another feature in the training file is the addition of sigmoid activation to the network by modifying it's structure at runtime. This is done with a line in the training section calling add_module on a layer of the network. This works best for training although the definition of the model now doesn't strictly match what it is in the generator section.

The generator and discriminator networks were both trained with the Adam optimizer with a learning rate of 0.0002 and betas values [0.5, 0.999] . These have been emperically found to be good values for the optimizer and this GAN problem.

Inference

The included inference.json generates a set number of png samples from the network and saves these to the directory ./outputs . The output directory can be changed by setting the output_dir value, and the number of samples changed by setting the num_samples value. The following command line assumes it is invoked in the bundle directory:

python -m monai.bundle run inferring --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf --bundle_root .

Note this script uses postprocessing to apply the sigmoid activation the model's outputs and to save the results to image files.

Export

The generator can be exported to a Torchscript bundle with the following:

python -m monai.bundle ckpt_export network_def --filepath mednist_gan.ts --ckpt_file models/model.pt --meta_file configs/metadata.json --config_file configs/inference.json

The model can be loaded without MONAI code after this operation. For example, an image can be generated from a set of random values with:

import torch
net = torch.jit.load("mednist_gan.ts")
latent = torch.rand(1, 64)
img = net(latent)  # (1,1,64,64)

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

MedNIST Hand Image Registration

Based on the tutorial of 2D registration

Downloading the Dataset

Download the dataset from here and extract the contents to a convenient location.

The MedNIST dataset was gathered from several sets from TCIA , the RSNA Bone Age Challenge , and the NIH Chest X-ray dataset .

The dataset is kindly made available by Dr. Bradley J. Erickson M.D., Ph.D. (Department of Radiology, Mayo Clinic) under the Creative Commons CC BY-SA 4.0 license .

If you use the MedNIST dataset, please acknowledge the source.

Training

Training with same-subject image inputs

python -m monai.bundle run training --config_file configs/train.yaml --dataset_dir "/workspace/data/MedNIST/Hand"

Training with cross-subject image inputs

python -m monai.bundle run training \
  --config_file configs/train.yaml \
  --dataset_dir "/workspace/data/MedNIST/Hand" \
  --cross_subjects True

Training from an existing checkpoint file, for example, models/model_key_metric=-0.0734.pt :

python -m monai.bundle run training --config_file configs/train.yaml [...omitting other args] --ckpt "models/model_key_metric=-0.0734.pt"

Inference

The following figure shows an intra-subject ( --cross_subjects False ) model inference results (Fixed, moving and predicted images from left to right)

The command shows an inference workflow with the checkpoint "models/model_key_metric=-0.0890.pt" and using device "cuda:1" :

python -m monai.bundle run eval \
  --config_file configs/inference.yaml \
  --ckpt "models/model_key_metric=-0.0890.pt" \
  --logging_file configs/logging.conf \
  --device "cuda:1"

Fine-tuning for cross-subject alignments

The following commands starts a finetuning workflow based on the checkpoint "models/model_key_metric=-0.0065.pt" for 5 epochs using the global mutual information loss.

python -m monai.bundle run training \
  --config_file configs/train.yaml \
  --cross_subjects True \
  --ckpt "models/model_key_metric=-0.0065.pt" \
  --lr 0.000001 \
  --trainer#loss_function "@mutual_info_loss" \
  --max_epochs 5

The following figure shows an inter-subject ( --cross_subjects True ) model inference results (Fixed, moving and predicted images from left to right)

Visualize the first pair of images for debugging (requires matplotlib )

python -m monai.bundle run display --config_file configs/train.yaml

python -m monai.bundle run display --config_file configs/train.yaml --cross_subjects True

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Model Overview

A neural architecture search algorithm for volumetric (3D) segmentation of the pancreas and pancreatic tumor from CT image. This model is trained using the neural network model from the neural architecture search algorithm, DiNTS [1].

Data

The training dataset is the Panceas Task from the Medical Segmentation Decathalon. Users can find more details on the datasets at http://medicaldecathlon.com/.

• Target: Liver and tumour
• Modality: Portal venous phase CT
• Size: 420 3D volumes (282 Training +139 Testing)
• Source: Memorial Sloan Kettering Cancer Center
• Challenge: Label unbalance with large (background), medium (pancreas) and small (tumour) structures.

Preprocessing

The data list/split can be created with the script scripts/prepare_datalist.py .

python scripts/prepare_datalist.py --path /path-to-Task07_Pancreas/ --output configs/dataset_0.json

Training configuration

The training was performed with at least 16GB-memory GPUs.

Actual Model Input: 96 x 96 x 96

Neural Architecture Search Configuration

The neural architecture search was performed with the following:

• AMP: True
• Optimizer: SGD
• Initial Learning Rate: 0.025
• Loss: DiceCELoss

Optimial Architecture Training Configuration

The training was performed with the following:

• AMP: True
• Optimizer: SGD
• (Initial) Learning Rate: 0.025
• Loss: DiceCELoss
• Note: If out-of-memory or program crash occurs while caching the data set, please change the cache_rate in CacheDataset to a lower value in the range (0, 1).

The segmentation of pancreas region is formulated as the voxel-wise 3-class classification. Each voxel is predicted as either foreground (pancreas body, tumour) or background. And the model is optimized with gradient descent method minimizing soft dice loss and cross-entropy loss between the predicted mask and ground truth segmentation.

Input

One channel - CT image

Output

Three channels - Label 2: pancreatic tumor - Label 1: pancreas - Label 0: everything else

Performance

Dice score is used for evaluating the performance of the model. This model achieves a mean dice score of 0.62.

Training Loss

The loss over 3200 epochs (the bright curve is smoothed, and the dark one is the actual curve)

Validation Dice

The mean dice score over 3200 epochs (the bright curve is smoothed, and the dark one is the actual curve)

Searched Architecture Visualization

Users can install Graphviz for visualization of searched architectures (needed in custom/decode_plot.py). The edges between nodes indicate global structure, and numbers next to edges represent different operations in the cell searching space. An example of searched architecture is shown as follows:

MONAI Bundle Commands

In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle. The CLI supports flexible use cases, such as overriding configs at runtime and predefining arguments in a file.

For more details usage instructions, visit the MONAI Bundle Configuration Page .

Execute model searching:

python -m scripts.search run --config_file configs/search.yaml

Execute multi-GPU model searching (recommended):

torchrun --nnodes=1 --nproc_per_node=8 -m scripts.search run --config_file configs/search.yaml

Execute training:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.yaml --logging_file configs/logging.conf

Override the train config to execute multi-GPU training:

torchrun --nnodes=1 --nproc_per_node=2 -m monai.bundle run training --meta_file configs/metadata.json --config_file "['configs/train.yaml','configs/multi_gpu_train.yaml']" --logging_file configs/logging.conf

Override the train config to execute evaluation with the trained model:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.yaml','configs/evaluate.yaml']" --logging_file configs/logging.conf

Execute inference:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.yaml --logging_file configs/logging.conf

Export checkpoint for TorchScript

python -m monai.bundle ckpt_export network_def --filepath models/model.ts --ckpt_file models/model.pt --meta_file configs/metadata.json --config_file configs/inference.yaml

References

[1] He, Y., Yang, D., Roth, H., Zhao, C. and Xu, D., 2021. Dints: Differentiable neural network topology search for 3d medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5841-5850).

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Description

A pre-trained model for classifying nuclei cells as the following types. - Other - Inflammatory - Epithelial - Spindle-Shaped

Model Overview

This model is trained using DenseNet121 over ConSeP dataset.

Data

The training dataset is from https://warwick.ac.uk/fac/cross_fac/tia/data/hovernet

wget https://warwick.ac.uk/fac/cross_fac/tia/data/hovernet/consep_dataset.zip
unzip -q consep_dataset.zip

Training configuration

The training was performed with the following:

• GPU: at least 12GB of GPU memory
• Actual Model Input: 4 x 128 x 128
• AMP: True
• Optimizer: Adam
• Learning Rate: 1e-4
• Loss: torch.nn.CrossEntropyLoss

Preprocessing

After downloading this dataset , python script data_process.py from scripts folder can be used to preprocess and generate the final dataset for training.

python scripts/data_process.py --input /path/to/data/CoNSeP --output /path/to/data/CoNSePNuclei

After generating the output files, please modify the dataset_dir parameter specified in configs/train.json and configs/inference.json to reflect the output folder which contains new dataset.json.

Class values in dataset are

• 1 = other
• 2 = inflammatory
• 3 = healthy epithelial
• 4 = dysplastic/malignant epithelial
• 5 = fibroblast
• 6 = muscle
• 7 = endothelial

As part of pre-processing, the following steps are executed.

• Crop and Extract each nuclei Image + Label (128x128) based on the centroid given in the dataset.
• Combine classes 3 & 4 into the epithelial class and 5,6 & 7 into the spindle-shaped class.
• Update the label index for the target nuclie based on the class value
• Other cells which are part of the patch are modified to have label idex = 255

Example dataset.json in output folder:

{
  "training": [
    {
      "image": "/workspace/data/CoNSePNuclei/Train/Images/train_1_3_0001.png",
      "label": "/workspace/data/CoNSePNuclei/Train/Labels/train_1_3_0001.png",
      "nuclei_id": 1,
      "mask_value": 3,
      "centroid": [
        64,
        64
      ]
    }
  ],
  "validation": [
    {
      "image": "/workspace/data/CoNSePNuclei/Test/Images/test_1_3_0001.png",
      "label": "/workspace/data/CoNSePNuclei/Test/Labels/test_1_3_0001.png",
      "nuclei_id": 1,
      "mask_value": 3,
      "centroid": [
        64,
        64
      ]
    }
  ]
}

Input and output formats

Input: 4 channels

• 3 RGB channels
• 1 signal channel (label mask)

Output: 4 channels

• 0 = Other
• 1 = Inflammatory
• 2 = Epithelial
• 3 = Spindle-Shaped

Scores

This model achieves the following F1 score on the validation data provided as part of the dataset:

• Train F1 score = 0.96
• Validation F1 score = 0.85

---

Confusion Metrics for Validation for individual classes are (at epoch 50):

Metric	Other	Inflammatory	Epithelial	Spindle-Shaped
Precision	0.5846	0.7143	0.9158	0.8399
Recall	0.2550	0.8441	0.9193	0.8106
F1-score	0.3551	0.7738	0.9175	0.8250

---

Confusion Metrics for Training for individual classes are (at epoch 50):

Metric	Other	Inflammatory	Epithelial	Spindle-Shaped
Precision	0.9059	0.9569	0.9754	0.9494
Recall	0.8370	0.9547	0.9790	0.9502
F1-score	0.8701	0.9558	0.9772	0.9498

Training Performance

A graph showing the training Loss and F1-score over 50 epochs.

Validation Performance

A graph showing the validation F1-score over 50 epochs.

commands example

Execute training:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf

Override the train config to execute multi-GPU training:

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run training --meta_file configs/metadata.json --config_file "['configs/train.json','configs/multi_gpu_train.json']" --logging_file configs/logging.conf

Please note that the distributed training related options depend on the actual running environment, thus you may need to remove --standalone , modify --nnodes or do some other necessary changes according to the machine you used. Please refer to pytorch's official tutorial for more details.

Override the train config to execute evaluation with the trained model:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf

Override the train config and evaluate config to execute multi-GPU evaluation:

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json','configs/multi_gpu_evaluate.json']" --logging_file configs/logging.conf

Execute inference:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf

Disclaimer

This is an example, not to be used for diagnostic purposes.

References

[1] S. Graham, Q. D. Vu, S. E. A. Raza, A. Azam, Y-W. Tsang, J. T. Kwak and N. Rajpoot. "HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images." Medical Image Analysis, Sept. 2019. [ doi ]

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Model Overview

A pre-trained model for simultaneous segmentation and classification of nuclei within multi-tissue histology images based on CoNSeP data. The details of the model can be found in [1].

Workflow

The model is trained to simultaneous segment and classify nuclei. Training is done via a two-stage approach. First initialized the model with pre-trained weights on the ImageNet dataset , trained only the decoders for the first 50 epochs, and then fine-tuned all layers for another 50 epochs. There are two training modes in total. If "original" mode is specified, it uses [270, 270] and [80, 80] for patch_size and out_size respectively. If "fast" mode is specified, it uses [256, 256] and [164, 164] for patch_size and out_size respectively. The results we show below are based on the "fast" model.

• We train the first stage with pre-trained weights from some internal data.

• The original author's repo also has pre-trained weights which is for non-commercial use. Each user is responsible for checking the content of models/datasets and the applicable licenses and determining if suitable for the intended use. The license for the pre-trained model is different than MONAI license. Please check the source where these weights are obtained from: https://github.com/vqdang/hover_net#data-format

PRETRAIN_MODEL_URL is "https://drive.google.com/u/1/uc?id=1KntZge40tAHgyXmHYVqZZ5d2p_4Qr2l5&export=download" which can be used in bash code below.

Data

The training data is from https://warwick.ac.uk/fac/cross_fac/tia/data/hovernet/ .

• Target: segment instance-level nuclei and classify the nuclei type
• Task: Segmentation and classification
• Modality: RGB images
• Size: 41 image tiles (2009 patches)

The provided labelled data was partitioned, based on the original split, into training (27 tiles) and testing (14 tiles) datasets.

After download the datasets, please run scripts/prepare_patches.py to prepare patches from tiles. Prepared patches are saved in your-concep-dataset-path /Prepared. The implementation is referring to https://github.com/vqdang/hover_net/blob/master/extract_patches.py . The command is like:

python scripts/prepare_patches.py -root your-concep-dataset-path

Training configuration

This model utilized a two-stage approach. The training was performed with the following:

• GPU: At least 24GB of GPU memory.
• Actual Model Input: 256 x 256
• AMP: True
• Optimizer: Adam
• Learning Rate: 1e-4
• Loss: HoVerNetLoss

Input

Input: RGB images

Output

Output: a dictionary with the following keys:

1. nucleus_prediction: predict whether or not a pixel belongs to the nuclei or background
2. horizontal_vertical: predict the horizontal and vertical distances of nuclear pixels to their centres of mass
3. type_prediction: predict the type of nucleus for each pixel

Model Performance

The achieved metrics on the validation data are:

Fast mode: - Binary Dice: 0.8293 - PQ: 0.4936 - F1d: 0.7480

Training Loss and Dice

stage1:

stage2:

Validation Dice

stage1:

stage2:

commands example

Execute training:

• Run first stage

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf --network_def#pretrained_url `PRETRAIN_MODEL_URL` --stage 0

• Run second stage

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf --network_def#freeze_encoder false --network_def#pretrained_url None --stage 1

Override the train config to execute multi-GPU training:

• Run first stage

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run training --meta_file configs/metadata.json --config_file "['configs/train.json','configs/multi_gpu_train.json']" --logging_file configs/logging.conf --train#dataloader#batch_size 8 --network_def#freeze_encoder true --network_def#pretrained_url `PRETRAIN_MODEL_URL` --stage 0

• Run second stage

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run training --meta_file configs/metadata.json --config_file "['configs/train.json','configs/multi_gpu_train.json']" --logging_file configs/logging.conf --train#dataloader#batch_size 4 --network_def#freeze_encoder false --network_def#pretrained_url None --stage 1

Override the train config to execute evaluation with the trained model:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf

Execute inference

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf

Disclaimer

This is an example, not to be used for diagnostic purposes.

References

[1] Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, Nasir Rajpoot, Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images, Medical Image Analysis, 2019 https://doi.org/10.1016/j.media.2019.101563

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Description

A pre-trained model for segmenting nuclei cells with user clicks/interactions.

Model Overview

This model is trained using BasicUNet over ConSeP dataset.

Data

The training dataset is from https://warwick.ac.uk/fac/cross_fac/tia/data/hovernet

wget https://warwick.ac.uk/fac/cross_fac/tia/data/hovernet/consep_dataset.zip
unzip -q consep_dataset.zip

Training configuration

The training was performed with the following:

• GPU: at least 12GB of GPU memory
• Actual Model Input: 5 x 128 x 128
• AMP: True
• Optimizer: Adam
• Learning Rate: 1e-4
• Loss: DiceLoss

Preprocessing

After downloading this dataset , python script data_process.py from scripts folder can be used to preprocess and generate the final dataset for training.

python scripts/data_process.py --input /path/to/data/CoNSeP --output /path/to/data/CoNSePNuclei

After generating the output files, please modify the dataset_dir parameter specified in configs/train.json and configs/inference.json to reflect the output folder which contains new dataset.json.

Class values in dataset are

• 1 = other
• 2 = inflammatory
• 3 = healthy epithelial
• 4 = dysplastic/malignant epithelial
• 5 = fibroblast
• 6 = muscle
• 7 = endothelial

As part of pre-processing, the following steps are executed.

• Crop and Extract each nuclei Image + Label (128x128) based on the centroid given in the dataset.
• Combine classes 3 & 4 into the epithelial class and 5,6 & 7 into the spindle-shaped class.
• Update the label index for the target nuclie based on the class value
• Other cells which are part of the patch are modified to have label idex = 255

Example dataset.json

{
  "training": [
    {
      "image": "/workspace/data/CoNSePNuclei/Train/Images/train_1_3_0001.png",
      "label": "/workspace/data/CoNSePNuclei/Train/Labels/train_1_3_0001.png",
      "nuclei_id": 1,
      "mask_value": 3,
      "centroid": [
        64,
        64
      ]
    }
  ],
  "validation": [
    {
      "image": "/workspace/data/CoNSePNuclei/Test/Images/test_1_3_0001.png",
      "label": "/workspace/data/CoNSePNuclei/Test/Labels/test_1_3_0001.png",
      "nuclei_id": 1,
      "mask_value": 3,
      "centroid": [
        64,
        64
      ]
    }
  ]
}

Input and output formats

Input: 5 channels

• 3 RGB channels
• +ve signal channel (this nuclei)
• -ve signal channel (other nuclei)

Output: 2 channels

• 0 = Background
• 1 = Nuclei

Scores

This model achieves the following Dice score on the validation data provided as part of the dataset:

• Train Dice score = 0.89
• Validation Dice score = 0.85

Training Performance

A graph showing the training Loss and Dice over 50 epochs.

Validation Performance

A graph showing the validation mean Dice over 50 epochs.

commands example

Execute training:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf

Override the train config to execute multi-GPU training:

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run training --meta_file configs/metadata.json --config_file "['configs/train.json','configs/multi_gpu_train.json']" --logging_file configs/logging.conf

Please note that the distributed training related options depend on the actual running environment, thus you may need to remove --standalone , modify --nnodes or do some other necessary changes according to the machine you used. Please refer to pytorch's official tutorial for more details.

Override the train config to execute evaluation with the trained model:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf

Override the train config and evaluate config to execute multi-GPU evaluation:

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json','configs/multi_gpu_evaluate.json']" --logging_file configs/logging.conf

Execute inference:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf

Disclaimer

This is an example, not to be used for diagnostic purposes.

References

[1] Koohbanani, Navid Alemi, et al. "NuClick: a deep learning framework for interactive segmentation of microscopic images." Medical Image Analysis 65 (2020): 101771. https://arxiv.org/abs/2005.14511.

[2] S. Graham, Q. D. Vu, S. E. A. Raza, A. Azam, Y-W. Tsang, J. T. Kwak and N. Rajpoot. "HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images." Medical Image Analysis, Sept. 2019. [ doi ]

[3] NuClick PyTorch Implementation

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Model Overview

A pre-trained model for automated detection of metastases in whole-slide histopathology images.

The model is trained based on ResNet18 [1] with the last fully connected layer replaced by a 1x1 convolution layer.

Data

All the data used to train, validate, and test this model is from Camelyon-16 Challenge . You can download all the images for "CAMELYON16" data set from various sources listed here .

Location information for training/validation patches (the location on the whole slide image where patches are extracted) are adopted from NCRF/coords .

Annotation information are adopted from NCRF/jsons .

• Target: Tumor
• Task: Detection
• Modality: Histopathology
• Size: 270 WSIs for training/validation, 48 WSIs for testing

Preprocessing

This bundle expects the training/validation data (whole slide images) reside in a {dataset_dir}/training/images . By default dataset_dir is pointing to /workspace/data/medical/pathology/ You can modify dataset_dir in the bundle config files to point to a different directory.

To reduce the computation burden during the inference, patches are extracted only where there is tissue and ignoring the background according to a tissue mask. Please also create a directory for prediction output. By default output_dir is set to eval folder under the bundle root.

Please refer to "Annotation" section of Camelyon challenge to prepare ground truth images, which are needed for FROC computation. By default, this data set is expected to be at /workspace/data/medical/pathology/ground_truths . But it can be modified in evaluate_froc.sh .

Training configuration

The training was performed with the following:

• Config file: train.config
• GPU: at least 16 GB of GPU memory.
• Actual Model Input: 224 x 224 x 3
• AMP: True
• Optimizer: Novograd
• Learning Rate: 1e-3
• Loss: BCEWithLogitsLoss
• Whole slide image reader: cuCIM (if running on Windows or Mac, please install OpenSlide on your system and change wsi_reader to "OpenSlide")

Input

The training pipeline is a json file (dataset.json) which includes path to each WSI, the location and the label information for each training patch.

Output

A probability number of the input patch being tumor or normal.

Inference on a WSI

Inference is performed on WSI in a sliding window manner with specified stride. A foreground mask is needed to specify the region where the inference will be performed on, given that background region which contains no tissue at all can occupy a significant portion of a WSI. Output of the inference pipeline is a probability map of size 1/stride of original WSI size.

Performance

FROC score is used for evaluating the performance of the model. After inference is done, evaluate_froc.sh needs to be run to evaluate FROC score based on predicted probability map (output of inference) and the ground truth tumor masks. This model achieve the 0.91 accuracy on validation patches, and FROC of 0.72 on the 48 Camelyon testing data that have ground truth annotations available.

MONAI Bundle Commands

In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle. The CLI supports flexible use cases, such as overriding configs at runtime and predefining arguments in a file.

For more details usage instructions, visit the MONAI Bundle Configuration Page .

Execute training

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf

Override the train config to execute multi-GPU training

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run training --meta_file configs/metadata.json --config_file "['configs/train.json','configs/multi_gpu_train.json']" --logging_file configs/logging.conf

Please note that the distributed training-related options depend on the actual running environment; thus, users may need to remove --standalone , modify --nnodes , or do some other necessary changes according to the machine used. For more details, please refer to pytorch's official tutorial .

Execute inference

CUDA_LAUNCH_BLOCKING=1 python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf

Evaluate FROC metric

cd scripts && source evaluate_froc.sh

Export checkpoint to TorchScript file

TorchScript conversion is currently not supported.

References

[1] He, Kaiming, et al, "Deep Residual Learning for Image Recognition." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. 2016. https://arxiv.org/pdf/1512.03385.pdf

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Prostate MRI zonal segmentation

Authors

Lisa C. Adams, Keno K. Bressem

Tags

Segmentation, MR, Prostate

Model Description

This model was trained with the UNet architecture [1] and is used for 3D volumetric segmentation of the anatomical prostate zones on T2w MRI images. The segmentation of the anatomical regions is formulated as a voxel-wise classification. Each voxel is classified as either central gland (1), peripheral zone (2), or background (0). The model is optimized using a gradient descent method that minimizes the focal soft-dice loss between the predicted mask and the actual segmentation.

Data

The model was trained in the prostate158 training data, which is available at https://doi.org/10.5281/zenodo.6481141. Only T2w images were used for this task.

Preprocessing

MRI images in the prostate158 dataset were preprocessed, including center cropping and resampling. When applying the model to new data, this preprocessing should be repeated.

Center cropping

T2w images were acquired with a voxel spacing of 0.47 x 0.47 x 3 mm and an axial FOV size of 180 x 180 mm. However, the prostate rarely exceeds an axial diameter of 100 mm, and for zonal segmentation, the tissue surrounding the prostate is not of interest and only increases the image size and thus the computational cost. Center-cropping can reduce the image size without sacrificing information.

The script center_crop.py allows to reproduce center-cropping as performed in the prostate158 paper.

python scripts/center_crop.py --file_name path/to/t2_image --out_name cropped_t2

Resampling

DWI and ADC sequences in prostate158 were resampled to the orientation and voxel spacing of the T2w sequence. As the zonal segmentation uses T2w images, no additional resampling is nessecary. However, the training script will perform additonal resampling automatically.

Performance

The model achives the following performance on the prostate158 test dataset:

	Rater 1				Rater 2
Metric	Transitional Zone	Peripheral Zone		Transitional Zone	Peripheral Zone
Dice Coefficient	0.877	0.754		0.875	0.730
Hausdorff Distance	18.3	22.8		17.5	33.2
Surface Distance	2.19	1.95		2.59	1.88

For more details, please see the original publication or official GitHub repository

System Configuration

The model was trained for 100 epochs on a workstaion with a single Nvidia RTX 3080 GPU. This takes approximatly 8 hours.

Limitations (Optional)

This training and inference pipeline was developed for research purposes only. This research use only software that has not been cleared or approved by FDA or any regulatory agency. The model is for research/developmental purposes only and cannot be used directly for clinical procedures.

Citation Info (Optional)

@article{ADAMS2022105817,
title = {Prostate158 - An expert-annotated 3T MRI dataset and algorithm for prostate cancer detection},
journal = {Computers in Biology and Medicine},
volume = {148},
pages = {105817},
year = {2022},
issn = {0010-4825},
doi = {https://doi.org/10.1016/j.compbiomed.2022.105817},
url = {https://www.sciencedirect.com/science/article/pii/S0010482522005789},
author = {Lisa C. Adams and Marcus R. Makowski and Günther Engel and Maximilian Rattunde and Felix Busch and Patrick Asbach and Stefan M. Niehues and Shankeeth Vinayahalingam and Bram {van Ginneken} and Geert Litjens and Keno K. Bressem},
keywords = {Prostate cancer, Deep learning, Machine learning, Artificial intelligence, Magnetic resonance imaging, Biparametric prostate MRI}
}

References

[1] Sakinis, Tomas, et al. "Interactive segmentation of medical images through fully convolutional neural networks." arXiv preprint arXiv:1903.08205 (2019).

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Description

A pre-trained model for training and inferencing volumetric (3D) kidney substructures segmentation from contrast-enhanced CT images (Arterial/Portal Venous Phase). Training pipeline is provided to support model fine-tuning with bundle and MONAI Label active learning.

A tutorial and release of model for kidney cortex, medulla and collecting system segmentation.

Authors: Yinchi Zhou (yinchi.zhou@vanderbilt.edu) | Xin Yu (xin.yu@vanderbilt.edu) | Yucheng Tang (yuchengt@nvidia.com) |

Model Overview

A pre-trained UNEST base model [1] for volumetric (3D) renal structures segmentation using dynamic contrast enhanced arterial or venous phase CT images.

Data

The training data is from the [ImageVU RenalSeg dataset] from Vanderbilt University and Vanderbilt University Medical Center. (The training data is not public available yet).

• Target: Renal Cortex | Medulla | Pelvis Collecting System
• Task: Segmentation
• Modality: CT (Artrial | Venous phase)
• Size: 96 3D volumes

The data and segmentation demonstration is as follow:

Method and Network

The UNEST model is a 3D hierarchical transformer-based semgnetation network.

Details of the architecture:

Training configuration

The training was performed with at least one 16GB-memory GPU.

Actual Model Input: 96 x 96 x 96

Input and output formats

Input: 1 channel CT image

Output: 4: 0:Background, 1:Renal Cortex, 2:Medulla, 3:Pelvicalyceal System

Performance

A graph showing the validation mean Dice for 5000 epochs.

This model achieves the following Dice score on the validation data (our own split from the training dataset):

Mean Valdiation Dice = 0.8523

Note that mean dice is computed in the original spacing of the input data.

commands example

Download trained checkpoint model to ./model/model.pt:

Add scripts component: To run the workflow with customized components, PYTHONPATH should be revised to include the path to the customized component:

export PYTHONPATH=$PYTHONPATH:"'<path to the bundle root dir>/scripts'"

Execute Training:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf

Execute inference:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf

More examples output

Disclaimer

This is an example, not to be used for diagnostic purposes.

References

[1] Yu, Xin, Yinchi Zhou, Yucheng Tang et al. "Characterizing Renal Structures with 3D Block Aggregate Transformers." arXiv preprint arXiv:2203.02430 (2022). https://arxiv.org/pdf/2203.02430.pdf

[2] Zizhao Zhang et al. "Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding." AAAI Conference on Artificial Intelligence (AAAI) 2022

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Model Overview

A pre-trained model for volumetric (3D) segmentation of the spleen from CT images.

This model is trained using the runner-up [1] awarded pipeline of the "Medical Segmentation Decathlon Challenge 2018" using the UNet architecture [2] with 32 training images and 9 validation images.

Data

The training dataset is the Spleen Task from the Medical Segmentation Decathalon. Users can find more details on the datasets at http://medicaldecathlon.com/.

• Target: Spleen
• Modality: CT
• Size: 61 3D volumes (41 Training + 20 Testing)
• Source: Memorial Sloan Kettering Cancer Center
• Challenge: Large-ranging foreground size

Training configuration

The segmentation of spleen region is formulated as the voxel-wise binary classification. Each voxel is predicted as either foreground (spleen) or background. And the model is optimized with gradient descent method minimizing Dice + cross entropy loss between the predicted mask and ground truth segmentation.

The training was performed with the following:

• GPU: at least 12GB of GPU memory
• Actual Model Input: 96 x 96 x 96
• AMP: True
• Optimizer: Adam
• Learning Rate: 1e-4
• Loss: DiceCELoss

Input

One channel - CT image

Output

Two channels - Label 1: spleen - Label 0: everything else

Performance

Dice score is used for evaluating the performance of the model. This model achieves a mean dice score of 0.96.

Training Loss

Validation Dice

MONAI Bundle Commands

In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle. The CLI supports flexible use cases, such as overriding configs at runtime and predefining arguments in a file.

For more details usage instructions, visit the MONAI Bundle Configuration Page .

Execute training:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf

Override the train config to execute multi-GPU training:

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run training --meta_file configs/metadata.json --config_file "['configs/train.json','configs/multi_gpu_train.json']" --logging_file configs/logging.conf

Please note that the distributed training-related options depend on the actual running environment; thus, users may need to remove --standalone , modify --nnodes , or do some other necessary changes according to the machine used. For more details, please refer to pytorch's official tutorial .

Override the train config to execute evaluation with the trained model:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf

Override the train config and evaluate config to execute multi-GPU evaluation:

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json','configs/multi_gpu_evaluate.json']" --logging_file configs/logging.conf

Execute inference:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf

References

[1] Xia, Yingda, et al. "3D Semi-Supervised Learning with Uncertainty-Aware Multi-View Co-Training." arXiv preprint arXiv:1811.12506 (2018). https://arxiv.org/abs/1811.12506.

[2] Kerfoot E., Clough J., Oksuz I., Lee J., King A.P., Schnabel J.A. (2019) Left-Ventricle Quantification Using Residual U-Net. In: Pop M. et al. (eds) Statistical Atlases and Computational Models of the Heart. Atrial Segmentation and LV Quantification Challenges. STACOM 2018. Lecture Notes in Computer Science, vol 11395. Springer, Cham. https://doi.org/10.1007/978-3-030-12029-0_40

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Model Overview

A pre-trained model for 3D segmentation of the spleen organ from CT images using DeepEdit.

DeepEdit is an algorithm that combines the power of two models in one single architecture. It allows the user to perform inference as a standard segmentation method (i.e., UNet) and interactively segment part of an image using clicks [2]. DeepEdit aims to facilitate the user experience and, at the same time, develop new active learning techniques.

The model was trained on 32 images and validated on 9 images.

Data

The training dataset is the Spleen Task from the Medical Segmentation Decathalon. Users can find more details on the datasets at http://medicaldecathlon.com/.

• Target: Spleen
• Modality: CT
• Size: 61 3D volumes (41 Training + 20 Testing)
• Source: Memorial Sloan Kettering Cancer Center
• Challenge: Large-ranging foreground size

Training configuration

The training as performed with the following: - GPU: at least 12GB of GPU memory - Actual Model Input: 128 x 128 x 128 - AMP: True - Optimizer: Adam - Learning Rate: 1e-4 - Loss: DiceCELoss

Input

Three channels - CT image - Spleen Segment - Background Segment

Output

Two channels - Label 1: spleen - Label 0: everything else

Performance

Dice score is used for evaluating the performance of the model. This model achieves a dice score of greater than 0.90, depending on the number of simulated clicks.

Training Dice

Training Loss

Validation Dice

MONAI Bundle Commands

In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle. The CLI supports flexible use cases, such as overriding configs at runtime and predefining arguments in a file.

For more details usage instructions, visit the MONAI Bundle Configuration Page .

Execute training:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf

Override the train config to execute multi-GPU training:

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run training --meta_file configs/metadata.json --config_file "['configs/train.json','configs/multi_gpu_train.json']" --logging_file configs/logging.conf

Please note that the distributed training-related options depend on the actual running environment; thus, users may need to remove --standalone , modify --nnodes , or do some other necessary changes according to the machine used. For more details, please refer to pytorch's official tutorial .

Override the train config to execute evaluation with the trained model:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf

Execute inference:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf

References

[1] Diaz-Pinto, Andres, et al. DeepEdit: Deep Editable Learning for Interactive Segmentation of 3D Medical Images. MICCAI Workshop on Data Augmentation, Labelling, and Imperfections. MICCAI 2022.

[2] Diaz-Pinto, Andres, et al. "MONAI Label: A framework for AI-assisted Interactive Labeling of 3D Medical Images." arXiv preprint arXiv:2203.12362 (2022).

[3] Sakinis, Tomas, et al. "Interactive segmentation of medical images through fully convolutional neural networks." arXiv preprint arXiv:1903.08205 (2019).

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Model Overview

A pre-trained Swin UNETR [1,2] for volumetric (3D) multi-organ segmentation using CT images from Beyond the Cranial Vault (BTCV) Segmentation Challenge dataset [3].

Data

The training data is from the BTCV dataset (Register through Synapse and download the Abdomen/RawData.zip ).

• Target: Multi-organs
• Task: Segmentation
• Modality: CT
• Size: 30 3D volumes (24 Training + 6 Testing)

Preprocessing

The dataset format needs to be redefined using the following commands:

unzip RawData.zip
mv RawData/Training/img/ RawData/imagesTr
mv RawData/Training/label/ RawData/labelsTr
mv RawData/Testing/img/ RawData/imagesTs

Training configuration

The training as performed with the following: - GPU: At least 32GB of GPU memory - Actual Model Input: 96 x 96 x 96 - AMP: True - Optimizer: Adam - Learning Rate: 2e-4

Input

1 channel - CT image

Output

14 channels: - 0: Background - 1: Spleen - 2: Right Kidney - 3: Left Kideny - 4: Gallbladder - 5: Esophagus - 6: Liver - 7: Stomach - 8: Aorta - 9: IVC - 10: Portal and Splenic Veins - 11: Pancreas - 12: Right adrenal gland - 13: Left adrenal gland

Performance

Dice score was used for evaluating the performance of the model. This model achieves a mean dice score of 0.8269

Training Loss

Validation Dice

MONAI Bundle Commands

In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle. The CLI supports flexible use cases, such as overriding configs at runtime and predefining arguments in a file.

For more details usage instructions, visit the MONAI Bundle Configuration Page .

Execute training:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf

Override the train config to execute multi-GPU training:

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run training --meta_file configs/metadata.json --config_file "['configs/train.json','configs/multi_gpu_train.json']" --logging_file configs/logging.conf

Please note that the distributed training-related options depend on the actual running environment; thus, users may need to remove --standalone , modify --nnodes , or do some other necessary changes according to the machine used. For more details, please refer to pytorch's official tutorial .

Override the train config to execute evaluation with the trained model:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf

Execute inference:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf

Export checkpoint to TorchScript file:

TorchScript conversion is currently not supported.

References

[1] Hatamizadeh, Ali, et al. "Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images." arXiv preprint arXiv:2201.01266 (2022). https://arxiv.org/abs/2201.01266.

[2] Tang, Yucheng, et al. "Self-supervised pre-training of swin transformers for 3d medical image analysis." arXiv preprint arXiv:2111.14791 (2021). https://arxiv.org/abs/2111.14791.

[3] Landman B, et al. "MICCAI multi-atlas labeling beyond the cranial vault–workshop and challenge." In Proc. of the MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge 2015 Oct (Vol. 5, p. 12).

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

2D Cardiac Valve Landmark Regressor

This network identifies 10 different landmarks in 2D+t MR images of the heart (2 chamber, 3 chamber, and 4 chamber) representing the insertion locations of valve leaflets into the myocardial wall. These coordinates are used in part of the construction of 3D FEM cardiac models suitable for physics simulation of heart functions.

Input images are individual 2D slices from the time series, and the output from the network is a (2, 10) set of 2D points in HW image coordinate space. The 10 coordinates correspond to the attachment point for these valves:

1. Mitral anterior in 2CH
2. Mitral posterior in 2CH
3. Mitral septal in 3CH
4. Mitral free wall in 3CH
5. Mitral septal in 4CH
6. Mitral free wall in 4CH
7. Aortic septal
8. Aortic free wall
9. Tricuspid septal
10. Tricuspid free wall

Landmarks which do not appear in a particular image are predicted to be (0, 0) or close to this location. The mitral valve is expected to appear in all three views. Landmarks are not provided for the pulmonary valve.

Example plot of landmarks on a single frame, see view_results.ipynb for visualising network output:

Training

The training script train.json is provided to train the network using a dataset of image pairs containing the MR image and a landmark image. This is done to reuse image-based transforms which do not currently operate on geometry. A number of other transforms are provided in valve_landmarks.py to implement Fourier-space dropout, image shifting which preserve landmarks, and smooth-field deformation applied to images and landmarks.

The dataset used for training unfortunately cannot be made public, however the training script can be used with any NPZ file containing the training image stack in key trainImgs and landmark image stack in trainLMImgs , plus testImgs and testLMImgs containing validation data. The landmark images are defined as 0 for every non-landmark pixel, with landmark pixels contaning the following values for each landmark type:

• 10: Mitral anterior in 2CH
• 15: Mitral posterior in 2CH
• 20: Mitral septal in 3CH
• 25: Mitral free wall in 3CH
• 30: Mitral septal in 4CH
• 35: Mitral free wall in 4CH
• 100: Aortic septal
• 150: Aortic free wall
• 200: Tricuspid septal
• 250: Tricuspid free wall

The following command will train with the default NPZ filename ./valvelandmarks.npz , assuming the current directory is the bundle directory:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file "['configs/train.json', 'configs/common.json']" \
    --bundle_root . --dataset_file ./valvelandmarks.npz --output_dir /path/to/outputs

Inference

The included inference.json script will run inference on a directory containing Nifti files whose images have shape (256, 256, 1, N) for N timesteps. For each image the output in the output_dir directory will be a npy file containing a result array of shape (N, 2, 10) storing the 10 coordinates for each N timesteps. Invoking this script can be done as follows, assuming the current directory is the bundle directory:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/inference.json', 'configs/common.json']" \
    --bundle_root . --dataset_dir /path/to/data --output_dir /path/to/outputs

The provided test Nifti file can be placed in a directory which is then used as the dataset_dir value. This image was derived from the AMRG Cardiac Atlas dataset (AMRG Cardiac Atlas, Auckland MRI Research Group, Auckland, New Zealand). The results from this inference can be visualised by changing path values in view_results.ipynb .

Reference

The work for this model and its application is described in:

Kerfoot, E, King, CE, Ismail, T, Nordsletten, D & Miller, R 2021, Estimation of Cardiac Valve Annuli Motion with Deep Learning. in E Puyol Anton, M Pop, M Sermesant, V Campello, A Lalande, K Lekadir, A Suinesiaputra, O Camara & A Young (eds), Statistical Atlases and Computational Models of the Heart. MandMs and EMIDEC Challenges - 11th International Workshop, STACOM 2020, Held in Conjunction with MICCAI 2020, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12592 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 146-155, 11th International Workshop on Statistical Atlases and Computational Models of the Heart, STACOM 2020 held in Conjunction with MICCAI 2020, Lima, Peru, 4/10/2020. https://doi.org/10.1007/978-3-030-68107-4_15

License

This model is released under the MIT License. The license file is included with the model.

3 Label Ventricular Segmentation

This network segments cardiac ventricle in 2D short axis MR images. The left ventricular pool is class 1, left ventricular myocardium class 2, and right ventricular pool class 3. Full cycle segmentation with this network is possible although much of the training data is composed of segmented end-diastole images. The input to the network is single 2D images thus segmenting whole time-dependent volumes consists of multiple inference operations.

The network and training scheme are essentially identical to that described in:

Kerfoot E., Clough J., Oksuz I., Lee J., King A.P., Schnabel J.A. (2019) Left-Ventricle Quantification Using Residual U-Net. In: Pop M. et al. (eds) Statistical Atlases and Computational Models of the Heart. Atrial Segmentation and LV Quantification Challenges. STACOM 2018. Lecture Notes in Computer Science, vol 11395. Springer, Cham. https://doi.org/10.1007/978-3-030-12029-0_40

Data

The dataset used to train this network unfortunately cannot be made public as it contains unreleased image data from King's College London. Existing public datasets such as the Sunnybrook Cardiac Dataset and ACDC Challenge set can be used to train a similar network.

The train.json configuration assumes all data is stored in a single npz file with keys "images" and "segs" containing respectively the raw image data and their accompanying segmentations. The given network was training with stored volumes with shapes (9095, 256, 256) thus other data of differing spatial dimensions must be cropped to (256, 256) or zero-padded to that size. For the training data this was done as a preprocessing step but the original pixel values are otherwise unchanged from their original forms.

Training

The network is trained with this data in conjunction with a series of augmentations for regularisation and robustness. Many of the original images are smaller than the expected size of (256, 256) and so were zero-padded, the network can thus be expected to be robust against large amounts of empty space in the inputs. Rotation and zooming is also applied to force the network to learn different sizes and orientations of the heart in the field of view.

Free-form deformation is applied to vary the shape of the heart and its surrounding tissues which mimics to a degree deformation like what would be observed through the cardiac cycle. This of course does not replicate the heart moving through plane during the cycle or represent other observed changes but does provide enough variation that full-cycle segmentation is generally acceptable.

Smooth fields are used to vary contrast and intensity in localised regions to simulate some of the variation in image quality caused by acquisition artefacts. Guassian noise is also added to simulate poor quality acquisition. These together force the network to learn to deal with a wider variation of image quality and partially to account for the difference between scanner vendors.

Training is invoked with the following command line:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf --bundle_root .

The dataset file is assumed to be allimages3label.npz but can be changed by setting the dataset_file value to your own file.

Inference

An example notebook visualise.ipynb demonstrates using the network directly with input images. Inference of 3D volumes only can be accomplished with the inference.json configuration:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf --dataset_dir dataset --output_dir ./output/ --bundle_root .

License

This model is released under the MIT License. The license file is included with the model.

Model Overview

Body CT segmentation models are evolving. Starting from abdominal multi-organ segmentation model [1]. Now the community is developing hundreds of target anatomies. In this bundle, we provide re-trained models for (3D) segmentation of 104 whole-body segments.

This model is trained using the SegResNet [3] network. The model is trained using TotalSegmentator datasets [2].

Figure source from the TotalSegmentator [2].

MONAI Label Showcase

• We highlight the use of this bundle to use and visualize in MONAI Label + 3D Slicer integration.

Data

The training set is the 104 whole-body structures from the TotalSegmentator released datasets. Users can find more details on the datasets at https://github.com/wasserth/TotalSegmentator. All rights and licenses are reserved to the original authors.

• Target: 104 structures
• Modality: CT
• Source: TotalSegmentator
• Challenge: Large volumes of structures in CT images

Preprocessing

To use the bundle, users need to download the data and merge all annotated labels into one NIFTI file. Each file contains 0-104 values, each value represents one anatomy class. A sample set is provided with this link .

Training Configuration

The segmentation of 104 tissues is formulated as voxel-wise multi-label segmentation. The model is optimized with the gradient descent method minimizing Dice + cross-entropy loss between the predicted mask and ground truth segmentation.

The training was performed with the following:

• GPU: 32 GB of GPU memory
• Actual Model Input: 96 x 96 x 96
• AMP: True
• Optimizer: AdamW
• Learning Rate: 1e-4
• Loss: DiceCELoss

Input

One channel - CT image

Output

105 channels - Label 0: Background (everything else) - label 1-105: Foreground classes (104)

High-Resolution and Low-Resolution Models

We retrained two versions of the totalSegmentator models, following the original paper and implementation. To meet multiple demands according to computation resources and performance, we provide a 1.5 mm model and a 3.0 mm model, both models are trained with 104 foreground output channels.

In this bundle, we configured a parameter called highres , users can set it to true when using 1.5 mm model, and set it to false to use the 3.0 mm model. The high-resolution model is named model.pt by default, the low-resolution model is named model_lowres.pt .

In MONAI Label use case, users can set the parameter in 3D Slicer plugin to control which model to infer and train.

• Pretrained Checkpoints
• 1.5 mm model: Download link
• 3.0 mm model: Download link

Resource Requirements and Latency Benchmarks

Latencies and memory performance of using the bundle with MONAI Label:

Tested Image Dimension: (512, 512, 397) , the slice thickness is 1.5mm in this case. After resample to 1.5 isotropic resolution, the dimension is (287, 287, 397)

1.5 mm (highres) model (Single Model with 104 foreground classes)

Benchmarking on GPU: Memory: 28.73G

• ++ Latencies => Total: 6.0277; Pre: 1.6228; Inferer: 4.1153; Invert: 0.0000; Post: 0.0897; Write: 0.1995

Benchmarking on CPU: Memory: 26G

• ++ Latencies => Total: 38.3108; Pre: 1.6643; Inferer: 30.3018; Invert: 0.0000; Post: 6.1656; Write: 0.1786

3.0 mm (lowres) model (single model with 104 foreground classes)

GPU: Memory: 5.89G

• ++ Latencies => Total: 1.9993; Pre: 1.2363; Inferer: 0.5207; Invert: 0.0000; Post: 0.0358; Write: 0.2060

CPU: Memory: 2.3G

• ++ Latencies => Total: 6.6138; Pre: 1.3192; Inferer: 3.6746; Invert: 0.0000; Post: 1.4431; Write: 0.1760

Performance

• 1.5 mm Model Training

• Training Accuracy

• Validation Dice

MONAI Bundle Commands

In addition to the Pythonic APIs, a few command line interfaces (CLI) are provided to interact with the bundle. The CLI supports flexible use cases, such as overriding configs at runtime and predefining arguments in a file.

For more details usage instructions, visit the MONAI Bundle Configuration Page .

Execute training

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf

Override the train config to execute multi-GPU training

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run training --meta_file configs/metadata.json --config_file "['configs/train.json','configs/multi_gpu_train.json']" --logging_file configs/logging.conf

Please note that the distributed training-related options depend on the actual running environment; thus, users may need to remove --standalone , modify --nnodes , or do some other necessary changes according to the machine used. For more details, please refer to pytorch's official tutorial .

Override the train config to execute evaluation with the trained model

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json']" --logging_file configs/logging.conf

Override the train config and evaluate config to execute multi-GPU evaluation

torchrun --standalone --nnodes=1 --nproc_per_node=2 -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file "['configs/train.json','configs/evaluate.json','configs/multi_gpu_evaluate.json']" --logging_file configs/logging.conf

Execute inference

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf

Execute inference with Data Samples

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf --datalist "['sampledata/imagesTr/s0037.nii.gz','sampledata/imagesTr/s0038.nii.gz']"

References

[1] Tang, Y., Gao, R., Lee, H.H., Han, S., Chen, Y., Gao, D., Nath, V., Bermudez, C., Savona, M.R., Abramson, R.G. and Bao, S., 2021. High-resolution 3D abdominal segmentation with random patch network fusion. Medical image analysis, 69, p.101894.

[2] Wasserthal, J., Meyer, M., Breit, H.C., Cyriac, J., Yang, S. and Segeroth, M., 2022. TotalSegmentator: robust segmentation of 104 anatomical structures in CT images. arXiv preprint arXiv:2208.05868.

[3] Myronenko, A., Siddiquee, M.M.R., Yang, D., He, Y. and Xu, D., 2022. Automated head and neck tumor segmentation from 3D PET/CT. arXiv preprint arXiv:2209.10809.

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Description

Detailed whole brain segmentation is an essential quantitative technique in medical image analysis, which provides a non-invasive way of measuring brain regions from a clinical acquired structural magnetic resonance imaging (MRI). We provide the pre-trained model for training and inferencing whole brain segmentation with 133 structures. Training pipeline is provided to support active learning in MONAI Label and training with bundle.

A tutorial and release of model for whole brain segmentation using the 3D transformer-based segmentation model UNEST.

Authors: Xin Yu (xin.yu@vanderbilt.edu)

Yinchi Zhou (yinchi.zhou@vanderbilt.edu) | Yucheng Tang (yuchengt@nvidia.com)

-------------------------------------------------------------------------------------

Fig.1 - The demonstration of T1w MRI images registered in MNI space and the whole brain segmentation labels with 133 classes

Model Overview

A pre-trained UNEST base model [1] for volumetric (3D) whole brain segmentation with T1w MR images. To leverage information across embedded sequences, ”shifted window” transformers are proposed for dense predictions and modeling multi-scale features. However, these attempts that aim to complicate the self-attention range often yield high computation complexity and data inefficiency. Inspired by the aggregation function in the nested ViT, we propose a new design of a 3D U-shape medical segmentation model with Nested Transformers (UNesT) hierarchically with the 3D block aggregation function, that learn locality behaviors for small structures or small dataset. This design retains the original global self-attention mechanism and achieves information communication across patches by stacking transformer encoders hierarchically.

Fig.2 - The network architecture of UNEST Base model

Data

The training data is from the Vanderbilt University and Vanderbilt University Medical Center with public released OASIS and CANDI datsets. Training and testing data are MRI T1-weighted (T1w) 3D volumes coming from 3 different sites. There are a total of 133 classes in the whole brain segmentation task. Among 50 T1w MRI scans from Open Access Series on Imaging Studies (OASIS) (Marcus et al., 2007) dataset, 45 scans are used for training and the other 5 for validation. The testing cohort contains Colin27 T1w scan (Aubert-Broche et al., 2006) and 13 T1w MRI scans from the Child and Adolescent Neuro Development Initiative (CANDI) (Kennedy et al., 2012). All data are registered to the MNI space using the MNI305 (Evans et al., 1993) template and preprocessed follow the method in (Huo et al., 2019). Input images are randomly cropped to the size of 96 × 96 × 96.

Important

The brain MRI images for training are registered to Affine registration from the target image to the MNI305 template using NiftyReg. The data should be in the MNI305 space before inference.

If your images are already in MNI space, skip the registration step.

You could use any resitration tool to register image to MNI space. Here is an example using ants. Registration to MNI Space: Sample suggestion. E.g., use ANTS or other tools for registering T1 MRI image to MNI305 Space.

pip install antspyx

#Sample ANTS registration

import ants
import sys
import os

fixed_image = ants.image_read('<fixed_image_path>')
moving_image = ants.image_read('<moving_image_path>')
transform = ants.registration(fixed_image,moving_image,'Affine')

reg3t = ants.apply_transforms(fixed_image,moving_image,transform['fwdtransforms'][0])
ants.image_write(reg3t,output_image_path)

Training configuration

The training and inference was performed with at least one 24GB-memory GPU.

Actual Model Input: 96 x 96 x 96

Input and output formats

Input: 1 channel T1w MRI image in MNI305 Space.

commands example

Download trained checkpoint model to ./model/model.pt:

Add scripts component: To run the workflow with customized components, PYTHONPATH should be revised to include the path to the customized component:

export PYTHONPATH=$PYTHONPATH: '<path to the bundle root dir>/scripts'

Execute Training:

python -m monai.bundle run training --meta_file configs/metadata.json --config_file configs/train.json --logging_file configs/logging.conf

Execute inference:

python -m monai.bundle run evaluating --meta_file configs/metadata.json --config_file configs/inference.json --logging_file configs/logging.conf

More examples output

Fig.3 - The output prediction comparison with variant and ground truth

Training/Validation Benchmarking

A graph showing the training accuracy for fine-tuning 600 epochs.

With 10 fine-tuned labels, the training process converges fast.

Complete ROI of the whole brain segmentation

133 brain structures are segmented.

#1	#2	#3	#4
0: background	1 : 3rd-Ventricle	2 : 4th-Ventricle	3 : Right-Accumbens-Area
4 : Left-Accumbens-Area	5 : Right-Amygdala	6 : Left-Amygdala	7 : Brain-Stem
8 : Right-Caudate	9 : Left-Caudate	10 : Right-Cerebellum-Exterior	11 : Left-Cerebellum-Exterior
12 : Right-Cerebellum-White-Matter	13 : Left-Cerebellum-White-Matter	14 : Right-Cerebral-White-Matter	15 : Left-Cerebral-White-Matter
16 : Right-Hippocampus	17 : Left-Hippocampus	18 : Right-Inf-Lat-Vent	19 : Left-Inf-Lat-Vent
20 : Right-Lateral-Ventricle	21 : Left-Lateral-Ventricle	22 : Right-Pallidum	23 : Left-Pallidum
24 : Right-Putamen	25 : Left-Putamen	26 : Right-Thalamus-Proper	27 : Left-Thalamus-Proper
28 : Right-Ventral-DC	29 : Left-Ventral-DC	30 : Cerebellar-Vermal-Lobules-I-V	31 : Cerebellar-Vermal-Lobules-VI-VII
32 : Cerebellar-Vermal-Lobules-VIII-X	33 : Left-Basal-Forebrain	34 : Right-Basal-Forebrain	35 : Right-ACgG--anterior-cingulate-gyrus
36 : Left-ACgG--anterior-cingulate-gyrus	37 : Right-AIns--anterior-insula	38 : Left-AIns--anterior-insula	39 : Right-AOrG--anterior-orbital-gyrus
40 : Left-AOrG--anterior-orbital-gyrus	41 : Right-AnG---angular-gyrus	42 : Left-AnG---angular-gyrus	43 : Right-Calc--calcarine-cortex
44 : Left-Calc--calcarine-cortex	45 : Right-CO----central-operculum	46 : Left-CO----central-operculum	47 : Right-Cun---cuneus
48 : Left-Cun---cuneus	49 : Right-Ent---entorhinal-area	50 : Left-Ent---entorhinal-area	51 : Right-FO----frontal-operculum
52 : Left-FO----frontal-operculum	53 : Right-FRP---frontal-pole	54 : Left-FRP---frontal-pole	55 : Right-FuG---fusiform-gyrus
56 : Left-FuG---fusiform-gyrus	57 : Right-GRe---gyrus-rectus	58 : Left-GRe---gyrus-rectus	59 : Right-IOG---inferior-occipital-gyrus ,
60 : Left-IOG---inferior-occipital-gyrus	61 : Right-ITG---inferior-temporal-gyrus	62 : Left-ITG---inferior-temporal-gyrus	63 : Right-LiG---lingual-gyrus
64 : Left-LiG---lingual-gyrus	65 : Right-LOrG--lateral-orbital-gyrus	66 : Left-LOrG--lateral-orbital-gyrus	67 : Right-MCgG--middle-cingulate-gyrus
68 : Left-MCgG--middle-cingulate-gyrus	69 : Right-MFC---medial-frontal-cortex	70 : Left-MFC---medial-frontal-cortex	71 : Right-MFG---middle-frontal-gyrus
72 : Left-MFG---middle-frontal-gyrus	73 : Right-MOG---middle-occipital-gyrus	74 : Left-MOG---middle-occipital-gyrus	75 : Right-MOrG--medial-orbital-gyrus
76 : Left-MOrG--medial-orbital-gyrus	77 : Right-MPoG--postcentral-gyrus	78 : Left-MPoG--postcentral-gyrus	79 : Right-MPrG--precentral-gyrus
80 : Left-MPrG--precentral-gyrus	81 : Right-MSFG--superior-frontal-gyrus	82 : Left-MSFG--superior-frontal-gyrus	83 : Right-MTG---middle-temporal-gyrus
84 : Left-MTG---middle-temporal-gyrus	85 : Right-OCP---occipital-pole	86 : Left-OCP---occipital-pole	87 : Right-OFuG--occipital-fusiform-gyrus
88 : Left-OFuG--occipital-fusiform-gyrus	89 : Right-OpIFG-opercular-part-of-the-IFG	90 : Left-OpIFG-opercular-part-of-the-IFG	91 : Right-OrIFG-orbital-part-of-the-IFG
92 : Left-OrIFG-orbital-part-of-the-IFG	93 : Right-PCgG--posterior-cingulate-gyrus	94 : Left-PCgG--posterior-cingulate-gyrus	95 : Right-PCu---precuneus
96 : Left-PCu---precuneus	97 : Right-PHG---parahippocampal-gyrus	98 : Left-PHG---parahippocampal-gyrus	99 : Right-PIns--posterior-insula
100 : Left-PIns--posterior-insula	101 : Right-PO----parietal-operculum	102 : Left-PO----parietal-operculum	103 : Right-PoG---postcentral-gyrus
104 : Left-PoG---postcentral-gyrus	105 : Right-POrG--posterior-orbital-gyrus	106 : Left-POrG--posterior-orbital-gyrus	107 : Right-PP----planum-polare
108 : Left-PP----planum-polare	109 : Right-PrG---precentral-gyrus	110 : Left-PrG---precentral-gyrus	111 : Right-PT----planum-temporale
112 : Left-PT----planum-temporale	113 : Right-SCA---subcallosal-area	114 : Left-SCA---subcallosal-area	115 : Right-SFG---superior-frontal-gyrus
116 : Left-SFG---superior-frontal-gyrus	117 : Right-SMC---supplementary-motor-cortex	118 : Left-SMC---supplementary-motor-cortex	119 : Right-SMG---supramarginal-gyrus
120 : Left-SMG---supramarginal-gyrus	121 : Right-SOG---superior-occipital-gyrus	122 : Left-SOG---superior-occipital-gyrus	123 : Right-SPL---superior-parietal-lobule
124 : Left-SPL---superior-parietal-lobule	125 : Right-STG---superior-temporal-gyrus	126 : Left-STG---superior-temporal-gyrus	127 : Right-TMP---temporal-pole
128 : Left-TMP---temporal-pole	129 : Right-TrIFG-triangular-part-of-the-IFG	130 : Left-TrIFG-triangular-part-of-the-IFG	131 : Right-TTG---transverse-temporal-gyrus
132 : Left-TTG---transverse-temporal-gyrus

Bundle Integration in MONAI Lable

The inference and training pipleine can be easily used by the MONAI Label server and 3D Slicer for fast labeling T1w MRI images in MNI space.

Disclaimer

This is an example, not to be used for diagnostic purposes.

References

[1] Yu, Xin, Yinchi Zhou, Yucheng Tang et al. Characterizing Renal Structures with 3D Block Aggregate Transformers. arXiv preprint arXiv:2203.02430 (2022). https://arxiv.org/pdf/2203.02430.pdf

[2] Zizhao Zhang et al. Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding. AAAI Conference on Artificial Intelligence (AAAI) 2022

[3] Huo, Yuankai, et al. 3D whole brain segmentation using spatially localized atlas network tiles. NeuroImage 194 (2019): 105-119.

License

Copyright (c) MONAI Consortium

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.