Helsinki Tomography Challenge 2022

What is limited angle tomography?

Computed tomography means reconstructing the internal structure of a physical body using X-ray images of the body taken from different directions. Mathematically, the problem is to recover a non-negative function from a collection of line integrals. Reconstruction of the original, full object requires that measurements are obtained continuously at least 180° around the object.

In limited-angle tomography reconstruction, the object is imaged using only limited angle interval of X-ray projections. The significance of this is that the reconstruction must be computed from an incomplete set of line integrals, a highly ill-posed and challenging task.

Challenge description

The purpose of the challenge is to recover the shapes of 2D targets imaged with limited-angle tomography, collected in the Industrial Mathematics Computed Tomography Laboratory at the University of Helsinki, Finland. The experimental setup, targets, and measurement protocol are described in the following sections.

The outcome of the challenge should be an algorithm which takes in the X-ray data, i.e., the sinogram and it’s associated metadata about the measurement geometry, and produces a reconstruction which has been segmented into two components: air and plastic.

Organising Committee

Rules

The rules and information about the HTC2022 can also be found in this pdf . (Updated 28.10.2022)

How to enter the competition

To enter the HTC2022 competition:Register before 23:59 EET (Eastern European Time) on September 30, 2022, using this electronic form..Send your submission to htc2022(“at”)fips.fi before November 4, 2022 23:59 EET (Eastern European Time). What needs to be submitted? See below for instructions.

Only submissions that fulfill the requirements listed below will be accepted.

Requirements of the competition

What needs to be submitted? Briefly, the codes must be in Matlab or Python 3.X and the algorithms must be shared with us as a private GitHub repository at latest on deadline. Check the following subsections for detailed instructions. Only submissions that fulfill the requirements listed below will be accepted.

The teams can submit more than one reconstruction algorithm to the challenge, however, each algorithm must be in a separate repository. The maximum number of algorithms is the number of members of the team. Your team do not need to register multiple times in case you decide to submit more than one algorithm to the challenge. The team can send a single email with the links to all the repositories.

After the deadline, there is a brief period during which we can troubleshoot the codes together with the competing teams. This is to ensure that we are able to run the codes. The troubleshoot communication is done mainly via ‘Issues’ section of the submitted repository, so pay attention to any activities in the repository after the deadline.

Special situations: The spirit of the competition is that the algorithm is a general-purpose algorithm, capable of reconstructing limited-angle tomography images of the targets. The organizing committee has the right to disqualify an algorithm trying to violate that spirit.

Conflict of interest: researchers affiliated with the Department of Mathematics and Statistics of University of Helsinki will not be added to the leaderboard and cannot win the competition.

Deadline

Deadline: November 4, 2022 23:59 EET (Eastern European Time)

The algorithms must be shared with us as a private GitHub repository at latest on the deadline. The codes should be in Matlab or Python3.

After the deadline there is a brief period during which we can troubleshoot the codes together with the competing teams. This is to ensure that we are able to run the codes.

Github repository

Competitors can update the contents of the shared repository as many times as needed before the deadline. We will consider only the latest release of your repository on Github.

Attention: Simple commits to the main branch will not be considered. You MUST also create a release. You can find Github’s documentation on how to create releases here. If the latest release does not work we will not accept older versions.

Your repository must contain a README.md file with at least the following sections:

• Authors, institution, location.
• Brief description of your algorithm and a mention of the competition.
• Installation instructions, including any requirements.
  • Matlab users: Please specify any toolboxes used.
  • Python users: Please specify any modules used. If you use Anaconda, please add to the repository an environment.yml file capable of creating an environment than can run your code (instructions). Otherwise, please add a requirements.txt file generated with pip freeze (instructions)
• Usage instructions.
• Show few examples.

If your algorithm requires uploading large files to Github (e.g. with trained coefficients of a neural network), you can use Git Large File Storage (preferable way) or store them in another server and add the link to the Github installation instructions.

The teams can submit more than one algorithm to the challenge, each algorithm must be in a separate repository. The maximum number of algorithms is the number of members of the team. The teams don’t need to register multiple times in case they decide to submit more than one algorithm to the challenge.

Your code on Github

The repository must contain a main routine that we can run to apply your algorithm automatically to every image in a given directory. This is the file we will run to evaluate your code. Give it an easy to identify name like main.m or main.py.

Important: The input directory contains only the test dataset. No training dataset is provided to your code during the assessment. Therefore, any training procedures must be performed by your team before the submission.

Your main routine must require three input arguments:

1. (string) Folder where the input image files are located
2. (string) Folder where the output images must be stored
3. (int) Difficulty category number. Values between 1 and 7

Below are the expected formats of the main routines in python and Matlab:

Matlab: The main function must be a callable function:

function main(inputFolder,outputFolder,categoryNbr)
...

your code comes here
...

Example calling the function:

>> main('path/to/input/files', 'path/to/output/files', 3)

Python: The main function must be a callable function from the command line. To achieve this you can use sys.argv or argparse module.

Example calling the function:

$ python3 main.py path/to/input/files path/to/output/files 3

The main routine must produce a reconstructed PNG file in the output folder for each image in the input folder. The output PNG images must have dimensions 512 x 512 pixels and the same filename apart from the extension. All images in the input directory belong to the same difficulty category, specified by the input argument. (Updated 28.10.2022)

The teams are allowed to use freely available python modules or Matlab toolboxes. Toolboxes, libraries and modules with paid licenses can also be used if the organizing committee also have the license. For example, the most usual Matlab toolboxes for image processing and deconvolution can be used (Image processing toolbox, wavelet toolbox, PDE toolbox, computer vision toolbox, deep learning toolbox, optimization toolbox). The teams can contact us to check if other toolboxes are available.

Scores and leaderboard

The scores and leaderboard are constructed step-wise as follows:

• All teams start with difficulty level 1. The reconstructions of the three samples (A, B, and C) of this level will be assessed quantitatively following the criteria described below, and their scores will be summed, forming the total score of the first level

• The team with the highest score S1 will be used as reference for the cut-off score of this level: any team with score S1 at least 25% of the highest score will pass to the next level.
• The same procedure is repeated for all difficulty levels, up to level 7, but considering only the teams that passed the cut-off score of the previous level:

• The cut-off of the levels stays fixed at 25% of the maximum at each level.
• Denote by Nmax the hardest level that at least one team could enter. If there is only one team, they win. If there are several teams competing in level Nmax they are ordered in the leaderboard according to their scores at that level.
• In case of a tie in level Nmax, the previous levels, starting from Nmax-1, will be compared until one of the competitors win. If the tie persists, the organizing committee will make the final decision on the winner.

Note: If one team submits more than one algorithm to the competition, then each submission will be temporarily assumed to belong to different ‘virtual’ teams when computing the scores and cut-offs. However, this team cannot be in more than one position in the leaderboard. In this situation, the organizing committee will consider only the highest performance algorithm when ranking the winners.

Open science spirit

Finally, the competitors must make their GitHub repositories public at latest on November 30, 2022. In the spirit of open science, only a public code can win HTC2022.

Limited angle tomography data for the challenge

The actual challenge data consists of 21 phantoms, arranged into seven groups of gradually increasing difficulty, with each level containing three different phantoms, labeled A, B, and C. As the difficulty level increases, the number of holes increases and their shapes become increasingly complex. Furthermore, the view-angle is reduced as the difficulty level increases, starting with a 90-degree field-of-view at level 1, and reducing by 10 degrees at each increasing level of difficulty. Each target is assigned to a single group, therefore, each target is used only once.

The limited data is then passed as input to the submitted algorithms for assessment of the reconstructions. See code examples for more details.

The targets have been scanned using full-angle tomography, and have been appropriately subsampled to create the challenge data. This enables comparison of the limited-angle reconstruction to the ground truth obtainable from the full-angle data. The ground truth is obtained using the segmentation procedure described in a later section.

Each group is specified in the table below. In this table, angular range specifies the view-angle in the limited-angle data. The view-angles in the challenge data will not all begin from 0 degrees.

The test dataset will be made public by the end of the competition.

Get the data

The training dataset is available here:

Note: The publicly available data will not be used by the committee for measuring the quality of the algorithms submitted to the challenge. These are reserved for developing the algorithms. We measured some extra data for that. The PSFs are the same as in the categories of the public dataset. However, the targets are slightly different in a way that will be made public only after the deadline.

Phantoms

The targets are homogenous acrylic disc phantoms of 70 mm in diameter, with holes of varying shapes made with a laser cutter. Each disk has a different number of irregular holes in random locations. Figure 1 below shows few examples.

Figure 1: Target examples. Note that examples are provided to the competitors as the training set and, therefore, do not belong to the test set used to evaluate the submissions.

The dataset collected for the HTC2022 challenge consists of two separate sets, with identical experimental setup and settings. One set is provided to the competitors as training set for algorithm development, and the other will be used by the organizers to test the reconstruction algorithms. The test set will be made public after the end of the competition.

Training dataset

The training set consists of five phantoms with full angular data. These are designed to facilitate algorithm development and benchmarking for the challenge itself. Four of the training phantoms contain holes. A fifth training phantom is a solid disc with no holes.

We encourage subsampling these datasets to create limited-data sinograms and comparing the reconstruction results to the ground truth obtainable from the full-data sinograms. Note that the phantoms are not all identically centered. Training data for each difficulty group can be created by subsampling these datasets to create limited-data sinograms that match the angular range of each group of the actual challenge data (See Table 1 below).

Note: As the orientation of CT reconstructions can depend on the tools used, we have included example reconstructions for each of the training phantoms to demonstrate how the reconstructions obtained from the sinograms and the specified geometry should be oriented. These reconstructions have been computed using the filtered back-projection algorithm provided by the ASTRA Toolbox.

We have also included segmentation examples of the reconstructions to demonstrate the desired format for the final competition entries. The segmentation images were obtained by the following steps (using this Python code):

1. Set all negative pixel values in the reconstruction to zero
2. Determine a threshold level using Otsu’s method
3. Globally threshold the image using the threshold level
4. Perform a morphological closing on the image using a disc with a radius of 3 pixels.

The competitors do not need to follow the above segmentation procedure, and are encouraged to explore various segmentation techniques for the limited-angle reconstructions.

The competitors are encouraged to generate extra training data using simulations. The organizing committee will not provide the code to generate new targets before the end of the competition.

Testing dataset

The test set will be made public after the end of the competition.

Table 1: Limited-angle tomography difficulty groups

Data format

The dataset is shared using the MATLAB .mat files (version 7.3). Each individual measurement dataset consists of data structure containing the sinogram and its associated metadata, including measurement geometry. In other words, each file contains the measurements for one tomographic image.

Python users can load this type of file into their code using the mat73 module. Please refer to this link on how to install and use the module.

Data collection

The challenge data was measured at the Industrial Mathematics Computed Tomography Laboratory at the University of Helsinki. The measurement device is a cone-beam computed tomography scanner designed and constructed in-house. The scanner consists of an X-ray source, a rotating sample holder, and an X-ray detector (Figure 2).

The data has already been pre-processed with background and flat-field corrections, and compensated for a slightly misaligned center of rotation in the cone-beam computed tomography scanner. The log-transforms from intensity measurements to attenuation data have also been already computed.

We highlight the following geometric definitions, needed to properly specify the X-ray projection operator for the measurement setup:

• Dsd: Distance from source to detector
• Dso: Distance from source to origin
• Dsd: Distance from origin to detector

The X-ray detector data was binned by a factor of four after the measurements, giving a pixel size of 0.2 millimeters.