Helsinki Deblur Challenge 2021

The Finnish Inverse Problems Society (FIPS) proudly presents the Helsinki Deblur Challenge 2021 (HDC2021). We invite all scientists and research groups to test their deconvolution algorithms on our real-world data.

Can you make it the top of the leaderboard on September 30, 2021, when the challenge closes?

The Grand Prize

The top participants of the challenge will be invited to a minisymposium at the Inverse Days Conference organized by the Finnish Inverse Problems Society (FIPS) to be held in Tampere, Finland, in December 2021.

The journal Inverse Problems and Imaging will run a special issue publishing articles from participants of the HDC2021 competition.

On top of eternal glory as the Supreme Deconvolver, the winner also receives the Ultimate Imaging Device. It is a vintage viewfinder camera model Konica Auto S2 shown in Figure 1.

Load a roll of legendary Kodak Tri-X film into this beautifully engineered classic camera and shoot away! You will surely need a top-notch deconvolution algorithm to deblur the resulting images. Why? Although the Hexanon 45mm lens has excellent quality (and an impressive maximal aperture 1.8), the manual focus gives no visual feedback whatsoever in the viewfinder. You will be sure to misfocus most of your photos!

About

What is deconvolution?

Figure 1: Left: original, properly focused photograph. Right: blurred (misfocused) and noisy photograph.

Image deconvolution is a fundamental inverse problem, where an unknown picture is blurred by a point spread function (PSF) and further degraded by random noise. In practice, images are often blurred by camera shake (curve-like PSF), by poor focusing (disc-like PSF) or by bad optical quality (more complicated PSF). See Figure 1 for an example of blur resulting from misfocus. The goal of deconvolution is to recover the original picture from the blurred and noisy image.

On one hand, deconvolution is a simple inverse problem since it is linear and can be accurately described with the help of the Fourier transform (FT) as severe damping of high-frequency components of the image. This is because FT turns convolution into point-wise multiplication in the frequency domain, and because the FTs of typical PSFs are close to zero at high frequencies.

On the other hand, image deconvolution is very ill-posed and difficult to solve when the PSF has wide extent and decays rapidly in the frequency domain.

The HDC2021 competition is based on real-world disc-like PSFs of increasing sizes, realized by misfocusing a digital camera. We produced a large set of photographic data with both sharp and blurred image of the same target. This allows development and testing of deconvolution methods, including approaches based on supervised machine learning.

In a data challenge it is important to have an objective and quantitative measure for reconstruction quality. We chose strings of random characters of the latin (roman) alphabet as targets in our images. The success of a deconvolution algorithm can then be measured by applying an Optical Character Recognition (OCR) software that turns the photograph into a text file. The resulting text can be compared to the original text, leading to a discrete error measure that is same for all participants of the challenge.

Challenge description

The purpose of the challenge is to recover the shapes of 2D targets imaged with limited-angle tomography, collected in the Industrial Mathematics Computed Tomography Laboratory at the University of Helsinki, Finland. The experimental setup, targets, and measurement protocol are described in the following sections.

The outcome of the challenge should be an algorithm which takes in the X-ray data, i.e., the sinogram and it’s associated metadata about the measurement geometry, and produces a reconstruction which has been segmented into two components: air and plastic.

Organising Committee

Samuli Siltanen | University of Helsinki, Finland
Markus Juvonen | University of Helsinki, Finland
Fernando Moura | University of Helsinki, Finland; Federal University of ABC, Brazil

Registration

How to register

To enter the HTC2022 competition:Register before 23:59 EET (Eastern European Time) on September 30, 2022, using this electronic form.

Rules

Rules

The rules and information about the HDC2021 can also be found in this pdf . (Updated 24.06.2021)

How to enter the competition

To enter the HDC2021 competition:Register before 23:59 EET (Eastern European Time) on July 31, 2021, using this electronic form. (We accept late registrations sent by email before 23:59 EET (Eastern European Time) on August 31, 2021.)Send your submission to hdc2021 (“at”) fips.fi before September 30, 2021 23:59 EET. What needs to be submitted? See below for instructions.

Only submissions that fulfill the requirements listed below will be accepted.

Requirements of the competition

The data is arranged into categories of gradually increasing difficulty, with the last ones being pretty much impossible. The HDC2021 competition is structured accordingly step-wise like this:

  1. All teams start with Category 0 containing only slightly blurred images. If their algorithm leads to 70% or more of the characters in Category 0 being identified correctly by the OCR, and passes the “sanity check”, they earn the right to compete in Category 1. (Below we describe the OCR procedure and explain the sanity check.)
  2. Recursion: once a team’s algorithm reaches at least 70% of correctly identified characters in Category N, they earn the right to compete in Category N+1. The algorithm is required to have an input parameter specifying the blur category. The sanity check may be applied at all categories.
  3. Denote by Nmax the largest category number that at least one team could enter. If there is only one team, they win. If there are several teams competing in Category Nmax, they are ordered in the leaderboard according to the number of characters correctly identified in Category Nmax material.
  4. In case of a tie in Category Nmax, the previous categories, starting from Nmax-1, will be compared until one of the competitors win. If there is still a tie, the organizing committee will increase the percentage threshold to make the final decision on the winner. If the tie persists, the organizing committee will make the final decision on the winner.

The spirit of the competition is that the same algorithm is used at all blur categories, and that it is a general-purpose algorithm, capable of deblurring also other kinds of images than text images. The organizing committee has the right to disqualify an algorithm trying to violate that spirit.

Conflict of interest: researchers affiliated with the Department of Mathematics and Statistics of University of Helsinki will not be added to the leaderboard and cannot win the competition.

OCR and string matching method

All deconvoluted images will be quantitatively measured by applying the tesseract OCR via pytesseract Python module, followed by a comparison with the true text using the Levenshtein distance between the strings, implemented in the FuzzyWuzzy Python module. The same set of parameters in pytesseract and FuzzyWuzzy will be used to assess all deconvoluted images.

Note: Only the middle text line out of the three will be measured.

The basic code that runs the OCR and quantifies the transcription can be downloaded here .

The function requires two arguments:

  1. The file name (relative or full path) of the deconvoluted image (output from your algorithm).
  2. The file name (relative or full path) of a text file with the expected text contents.

The code runs in Python 3. Please check requirements.txt in the downloadable zip file for the version numbers of the modules.

Sanity check

The sanity check basically means that the algorithm should serve as a deconvolution method for general images, not only for images of text. This is to prevent approaches that always output text, regardless of the input. The sanity test image material contains some technical targets and some natural images (See examples in Figure 1 & 2 on this page (New 29.07.2021)), and the threshold for passing is very low. It’s enough to have some sort of deconvolution effect, even a poor one, instead of always producing images of text.

Figure 1: Sanity check example of a natural image of a bird with 3 different levels of blur.
Figure 2: Sanity check example of a natural image of a building with 3 different levels of blur.

Deadline

Deadline: September 30, 2021 23:59 EET (Eastern European Time)

The algorithms must be shared with us as a private GitHub repository at latest on the deadline. The codes should be in Matlab or Python3.

After the deadline there is a brief period during which we can troubleshoot the codes together with the competing teams. This is to ensure that we are able to run the codes.

Github repository

Competitors can update the contents of the shared repository as many times as needed before the deadline. We will consider only the latest release of your repository on Github.

Attention: Simple commits to the main branch will not be considered. You MUST also create a release. You can find Github’s documentation on how to create releases here. If the latest release does not work we will not accept older versions.

Your repository must contain a README.md file with at least the following sections:

  • Authors, institution, location.
  • Brief description of your algorithm and a mention of the competition.
  • Installation instructions, including any requirements.
  • Usage instructions.
  • Show a few examples.

The teams can submit more than one deconvolution algorithm to the challenge, each algorithm must be in a separate repository. The maximum number of algorithms is the number of members of the team. The teams don’t need to register multiple times in case they decide to submit more than one algorithm to the challenge. (New 24.06.2021)

Your code on Github

The repository must contain a main routine that we can run to apply your algorithm automatically to every image in a given directory. This is the file we will run to evaluate your code. Give it an easy to identify name like main.m or main.py.

Your main routine must require three input arguments:

  1. (string) Folder where the input image files are located
  2. (string) Folder where the output images must be stored
  3. (int) Blur category number. Values between 0 and 19

Below are the expected formats of the main routines in python and Matlab:

Matlab: The main function must be a callable function:

function main(inputFolder,outputFolder,categoryNbr)
...

your code comes here
...

Example calling the function:

>> main('path/to/input/files', 'path/to/output/files', 3)

Python: The main function must be a callable function from the command line. To achieve this you can use sys.argv or argparse module.

Example calling the function:

$ python3 main.py path/to/input/files path/to/output/files 3

The main routine must produce a deconvolved PNG file in the output folder for each image in the input folder. The output PNG images must have the same dimensions as the input files and the same filename apart from the extension. All images in the input directory belong to the same blur category, specified by the input argument.

The teams are allowed to use freely available python modules or Matlab toolboxes. Toolboxes, libraries and modules with paid licenses can also be used if the organizing committee also have the license. For example, the most usual Matlab toolboxes for image processing and deconvolution can be used (Image processing toolbox, wavelet toolbox, PDE toolbox, computer vision toolbox, deep learning toolbox, optimization toolbox). The teams can contact us to check if other toolboxes are available.

Scores and leaderboard

The scores and leaderboard are constructed step-wise as follows:

  • All teams start with difficulty level 1. The reconstructions of the three samples (A, B, and C) of this level will be assessed quantitatively following the criteria described below, and their scores will be summed, forming the total score of the first level
  • The team with the highest score S1 will be used as reference for the cut-off score of this level: any team with score S1 at least 25% of the highest score will pass to the next level.
  • The same procedure is repeated for all difficulty levels, up to level 7, but considering only the teams that passed the cut-off score of the previous level:
  • The cut-off of the levels stays fixed at 25% of the maximum at each level.
  • Denote by Nmax the hardest level that at least one team could enter. If there is only one team, they win. If there are several teams competing in level Nmax they are ordered in the leaderboard according to their scores at that level.
  • In case of a tie in level Nmax, the previous levels, starting from Nmax-1, will be compared until one of the competitors win. If the tie persists, the organizing committee will make the final decision on the winner.

Note: If one team submits more than one algorithm to the competition, then each submission will be temporarily assumed to belong to different ‘virtual’ teams when computing the scores and cut-offs. However, this team cannot be in more than one position in the leaderboard. In this situation, the organizing committee will consider only the highest performance algorithm when ranking the winners.

Open science spirit

Finally, the competitors must make their GitHub repositories public at latest on October 31, 2021. In the spirit of open science, only a public code can win HDC2021.

Data

How the data was measured

We showed random strings of text to two digital cameras simultaneously using an e-ink display and a beamsplitter arrangement (half-transparent mirror at 45° angle). One of the cameras was properly focused while the other one was progressively misfocused, and sporting a neutral density filter, resulting in blurred and noisy images.

Figure 1: Diagram of the experiment setup. The mirror is half-transparent. Note that the image recorded by Camera 2 is fipped due to the mirror.

We recorded also images of dots and other technical targets with the same camera settings that we used for the actual targets. This makes it easier for the competitors to estimate the PSFs empirically.

Get the data

The test dataset is now available here https://doi.org/10.5281/zenodo.5713637

The New version 1.1 of the data with 10 additional steps is available here https://doi.org/10.5281/zenodo.4916176

The Old version 1.0 of the data is still available here https://doi.org/10.5281/zenodo.4772280

You can find all the details of the measurement of the public dataset in this ArXiv preprint: http://arxiv.org/abs/2105.10233

Note: The publicly available dataset will not be used by the committee for measuring the quality of the algorithms submitted to the challenge. These are reserved for developing the algorithms. We measured some extra data for that. The PSFs are the same as in the categories of the public dataset. However, the targets are slightly different in a way that will be made public only after the deadline.

Figure 2: A sample set of 10 image pairs. One pair for each amount of blur.

News

Update 11.11.2021 | Test dataset now published! Check ‘Data’ section. Samples of the deconvolved images are now available. Check ‘Results’ section.
Update 4.11.2021 | Results published! Check ‘Results’ section.
Update 27.8.2021 | Added more FAQ questions and answers.
Update 29.7.2021 | Examples of the sanity check images added to the rules page.arified the registration and submission deadlines on the main page and fixed the link to pdf on the rules page.
Update 9.6.2021 | Additional 10 steps added to the data! The new version 1.1 of the data on Zenodo can be found here and on the data page.
Update 1.6.2021 | The code that runs the OCR and quantifies the transcription can now be downloaded (on the rules page).
Update 27.5.2021 | Registration form opened (on the rules page).
Update 27.5.2021 | Minor modifications and clarifications added to the rules.

Results

Winners

1st place: Theophil Trippe, Jan Macdonald & Maximilian März from TU Berlin and Martin Genzel from Utrecht University. GitHub_A

2nd place: Ji Li, Department of Mathematics, National University of Singapore. GitHub_B

3rd place: Daniël M. Pelt, LIACS, Leiden University, Leiden, The Netherlands. GitHub

All Registered Teams

This is the list of teams that registered for the challenge.

Together with the names, the links to the Github repositories submitted to the challenge.

  • 01 – Leiden University, LIACS – The Netherlands. GitHub
  • 02 – Indian Institute of Technology Madras, Chennai; Department of Engineering Design – India
  • 03 – DAMTP, University of Cambridge – United Kingdom04 – Dipartimento di Sciente Fisiche, Informatiche e Matematiche Università degli Studi di Modena e Reggio Emilia – Italy. GitHub
  • 05 – University of Helsinki, The Department of Mathematics and Statistics – Finland
  • 06 – Heinrich Heine University Düsseldorf, Department of Computer Science – Germany. GitHub
  • 07 – Lomonosov Moscow State University, Faculty of Computational Mathematics and Cybernetics – Russia
  • 08 – Xiangtan University – China
  • 09 – University of Campinas (UNICAMP), School of Electrical and Computer Engineering – Brazil. GitHub_AGitHub_B
  • 10 – Mediterranean Institute of Technology, Software and Computer Systems Engineering – Tunisia
  • 11 – Center for Industrial Mathematics (ZeTeM) University of Bremen (et al.) – Germany/United Kingdom. GitHub_AGitHub_BGitHub_C
  • 12 – National University of Singapore, Department of Mathematics – Singapore. GitHub_AGitHub_B
  • 13 – Federal University of ABC; Center for Engineering, Modeling and Applied Social Sciences – Brazil. GitHub
  • 14 – University of Eastern Finland, Applied Physics – Finland
  • 15 – Technische Universität Berlin, Institute of Mathematics. Utrecht University, Mathematical Institute – Germany. GitHub_AGitHub_B
  • 16 – Technical University of Denmark, DTU Compute – Denmark. GitHub_AGitHub_B
  • 17 – Argonne National Laboratory, X-ray Science Division – United States of America

Results

Figure 1: An example result of deconvolving the most difficult level.

You can download here some of the deconvolution results. You can also click the image below to download.

The table below shows the OCR scores obtained from the submitted algorithms using the test dataset. The table presents the average OCR scores in each blur category. Team ID numbers follow the list above. Those that submitted more than one algorithm were marked with suffix _A, _B, _C.

The red cells mark the first category where the algorithms did not reach the minimum threshold of 70%. The stop category of the algorithm is the last category before the red cell. The stop category for the teams that submitted more than one algorithm is the best among their submissions.

The last column shows the stop category for each team

Test dataset

The test dataset is different from the training set to assess robustness. It was collected the same way but the strings now include number characters. Each category contains 40 pictures of the e-ink display showing text (20 for each font type), together with pictures of the e-ink display showing images, serving for the sanity check.

The test dataset is now available here https://doi.org/10.5281/zenodo.5713637

FAQ

Contact

To contact the HDC2021 organizers send an e-mail to hdc2021 (“at”) fips.fi

Scroll to Top