Given the scientific impact of challenges, it is surprising that there is a huge discrepancy between their impact and quality control as demonstrated by a study on biomedical image analysis competitions 2. Today, the performance of algorithms on challenge data is essential, not only for the acceptance of a paper, but also for the individuals’ scientific careers and the opportunity that algorithms might be translated to a clinical setting. The results of these international competitions are commonly published in prestigious journals 3, 4, 5, 6, 7, 8, and challenge winners are sometimes awarded with huge amounts of prize money. In the last couple of years, grand challenges have evolved as the standard to validate biomedical image analysis methods in a comparative manner 1, 2. Our framework could thus become an important tool for analyzing and visualizing challenge results in the field of biomedical image analysis and beyond. This is demonstrated by the experiments performed in the specific context of biomedical image analysis challenges. Our approach offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, which cannot be revealed by commonly applied visualization techniques. Given these shortcomings, the contribution of this paper is two-fold: (1) we present a set of methods to comprehensively analyze and visualize the results of single-task and multi-task challenges and apply them to a number of simulated and real-life challenges to demonstrate their specific strengths and weaknesses (2) we release the open-source framework challengeR as part of this work to enable fast and wide adoption of the methodology proposed in this paper. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature. While the number of these international competitions is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Grand challenges have become the de facto standard for benchmarking image analysis algorithms.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |