Benchmarking & Sample Datasets – WG5

Mission and Workplan

WG5 aims to identify common BioImage analysis problems and benchmark existing solutions (BioImage analysis workflows).

WG5 will create a complete online framework allowing to compare existing BioImage analysis workflows by enabling to run these workflows online on community validated microscopy datasets, and benchmark the results by comparing them to ground truth object annotations (e.g. cells, vessels, spots, etc.). To help users selecting the right tool, these benchmarking results will be referenced from WG4 webtool (BISE). This tool has the potential to boost the development of BioImage analysis solutions, and to trigger exchanges between open source projects and companies. It can also be used as a reference that helps reproduce the image analysis protocols referenced in scientific publications.

Firstly, WG5 will define a taxonomy of common BioImage  analysis problems and organize them in classes. WG5 will then gather annotated datasets: real images sourced from existing sample data collections, synthetic images generated, and new datasets collected from open calls. Object annotation might also be carried out collaboratively to annotate existing images. Compatible licensing models for sharing images, annotations and workflows will be devised.

WG5 will then define standards for the interoperability of image analysis software to run benchmarks of workflows coming from various software packages. Some standards will also be specified for the expected output formats of workflows and ground truth annotations (problem class dependent). Finally, relevant benchmarking metrics will be selected for each BioImage analysis problem class to assess correctness, robustness, efficiency, flexibility and usability of the workflows.

Based on this information, a web-based platform enabling running such benchmarks and reporting the results will be implemented. It will be based on existing solutions from similar initiatives, extended by internal development and outsourcing. Cloud computing and storage solution will be provided by NeuBIAS as a WG5 managed server. The specification, implementation and maintenance of the software infrastructure will be supported by WG5.

The sample datasets collection will be made publicly available as part of the benchmarking webtool. This resource will provide public access to a set of standard, annotated datasets reflecting common BIAS tasks. These will be critical for fair benchmarking of solutions, and will produce a number of additional benefits including use for illustrative purposes: i.e. to support teaching and knowledge dissemination (in particular, WG1 and WG6).

WG5 webtool presentation poster