The BioCreative V.5 evaluation workshop: tasks, organization, sessions and topics

Abstract

The BioCreative (Critical Assessment of Information Extraction in Biology) is a community-wide effort with the aim of evaluating biomedical text mining and information extraction tools. It is organized in the form of shared tasks or challenges. The evaluation workshop linked to each BioCreative event serves to analyze the results obtained for each track, and to present the used Gold Standard datasets/evaluation settings. For each track, top scoring teams were invited to present a system’s description talk to uncover the used methodology and system’s implementation. Building on the achievements of previous BioCreative evaluations and workshops (BioCreative I, II, II.5, III, 2012 workshop, IV and V [1–5]) we have organized the BioCreative V.5 challenge evaluation workshop in Barcelona, Spain on April 26th-27th, 2017. To promote synergies, this workshop was co-located with the ELIXIR-EXCELERATE workshop on text mining infrastructure requirements, with a particular emphasis on the role of Text Mining systems for Data Curation and Knowledge Management in the Life Sciences domain. The goal of BioCreative V.5 was to address some of the major barriers to the adoption and use of text mining tools, related to assessment, accessibility, interoperability, robustness and integration. Two traditional BioCreative tracks focused on monitoring progress on the recognition of relevant bio-entities (chemicals – CEMP track and genes/proteins – GPRO track). A novel track called TIPS (Technical interoperability and performance of annotation servers) focused on the technical aspects of the evaluation of continuous text annotation web services. The topics that were addressed during the workshop included: (1) continuous evaluation/stability of text mining tools, (2) enabling of interoperability of multiple text annotation systems at the technical level (design of compatible annotation schemas), (3) extraction of textual content from heterogeneous document sources, and (4) visualization and comparative assessment of automatic and manual annotations. Through the contribution of invited speakers and panel sessions, we also discussed aspects related to annotation formats, technical integration of text mining components, the experience of text mining techniques for the DARPA Big Mechanism program and the use of text processing web services and workflows. During the collocated workshops, the OpenMinted project and the open call for tenders were presented.