Main evaluation results
| Participant | Recall | Precision | F1 |
| TEES-2.1 | 0.35 | 0.82 | 0.49 |
| IRISA-TexMex | 0.44 | 0.46 | 0.45 |
| Boun | 0.23 | 0.38 | 0.29 |
| LIMSI | 0.04 | 0.29 | 0.07 |
| Participant | Recall | Precision | F1 |
| TEES-2.1 | 0.12 | 0.18 | 0.14 |
| LIMSI | 0.04 | 0.12 | 0.06 |
Evaluation algorithm
The evaluation performs a pairing between each relation in the reference with a predicted relation. The pairing maximizes the following score for relations of type Localization:
B . J
- B is the Bacterium boundaries match. It is equal to 1 if the Bacterium arguments of the reference and the prediction have the exact same boundaries, otherwise 0.
- J is the Localization boundaries match. It is the Jaccard index between the Localization arguments of the reference and the predition.
For relations of type PartOf, the score is 1 if the Host arguments overlaps and if the Part arguments overlap, otherwise 0. Boundaries are not taken into account in PartOf relations in order to not penalize boundaries mismatches twice; boundaries are already factored in the score of Localization relations.
In the case of equivalence between entities, the pairing uses the equivalent entity that maximizes the score. If several equivalent (and redundant) relations are found then the one that has the highest score is used.
The Recall is the sum of the scores of reference to prediction pairing divided by the number of relations in the reference.
The Precision is the sum of the scores of prediction to reference pairing divided by the number of relations in the prediction.
The F1 is the harmonic mean of Recall and Precision.
Alternate evaluations
Description of parameters
- Localization only: PartOf relations have been removed from the pairing, the evaluation only measures the accuracy of Localization relations.
- PartOf only: Localization relations have been removed from the pairing, the evaluation only measures the accuracy of PartOf relations.
- No boundaries: the Recall and the Precision are computed from the number of pairings, as if J has been removed from the score formula. Note however that the pairing still maximizes "B . J"; only the scores are altered.
- Relaxed bacteria: B has been redifined as: 1 if the Bacterium arguments overlap, otherwise 0. Both scores and pairing are affected. The pairing maximizes the overlap between Bacterium arguments.
No boundaries
| Participant | Recall | Precision | F1 |
| TEES-2.1 | 0.14 | 0.21 | 0.17 |
| LIMSI | 0.04 | 0.12 | 0.06 |
Relaxed bactreria
| Participant | Recall | Precision | F1 |
| TEES-2.1 | 0.28 | 0.52 | 0.36 |
| LIMSI | 0.07 | 0.71 | 0.10 |
No boundaries, relaxed bacteria
| Participant | Recall | Precision | F1 |
| TEES-2.1 | 0.37 | 0.64 | 0.47 |
| LIMSI | 0.07 | 0.72 | 0.12 |
PartOf only
| Participant | Recall | Precision | F1 |
| TEES-2.1 | 0.27 | 0.77 | 0.40 |
| LIMSI | 0.03 | 0.17 | 0.05 |
Localization only
| Participant | Recall | Precision | F1 |
| TEES-2.1 | 0.05 | 0.07 | 0.06 |
| LIMSI | 0.04 | 0.12 | 0.06 |
Localization only, no boundaries
| Participant | Recall | Precision | F1 |
| TEES-2.1 | 0.08 | 0.10 | 0.09 |
| LIMSI | 0.04 | 0.12 | 0.06 |
Localization only, relaxed bacteria
| Participant | Recall | Precision | F1 |
| TEES-2.1 | 0.29 | 0.47 | 0.35 |
| LIMSI | 0.08 | 0.81 | 0.15 |
Localization, no boundaries, relaxed bacteria
| Participant | Recall | Precision | F1 |
| TEES-2.1 | 0.41 | 0.61 | 0.49 |
| LIMSI | 0.09 | 0.82 | 0.15 |