Task 1

Main evaluation results

Legend

Evaluation algorithm

The evaluation performs a pairing between each reference habitat to a predicted habitat. The pairing maximizes a score defined as:

J . W

J is the Jaccard index between the reference and predicted entity as defined in [Bossy et al, 2012]. J measures the boundaries accuracy of the predicted entity.

W is the semantic similarity between ontolgy concepts attributed to the reference entity and to the predicted entity. We use the semantic similarity described in [Wang et al, 2006]. This similarity is exclusively based on the is-a relationships between concepts, we set the wis-a parameter to 0.65 in order to penalize favor ancestor/descendent predictions rather than sibling predictions.

Habitat entities in the reference that have no corresponding entity in the prediction are Deletions (D column).

Habitat entities in the prediction that have no corresponding entity in the reference are Insertions (I column).

The sum of the scores for all successful pairings is the Matches (M column). The difference between the number of pairings and the Matches is the Substitutions (S column).

Entity boundaries evaluation

In this evaluation, the Matches are re-defined as the sum of the J component of the score for each pairing. In this way the scores measure the boundaries accuracy of predicted entities, without taking into account the semantic categorization.

Note however that the pairing still maximizes J.W. Therefore columns I, D and P remain unchanged.

Ontology categorization evaluation

In this evaluation, the Matches are re-defined as the sum of the W component of the score for each pairing. In this way the scores measure the semantic categorization accuracy of predicted entities, without taking into account the entities boundaries.

Note however that the pairing still maximizes J.W. Therefore columns I, D and P remain unchanged.

In the following evaluations, the semantic weight attributed to the is-a relation has been altered:

w = 1 --> With a weight of 1, the score approches a "Manhattan distance" between the reference category and the predicted category; it is nearly equivalent to step counting semantic distances. It is more forgiving if the prediction is "in the vicinity" of the references, even though it is not an ancestor or descedent. It is more severe for predictions that further from the reference.

w = 0.1 --> With a weight of 0.1, the score favours predictions in the "lineage" of the reference, that is to say ancestors and descendants. It severly penalizes predictions of siblings. However, since the ontology root is the ancestor of all possible concepts, this score does not penalize predictions that are too general.

w = 0.8 --> 0.8 is the value recommended by the authors of the semantic distance. It is shown for reference and bears no particular interest for the task.