Evaluation

Each team will be assigned three ranking numbers based on the three criteria below, one ranking number per criterion, using a standard competition ranking. The sum of these numbers will be used for the final ranking. Matlab code for calculating the evaluation metrics is available here.

Detection

The ground truth for each segmented object is the object in the manual annotation that has maximum overlap with that segmented object.

A segmented glandular object that intersects with at least 50% of its ground truth will be considered as true positive, otherwise it will be considered as false positive. A ground truth glandular object that has no corresponding segmented object or has less than 50% of its area overlapped by its corresponding segmented object will be considered as false negative.

Let

$TP$ be the number of true positives,
$FP$ be the number of false positives,
$FN$ be the number of false negatives.

A metric for gland detection is the F1-score, defined by

$\mathrm{F1score} = \frac{2\cdot\mathrm{Precision}\cdot\mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}$

where

$\mathrm{Precision} = TP/(TP + FP)$ , $\mathrm{Recall} = TP/(TP + FN)$

Segmentation

Given $G$ a set of pixels annotated as a ground truth obect and $S$ a set of pixels segmented as a glandular object, Dice index is defined as follows

$\mathrm{Dice}(G,S) = \frac{2|G\cap{S}|}{|G|+|S|}$

Further, let

$\mathcal{G}_a$ denote a set of ground truth objects in image $a$ .
$\mathcal{S}_a$ denote a set of segmented objects in image $a$ .
$S_i \in \mathcal{S}_a$ denote the $i$ th segmented object in image $a$ .
$G_i \in \mathcal{G}_a$ denote a ground truth object that maximally overlaps $S_i$ in image $a$ .
$\tilde{G}_i \in \mathcal{G}_a$ denote the $i$ th ground truth object in image $a$ .
$\tilde{S}_i \in \mathcal{S}_a$ denote a segmented object that maximally overlaps $\tilde{G}_i$ in image $a$ .
$\mathcal{G} = \cup_a \mathcal{G}_a$ a set of all ground truth objects.
$\mathcal{S} = \cup_a \mathcal{S}_a$ a set of all segmented objects.
$n_S$ denote the total number of segmented objects in $\mathcal{S}$ .
$n_G$ denote the total number of ground truth objects in $\mathcal{G}$ .

We define the object-level Dice index as

$\mathrm{Dice}_\text{object}(G,S)=\frac{1}{2}\left[\sum_{i=1}^{n_S}\omega_i\mathrm{Dice}(G_i,S_i)+\sum_{i=1}^{n_G}\tilde{\omega}_i\mathrm{Dice}(\tilde{G}_i,\tilde{S}_i)\right],$

where

$\omega_i=|S_i|/\sum_{j=1}^{n_S}|S_j|,$ $\tilde{\omega}_i=|\tilde{G}_i|/\sum_{j=1}^{n_G}|\tilde{G}_j|.$

The object-level Dice index will be used to evaluate the performance of segmentation.

Shape Similarity

Let $G$ denote a set of pixels annotated as ground truth and $S$ denote a set of pixels segmented as glandular objects. A Hausdorff distance between $G$ and $S$ is defined as

$\mathrm{H}(G,S) = \max\{\sup_{x\in{G}}\inf_{y\in{S}}\|x-y\|,\sup_{y\in{S}}\inf_{x\in{G}}\|x-y\|\}$ .

Now, let

$\mathcal{G}_a$ denote a set of ground truth objects in image $a$ .
$\mathcal{S}_a$ denote a set of segmented objects in image $a$ .
$S_i \in \mathcal{S}_a$ denote the $i$ th segmented object in image $a$ .
$G_i \in \mathcal{G}_a$ denote a ground truth object that maximally overlaps $S_i$ in image $a$ . If there is no ground truth object overlapping $S_i$ , $G_i$ is defined as ground truth object $G \in \mathcal{G}_a$ that has the minimum Hausdorff distance from $S_i$ .
$\tilde{G}_i \in \mathcal{G}_a$ denote the $i$ th ground truth object in image $a$ .
$\tilde{S}_i \in \mathcal{S}_a$ denote a segmented object that maximally overlaps $\tilde{G}_i$ in image $a$ . If there is no segmented object overlapping $\tilde{G}_i$ , $\tilde{S}_i$ is defined as segmented object $\tilde{S} \in \mathcal{S}_a$ that has the minimum Hausdorff distance from $\tilde{G}_i$ .
$\mathcal{G} = \cup_a \mathcal{G}_a$ a set of all ground truth objects.
$\mathcal{S} = \cup_a \mathcal{S}_a$ a set of all segmented objects.
$n_S$ denote the total number of segmented objects in $\mathcal{S}$ .
$n_G$ denote the total number of ground truth objects in $\mathcal{G}$ .

We measure the shape similarity between all segmented objects in $\mathcal{S}$ and all ground truth objects in $\mathcal{G}$ using the object-level Hausdorff distance

$\mathrm{H}_\text{object}(\mathcal{S},\mathcal{G}) = \frac{1}{2} \left[ \sum_{i = 1}^{n_s} \omega_i H(G_i,S_i) + \sum_{i=1}^{n_G} \tilde{\omega}_i H(\tilde{G}_i,\tilde{S}_i)\right],$

where

$\omega_i=|S_i|/\sum_{j=1}^{n_S}|S_j|,$ $\tilde{\omega}_i=|\tilde{G}_i|/\sum_{j=1}^{n_G}|\tilde{G}_j|.$