# Evaluation

Each team will be assigned three ranking numbers based on the three criteria below, one ranking number per criterion, using a standard competition ranking. The sum of these numbers will be used for the final ranking. Matlab code for calculating the evaluation metrics is available here.

### Detection

The ground truth for each segmented object is the object in the manual annotation that has maximum overlap with that segmented object.

A segmented glandular object that intersects with at least 50% of its ground truth will be considered as true positive, otherwise it will be considered as false positive. A ground truth glandular object that has no corresponding segmented object or has less than 50% of its area overlapped by its corresponding segmented object will be considered as false negative.

Let

• $TP$ be the number of true positives,
• $FP$ be the number of false positives,
• $FN$ be the number of false negatives.

A metric for gland detection is the F1-score, defined by

$\mathrm{F1score} = \frac{2\cdot\mathrm{Precision}\cdot\mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}$

where

$\mathrm{Precision} = TP/(TP + FP)$, $\mathrm{Recall} = TP/(TP + FN)$

### Segmentation

Given $G$ a set of pixels annotated as a ground truth obect and $S$ a set of pixels segmented as a glandular object, Dice index is defined as follows

$\mathrm{Dice}(G,S) = \frac{2|G\cap{S}|}{|G|+|S|}$

Further, let

• $\mathcal{G}_a$ denote a set of ground truth objects in image $a$.
• $\mathcal{S}_a$ denote a set of segmented objects in image $a$.
• $S_i \in \mathcal{S}_a$ denote the $i$th segmented object in image $a$.
• $G_i \in \mathcal{G}_a$ denote a ground truth object that maximally overlaps $S_i$ in image $a$.
• $\tilde{G}_i \in \mathcal{G}_a$ denote the $i$th ground truth object in image $a$.
• $\tilde{S}_i \in \mathcal{S}_a$ denote a segmented object that maximally overlaps $\tilde{G}_i$ in image $a$.
• $\mathcal{G} = \cup_a \mathcal{G}_a$ a set of all ground truth objects.
• $\mathcal{S} = \cup_a \mathcal{S}_a$ a set of all segmented objects.
• $n_S$ denote the total number of segmented objects in $\mathcal{S}$.
• $n_G$ denote the total number of ground truth objects in $\mathcal{G}$.

We define the object-level Dice index as

$\mathrm{Dice}_\text{object}(G,S)=\frac{1}{2}\left[\sum_{i=1}^{n_S}\omega_i\mathrm{Dice}(G_i,S_i)+\sum_{i=1}^{n_G}\tilde{\omega}_i\mathrm{Dice}(\tilde{G}_i,\tilde{S}_i)\right],$

where

$\omega_i=|S_i|/\sum_{j=1}^{n_S}|S_j|,$ $\tilde{\omega}_i=|\tilde{G}_i|/\sum_{j=1}^{n_G}|\tilde{G}_j|.$

The object-level Dice index will be used to evaluate the performance of segmentation.

### Shape Similarity

Let $G$ denote a set of pixels annotated as ground truth and $S$ denote a set of pixels segmented as glandular objects. A Hausdorff distance between $G$ and $S$ is defined as

$\mathrm{H}(G,S) = \max\{\sup_{x\in{G}}\inf_{y\in{S}}\|x-y\|,\sup_{y\in{S}}\inf_{x\in{G}}\|x-y\|\}$.

Now, let

• $\mathcal{G}_a$ denote a set of ground truth objects in image $a$.
• $\mathcal{S}_a$ denote a set of segmented objects in image $a$.
• $S_i \in \mathcal{S}_a$ denote the $i$th segmented object in image $a$.
• $G_i \in \mathcal{G}_a$ denote a ground truth object that maximally overlaps $S_i$ in image $a$. If there is no ground truth object overlapping $S_i$, $G_i$ is defined as ground truth object $G \in \mathcal{G}_a$ that has the minimum Hausdorff distance from $S_i$.
• $\tilde{G}_i \in \mathcal{G}_a$ denote the $i$th ground truth object in image $a$.
• $\tilde{S}_i \in \mathcal{S}_a$ denote a segmented object that maximally overlaps $\tilde{G}_i$ in image $a$. If there is no segmented object overlapping $\tilde{G}_i$, $\tilde{S}_i$ is defined as segmented object $\tilde{S} \in \mathcal{S}_a$ that has the minimum Hausdorff distance from $\tilde{G}_i$.
• $\mathcal{G} = \cup_a \mathcal{G}_a$ a set of all ground truth objects.
• $\mathcal{S} = \cup_a \mathcal{S}_a$ a set of all segmented objects.
• $n_S$ denote the total number of segmented objects in $\mathcal{S}$.
• $n_G$ denote the total number of ground truth objects in $\mathcal{G}$.

We measure the shape similarity between all segmented objects in $\mathcal{S}$ and all ground truth objects in $\mathcal{G}$ using the object-level Hausdorff distance

$\mathrm{H}_\text{object}(\mathcal{S},\mathcal{G}) = \frac{1}{2} \left[ \sum_{i = 1}^{n_s} \omega_i H(G_i,S_i) + \sum_{i=1}^{n_G} \tilde{\omega}_i H(\tilde{G}_i,\tilde{S}_i)\right],$

where

$\omega_i=|S_i|/\sum_{j=1}^{n_S}|S_j|,$ $\tilde{\omega}_i=|\tilde{G}_i|/\sum_{j=1}^{n_G}|\tilde{G}_j|.$