ignite.contrib.metrics

Contribution module of metrics

class ignite.contrib.metrics.AveragePrecision(activation=None, output_transform=<function AveragePrecision.<lambda>>)

Computes Average Precision accumulating predictions and the ground-truth during an epoch and applying sklearn.metrics.average_precision_score .

Parameters

output_transform (callable, optional) – a callable that is used to transform the Engine’s process_function’s output into the form expected by the metric. This can be useful if, for example, you have a multi-output model and you want to compute the metric with respect to one of the outputs.

AveragePrecision expects y to be comprised of 0’s and 1’s. y_pred must either be probability estimates or confidence values. To apply an activation to y_pred, use output_transform as shown below:

def activated_output_transform(output):
    y_pred, y = output
    y_pred = torch.softmax(y_pred)
    return y_pred, y

avg_precision = AveragePrecision(activated_output_transform)

class ignite.contrib.metrics.ROC_AUC(output_transform=<function ROC_AUC.<lambda>>)

Computes Area Under the Receiver Operating Characteristic Curve (ROC AUC) accumulating predictions and the ground-truth during an epoch and applying sklearn.metrics.roc_auc_score .

Parameters

output_transform (callable, optional) – a callable that is used to transform the Engine’s process_function’s output into the form expected by the metric. This can be useful if, for example, you have a multi-output model and you want to compute the metric with respect to one of the outputs.

ROC_AUC expects y to be comprised of 0’s and 1’s. y_pred must either be probability estimates or confidence values. To apply an activation to y_pred, use output_transform as shown below:

def activated_output_transform(output):
    y_pred, y = output
    y_pred = torch.sigmoid(y_pred)
    return y_pred, y

roc_auc = ROC_AUC(activated_output_transform)

Regression metrics

Module ignite.contrib.metrics.regression provides implementations of metrics useful for regression tasks. Definitions of metrics are based on Botchkarev 2018, page 30 “Appendix 2. Metrics mathematical definitions”.

Complete list of metrics:

• CanberraMetric

• FractionalAbsoluteError

• FractionalBias

• GeometricMeanAbsoluteError

• GeometricMeanRelativeAbsoluteError

• ManhattanDistance

• MaximumAbsoluteError

• MeanAbsoluteRelativeError

• MeanError

• MeanNormalizedBias

• MedianAbsoluteError

• MedianAbsolutePercentageError

• MedianRelativeAbsoluteError

• WaveHedgesDistance

class ignite.contrib.metrics.regression.CanberraMetric(output_transform=<function Metric.<lambda>>)

Calculates the Canberra Metric.

$\text{CM} = \sum_{j=1}^n\frac{|A_j - P_j|}{A_j + P_j}$

where, $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1).

class ignite.contrib.metrics.regression.FractionalAbsoluteError(output_transform=<function Metric.<lambda>>)

Calculates the Fractional Absolute Error.

$\text{FAE} = \frac{1}{n}\sum_{j=1}^n\frac{2 |A_j - P_j|}{|A_j| + |P_j|}$

where, $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1).

class ignite.contrib.metrics.regression.FractionalBias(output_transform=<function Metric.<lambda>>)

Calculates the Fractional Bias:

$\text{FB} = \frac{1}{n}\sum_{j=1}^n\frac{2 (A_j - P_j)}{A_j + P_j}$,

where $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1).

class ignite.contrib.metrics.regression.GeometricMeanAbsoluteError(output_transform=<function Metric.<lambda>>)

Calculates the Geometric Mean Absolute Error.

$\text{GMAE} = \exp(\frac{1}{n}\sum_{j=1}^n\ln(|A_j - P_j|)$)

where, $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1).

class ignite.contrib.metrics.regression.GeometricMeanRelativeAbsoluteError(output_transform=<function Metric.<lambda>>)

Calculates the Geometric Mean Relative Absolute Error:

$\text{GMRAE} = \exp(\frac{1}{n}\sum_{j=1}^n \ln\frac{|A_j - P_j|}{|A_j - \bar{A}|})$

where $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1).

class ignite.contrib.metrics.regression.ManhattanDistance(output_transform=<function Metric.<lambda>>)

Calculates the Manhattan Distance:

$\text{MD} = \sum_{j=1}^n (A_j - P_j)$,

where $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1).

class ignite.contrib.metrics.regression.MaximumAbsoluteError(output_transform=<function Metric.<lambda>>)

Calculates the Maximum Absolute Error:

$\text{MaxAE} = \max_{j=1,n} \left( \lvert A_j-P_j \rvert \right)$,

where $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1).

class ignite.contrib.metrics.regression.MeanAbsoluteRelativeError(output_transform=<function Metric.<lambda>>)

Calculate Mean Absolute Relative Error:

$\text{MARE} = \frac{1}{n}\sum_{j=1}^n\frac{\left|A_j-P_j\right|}{\left|A_j\right|}$,

where $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in the reference Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1).

class ignite.contrib.metrics.regression.MeanError(output_transform=<function Metric.<lambda>>)

Calculates the Mean Error:

$\text{ME} = \frac{1}{n}\sum_{j=1}^n (A_j - P_j)$,

where $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in the reference Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1).

class ignite.contrib.metrics.regression.MeanNormalizedBias(output_transform=<function Metric.<lambda>>)

Calculates the Mean Normalized Bias:

$\text{MNB} = \frac{1}{n}\sum_{j=1}^n\frac{A_j - P_j}{A_j}$,

where $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in the reference Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1).

class ignite.contrib.metrics.regression.MedianAbsoluteError(output_transform=<function MedianAbsoluteError.<lambda>>)

Calculates the Median Absolute Error:

$\text{MdAE} = \text{MD}_{j=1,n} \left( |A_j - P_j| \right)$,

where $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1) and of type float32.

Warning

Current implementation stores all input data (output and target) in as tensors before computing a metric. This can potentially lead to a memory error if the input data is larger than available RAM.

class ignite.contrib.metrics.regression.MedianAbsolutePercentageError(output_transform=<function MedianAbsolutePercentageError.<lambda>>)

Calculates the Median Absolute Percentage Error:

$\text{MdAPE} = 100 \cdot \text{MD}_{j=1,n} \left( \frac{|A_j - P_j|}{|A_j|} \right)$,

where $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1) and of type float32.

Warning

Current implementation stores all input data (output and target) in as tensors before computing a metric. This can potentially lead to a memory error if the input data is larger than available RAM.

class ignite.contrib.metrics.regression.MedianRelativeAbsoluteError(output_transform=<function MedianRelativeAbsoluteError.<lambda>>)

Calculates the Median Relative Absolute Error:

$\text{MdRAE} = \text{MD}_{j=1,n} \left( \frac{|A_j - P_j|}{|A_j - \bar{A}|} \right)$,

where $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1) and of type float32.

Warning

Current implementation stores all input data (output and target) in as tensors before computing a metric. This can potentially lead to a memory error if the input data is larger than available RAM.

class ignite.contrib.metrics.regression.WaveHedgesDistance(output_transform=<function Metric.<lambda>>)

Calculates the Wave Hedges Distance.

$\text{WHD} = \sum_{j=1}^n\frac{|A_j - P_j|}{max(A_j, P_j)}$, where, $A_j$ is the ground truth and $P_j$ is the predicted value.

More details can be found in Botchkarev 2018.

• update must receive output of the form (y_pred, y).

• y and y_pred must be of same shape (N, ) or (N, 1).