ignite.metrics

Metrics provide a way to compute various quantities of interest in an online fashion without having to store the entire output history of a model.

In practice a user needs to attach the metric instance to an engine. The metric value is then computed using the output of the engine’s process_function:

def process_function(engine, batch):
    # ...
    return y_pred, y

engine = Engine(process_function)
metric = Accuracy()
metric.attach(engine, "accuracy")

If the engine’s output is not in the format y_pred, y, the user can use the output_transform argument to transform it:

def process_function(engine, batch):
    # ...
    return {'y_pred': y_pred, 'y_true': y, ...}

engine = Engine(process_function)

def output_transform(output):
    # `output` variable is returned by above `process_function`
    y_pred = output['y_pred']
    y = output['y_true']
    return y_pred, y  # output format is according to `Accuracy` docs

metric = Accuracy(output_transform=output_transform)
metric.attach(engine, "accuracy")

class ignite.metrics.Accuracy(output_transform=<function Metric.<lambda>>)

Calculates the accuracy.

• update must receive output of the form (y_pred, y).

• y_pred must be in the following shape (batch_size, num_categories, …) or (batch_size, …)

• y must be in the following shape (batch_size, …)

class ignite.metrics.BinaryAccuracy(*args, **kwargs)

Note: This metric is deprecated in favor of Accuracy.

class ignite.metrics.CategoricalAccuracy(*args, **kwargs)

Note: This metric is deprecated in favor of Accuracy.

class ignite.metrics.Loss(loss_fn, output_transform=<function Loss.<lambda>>, batch_size=<function Loss.<lambda>>)

Calculates the average loss according to the passed loss_fn.

Parameters

• loss_fn (callable) – a callable taking a prediction tensor, a target tensor, optionally other arguments, and returns the average loss over all observations in the batch.

• output_transform (callable) – a callable that is used to transform the ignite.engine.Engine’s process_function’s output into the form expected by the metric. This can be useful if, for example, you have a multi-output model and you want to compute the metric with respect to one of the outputs. The output is is expected to be a tuple (prediction, target) or (prediction, target, kwargs) where kwargs is a dictionary of extra keywords arguments.

• batch_size (callable) – a callable taking a target tensor that returns the first dimension size (usually the batch size).

class ignite.metrics.MeanAbsoluteError(output_transform=<function Metric.<lambda>>)

Calculates the mean absolute error.

• update must receive output of the form (y_pred, y).

class ignite.metrics.MeanPairwiseDistance(p=2, eps=1e-06, output_transform=<function MeanPairwiseDistance.<lambda>>)

Calculates the mean pairwise distance.

• update must receive output of the form (y_pred, y).

class ignite.metrics.MeanSquaredError(output_transform=<function Metric.<lambda>>)

Calculates the mean squared error.

• update must receive output of the form (y_pred, y).

class ignite.metrics.Metric(output_transform=<function Metric.<lambda>>)

Base class for all Metrics.

Parameters

output_transform (callable, optional) – a callable that is used to transform the ignite.engine.Engine’s process_function’s output into the form expected by the metric. This can be useful if, for example, you have a multi-output model and you want to compute the metric with respect to one of the outputs.

abstract compute()

Computes the metric based on it’s accumulated state.

This is called at the end of each epoch.

Returns

the actual quantity of interest

Return type

Any

Raises

NotComputableError – raised when the metric cannot be computed

abstract reset()

Resets the metric to to it’s initial state.

This is called at the start of each epoch.

abstract update(output)

Updates the metric’s state using the passed batch output.

This is called once for each batch.

Parameters

output – the is the output from the engine’s process function

class ignite.metrics.Precision(average=False, output_transform=<function Precision.<lambda>>)

Calculates precision.

• update must receive output of the form (y_pred, y).

If average is True, returns the unweighted average across all classes. Otherwise, returns a tensor with the precision for each class.

class ignite.metrics.Recall(average=False, output_transform=<function Recall.<lambda>>)

Calculates recall.

• update must receive output of the form (y_pred, y).

If average is True, returns the unweighted average across all classes. Otherwise, returns a tensor with the recall for each class.

class ignite.metrics.RootMeanSquaredError(output_transform=<function Metric.<lambda>>)

Calculates the root mean squared error.

• update must receive output of the form (y_pred, y).

class ignite.metrics.TopKCategoricalAccuracy(k=5, output_transform=<function TopKCategoricalAccuracy.<lambda>>)

Calculates the top-k categorical accuracy.

• update must receive output of the form (y_pred, y).

class ignite.metrics.EpochMetric(compute_fn, output_transform=<function EpochMetric.<lambda>>)

Class for metrics that should be computed on the entire output history of a model. Model’s output and targets are restricted to be of shape (batch_size, n_classes). Output datatype should be float32. Target datatype should be long.

Warning

Current implementation stores all input data (output and target) in as tensors before computing a metric. This can potentially lead to a memory error if the input data is larger than available RAM.

• update must receive output of the form (y_pred, y).

If target shape is (batch_size, n_classes) and n_classes > 1 than it should be binary: e.g. [[0, 1, 0, 1], ]

Parameters

• compute_fn (callable) – a callable with the signature (torch.tensor, torch.tensor) takes as the input predictions and targets and returns a scalar.

• output_transform (callable, optional) – a callable that is used to transform the ignite.engine.Engine’s process_function’s output into the form expected by the metric. This can be useful if, for example, you have a multi-output model and you want to compute the metric with respect to one of the outputs.

class ignite.metrics.RunningAverage(src=None, alpha=0.98, output_transform=None)

Compute running average of a metric or the output of process function.

Parameters

• src (Metric or None) – input source: an instance of ignite.metrics.Metric or None. The latter corresponds to engine.state.output which holds the output of process function.

• alpha (float, optional) – running average decay factor, default 0.98

• output_transform (Callable, optional) – a function to use to transform the output if src is None and corresponds the output of process function. Otherwise it should be None.

Examples:

alpha = 0.98
acc_metric = RunningAverage(CategoricalAccuracy(output_transform=lambda x: [x[1], x[2]]), alpha=alpha)
acc_metric.attach(trainer, 'running_avg_accuracy')

avg_output = RunningAverage(output_transform=lambda x: x[0], alpha=alpha)
avg_output.attach(trainer, 'running_avg_loss')

@trainer.on(Events.ITERATION_COMPLETED)
def log_running_avg_metrics(engine):
    print("running avg accuracy:", engine.state.metrics['running_avg_accuracy'])
    print("running avg loss:", engine.state.metrics['running_avg_loss'])