flambe.metric

Package Contents

class flambe.metric.Metric[source]

Bases: flambe.compile.Component

Base Metric interface.

Objects implementing this interface should take in a sequence of examples and provide as output a processd list of the same size.

compute(self, pred: torch.Tensor, target: torch.Tensor)

Computes the metric over the given prediction and target.

Parameters:
  • pred (torch.Tensor) – The model predictions
  • target (torch.Tensor) – The ground truth targets
Returns:

The computed metric

Return type:

torch.Tensor

aggregate(self, state: dict, *args, **kwargs)

Aggregates by simply storing preds and targets

Parameters:
  • state (dict) – the metric state
  • args (the pred, target tuple) –
Returns:

the state dict

Return type:

dict

finalize(self, state: Dict)

Finalizes the metric computation

Parameters:state (dict) – the metric state
Returns:The final score.
Return type:float
__call__(self, *args, **kwargs)

Makes Featurizer a callable.

__str__(self)

Return the name of the Metric (for use in logging).

class flambe.metric.MultiLabelCrossEntropy(weight: Optional[torch.Tensor] = None, ignore_index: Optional[int] = None, reduction: str = 'mean')[source]

Bases: flambe.metric.metric.Metric

__str__(self)

Return the name of the Metric (for use in logging).

compute(self, pred: torch.Tensor, target: torch.Tensor)

Computes the multilabel cross entropy loss.

Parameters:
  • pred (torch.Tensor) – input logits of shape (B x N)
  • target (torch.LontTensor) – target tensor of shape (B x N)
Returns:

loss – Multi label cross-entropy loss, of shape (B)

Return type:

torch.Tensor

class flambe.metric.MultiLabelNLLLoss(weight: Optional[torch.Tensor] = None, ignore_index: Optional[int] = None, reduction: str = 'mean')[source]

Bases: flambe.metric.metric.Metric

__str__(self)

Return the name of the Metric (for use in logging).

compute(self, pred: torch.Tensor, target: torch.Tensor)

Computes the Negative log likelihood loss for multilabel.

Parameters:
  • pred (torch.Tensor) – input logits of shape (B x N)
  • target (torch.LontTensor) – target tensor of shape (B x N)
Returns:

loss – Multi label negative log likelihood loss, of shape (B)

Return type:

torch.float

class flambe.metric.Accuracy[source]

Bases: flambe.metric.metric.AverageableMetric

compute(self, pred: torch.Tensor, target: torch.Tensor)

Computes the loss.

Parameters:
  • pred (Tensor) – input logits of shape (B x N)
  • target (LontTensor) – target tensor of shape (B) or (B x N)
Returns:

accuracy – single label accuracy, of shape (B)

Return type:

torch.Tensor

class flambe.metric.Perplexity[source]

Bases: flambe.metric.Metric

Token level perplexity, computed a exp(cross_entropy).

compute(self, pred: torch.Tensor, target: torch.Tensor)

Compute the preplexity given the input and target.

Parameters:
  • pred (torch.Tensor) – input logits of shape (B x N)
  • target (torch.LontTensor) – target tensor of shape (B)
Returns:

Output perplexity

Return type:

torch.float

aggregate(self, state: dict, *args, **kwargs)

Aggregates by only storing entropy per sample

Parameters:
  • state (dict) – the metric state
  • args (the pred, target tuple) –
Returns:

the state dict

Return type:

dict

finalize(self, state: Dict)

Finalizes the metric computation

Parameters:state (dict) – the metric state
Returns:The final score.
Return type:float
class flambe.metric.BPC[source]

Bases: flambe.metric.dev.perplexity.Perplexity

Bits per character. Computed as log_2(perplexity)

Inherits from Perplexity to share aggregate functionality.

compute(self, pred: torch.Tensor, target: torch.Tensor)

Compute the bits per character given the input and target.

Parameters:
  • pred (torch.Tensor) – input logits of shape (B x N)
  • target (torch.LontTensor) – target tensor of shape (B)
Returns:

Output perplexity

Return type:

torch.float

finalize(self, state: Dict)

Finalizes the metric computation

Parameters:state (dict) – the metric state
Returns:The final score.
Return type:float
class flambe.metric.AUC(max_fpr=1.0)[source]

Bases: flambe.metric.metric.Metric

__str__(self)

Return the name of the Metric (for use in logging).

compute(self, pred: torch.Tensor, target: torch.Tensor)

Compute AUC at the given max false positive rate.

Parameters:
  • pred (torch.Tensor) – The model predictions of shape numsamples
  • target (torch.Tensor) – The binary targets of shape numsamples
Returns:

The computed AUC

Return type:

torch.Tensor

class flambe.metric.MultiClassAUC[source]

Bases: flambe.metric.dev.auc.AUC

N-Ary (Multiclass) AUC for k-way classification

compute(self, pred: torch.Tensor, target: torch.Tensor)

Compute multiclass AUC at the given max false positive rate.

Parameters:
  • pred (torch.Tensor) – The model predictions of shape numsamples x numclasses
  • target (torch.Tensor) –
    The binary targets of shape:
    • numsamples. In this case the elements index into the different classes
    • numsamples x numclasses. This implementation only considers the indices of the max values as positive labels
Returns:

The computed AUC

Return type:

torch.Tensor

class flambe.metric.BinaryPrecision(threshold: float = 0.5, positive_label: int = 1)[source]

Bases: flambe.metric.dev.binary.BinaryMetric

Compute Binary Precision.

An example is considered negative when its score is below the specified threshold. Binary precition is computed as follows:

` |True positives| / |True Positives| + |False Positives| `

compute_binary(self, pred: torch.Tensor, target: torch.Tensor)

Compute binary precision.

Parameters:
  • pred (torch.Tensor) – Predictions made by the model. It should be a probability 0 <= p <= 1 for each sample, 1 being the positive class.
  • target (torch.Tensor) – Ground truth. Each label should be either 0 or 1.
Returns:

The computed binary metric

Return type:

torch.float

__str__(self)

Return the name of the Metric (for use in logging).

class flambe.metric.BinaryRecall(threshold: float = 0.5, positive_label: int = 1)[source]

Bases: flambe.metric.dev.binary.BinaryMetric

Compute binary recall.

An example is considered negative when its score is below the specified threshold. Binary precition is computed as follows:

` |True positives| / |True Positives| + |False Negatives| `

compute_binary(self, pred: torch.Tensor, target: torch.Tensor)

Compute binary recall.

Parameters:
  • pred (torch.Tensor) – Predictions made by the model. It should be a probability 0 <= p <= 1 for each sample, 1 being the positive class.
  • target (torch.Tensor) – Ground truth. Each label should be either 0 or 1.
Returns:

The computed binary metric

Return type:

torch.float

__str__(self)

Return the name of the Metric (for use in logging).

class flambe.metric.BinaryAccuracy[source]

Bases: flambe.metric.dev.binary.BinaryMetric

Compute binary accuracy.

` |True positives + True negatives| / N `

compute_binary(self, pred: torch.Tensor, target: torch.Tensor)

Compute binary accuracy.

Parameters:
  • pred (torch.Tensor) – Predictions made by the model. It should be a probability 0 <= p <= 1 for each sample, 1 being the positive class.
  • target (torch.Tensor) – Ground truth. Each label should be either 0 or 1.
Returns:

The computed binary metric

Return type:

torch.float

class flambe.metric.F1(threshold: float = 0.5, positive_label: int = 1, eps: float = 1e-08)[source]

Bases: flambe.metric.dev.binary.BinaryMetric

compute_binary(self, pred: torch.Tensor, target: torch.Tensor)

Compute F1. Score, the harmonic mean between precision and recall.

Parameters:
  • pred (torch.Tensor) – Predictions made by the model. It should be a probability 0 <= p <= 1 for each sample, 1 being the positive class.
  • target (torch.Tensor) – Ground truth. Each label should be either 0 or 1.
Returns:

The computed binary metric

Return type:

torch.float

class flambe.metric.Recall(top_k: int = 1)[source]

Bases: flambe.metric.metric.AverageableMetric

__str__(self)

Return the name of the Metric (for use in logging).

compute(self, pred: torch.Tensor, target: torch.Tensor)

Computes the recall @ k.

Parameters:
  • pred (Tensor) – input logits of shape (B x N)
  • target (LongTensor) – target tensor of shape (B) or (B x N)
Returns:

recall – single label recall, of shape (B)

Return type:

torch.Tensor