art.metrics
¶
Module providing metrics and verifications.
Clique Method Robustness Verification¶
-
class
art.metrics.
RobustnessVerificationTreeModelsCliqueMethod
(classifier: ClassifierDecisionTree)¶ Robustness verification for decision-tree-based models. Following the implementation in https://github.com/chenhongge/treeVerification (MIT License, 9 August 2019)
Paper link: https://arxiv.org/abs/1906.03849-
__init__
(classifier: ClassifierDecisionTree) → None¶ Create robustness verification for a decision-tree-based classifier.
- Parameters
classifier – A trained decision-tree-based classifier.
-
__weakref__
¶ list of weak references to the object (if defined)
-
verify
(x: numpy.ndarray, y: numpy.ndarray, eps_init: float, norm: int = inf, nb_search_steps: int = 10, max_clique: int = 2, max_level: int = 2) → Tuple[float, float]¶ Verify the robustness of the classifier on the dataset (x, y).
- Return type
Tuple
- Parameters
x (
ndarray
) – Feature data of shape (nb_samples, nb_features).y (
ndarray
) – Labels, one-vs-rest encoding of shape (nb_samples, nb_classes).eps_init (
float
) – Attack budget for the first search step.norm (
int
) – The norm to apply epsilon.nb_search_steps (
int
) – The number of search steps.max_clique (
int
) – The maximum number of nodes in a clique.max_level (
int
) – The maximum number of clique search levels.
- Returns
A tuple of the average robustness bound and the verification error at eps.
-
Loss Sensitivity¶
-
art.metrics.
loss_sensitivity
(classifier: CLASSIFIER_LOSS_GRADIENTS_TYPE, x: numpy.ndarray, y: numpy.ndarray) → numpy.ndarray¶ Local loss sensitivity estimated through the gradients of the prediction at points in x.
Paper link: https://arxiv.org/abs/1706.05394- Return type
ndarray
- Parameters
classifier – A trained model.
x (
ndarray
) – Data sample of shape that can be fed into classifier.y (
ndarray
) – Labels for sample x, one-hot encoded.
- Returns
The average loss sensitivity of the model.
Empirical Robustness¶
-
art.metrics.
empirical_robustness
(classifier: CLASSIFIER_TYPE, x: numpy.ndarray, attack_name: str, attack_params: Optional[Dict[str, Any]] = None) → Union[float, numpy.ndarray]¶ Compute the Empirical Robustness of a classifier object over the sample x for a given adversarial crafting method attack. This is equivalent to computing the minimal perturbation that the attacker must introduce for a successful attack.
Paper link: https://arxiv.org/abs/1511.04599- Parameters
classifier – A trained model.
x (
ndarray
) – Data sample of shape that can be fed into classifier.attack_name (
str
) – A string specifying the attack to be used. Currently supported attacks are {fgsm’, `hsj} (Fast Gradient Sign Method, Hop Skip Jump).attack_params – A dictionary with attack-specific parameters. If the attack has a norm attribute, then it will be used as the norm for calculating the robustness; otherwise the standard Euclidean distance is used (norm=2).
- Returns
The average empirical robustness computed on x.
CLEVER¶
-
art.metrics.
clever_u
(classifier: CLASSIFIER_CLASS_LOSS_GRADIENTS_TYPE, x: numpy.ndarray, nb_batches: int, batch_size: int, radius: float, norm: int, c_init: float = 1.0, pool_factor: int = 10) → float¶ Compute CLEVER score for an untargeted attack.
Paper link: https://arxiv.org/abs/1801.10578- Return type
float
- Parameters
classifier – A trained model.
x (
ndarray
) – One input sample.nb_batches (
int
) – Number of repetitions of the estimate.batch_size (
int
) – Number of random examples to sample per batch.radius (
float
) – Radius of the maximum perturbation.norm (
int
) – Current support: 1, 2, np.inf.c_init (
float
) – initialization of Weibull distribution.pool_factor (
int
) – The factor to create a pool of random samples with size pool_factor x n_s.
- Returns
CLEVER score.
-
art.metrics.
clever_t
(classifier: CLASSIFIER_CLASS_LOSS_GRADIENTS_TYPE, x: numpy.ndarray, target_class: int, nb_batches: int, batch_size: int, radius: float, norm: int, c_init: float = 1.0, pool_factor: int = 10) → float¶ Compute CLEVER score for a targeted attack.
Paper link: https://arxiv.org/abs/1801.10578- Return type
float
- Parameters
classifier – A trained model.
x (
ndarray
) – One input sample.target_class (
int
) – Targeted class.nb_batches (
int
) – Number of repetitions of the estimate.batch_size (
int
) – Number of random examples to sample per batch.radius (
float
) – Radius of the maximum perturbation.norm (
int
) – Current support: 1, 2, np.inf.c_init (
float
) – Initialization of Weibull distribution.pool_factor (
int
) – The factor to create a pool of random samples with size pool_factor x n_s.
- Returns
CLEVER score.
Wasserstein Distance¶
-
art.metrics.
wasserstein_distance
(u_values: numpy.ndarray, v_values: numpy.ndarray, u_weights: Optional[numpy.ndarray] = None, v_weights: Optional[numpy.ndarray] = None) → numpy.ndarray¶ Compute the first Wasserstein distance between two 1D distributions.
- Return type
ndarray
- Parameters
u_values (
ndarray
) – Values of first distribution with shape (nb_samples, feature_dim_1, …, feature_dim_n).v_values (
ndarray
) – Values of second distribution with shape (nb_samples, feature_dim_1, …, feature_dim_n).u_weights – Weight for each value. If None, equal weights will be used.
v_weights – Weight for each value. If None, equal weights will be used.
- Returns
The Wasserstein distance between the two distributions.