tests.utils

Module providing convenience functions specifically for unit tests.

Test Base Classes

class tests.utils.TestBase(methodName='runTest')

This class implements the base class for all unit tests.

class tests.utils.ExpectedValue(value, decimals)

Trained Models for Unittests, MNIST

tests.utils.get_image_classifier_tf(from_logits=False, load_init=True, sess=None)
tests.utils.get_image_classifier_tf_v1(from_logits=False, load_init=True, sess=None)

Standard TensorFlow classifier for unit testing.

The following hyper-parameters were used to obtain the weights and biases: learning_rate: 0.01 batch size: 10 number of epochs: 2 optimizer: tf.train.AdamOptimizer

Parameters
  • from_logits (bool) – Flag if model should predict logits (True) or probabilities (False).

  • load_init (bool) – Load the initial weights if True.

  • sess (tf.Session) – Computation session.

Returns

TensorFlowClassifier, tf.Session()

tests.utils.get_image_classifier_tf_v2(from_logits=False)

Standard TensorFlow v2 classifier for unit testing.

The following hyper-parameters were used to obtain the weights and biases: learning_rate: 0.01 batch size: 10 number of epochs: 2 optimizer: tf.train.AdamOptimizer

Returns

TensorFlowV2Classifier

tests.utils.get_image_classifier_kr(loss_name='categorical_crossentropy', loss_type='function_losses', from_logits=False, load_init=True)

Standard Keras classifier for unit testing

The weights and biases are identical to the TensorFlow model in get_classifier_tf().

Parameters
  • loss_name (str) – The name of the loss function.

  • loss_type (str) – The type of loss function definitions: label (loss function defined by string of its name), function_losses (loss function imported from keras.losses), function_backend (loss function imported from keras.backend)

  • from_logits (bool) – Flag if model should predict logits (True) or probabilities (False).

  • load_init (bool) – Load the initial weights if True.

Returns

KerasClassifier, tf.Session()

tests.utils.get_image_classifier_kr_tf(loss_name='categorical_crossentropy', loss_type='function', from_logits=False)

Standard Keras classifier for unit testing

The weights and biases are identical to the TensorFlow model in get_classifier_tf().

Parameters
  • loss_name (str) – The name of the loss function.

  • loss_type (str) – The type of loss function definitions: label (loss function defined by string of its name), function_losses (loss function), class (loss function generator)

  • from_logits (bool) – Flag if model should predict logits (True) or probabilities (False).

Returns

KerasClassifier

tests.utils.get_image_classifier_kr_functional(input_layer=1, output_layer=1)
tests.utils.get_image_classifier_kr_tf_functional(input_layer=1, output_layer=1)

Standard Keras_tf classifier for unit testing built with a functional model

Returns

KerasClassifier

tests.utils.get_image_classifier_kr_tf_with_wildcard()

Standard Tensorflow-Keras binary classifier for unit testing

Returns

KerasClassifier

tests.utils.get_image_classifier_kr_tf_binary()

Standard Tensorflow-Keras binary classifier for unit testing

Returns

KerasClassifier

tests.utils.get_image_classifier_pt(from_logits=False, load_init=True)

Standard PyTorch classifier for unit testing.

Parameters
  • from_logits (bool) – Flag if model should predict logits (True) or probabilities (False).

  • load_init (bool) – Load the initial weights if True.

Returns

PyTorchClassifier

tests.utils.get_classifier_bb(defences=None)

Standard BlackBox classifier for unit testing

Returns

BlackBoxClassifier

tests.utils.get_image_classifier_mxnet_custom_ini()
tests.utils.get_gan_inverse_gan_ft()
tests.utils.get_attack_classifier_pt(num_features)

PyTorch classifier for testing membership inference attacks.

Parameters

num_features (int) – The number of features in the attack model.

Returns

Model for attack.

Return type

PyTorchClassifier

tests.utils.check_adverse_example_x(x_adv, x_original, max=1.0, min=0.0, bounded=True)

Performs basic checks on generated adversarial inputs (whether x_test or x_train) :type bounded: bool :type min: float :type max: float :param x_adv: :param x_original: :param max: :param min: :param bounded: :return:

tests.utils.check_adverse_predicted_sample_y(y_pred_adv, y_non_adv)
tests.utils.is_valid_framework(framework)

Trained Models for Unittests, Iris

tests.utils.get_tabular_classifier_tf(load_init=True, sess=None)
tests.utils.get_tabular_classifier_tf_v1(load_init=True, sess=None)

Standard TensorFlow classifier for unit testing.

The following hyper-parameters were used to obtain the weights and biases:

  • learning_rate: 0.01

  • batch size: 5

  • number of epochs: 200

  • optimizer: tf.train.AdamOptimizer

The model is trained of 70% of the dataset, and 30% of the training set is used as validation split.

Parameters
  • load_init (bool) – Load the initial weights if True.

  • sess (tf.Session) – Computation session.

Returns

The trained model for Iris dataset and the session.

Return type

tuple(TensorFlowClassifier, tf.Session)

tests.utils.get_tabular_classifier_tf_v2()

Standard TensorFlow v2 classifier for unit testing.

The following hyper-parameters were used to obtain the weights and biases:

  • learning_rate: 0.01

  • batch size: 5

  • number of epochs: 200

  • optimizer: tf.train.AdamOptimizer

The model is trained of 70% of the dataset, and 30% of the training set is used as validation split.

Returns

The trained model for Iris dataset and the session.

Return type

TensorFlowV2Classifier

tests.utils.get_tabular_classifier_scikit_list(clipped=False)
tests.utils.get_tabular_classifier_kr(load_init=True)

Standard Keras classifier for unit testing on Iris dataset. The weights and biases are identical to the TensorFlow model in get_iris_classifier_tf.

Parameters

load_init (bool) – Load the initial weights if True.

Returns

The trained model for Iris dataset and the session.

Return type

tuple(KerasClassifier, tf.Session)

tests.utils.get_tabular_classifier_pt(load_init=True)

Standard PyTorch classifier for unit testing on Iris dataset.

Parameters

load_init (bool) – Load the initial weights if True.

Returns

Trained model for Iris dataset.

Return type

PyTorchClassifier

Random Number Generators

tests.utils.master_seed(seed=1234, set_random=True, set_numpy=True, set_tensorflow=False, set_mxnet=False, set_torch=False)

Set the seed for all random number generators used in the library. This ensures experiments reproducibility and stable testing.

Parameters
  • seed (int) – The value to be seeded in the random number generators.

  • set_random (bool) – The flag to set seed for random.

  • set_numpy (bool) – The flag to set seed for numpy.

  • set_tensorflow (bool) – The flag to set seed for tensorflow.

  • set_mxnet (bool) – The flag to set seed for mxnet.

  • set_torch (bool) – The flag to set seed for torch.