tests.utils¶
Module providing convenience functions specifically for unit tests.
Test Base Classes¶
- class tests.utils.TestBase(methodName='runTest')¶
This class implements the base class for all unit tests.
- class tests.utils.ExpectedValue(value, decimals)¶
Trained Models for Unittests, MNIST¶
- tests.utils.get_image_classifier_tf(from_logits=False, load_init=True, sess=None, framework=None)¶
- tests.utils.get_image_classifier_tf_v1(from_logits=False, load_init=True, sess=None)¶
Standard TensorFlow classifier for unit testing.
The following hyper-parameters were used to obtain the weights and biases: learning_rate: 0.01 batch size: 10 number of epochs: 2 optimizer: tf.train.AdamOptimizer
- Parameters
from_logits (bool) – Flag if model should predict logits (True) or probabilities (False).
load_init (bool) – Load the initial weights if True.
sess (tf.Session) – Computation session.
- Returns
TensorFlowClassifier, tf.Session()
- tests.utils.get_image_classifier_tf_v2(from_logits=False)¶
Standard TensorFlow v2 classifier for unit testing.
The following hyper-parameters were used to obtain the weights and biases: learning_rate: 0.01 batch size: 10 number of epochs: 2 optimizer: tf.train.AdamOptimizer
- Returns
TensorFlowV2Classifier
- tests.utils.get_image_classifier_kr(loss_name='categorical_crossentropy', loss_type='function_losses', from_logits=False, load_init=True)¶
Standard Keras classifier for unit testing
The weights and biases are identical to the TensorFlow model in get_classifier_tf().
- Parameters
loss_name (str) – The name of the loss function.
loss_type (str) – The type of loss function definitions: label (loss function defined by string of its name), function_losses (loss function imported from keras.losses), function_backend (loss function imported from keras.backend)
from_logits (bool) – Flag if model should predict logits (True) or probabilities (False).
load_init (bool) – Load the initial weights if True.
- Returns
KerasClassifier, tf.Session()
- tests.utils.get_image_classifier_kr_tf(loss_name='categorical_crossentropy', loss_type='function', from_logits=False)¶
Standard Keras classifier for unit testing
The weights and biases are identical to the TensorFlow model in get_classifier_tf().
- Parameters
loss_name (str) – The name of the loss function.
loss_type (str) – The type of loss function definitions: label (loss function defined by string of its name), function_losses (loss function), class (loss function generator)
from_logits (bool) – Flag if model should predict logits (True) or probabilities (False).
- Returns
KerasClassifier
- tests.utils.get_image_classifier_kr_functional(input_layer=1, output_layer=1)¶
- tests.utils.get_image_classifier_kr_tf_functional(input_layer=1, output_layer=1)¶
Standard Keras_tf classifier for unit testing built with a functional model
- Returns
KerasClassifier
- tests.utils.get_image_classifier_kr_tf_with_wildcard()¶
Standard TensorFlow-Keras binary classifier for unit testing
- Returns
KerasClassifier
- tests.utils.get_image_classifier_kr_tf_binary()¶
Standard TensorFlow-Keras binary classifier for unit testing
- Returns
KerasClassifier
- tests.utils.get_image_classifier_pt(from_logits=False, load_init=True, use_maxpool=True)¶
Standard PyTorch classifier for unit testing.
- Parameters
from_logits (bool) – Flag if model should predict logits (True) or probabilities (False).
load_init (bool) – Load the initial weights if True.
use_maxpool (bool) – If to use a classifier with maxpool or not
- Returns
PyTorchClassifier
- tests.utils.get_classifier_bb(defences=None)¶
Standard BlackBox classifier for unit testing
- Returns
BlackBoxClassifier
- tests.utils.get_image_classifier_mxnet_custom_ini()¶
- tests.utils.get_gan_inverse_gan_ft()¶
- tests.utils.get_attack_classifier_pt(num_features)¶
PyTorch classifier for testing membership inference attacks.
- Parameters
num_features (int) – The number of features in the attack model.
- Returns
Model for attack.
- Return type
- tests.utils.check_adverse_example_x(x_adv, x_original, max=1.0, min=0.0, bounded=True)¶
Performs basic checks on generated adversarial inputs (whether x_test or x_train) :type bounded:
bool:type min:float:type max:float:param x_adv: :param x_original: :param max: :param min: :param bounded: :return:
- tests.utils.check_adverse_predicted_sample_y(y_pred_adv, y_non_adv)¶
- tests.utils.is_valid_framework(framework)¶
Trained Models for Unittests, Iris¶
- tests.utils.get_tabular_classifier_tf(load_init=True, sess=None)¶
- tests.utils.get_tabular_classifier_tf_v1(load_init=True, sess=None)¶
Standard TensorFlow classifier for unit testing.
The following hyper-parameters were used to obtain the weights and biases:
learning_rate: 0.01
batch size: 5
number of epochs: 200
optimizer: tf.train.AdamOptimizer
The model is trained of 70% of the dataset, and 30% of the training set is used as validation split.
- Parameters
load_init (bool) – Load the initial weights if True.
sess (tf.Session) – Computation session.
- Returns
The trained model for Iris dataset and the session.
- Return type
tuple(TensorFlowClassifier, tf.Session)
- tests.utils.get_tabular_classifier_tf_v2()¶
Standard TensorFlow v2 classifier for unit testing.
The following hyper-parameters were used to obtain the weights and biases:
learning_rate: 0.01
batch size: 5
number of epochs: 200
optimizer: tf.train.AdamOptimizer
The model is trained of 70% of the dataset, and 30% of the training set is used as validation split.
- Returns
The trained model for Iris dataset and the session.
- Return type
TensorFlowV2Classifier
- tests.utils.get_tabular_classifier_scikit_list(clipped=False, model_list_names=None)¶
- tests.utils.get_tabular_classifier_kr(load_init=True)¶
Standard Keras classifier for unit testing on Iris dataset. The weights and biases are identical to the TensorFlow model in get_iris_classifier_tf.
- Parameters
load_init (bool) – Load the initial weights if True.
- Returns
The trained model for Iris dataset and the session.
- Return type
tuple(KerasClassifier, tf.Session)
- tests.utils.get_tabular_classifier_pt(load_init=True)¶
Standard PyTorch classifier for unit testing on Iris dataset.
- Parameters
load_init (bool) – Load the initial weights if True.
- Returns
Trained model for Iris dataset.
- Return type
Random Number Generators¶
- tests.utils.master_seed(seed=1234, set_random=True, set_numpy=True, set_tensorflow=False, set_mxnet=False, set_torch=False)¶
Set the seed for all random number generators used in the library. This ensures experiments reproducibility and stable testing.
- Parameters
seed (int) – The value to be seeded in the random number generators.
set_random (bool) – The flag to set seed for random.
set_numpy (bool) – The flag to set seed for numpy.
set_tensorflow (bool) – The flag to set seed for tensorflow.
set_mxnet (bool) – The flag to set seed for mxnet.
set_torch (bool) – The flag to set seed for torch.