Skip to content

model_tests.FEAT.MinMaxMetricThreshold

MinMaxMetricThreshold Objects

@dataclass
class MinMaxMetricThreshold(ModelTest)

Test if all the subgroups for a given attribute meets a certain level of expected, performance. If fpr / fnr is used as a metric, they have to be lower than the threshold. If tpr / tnr is used as a metric, they have to be greater than the threshold.

The test also stores a dataframe showing the results of each groups and ROC curve plots for every subgroup along with the points which maximises tpr-fpr.

Arguments:

  • attr - Column name of the protected attribute.
  • metric - Type of performance metric for the test, choose from 'fpr' - false positive rate, 'tpr' - true positive rate, 'fnr' - false negative rate, 'tnr' - true negative rate.
  • threshold - Threshold for the test. To pass, fpr / fnr has to be lower than the threshold or tpr/tnr has to be greater than the threshold.
  • proba_threshold - Probability threshold for the output to be classified as 1. By default, it is 0.5.
  • test_name - Name of the test, default is 'ROC/Min Max Threshold Test'.
  • test_desc - Description of the test. If none is provided, an automatic description will be generated based on the rest of the arguments passed in.

get_result

def get_result(df_test_with_output: pd.DataFrame) -> pd.DataFrame

Test if at the current probability thresholds, for a particular attribute, the fpr/tpr of its groups passes the maximum/mininum specified metric thresholds.

Arguments:

  • df_test_with_output - Dataframe containing protected attributes with "prediction_probas" and "truth" column.

Returns:

Dataframe with the results of each group.

plot

def plot(save_plots: bool = True)

Plots ROC curve for every group in the attribute and mark the optimal probability threshold, the point which maximises tpr-fpr.

Arguments:

  • save_plots - If True, saves the plots to the class instance.

run

def run(df_test_with_output: pd.DataFrame) -> bool

Runs test by calculating result and evaluating if it passes a defined condition.

Arguments:

  • df_test_with_output - Dataframe containing protected attributes with "prediction_probas" and "truth" column. protected attribute should not be encoded.