\vspace{-0.5cm}
\section{EXPERIMENTAL EVALUATION}

In this section we empirically analyze our proposed PE-LAC algorithm in terms of both
classification accuracy and model size/interpretability. 
Our evaluation is based on a direct comparison against the original LAC algorithm~\cite{lac}.
We also employ a baseline classifier based on closed rules, which we call CL-LAC (Closed Lazy Associative
Classification), so that we can compare PE-LAC against a strategy based on condensed representations.
It is worth mentioning that most of the results can be also compared against other algorithms since 
we used the same evaluation methodology and the same datasets as in \cite{lac}. 
Finally, we also evaluate the effectiveness of our proposed strategy when used to filter closed rules,
an algorithm we call as CL-PE-LAC (Closed Pareto-Efficient Lazy Associative Classification).
We first discuss the evaluation methodology and significance tests, and then we present our results.

\begin{table}[tb]
\caption{Datasets.}
\label{table.caracterizao.allbases}
\centering
\begin{center}
\begin{tabular}{|l|r|r|r|r|r|r|r|}\hline
\multicolumn{1}{|c|}{\multirow{2}{*}{\textbf{Dataset}}} & \multirow{2}{*}{\textbf{\# attributes}} & \multirow{2}{*}{\textbf{\# instances}} & \multicolumn{5}{c|}{\textbf{Class Frequency}} \\ \cline{4-8}
& & &\textbf{ 0} & \textbf{ 1} & \textbf{ 2} & \textbf{ 3} & \textbf{4}\\\hline\hline
UCI-austra & 14 &   621 & 0.55 & 0.45 & - & - & -\\\hline
UCI-breast & 10 &  629 & 0.66 & 0.34 & - & - & -\\\hline
UCI-cleve & 13 &   272 & 0.55 & 0.45 & - & - & -\\\hline
UCI-crx & 15 &   621 & 0.55 & 0.45 & - & - & -\\\hline
UCI-diabetes & 8 &  691 & 0.64 &  0.36 & - & - & -\\\hline
UCI-german & 20  &  900 & 0.70 & 0.30 & - & - & -\\\hline
UCI-heart & 13  &  243 & 0.42 & 0.58 & - & - & -\\\hline
UCI-hepati & 19  &  139 & 0.79 & 0.21 & - & - & -\\\hline
UCI-horse & 22  &  331 & 0.36 & 0.64 & - & - & -\\\hline
UCI-ionosphere & 34  &  315 & 0.65 &  0.35 & - & - & -\\\hline
Twitter-elections & [1,45]   &  66,643 & 0.30 & 0.70 & - & - & -\\\hline
Twitter-worldcup & [4,34]  &  3,215 & 0.80  & 0.20 & - & - & -\\\hline
Kaggle-titanic  & 8  &  891 & 0.62 & 0.38 & - & - & -\\\hline
Twitter-times & [8,29]    &  5,616 & 0.04 & 0.04 & 0.72 & 0.12 & 0.08 \\\hline
\end{tabular}
\end{center}
\end{table}
\vspace{-0.5cm}
\subsection{Evaluation Methodology}

We used several datasets in our experiments. Specifically,
we used 10 benchmark datasets coming from the UCI repository~\cite{uci},
3 datasets from Twitter sentiment classification~\cite{sigir}, and one dataset from Kaggle~\cite{kaggle}. 
Table~\ref{table.caracterizao.allbases} shows information about these datasets. 
Some datasets are structured, and in this case all instances have a fixed number of attributes. 
Other datasets are obtained from applications that involve textual data (i.e., tweets),
and thus instances may have different number of attributes. 
In all the experiments with the aforementioned datasets we used 5-fold cross-validation and the final results of each experiment represent the average of the five runs.
Classification
accuracy is assessed through the conventional
precision, recall and $F_1$ measures. Precision $p$ is defined as the
proportion of correctly classified test instances in the test-set.
Recall $r$ is defined as the proportion of correctly
classified test instances out of all the instances having the target
class. $F_1$ is a combination of precision and recall defined as
the harmonic mean $\frac{2pr}{p+r}$.  Macro- and
micro-averaging~\cite{yang} were applied to $F_1$ to get
single performance values.
Micro-$F_1$ essentially corresponds to accuracy, while Macro-$F_1$
is the average of the $F_1$ values over all classes. Finally, model interpretability is assessed by the number of rules within the classifier.

\subsection{Significance Tests}

The objective of the PE-LAC algorithm is to decrease the number of rules in $\mathcal{R}$ while keeping classification accuracy (i.e., Macro- and Micro-$F_1$) approximately the same. In order to check if the objective was reached we will employ three significance tests for paired-comparison and two group-based hypothesis test. Tests for paired-comparison include: (i) T-Test (parametric), (ii) Wilcoxon (non-parametric), and (iii) Permutation (non-parametric). Group-based hypothesis tests include: (i) ANOVA (parametric), and (ii) Kruskal-Wallis (non-parametric).
%Table~\ref{table.test.premissas} presents a brief description of the aforementioned significance tests.
Additional details about these tests can be found
in~\cite{Wasserman2010}.

%\begin{table}[tb]
%\centering
%\caption{Significance Tests.}
%\label{table.test.premissas}
%\begin{center}
%\begin{tabular}{|l|p{7.0cm}|} \hline
%\multicolumn{1}{|c|}{\textbf{Significance Test}} & \multicolumn{1}{|c|}{\textbf{Description}} \\ \hline \hline
%Paired T-Test & Assumes Normal distribution. \\
% & Works better when the sample size is $\geq 30$. \\ 
%  & Well-known and largely used. \\ \hline
% Wilcoxon &  Does not assume Normal distribution.\\ 
%		    & Assumes that samples belong to the same population.\\ 
%		   &  Works well for small sample sizes.\\ \hline
%Permutation & Does not assume Normal distribution.\\
%		   &  Works well for small sample sizes.\\ \hline \hline
%ANOVA & Assumes Normal distribution.\\ \hline
%Kruskal-Wallis & Does not assume Normal distribution. \\ \hline
%\end{tabular}
%\end{center}
%\end{table}

We always consider two options: $\bm{H_0}$ is the null-hypothesis, and $\bm{H_1}$ is the alternate hypothesis. In our case, the null hypotheses that may be accepted or rejected are:

\begin{enumerate}
 \item Both LAC and PE-LAC algorithms learn classifiers that achieve the same Micro-$F_1$ values.
 \item Both LAC and PE-LAC algorithms learn classifier that achieve the same Macro-$F_1$ values.
 \item Both LAC and PE-LAC algorithms learn classifiers composed of the same number of rules.
\end{enumerate}

The null-hypothesis is only rejected if its probability is very low~\cite{Wasserman2010}.
\begin{wraptable}[9]{r}{8cm}
\centering
\caption{$p$-value Scale.}
\label{table.pvalue}
\begin{center}
\begin{tabular}{ll}
\textbf{$p$-value} & \textbf{Evidence} \\ \hline
$\bm{< 0.01}$ &  Strong evidence against $\bm{H_0}$. \\
$\bm{0.01 - 0.10}$ & Weak evidence against $\bm{H_0}$. \\
$\bm{> 0.10}$ & No evidence against $\bm{H_0}$. \\ 
\end{tabular}
\end{center}
\end{wraptable}
In order to assess the evidence against
$\bm{H_0}$ we use a $p$-value, which is given as $\mathbb{P}(T > t_{obs}|H_o)$, where $T$ is a given statistics and
$t_{obs}$ is the observed value for such statistics (i.e., Macro-$F_1$, Micro-$F_1$, or the number of rules within the classifier). Typically, researchers have use the scale shown in Table~\ref{table.pvalue}.
The case in which $\bm{H_0}$ is rejected, but is indeed true, is called Type-I error. Similarly, a Type-II error occurs if $\bm{H_1}$ is true but
$\bm{H_0}$ was accepted.

\begin{table}[tb]
\caption{PE-LAC: Performance according to $\eta$.}
\label{table.results.layers}
\centering
\begin{center}
\begin{tabular}{|l|l|c|rrrrrrrrrr|}\hline
& \textbf{Objective} & \begin{sideways}\textbf{\# frontiers ($\eta$)}\end{sideways}&\begin{sideways}\textbf{UCI-austra}\end{sideways} &
\begin{sideways}\textbf{UCI-breast}\end{sideways} &
 \begin{sideways}\textbf{UCI-cleve}\end{sideways} &
 \begin{sideways}\textbf{UCI-crx}\end{sideways} &
 \begin{sideways}\textbf{UCI-diabetes}\end{sideways} &
 \begin{sideways}\textbf{UCI-german}\end{sideways} &
 \begin{sideways}\textbf{UCI-heart}\end{sideways} &
 \begin{sideways}\textbf{UCI-hepati}\end{sideways} &
 \begin{sideways}\textbf{UCI-horse}\end{sideways} &
 \begin{sideways}\textbf{UCI-ionosphere}\end{sideways} 
\\ \hline \hline

  \multirow{4}{*}{\begin{sideways}\textbf{Conf.}\end{sideways}}  & \multirow{2}{*}{Micro-$F_1$} & 1 & 0.82  & 0.95 & 0.80 &0.82 & 0.69 & 0.71  & 0.83 & 0.82 & 0.78  & 0.91  \\
  &   & 2 & 0.85  & 0.96  & 0.83 & 0.84  & 0.75  & 0.71  & 0.85 & 0.83 &  0.79  & 0.93 \\
  &   & 3 & 0.86  & 0.97 & 0.83 &0.86 & 0.78 & 0.72  &0.85 &0.83 & 0.82  &0.94   \\
  &   & 4 & 0.87  & 0.97 & 0.84 &0.86 & 0.78 & 0.71  &0.85 &0.84 & 0.83  &0.94   \\ \hline \hline
 
 \multirow{4}{*}{\begin{sideways}\textbf{Conf.}\end{sideways}}  & \multirow{2}{*}{\#Rules} & 1 & 3  & 7  & 3 & 3  & 2  & 5  & 3 & 21 &  15  & 103 \\
  &   & 2 & 5  & 8  & 5 & 6  & 6  & 6  & 5 & 24 &  17  & 107 \\
  &   & 3 & 8  & 9  & 7 & 10  & 9  & 8   & 8 &27 &  20  & 111  \\
  &   & 4 & 11  & 11  & 10 & 13  & 12  & 10   & 10 & 31 &  23  & 115  \\ \hline

\end{tabular}
\end{center}
\caption{PE-LAC: Performance according to criteria ($1=$confidence, $2=$support, $3=$added value, $4=$Yules'Q).}
\label{table.results.criteria}
\centering
\begin{center}
\begin{tabular}{|l|l|r|r|r|r|r|r|r|r|r|r|r|}\hline
\multicolumn{1}{|c|}{\multirow{2}{*}{\textbf{Dataset}}} & \multicolumn{1}{c}{\multirow{2}{*}{\textbf{Objective}}} & \multicolumn{11}{|c|}{\textbf{Criteria}} \\ \cline{3-13}
  &   & 1,2 & 1,3 & \textbf{1,4} & 2,3 & 2,4 & 3,4 & 1,2,3 & 1,2,4 & 1,3,4 & 2,3,4 & 1,2,3,4 \\ \hline \hline 
  
\multirow{3}{*}{\begin{sideways}\textbf{austra}\end{sideways}} & \#Rules & 24  & 4 & 8  & 25 &  12 & 8  & 26 & 26 & 8  & 26  & 27 \\ \cline{2-13}
   & Micro-$F_1$ &  0.83 &  0.84 & 0.86   & 0.86  & 0.79  & 0.85  & 0.86  &  0.86 &  0.86  & 0.86  &  0.86 \\ \cline{2-13}
   & Macro-$F_1$ &  0.82 &  0.83 & 0.85   & 0.85  & 0.77  & 0.85   & 0.85 & 0.85  &  0.86  & 0.85 &  0.85\\ \hline \hline
   
   
\multirow{3}{*}{\begin{sideways}\textbf{breast}\end{sideways}} & \#Rules & 25 & 9 & 9  & 26 & 24 & 10  & 26 & 25 & 10  & 26 & 26 \\ \cline{2-13}
   & Micro-$F_1$ & 0.96 & 0.96 & 0.97  &  0.97 &  0.96 & 0.95  & 0.97 &  0.96 & 0.95  & 0.97  & 0.97 \\ \cline{2-13}
   & Macro-$F_1$ & 0.95  & 0.95 & 0.96  & 0.97 & 0.95 &  0.94 &  0.97 & 0.96  & 0.95  & 0.97 & 0.97 \\ \hline \hline
   

\multirow{3}{*}{\begin{sideways}\textbf{cleve}\end{sideways}} & \#Rules & 32 &  5 &  7 & 34 &  23 &  7  & 37 & 36  & 7  & 35  & 37  \\ \cline{2-13}
   & Micro-$F_1$ & 0.79 &  0.82 &  0.83 & 0.81 & 0.82 & 0.81  & 0.81 & 0.80 & 0.82  & 0.80  & 0.81 \\ \cline{2-13}
   & Macro-$F_1$ & 0.78 & 0.81 &  0.83 & 0.80 &  0.81 &  0.80 & 0.80 &  0.79 & 0.81  & 0.79 &  0.80 \\ \hline 

\end{tabular}
\end{center}
\end{table}

\subsection{Results}

The first experiment is devoted to evaluate how $\eta$ impacts classification accuracy. For this experiment we used the UCI datasets, and varied $\eta$ from 1 to 4 frontiers. Further, due to lack of space we only consider confidence as the $\theta$-statistics used in Equation 1. Micro-$F_1$ numbers may suffer when $\eta=1$, indicating that the small number of rules makes the classifier less robust to noise. Clearly, Micro-$F_1$ stabilizes for $\eta$ values around 3 frontiers. The number of rules, as expected, increases with $\eta$. Unless otherwise stated, for now on we will fix $\eta=3$ in all remaining experiments.

Table~\ref{table.results.criteria} shows Macro- and Micro-$F_1$ numbers, as well as the number of rules within the classifier, for different rule-utility spaces. Due to lack of space we limit our analysis to three datasets, namely: UCI-austra, UCI-breast, and UCI-cleve. Although different criteria are used to build the rule-utility space, we only consider confidence as the $\theta$-statistics used in Equation 1. Overall, the best rule-utility space is the one composed by confidence and Yules'Q. Together, these two criteria show all properties discussed in Table I. Further, the number of rules within the classifier increases with the number of dimensions. Therefore, for now on we employ confidence and Yules'Q as the dimensions of the rule-utility space.

Accuracy numbers as well as model size for LAC and PE-LAC algorithms are shown in Table~\ref{table.results.partial1}. 
A quick analysis indicates that we have reached our main objective of learning smaller classifiers without hurting classification performance. In most of the cases, the average number of rules within $\mathcal{R}_x$ was decreased by two orders of magnitude. For instance, for the UCI-austra dataset the average number of rules decreases from 210 to only 8. Also, in most of the cases, Macro- and Micro-$F_1$ numbers obtained by LAC, CL-LAC, PE-LAC, and CL-PE-LAC are all very similar. In some cases, however, we observe large variations, in many cases favoring PE-LAC.



\begin{table}[tb]
\renewcommand{\tabcolsep}{.80mm}
\renewcommand{\arraystretch}{1}
\caption{Accuracy numbers and model size/interpretability. Darker cells indicate better results.}
\label{table.results.partial1}
\centering
\begin{center}
\begin{tabular}{|l|l|l|rrrrrrrrrrrrrr|}\hline
 
& \textbf{Objective} & \textbf{Algorithm} & \begin{sideways}\textbf{UCI-austra}\end{sideways} & 
\begin{sideways}\textbf{UCI-breast}\end{sideways} & 
 \begin{sideways}\textbf{UCI-cleve}\end{sideways} & 
 \begin{sideways}\textbf{UCI-crx}\end{sideways} & 
 \begin{sideways}\textbf{UCI-diabetes}\end{sideways} & 
 \begin{sideways}\textbf{UCI-german}\end{sideways} & 
 \begin{sideways}\textbf{UCI-heart}\end{sideways} & 
 \begin{sideways}\textbf{UCI-hepati}\end{sideways} & 
 \begin{sideways}\textbf{UCI-horse}\end{sideways} & 
 \begin{sideways}\textbf{UCI-ionosphere}\end{sideways}  &
 \begin{sideways}\textbf{Twitter-elections}\end{sideways}  &
 \begin{sideways}\textbf{Twitter-worldcup}\end{sideways}  &
 \begin{sideways}\textbf{Kaggle-titanic}\end{sideways}  &
 \begin{sideways}\textbf{Twitter-times}\end{sideways}  
\\ \hline \hline 

  \multirow{4}{*}{\begin{sideways}\textbf{Any}\end{sideways}}  & \multirow{4}{*}{\#Rules} & LAC & 210 & 110 & 181 & 239 & 72 & 419 & 181 & 377 & 502 & 1181 & 201 & 197 & 70 & 1286  \\ 
  &   & CL-LAC &  \cellcolor{blue!15} 30 & \cellcolor{blue!15} 22 & \cellcolor{blue!15} 28 & \cellcolor{blue!15} 32 & \cellcolor{blue!15} 16 & \cellcolor{blue!15} 42 & \cellcolor{blue!15} 28 & \cellcolor{blue!15} 39 & \cellcolor{blue!15} 45 & \cellcolor{blue!30} 69 & \cellcolor{blue!30} 28 & \cellcolor{blue!30} 28 & \cellcolor{blue!15} 18 & \cellcolor{blue!15} 110\\ 
  &   & PE-LAC & \cellcolor{blue!30}  8 & \cellcolor{blue!30} 9 & \cellcolor{blue!30} 7 & \cellcolor{blue!30} 10 & \cellcolor{blue!30} 9 & \cellcolor{blue!30} 8 & \cellcolor{blue!30} 8 & \cellcolor{blue!30} 27 & \cellcolor{blue!30} 20 & \cellcolor{blue!15} 111 & \cellcolor{blue!15} 49 & \cellcolor{blue!15} 60 & \cellcolor{blue!30} 7 & \cellcolor{blue!30} 108 \\ 
 &   & CL-PE-LAC &  \cellcolor{blue!50} 6 & \cellcolor{blue!50} 4 & \cellcolor{blue!50} 7 & \cellcolor{blue!50} 6 & \cellcolor{blue!50} 6 & \cellcolor{blue!50} 7 & \cellcolor{blue!50} 6 & \cellcolor{blue!50} 8 & \cellcolor{blue!50} 6 & \cellcolor{blue!50} 4 & \cellcolor{blue!50} 6 & \cellcolor{blue!50} 8 & \cellcolor{blue!50} 5 & \cellcolor{blue!50} 9 \\ \hline \hline 

\multirow{8}{*}{\begin{sideways}\textbf{ Conf. }\end{sideways}} & \multirow{4}{*}{Micro-$F_1$}   & LAC & \cellcolor{blue!30} 0.84 & \cellcolor{blue!50} 0.97 & \cellcolor{blue!30} 0.82 & \cellcolor{blue!30} 0.83 & \cellcolor{blue!30} 0.7 & \cellcolor{blue!30} 0.7 & \cellcolor{blue!30} 0.84 & \cellcolor{blue!15} 0.79 & \cellcolor{blue!15} 0.74 & \cellcolor{blue!15} 0.9 & \cellcolor{blue!15} 0.94 & \cellcolor{blue!50} 0.96 & \cellcolor{blue!30} 0.75 & \cellcolor{blue!30} 0.95 \\
 & & CL-LAC &  0.81 &  0.96 &  0.79 &  0.8 &  0.65 & \cellcolor{blue!30} 0.7 &  0.81 & \cellcolor{blue!15} 0.79 &  0.65 &  0.89 &  0.87 & \cellcolor{blue!50} 0.96 &  0.71 &  0.83 \\
 & & PE-LAC & \cellcolor{blue!50} 0.86 & \cellcolor{blue!50} 0.97 & \cellcolor{blue!50} 0.83 & \cellcolor{blue!50} 0.86 & \cellcolor{blue!50} 0.78 & \cellcolor{blue!50} 0.72 & \cellcolor{blue!50} 0.85 & \cellcolor{blue!50} 0.83 & \cellcolor{blue!50} 0.82 & \cellcolor{blue!50} 0.94 & \cellcolor{blue!50} 0.96 & \cellcolor{blue!50} 0.96 & \cellcolor{blue!50} 0.77 & \cellcolor{blue!50} 0.96 \\
 & & CL-PE-LAC & \cellcolor{blue!15} 0.83 & \cellcolor{blue!50} 0.97 & \cellcolor{blue!15} 0.81 & \cellcolor{blue!15} 0.81 & \cellcolor{blue!30} 0.7 & \cellcolor{blue!30} 0.7 & \cellcolor{blue!15} 0.83 & \cellcolor{blue!30} 0.82 & \cellcolor{blue!30} 0.79 & \cellcolor{blue!30} 0.91 & \cellcolor{blue!30} 0.95 &  0.95 & \cellcolor{blue!15} 0.73 & \cellcolor{blue!15} 0.92 \\  \hhline{~*{16}{-}} %\cline{2-16}

& \multirow{4}{*}{Macro-$F_1$} 
 & LAC & \cellcolor{blue!30} 0.83 & \cellcolor{blue!50} 0.97 & \cellcolor{blue!30} 0.81 & \cellcolor{blue!30} 0.82 & \cellcolor{blue!30} 0.57 & \cellcolor{blue!15} 0.41 & \cellcolor{blue!30} 0.83 & \cellcolor{blue!15} 0.44 & \cellcolor{blue!15} 0.64 & \cellcolor{blue!15} 0.88 & \cellcolor{blue!15} 0.92 & \cellcolor{blue!50} 0.93 & \cellcolor{blue!30} 0.7 & \cellcolor{blue!30} 0.7 \\
 & & CL-LAC &  0.8 & \cellcolor{blue!15} 0.96 &  0.78 & \cellcolor{blue!15} 0.79 &  0.4 & \cellcolor{blue!15} 0.41 &  0.79 & \cellcolor{blue!15} 0.44 &  0.41 &  0.86 &  0.82 &  0.92 &  0.63 &  0.35 \\
 & & PE-LAC & \cellcolor{blue!50} 0.85 & \cellcolor{blue!15} 0.96 & \cellcolor{blue!50} 0.83 & \cellcolor{blue!50} 0.86 & \cellcolor{blue!50} 0.74 & \cellcolor{blue!50} 0.5 & \cellcolor{blue!50} 0.84 & \cellcolor{blue!50} 0.65 & \cellcolor{blue!50} 0.78 & \cellcolor{blue!50} 0.93 & \cellcolor{blue!50} 0.95 & \cellcolor{blue!50} 0.93 & \cellcolor{blue!50} 0.75 & \cellcolor{blue!50} 0.77 \\
 & & CL-PE-LAC & \cellcolor{blue!15} 0.82 & \cellcolor{blue!50} 0.97 & \cellcolor{blue!15} 0.8 & \cellcolor{blue!15} 0.79 & \cellcolor{blue!30} 0.57 & \cellcolor{blue!30} 0.46 & \cellcolor{blue!15} 0.82 & \cellcolor{blue!30} 0.55 & \cellcolor{blue!30} 0.75 & \cellcolor{blue!30} 0.89 & \cellcolor{blue!30} 0.93 & \cellcolor{blue!50} 0.93 & \cellcolor{blue!15} 0.67 & \cellcolor{blue!15} 0.59 \\ \hline \hline

\multirow{8}{*}{\begin{sideways}\textbf{ Support }\end{sideways}} & \multirow{4}{*}{Micro-$F_1$}   & LAC & \cellcolor{blue!30} 0.8 & \cellcolor{blue!15} 0.93 & \cellcolor{blue!30} 0.8 & \cellcolor{blue!30} 0.8 & \cellcolor{blue!30} 0.65 & \cellcolor{blue!50} 0.7 & \cellcolor{blue!30} 0.81 & \cellcolor{blue!50} 0.79 & \cellcolor{blue!15} 0.66 & \cellcolor{blue!30} 0.79 & \cellcolor{blue!15} 0.72 & \cellcolor{blue!50} 0.96 & \cellcolor{blue!30} 0.7 & \cellcolor{blue!15} 0.72 \\
 & & CL-LAC &  0.72 &  0.89 &  0.75 &  0.72 & \cellcolor{blue!15} 0.64 & \cellcolor{blue!50} 0.7 &  0.74 & \cellcolor{blue!50} 0.79 &  0.64 &  0.75 &  0.7 & \cellcolor{blue!15} 0.8 &  0.65 & \cellcolor{blue!15} 0.72 \\
 & & PE-LAC & \cellcolor{blue!50} 0.86 & \cellcolor{blue!50} 0.95 & \cellcolor{blue!50} 0.82 & \cellcolor{blue!50} 0.85 & \cellcolor{blue!50} 0.69 & \cellcolor{blue!50} 0.7 & \cellcolor{blue!50} 0.82 &  0.78 & \cellcolor{blue!50} 0.83 & \cellcolor{blue!50} 0.92 & \cellcolor{blue!50} 0.89 & \cellcolor{blue!30} 0.81 & \cellcolor{blue!50} 0.78 & \cellcolor{blue!50} 0.82 \\
 & & CL-PE-LAC & \cellcolor{blue!15} 0.79 & \cellcolor{blue!30} 0.94 & \cellcolor{blue!15} 0.77 & \cellcolor{blue!15} 0.77 & \cellcolor{blue!15} 0.64 & \cellcolor{blue!50} 0.7 & \cellcolor{blue!15} 0.75 & \cellcolor{blue!50} 0.79 & \cellcolor{blue!30} 0.79 & \cellcolor{blue!30} 0.79 & \cellcolor{blue!30} 0.77 &  0.77 & \cellcolor{blue!15} 0.68 & \cellcolor{blue!30} 0.75 \\  \hhline{~*{16}{-}} %\cline{2-16}

& \multirow{4}{*}{Macro-$F_1$} 
 & LAC & \cellcolor{blue!30} 0.78 & \cellcolor{blue!15} 0.92 & \cellcolor{blue!30} 0.79 & \cellcolor{blue!30} 0.78 & \cellcolor{blue!15} 0.4 & \cellcolor{blue!30} 0.41 & \cellcolor{blue!30} 0.8 & \cellcolor{blue!30} 0.44 & \cellcolor{blue!15} 0.65 & \cellcolor{blue!30} 0.72 & \cellcolor{blue!15} 0.46 & \cellcolor{blue!50} 0.92 & \cellcolor{blue!30} 0.58 & \cellcolor{blue!15} 0.17 \\
 & & CL-LAC &  0.67 &  0.87 &  0.73 &  0.68 &  0.39 & \cellcolor{blue!30} 0.41 &  0.69 & \cellcolor{blue!30} 0.44 &  0.39 &  0.63 &  0.41 &  0.45 &  0.47 & \cellcolor{blue!15} 0.17 \\
 & & PE-LAC & \cellcolor{blue!50} 0.86 & \cellcolor{blue!50} 0.94 & \cellcolor{blue!50} 0.81 & \cellcolor{blue!50} 0.85 & \cellcolor{blue!30} 0.65 & \cellcolor{blue!50} 0.53 & \cellcolor{blue!50} 0.81 & \cellcolor{blue!50} 0.49 & \cellcolor{blue!50} 0.81 & \cellcolor{blue!50} 0.91 & \cellcolor{blue!50} 0.87 & \cellcolor{blue!30} 0.77 & \cellcolor{blue!50} 0.74 & \cellcolor{blue!50} 0.34 \\
 & & CL-PE-LAC & \cellcolor{blue!15} 0.77 & \cellcolor{blue!30} 0.93 & \cellcolor{blue!15} 0.75 & \cellcolor{blue!15} 0.75 & \cellcolor{blue!50} 0.89 & \cellcolor{blue!30} 0.41 & \cellcolor{blue!15} 0.72 & \cellcolor{blue!30} 0.44 & \cellcolor{blue!30} 0.77 & \cellcolor{blue!30} 0.72 & \cellcolor{blue!30} 0.69 & \cellcolor{blue!15} 0.69 & \cellcolor{blue!15} 0.54 & \cellcolor{blue!30} 0.22 \\ \hline \hline

\multirow{8}{*}{\begin{sideways}\textbf{ Added }\end{sideways}} & \multirow{4}{*}{Micro-$F_1$}   & LAC & \cellcolor{blue!30} 0.86 & \cellcolor{blue!50} 0.97 & \cellcolor{blue!30} 0.84 & \cellcolor{blue!50} 0.86 & \cellcolor{blue!15} 0.73 & \cellcolor{blue!30} 0.69 & \cellcolor{blue!50} 0.86 &  0.77 & \cellcolor{blue!15} 0.74 & \cellcolor{blue!30} 0.91 & \cellcolor{blue!30} 0.94 & \cellcolor{blue!30} 0.94 & \cellcolor{blue!30} 0.77 & \cellcolor{blue!50} 0.96 \\
 & & CL-LAC & \cellcolor{blue!15} 0.85 & \cellcolor{blue!15} 0.96 & \cellcolor{blue!50} 0.85 & \cellcolor{blue!15} 0.84 &  0.72 & \cellcolor{blue!15} 0.66 & \cellcolor{blue!15} 0.84 & \cellcolor{blue!15} 0.78 &  0.73 &  0.9 &  0.92 & \cellcolor{blue!30} 0.94 &  0.72 &  0.94 \\
 & & PE-LAC & \cellcolor{blue!50} 0.87 & \cellcolor{blue!15} 0.96 & \cellcolor{blue!30} 0.84 & \cellcolor{blue!50} 0.86 & \cellcolor{blue!50} 0.77 & \cellcolor{blue!50} 0.72 & \cellcolor{blue!50} 0.86 & \cellcolor{blue!30} 0.81 & \cellcolor{blue!50} 0.8 & \cellcolor{blue!30} 0.91 & \cellcolor{blue!50} 0.96 & \cellcolor{blue!50} 0.95 & \cellcolor{blue!50} 0.79 & \cellcolor{blue!50} 0.96 \\
 & & CL-PE-LAC &  0.84 & \cellcolor{blue!50} 0.97 &  0.81 & \cellcolor{blue!15} 0.84 & \cellcolor{blue!30} 0.74 & \cellcolor{blue!15} 0.66 &  0.83 & \cellcolor{blue!50} 0.82 & \cellcolor{blue!30} 0.79 & \cellcolor{blue!50} 0.92 & \cellcolor{blue!30} 0.94 & \cellcolor{blue!30} 0.94 & \cellcolor{blue!15} 0.74 & \cellcolor{blue!50} 0.96 \\  \hhline{~*{16}{-}} %\cline{2-16}

& \multirow{4}{*}{Macro-$F_1$} 
 & LAC & \cellcolor{blue!30} 0.86 & \cellcolor{blue!50} 0.97 & \cellcolor{blue!30} 0.83 & \cellcolor{blue!50} 0.86 & \cellcolor{blue!15} 0.72 & \cellcolor{blue!50} 0.67 & \cellcolor{blue!50} 0.85 & \cellcolor{blue!30} 0.72 & \cellcolor{blue!15} 0.73 & \cellcolor{blue!30} 0.9 & \cellcolor{blue!30} 0.94 & \cellcolor{blue!15} 0.91 & \cellcolor{blue!30} 0.76 & \cellcolor{blue!15} 0.76 \\
 & & CL-LAC & \cellcolor{blue!15} 0.85 & \cellcolor{blue!15} 0.96 & \cellcolor{blue!50} 0.84 & \cellcolor{blue!15} 0.83 & \cellcolor{blue!15} 0.72 & \cellcolor{blue!15} 0.64 & \cellcolor{blue!15} 0.84 & \cellcolor{blue!30} 0.72 & \cellcolor{blue!15} 0.73 &  0.89 &  0.91 &  0.9 &  0.71 &  0.72 \\
 & & PE-LAC & \cellcolor{blue!50} 0.87 & \cellcolor{blue!15} 0.96 & \cellcolor{blue!30} 0.83 & \cellcolor{blue!50} 0.86 & \cellcolor{blue!50} 0.75 & \cellcolor{blue!30} 0.65 & \cellcolor{blue!50} 0.85 &  0.71 & \cellcolor{blue!50} 0.78 & \cellcolor{blue!30} 0.9 & \cellcolor{blue!50} 0.95 & \cellcolor{blue!30} 0.92 & \cellcolor{blue!50} 0.77 & \cellcolor{blue!50} 0.78 \\
 & & CL-PE-LAC &  0.83 & \cellcolor{blue!50} 0.97 &  0.81 & \cellcolor{blue!15} 0.83 & \cellcolor{blue!30} 0.73 & \cellcolor{blue!15} 0.64 &  0.82 & \cellcolor{blue!50} 0.73 & \cellcolor{blue!50} 0.78 & \cellcolor{blue!50} 0.91 & \cellcolor{blue!15} 0.93 & \cellcolor{blue!50} 0.93 & \cellcolor{blue!15} 0.73 & \cellcolor{blue!30} 0.77 \\ \hline \hline

\multirow{8}{*}{\begin{sideways}\textbf{ Yules'Q }\end{sideways}} & \multirow{4}{*}{Micro-$F_1$}   & LAC & \cellcolor{blue!50} 0.86 & \cellcolor{blue!50} 0.97 & \cellcolor{blue!30} 0.83 & \cellcolor{blue!50} 0.86 & \cellcolor{blue!50} 0.75 & \cellcolor{blue!50} 0.73 & \cellcolor{blue!30} 0.85 & \cellcolor{blue!30} 0.81 & \cellcolor{blue!15} 0.74 & \cellcolor{blue!30} 0.91 & \cellcolor{blue!50} 0.96 & \cellcolor{blue!50} 0.95 & \cellcolor{blue!50} 0.78 & \cellcolor{blue!50} 0.97 \\
 & & CL-LAC & \cellcolor{blue!50} 0.86 & \cellcolor{blue!50} 0.97 & \cellcolor{blue!50} 0.84 & \cellcolor{blue!15} 0.85 & \cellcolor{blue!50} 0.75 & \cellcolor{blue!15} 0.67 & \cellcolor{blue!50} 0.86 & \cellcolor{blue!30} 0.81 &  0.71 &  0.88 &  0.94 &  0.93 & \cellcolor{blue!15} 0.75 & \cellcolor{blue!15} 0.95 \\
 & & PE-LAC & \cellcolor{blue!50} 0.86 & \cellcolor{blue!50} 0.97 & \cellcolor{blue!30} 0.83 & \cellcolor{blue!50} 0.86 & \cellcolor{blue!15} 0.74 & \cellcolor{blue!30} 0.72 & \cellcolor{blue!30} 0.85 & \cellcolor{blue!30} 0.81 & \cellcolor{blue!50} 0.82 & \cellcolor{blue!50} 0.94 & \cellcolor{blue!50} 0.96 & \cellcolor{blue!50} 0.95 & \cellcolor{blue!50} 0.78 & \cellcolor{blue!30} 0.96 \\
 & & CL-PE-LAC &  0.83 & \cellcolor{blue!50} 0.97 &  0.81 &  0.81 & \cellcolor{blue!15} 0.74 & \cellcolor{blue!15} 0.67 &  0.84 & \cellcolor{blue!50} 0.83 & \cellcolor{blue!30} 0.79 & \cellcolor{blue!15} 0.9 & \cellcolor{blue!15} 0.95 & \cellcolor{blue!50} 0.95 &  0.74 &  0.93 \\  \hhline{~*{16}{-}} %\cline{2-16}

& \multirow{4}{*}{Macro-$F_1$} 
 & LAC & \cellcolor{blue!50} 0.85 & \cellcolor{blue!50} 0.97 & \cellcolor{blue!15} 0.82 & \cellcolor{blue!50} 0.85 & \cellcolor{blue!30} 0.73 & \cellcolor{blue!50} 0.7 & \cellcolor{blue!30} 0.84 & \cellcolor{blue!50} 0.74 & \cellcolor{blue!15} 0.73 & \cellcolor{blue!30} 0.89 & \cellcolor{blue!50} 0.95 & \cellcolor{blue!30} 0.92 & \cellcolor{blue!50} 0.76 & \cellcolor{blue!50} 0.8 \\
 & & CL-LAC & \cellcolor{blue!50} 0.85 & \cellcolor{blue!50} 0.97 & \cellcolor{blue!50} 0.83 & \cellcolor{blue!50} 0.85 & \cellcolor{blue!50} 0.74 & \cellcolor{blue!30} 0.65 & \cellcolor{blue!50} 0.85 & \cellcolor{blue!50} 0.74 &  0.7 &  0.86 & \cellcolor{blue!15} 0.94 &  0.89 & \cellcolor{blue!15} 0.72 & \cellcolor{blue!15} 0.74 \\
 & & PE-LAC & \cellcolor{blue!50} 0.85 &  0.96 & \cellcolor{blue!50} 0.83 & \cellcolor{blue!50} 0.85 & \cellcolor{blue!30} 0.73 &  0.57 & \cellcolor{blue!30} 0.84 &  0.68 & \cellcolor{blue!50} 0.79 & \cellcolor{blue!50} 0.93 & \cellcolor{blue!50} 0.95 & \cellcolor{blue!30} 0.92 & \cellcolor{blue!50} 0.76 & \cellcolor{blue!30} 0.77 \\
 & & CL-PE-LAC &  0.83 & \cellcolor{blue!50} 0.97 &  0.8 &  0.8 & \cellcolor{blue!30} 0.73 & \cellcolor{blue!30} 0.65 &  0.83 & \cellcolor{blue!15} 0.73 & \cellcolor{blue!30} 0.77 & \cellcolor{blue!15} 0.88 & \cellcolor{blue!15} 0.94 & \cellcolor{blue!50} 0.94 &  0.71 &  0.65 \\ \hline 


\end{tabular}
\end{center}
\end{table}

A deeper analysis is necessary in order to verify if the objective of decreasing the number of rules without hurting classification effectiveness was indeed reached. Thus, we apply three paired significance tests, as shown 
in Table~\ref{table.testt.paired}. We evaluate three statistics: Micro-$F_1$, Macro-$F_1$, and the average number of rules within $\mathcal{R}_x$ and $\mathcal{R}^{*}_x$, using the numbers shown in Table~\ref{table.results.partial1} and comparing LAC against PE-LAC under the same $\theta$ statistics 
(i.e., confidence, support, added-value, or Yules'Q).
Clearly, null-hypotheses (1) and (2) are accepted (i.e., both LAC and PE-LAC learn classifiers that achieve the same Micro- and Macro-$F_1$ values), while null-hypothesis (3) is rejected (accepting the alternate hypothesis that LAC and PE-LAC learn classifiers with different number of rules).
Although the T-Test is a parametric test, and therefore not the ideal test for small sample sizes,
it presented $p$-values that are similar to those presented by Wilcoxon and Permutation.

\begin{table}[tb]
\caption{Paired tests: LAC vs. PE-LAC.}
\label{table.testt.paired}
\centering
\begin{center}
\begin{tabular}{|l||r|r|r||r|r|r||r|r|r|}\hline
\multicolumn{1}{|c||}{\multirow{3}{*}{\textbf{$\theta$ statistics}}} & \multicolumn{9}{|c|}{\textbf{$p$-value}}\\ \cline{2-10}
  & \multicolumn{3}{|c||}{\textbf{Micro-$F_1$ (1)}} & \multicolumn{3}{|c||}{\textbf{Macro-$F_1$ (2)}} & \multicolumn{3}{|c|}{\textbf{\# Rules (3)}} \\ \cline{2-10}
 & \textbf{T-Test} & \textbf{Wilc.} & \textbf{Perm.}  & \textbf{T-Test} & \textbf{Wilc.} & \textbf{Perm.}  & \textbf{T-Test} & \textbf{Wilc.} & \textbf{Perm.} \\ \hline \hline
Confidence  & 0.42 & 0.38 & 0.42 & 0.29 & 0.31 & 0.28 & \color{blue}{\textbf{0.00}} & \color{blue}{\textbf{0.00}} & \color{blue}{\textbf{0.00}}\\ \hline
Support  & 0.13 & 0.07 & 0.12 & 0.16 & 0.12 & 0.15 & \color{blue}{\textbf{0.00}} & \color{blue}{\textbf{0.00}} & \color{blue}{\textbf{0.00}}\\ \hline
Added-Value & 0.63 &   0.63 & 0.63 &  0.84 & 0.77 & 0.84 & \color{blue}{\textbf{0.00}} & \color{blue}{\textbf{0.00}} & \color{blue}{\textbf{0.00}} \\ \hline     
Yules'Q     &  0.86 & 0.96 & 0.86 &  0.82 & 1.00 & 0.81 & \color{blue}{\textbf{0.00}} & \color{blue}{\textbf{0.00}} & \color{blue}{\textbf{0.00}}\\ \hline
\end{tabular}
\end{center}
\caption{Group-based tests: LAC, PE-LAC and both.}
\label{table.testt.set}
\centering
\begin{center}
\begin{tabular}{|l||r|r||r|r||r|r|}\hline
\multicolumn{1}{|c||}{\multirow{3}{*}{\textbf{Hypothesis}}} & \multicolumn{6}{|c|}{\textbf{$p$-value}}\\ \cline{2-7}
  & \multicolumn{2}{|c||}{\textbf{Micro-$F_1$}} & \multicolumn{2}{|c||}{\textbf{Macro-$F_1$}} & \multicolumn{2}{|c|}{\textbf{\# Rules}} \\ \cline{2-7}
  & \textbf{ANOVA} & \textbf{Kruskal-Wallis}  & \textbf{ANOVA} & \textbf{Kruskal-Wallis}  & \textbf{ANOVA} & \textbf{Kruskal-Wallis} \\ \hline \hline
(1) & 0.09 & 0.08 & \textbf{0.01} & \textbf{0.05} & 1.00  & 1.00 \\ \hline
(2) & 0.46 & 0.46 & 0.30 & 0.65 & 1.00 & 1.00  \\ \hline
(3) & 0.10 &0.10 & \textbf{0.00} & 0.14 &  \color{blue}{\textbf{0.00}}  & \color{blue}{\textbf{0.00}} \\ \hline
\end{tabular}
\end{center}
\end{table}


In addition to the paired tests, we also employ group-based significance tests in order to check the following hypotheses:

\begin{enumerate}
\item The LAC algorithm shows the same performance numbers (Micro-$F_1$, Macro-$F_1$ or number of rules), no matter the $\theta$ statistics used in Equation~\ref{eq1}.
\item The PE-LAC algorithm shows the same performance numbers (Micro-$F_1$, Macro-$F_1$ or number of rules), no matter the $\theta$ statistics used in Equation~\ref{eq1}.
\item Both LAC and PE-LAC algorithms show the same performance numbers (either in terms of Micro-$F_1$, Macro-$F_1$ or number of rules), no matter the $\theta$ statistics used in Equation~\ref{eq1}.
\end{enumerate}

The results obtained by the group-based significance tests are shown in Table~\ref{table.testt.set}. Considering the Micro-$F_1$,
all hypotheses were accepted. Considering the Macro-$F_1$ the hypotheses 1 and 3 were rejected, this happened because the support is not a good statistic when the data are unbalanced.
This indicates that the $\theta$ statistics used to weight the rules does not impact significantly the results, except by support.
The hypothesis that the classifiers built by LAC and PE-LAC have the same size were rejected as expected. 
Finally, in order to verify how the classifiers that are built by PE-LAC are much more interpretable than
\begin{wrapfigure}[18]{r}{12cm}
  \centering
  \subfloat[LAC, \#Rules=72]{\label{fig.rules.lac} \includegraphics[width=0.35\textwidth]{img/lac.png}}
   \qquad
  \subfloat[PE-LAC, \#Rules=4]{\label{fig.rules.lacp} \includegraphics[width=0.35\textwidth]{img/lacp.png}}
    \caption{Rules generated for one passenger in Titanic (a survivor).}
  \label{fig.rules}
\end{wrapfigure}
the corresponding 
classifiers built by LAC, we show in Fig.~\ref{fig.rules} the classifier $\mathcal{R}_x$ (built by LAC) and its 
corresponding counterpart $\mathcal{R}^{*}_x$ (built by PE-LAC) for one passenger in Titanic (a survivor). The rules
 shown in black indicate the death of the passenger, while the rules shown in blue indicate survival. While it is difficult to grasp some explanation from the classifier built by LAC,
we may easily capture the reasons that lead the passenger to survive in the classifier built by PE-LAC $-$ he has one sibling, and was hosted in a cabin.
