\section{Experiments}\label{experimentos}

In this section we describe the experiments we conducted using the two datasets. For each experiment we present the experimental design, describe the gold standard and the evaluation metrics used and discuss the results.

Our main goal with the experiments was to verify whether our time-aware enhanced strategy was more suitable to produce rankings for specific time points than a baseline that ignores the time factor. In other words, we would like to evaluate how effective our strategy with the temporal decaying function is for this specific task.

\subsection{MMA Experiments}\label{mma_exp}


%\indent
%Finally, in this Section we describe the experiments designed to verify the effectiveness of our proposed temporally-aware ranking method.

\subsubsection{Experimental Design}

For the MMA experiments, we built several fighting networks divided per category and varied the weighting method for the edges. Next we applied an edge-weighted version of the PageRank algorithm~\cite{white_algorithms_2003} to all networks to rank the athletes. We then compared the generated rankings with a gold standard to verify which weighting strategy produced the best results. More specifically we tested two weighting strategies:

\begin{enumerate}
\item Cumulative. No time-aware function is applied to the weight of the edges, which is our baseline and corresponds to the method proposed by~\citeN{radicchi_who_2011}: a PageRank variation that does not consider the time factor. Notice that this is the same as considering $\alpha=0$ in our edge weighting function described in Section~\ref{temporal}.
\item Time-aware. The time-aware function is applied to the weight of the edges (for this, we varied $\alpha$ from 0.05 to 0.35 with a 0.05 step).
\end{enumerate}

Overall, we built and compared the rankings produced by 56 different fighting networks: $7$ categories $\times$ $8$ variations of the edge weighting function (one cumulative and seven functions with temporal decay). 

\subsubsection{Gold Standard}\label{goldstd}
Since our ranking method and the baseline's \cite{radicchi_who_2011}, which completely ignores the temporal factors, produce ranked lists of the best athletes, we need an external list of the best fighters, ideally produced by a third-party, to serve as a gold standard with which we could compare our results. This existing rank should also be credible enough so that the results could be trusted.

Accordingly, we chose to use the USA Today / SB Nation MMA 2012 Consensus Ranking\footnote{\scriptsize{http://www.bloodyelbow.com/rankings}}, a ranking of the 25 best fighters of 2011 for each category built from the aggregation of other 20 most important rankings of the MMA community, which may use different criteria. This was supposed to be a consensual ranking, therefore being a good candidate for a gold standard. We have used the rankings published on October 27, 2011, and thus have only considered fights that happened up to that date in order to generate the rankings.

\subsubsection{Evaluation Metrics}\label{spearman}
To compare the rankings generated by our method and the baseline's with the respective gold standard, we used two metrics: Recall and the Spearman's Rank Correlation Coefficient\footnote{\scriptsize{http://mathworld.wolfram.com/SpearmanRankCorrelationCoefficient.html}}. Recall captures  how many of the top-25 best fighters in each category we are able to retrieve with each ranking method.

Spearman's Rank Correlation Coefficient captures how well two rankings correlate in terms of the positions each fighter appears in the respective rankings. It returns values between 1 and -1 for perfectly coincident and inverse rankings. Positive values indicate some correlation, negative values indicate negative correlation and a value of 0 means there is no correlation at all between the ranks.

One disadvantage of using the Spearman's Rank Correlation Coefficient to compare two ranks, in our case specifically, is that the gold standard considers only the top 25 positions while the ranking methods return a ranked list with all the fighters in the dataset, totalizing at least 50 fighters per category. As the set of all possible rank values is different, this may cause some anomalies in the computation of this metric. To minimize this problem, we implemented a small variation of this coefficient, in which fighters whose real position in the gold standard ranking are below the 25th, but that are not retrieved by the analyzed ranking methods in their top 25 results, are ordered according to the position they appear in the respective rankings and are then given rank positions 26, 27 and so on.  This small change smooths the distortions while at the same time  penalizes the method that retrieves more of those ``below 25''  fighters.

% although this is true for both: our solution and the baseline. Given that the conditions are equal, the methods are still comparable with this metric, so we use it to report the experimental results.

\subsubsection{Results}


Table \ref{results_networks} shows the results obtained for the 56 generated fighting networks. Each line corresponds to the results for one specific network.
Columns \textit{Weighting Class}, \textit{Fighters Compared}, $\alpha$, \textit{Recall} and \textit{Spearman's Coef} represent, respectively,
the fighting category, the number of fighters present in the gold standard in each category for which we have information in our dataset,
the respective $\alpha$ parameter for the time decaying strategies,
and Recall and Spearman's Coefficient results obtained by comparing the generated rank and the gold standard. 

\input{tabela_resultados.tex}



By analyzing the results, we can make some general observations:
\begin{itemize}
\item In all categories, our  time-aware alternatives outperformed the baseline ($\alpha=0$) according to the Recall and Spearman's Coefficient metrics. Recall figures for the respective best $\alpha$ value in each category varied between 0.74 and 0.96 which can be considered an excellent result. In fact,  in most cases our Recall results are equal or better than the baselines', with gains of up to 30\%. Notice also the Spearman's  Coefficient for the best $\alpha$ values are positively correlated with the gold standard ranking in five out of seven categories and that in some cases there are very significant correlations (e.g., for categories ``Heavyweight and ``Light Heavyweight'')\footnote{\scriptsize{Notice that in a few cases, the  Spearman Coefficient's results are lower than the inferior limit of -1. This occurs because the universe of possible ranking values are different between the generated ranks and the gold standard, although this problem has been alleviated by our small adaptation of the metric as explained in Section \ref{spearman}.}}.
\item In all cases but the Featherweight category, in which the overall results were not good, the  best results were obtained with $\alpha$ varying between $0.20$ and $0.30$ for the strategies that use the edge weighting function\footnote{\scriptsize{In fact, in the category Bantamweight, best results were obtained with $\alpha=0.35$ but these are basically the same as those obtained with $\alpha=0.30$.}}.
This indicates that in order to produce good results one could look only for values within a small range, reducing the problem of searching for the best parameters for the function.
\end{itemize}


If we take an even closer look at the rankings, we may see some interesting results. Tables~\ref{fig:res}(a) and \ref{fig:res}(b) show the ``top 10'' rankings produced by our proposed method and the baseline's for the  ``Light Heavyweight'' and  ``Heavyweight'' categories, respectively. In these figures, the first column corresponds to the fighter, the second to his rank position in the gold standard, the third to the rank position produced by our method, and the fourth to the square of the difference---as used by the Speaman Rank correlation metric---between the predicted rank position and the ``correct'' one. The fifth and sixth colums show similar information but for the baseline ranking. Differences in rank positions higher than 10 ($D^2 > 100$) are marked in bold.


\begin{table*}[tb]
\caption{Detailed results.}\label{fig:res}
\captionspace
  \centering
  \subtable[Ranking results for the Light Heavyweight category.]{
  \centering
  \begin{scriptsize} % small or scriptsize (extremo)
  \begin{tabular}{cccccc}
    \toprule
      \multirow{2}{*}{\textbf{Fighter}} & \textbf{Gold} & \multirow{2}{*}{\textbf{Ours}} & \multirow{2}{*}{$\mathbf{D^2}$} & \multirow{2}{*}{\textbf{Baseline}} & \multirow{2}{*}{$\mathbf{D^2}$} \tabularnewline
                                        & \textbf{Standard} &           & & & \tabularnewline
     \midrule
     Jon Jones & 1 & 1 & 0 & 6 & 25 \tabularnewline
     Mauricio Rua & 2 & 8 & 36 & 10 & 64 \tabularnewline
     Rashad Evans & 3 & 2 & 1 & 1 & 4 \tabularnewline
     Quinton Jackson & 4 & 4 & 0 & 5 & 1 \tabularnewline
     Lyoto Machida & 5 & 3 & 4 & 2 & 9 \tabularnewline
     Dan Henderson & 6 & 22 & \textbf{256} & 20 & \textbf{196} \tabularnewline
     Phil Davis & 7 & 7 & 0 & 19 & \textbf{144} \tabularnewline
     Forrest Griffin & 8 & 6 & 4 & 4 & 16 \tabularnewline
     Gegard Mousasi & 9 & 14 & 25 & 21 & \textbf{144} \tabularnewline
     Rafael Cavalcante & 10 & 9 & 1 & 22 & \textbf{144} \tabularnewline

    \bottomrule
    
  \end{tabular}
  \end{scriptsize}
  
%  \label{results_light_heavyweight}
}

\subtable[Ranking results for the Heavyweight category.]{
  \centering
  \begin{scriptsize} % small or scriptsize (extremo)
  \begin{tabular}{cccccc}
    \toprule
            \multirow{2}{*}{\textbf{Fighter}} & \textbf{Gold} & \multirow{2}{*}{\textbf{Ours}} & \multirow{2}{*}{$\mathbf{D^2}$} & \multirow{2}{*}{\textbf{Baseline}} & \multirow{2}{*}{$\mathbf{D^2}$} \tabularnewline
                                        & \textbf{Standard} &                 & & & \tabularnewline
     \midrule
     Cain Velasquez & 1 & 5 & 16 & 12 & \textbf{121} \tabularnewline
     Junior dos Santos & 2 & 3 & 1 & 13 & \textbf{121} \tabularnewline
     Alistair Overeem & 3 & 1 & 4 & 7 & 16 \tabularnewline
     Brock Lesnar & 4 & 18 & \textbf{196} & 16 & \textbf{144} \tabularnewline
     Fabricio Werdum & 5 & 10 & 25 & 5 & 0 \tabularnewline
     Frank Mir & 6 & 4 & 4 & 4 & 4 \tabularnewline
     Shane Carwin & 7 & 22 & \textbf{225} & 25 & \textbf{324} \tabularnewline
     Josh Barnett & 8 & 2 & 36 & 8 & 0 \tabularnewline
     Daniel Cormier & 9 & 8 & 1 & 26 & \textbf{289} \tabularnewline
     Antonio R. Nogueira & 10 & 9 & 1 & 2 & 64 \tabularnewline
    \bottomrule
  \end{tabular}
  \end{scriptsize}
%  \label{results_light_heavyweight}
}
{\vspace{-0.2cm}}
\end{table*}




%\begin{table*}[tb]
%  \subtable[Ranking results for the Light Heavyweight category.]{
%  \centering
%  \begin{scriptsize} % small or scriptsize (extremo)
%  \begin{tabular}{cccccc}
%    \toprule
%      \multirow{2}{*}{\textbf{Fighter}} & \textbf{Gold} & \multirow{2}{*}{\textbf{Ours}} & \multirow{2}{*}{$\mathbf{D^2}$} & \multirow{2}{*}{\textbf{Baseline}} & \multirow{2}{*}{$\mathbf{D^2}$} \tabularnewline
%                                        & \textbf{Standard} &           & & & \tabularnewline
%     \midrule
%     Jon Jones & 1 & 1 & 0 & 6 & 25 \tabularnewline
%     Mauricio Rua & 2 & 8 & 36 & 10 & 64 \tabularnewline
%     Rashad Evans & 3 & 2 & 1 & 1 & 4 \tabularnewline
%     Quinton Jackson & 4 & 4 & 0 & 5 & 1 \tabularnewline
%     Lyoto Machida & 5 & 3 & 4 & 2 & 9 \tabularnewline
%     Dan Henderson & 6 & 22 & \textbf{256} & 20 & \textbf{196} \tabularnewline
%     Phil Davis & 7 & 7 & 0 & 19 & \textbf{144} \tabularnewline
%     Forrest Griffin & 8 & 6 & 4 & 4 & 16 \tabularnewline
%     Gegard Mousasi & 9 & 14 & 25 & 21 & \textbf{144} \tabularnewline
%     Rafael Cavalcante & 10 & 9 & 1 & 22 & \textbf{144} \tabularnewline
%
%    \bottomrule
%    
%  \end{tabular}
%  \end{scriptsize}
%  
%%  \label{results_light_heavyweight}
%}
%\subtable[Ranking results for the Heavyweight category.]{
%  \centering
%  \begin{scriptsize} % small or scriptsize (extremo)
%  \begin{tabular}{cccccc}
%    \toprule
%            \multirow{2}{*}{\textbf{Fighter}} & \textbf{Gold} & \multirow{2}{*}{\textbf{Ours}} & \multirow{2}{*}{$\mathbf{D^2}$} & \multirow{2}{*}{\textbf{Baseline}} & \multirow{2}{*}{$\mathbf{D^2}$} \tabularnewline
%                                        & \textbf{Standard} &                 & & & \tabularnewline
%     \midrule
%     Cain Velasquez & 1 & 5 & 16 & 12 & \textbf{121} \tabularnewline
%     Junior dos Santos & 2 & 3 & 1 & 13 & \textbf{121} \tabularnewline
%     Alistair Overeem & 3 & 1 & 4 & 7 & 16 \tabularnewline
%     Brock Lesnar & 4 & 18 & \textbf{196} & 16 & \textbf{144} \tabularnewline
%     Fabricio Werdum & 5 & 10 & 25 & 5 & 0 \tabularnewline
%     Frank Mir & 6 & 4 & 4 & 4 & 4 \tabularnewline
%     Shane Carwin & 7 & 22 & \textbf{225} & 25 & \textbf{324} \tabularnewline
%     Josh Barnett & 8 & 2 & 36 & 8 & 0 \tabularnewline
%     Daniel Cormier & 9 & 8 & 1 & 26 & \textbf{289} \tabularnewline
%     Antonio R. Nogueira & 10 & 9 & 1 & 2 & 64 \tabularnewline
%    \bottomrule
%  \end{tabular}
%  \end{scriptsize}
%%  \label{results_light_heavyweight}
%}
%{\vspace{-0.2cm}}
%\caption{Detailed results.}\label{fig:res}
%\end{table*}


We can see in the those tables that our ranking and the gold standard have significant similarities. Moreover, when there is a difference in the rank position of a given fighter when compared to his position in the gold standard, this difference is usually small (see values in column $D^2$), with cases of exact prediction. Furthermore, these differences are in most cases much lower than those  produced by the baseline's ranking. Results in terms of recall are also very good. For the Heavyweight category, from 23 fighters present in the gold standard rank, we were are able to retrieve 22 with our proposed method, an excellent result (96\% of recall). For the Light Heavyweight category there are 19 out of 23 fighters in our ranking, also a significant result (83\% of recall). In sum, we can consider that the ranks produced by our time-aware ranking method are, in fact, very satisfactory in several cases.


\subsection{Tennis Experiments}\label{tennis_exp}
\subsubsection{Experimental Design}

In the second round of experiments we applied our method to the tennis dataset. We adopted the same experimental design, building match networks for different time spans instead of categories, and varying the weighting method for the edges. We next applied the PageRank algorithm to rank athletes and compared the generated rankings with a gold standard.

We generated networks considering eight points of interest for which we generated and compared rankings ranging from 1975 to 2010 with a five year step. Each point of interest corresponds to a time span having 1968 as its first year and the point of interest as the final year. Therefore, we generated networks with matches that occurred during the periods 1968-1975, 1968-1980 and so on. For each time span we tested two weighting strategies, as we did in the previous experiment:

\begin{enumerate}
\item Cumulative. No time-aware function is applied to the weight of the edges. Notice that this is the same as considering $\alpha=0$ in our edge weighting function described in Section~\ref{temporal}.
\item Time-aware. The time-aware function is applied to the weight of the edges (for this, we varied $\alpha$ from 0.25 to 1.75 with a 0.25 step). Notice that $\alpha$ is tested with higher values because of the larger time span of the networks.
\end{enumerate}

Therefore, we generated a total of 64 networks: $8$ time spans $\times$ $8$ variations of the edge weighting function (one cumulative and seven functions with temporal decay). 

In this second round of experiments we also wanted to verify whether generating networks considering just the matches that occurred near to a given point of interest could lead us to better rankings than considering all matches up to that point. For example, given the year 2000 as a point of interest, we would like to know whether building a network with matches that occurred during the years 1999 to 2000 would give us better rankings than considering matches that occurred during the whole period of 1968 to 2000. Therefore, we also generated networks that considered shorter periods (called time spans), including only matches that occurred near to a point of interest. More specifically, for each point of interest, we considered six time spans, which consisted of fixing the point of interest as the final year and varying the initial year in an one year step. For example, given 2000 as the year of interest, we built six networks, considering the matches that occurred during the time spans 1995-2000, 1996-2000, ..., 2000-2000. For each time span we varied the edge weighting strategy similarly as described above, but varying $\alpha$ from 0.05 to 0.35 with a 0.05 step.

In this way, for this experiment we generated 384 networks: $48$ time spans (8 points of interests $\times$ 6 time spans) $\times$ $8$ variations of the edge weighting function (one cumulative and seven functions with temporal decay). The eight points of interested in our experiment are the years of 1975, 1980, 1985, 1990, 1995, 2000, 2005 and 2010, which, as we will see below, correspond to the years that we want to contrast the rankings generated by our method for specific time spans with the corresponding ATP official ranking.

Thus, all together, we generated a total of 448 networks for the tennis dataset.

\subsubsection{Gold Standard}\label{goldstd}
The natural gold standard to compare the rankings of tennis players is the official ATP ranking. Thus, for each network we consider as the gold standard the ATP ranking of the respective year (point) of interest.

\subsubsection{Evaluation Metrics}\label{spearman}

Likewise the MMA experiments, we used the same metrics, Recall and the Spearman's Rank Correlation Coefficient, to compare the rankings generated by our method and the baseline's with the respective gold standard.

\subsubsection{Results}

Table~\ref{results_tennis} shows the best results obtained for the 448 tennis match networks, sorted by descending order of recall and Spearman's Coefficient values, considering the following strategies:

\begin{enumerate}
\item Full period network with cumulative edge weighting function. The network is built considering matches from the whole period, i.e., from the year 1968 to the point of interest. No time-aware function is applied to the weight of the edges. Notice that this is the same as considering $\alpha=0$ in our edge weighting function described in Section~\ref{temporal}.
\item Full period network with time-aware edge weighting function. The network is built considering matches from the whole period, i.e., from the year 1968 to the point of interest. The time-aware function is applied to the weight of the edges by varying $\alpha$ from 0.25 to 1.75 with a 0.25 step.
\item Time span network with cumulative edge weighting function. The network is built considering matches that occurred at most five years before the point of interest. No time-aware function is applied to the weight of the edges. Notice that this is the same as considering $\alpha=0$ in our edge weighting function described in Section~\ref{temporal}.
\item Time span network with time-aware edge weighting function. The network is built considering matches that occurred at most five years before the point of interest. The time-aware function is applied to the weight of the edges by varying $\alpha$ from 0.05 to 0.35 with a 0.05 step.
\end{enumerate}

By analyzing the results, we can make the following comments:

\begin{itemize}
\item Considering the cumulative edge weighting function in the full period network leads to poor results. This is expected, since ATP considers only the respective year's results to produce its final ranking, whereas that strategy considers all matches equally important, no matter when they occurred.
\item In all points of interest analyzed, the strategies that use the time-aware edge weighting function ($\alpha$ $>$ 0) outperform the ones with cumulative edge weighting function, although we may consider a tie in some cases.
\item In general, considering small time spans (two to three years) to produce a ranking leads to good results, i.e., the produced rankings are very close to the official ATP rankings.
\end{itemize}

\input{tabela_resultados_tenis.tex}
