\section{the OBAS Benchmark}
\label{sec:theOBASBenchmark}


In this section, we introduce OBAS, an OLAP Benchmark for Analysis Services, which enables the evaluation of analysis services and allows the creation of varying-sized data cubes. The main goals of OBAS are to become: (1) relevant by helping in the experimental evaluation of analysis services using a workload based on the MDX language, OLAP functions with an increasing number of computed items (i.e. members and sets), reliability and throughput metrics, and data cubes scalability; (2) portable by using a standard multidimensional query language because MDX is adopted in most of the currently available analysis services, and by using the XMLA (XML for Analysis) standard in the development of communication interfaces; and (3)  scalable because it enables the generation of  variable-sized datasets and the processing of a variable number of concurrent executions. 

The proposed OBAS benchmark is described in the following sections according to its characteristics: Section \ref{sec:DimensionalDataSchema} describes the schema used to generate dimensional data cubes, Section \ref{sec:Workload} details the queries used as workload and OLAP processing costs, Section \ref{sec:runningSettings} addresses running settings, Section \ref{sec:performanceMetrics} tackles a set of performance metrics to be used in the experiments and Section \ref{sec:theOBASSystemPrototype} describes the architecture of the OBAS system prototype.

\subsection{Dimensional Data Schema}
\label{sec:DimensionalDataSchema}

\begin{figure*}[b]
 \begin{center}
    \includegraphics[height=4.5cm,width=14.7cm]{Fig1}
    \caption{The OBAS Dimensional Data Schema}\label{fig1}
  \end{center}
\end{figure*}

Figure \ref{fig1} shows the dimensional data schema of OBAS and the number of tuples of each table calculated according to the scale factor SF. As a result, OBAS works with the metaphor of data cubes with increasing volumes of data. For the sake of simplicity, the data schema contains only the tables and attributes that are found in the SSB schema. The removal of attributes from the SSB schema improves the data schema legibility and was made because users do not often generate data cubes with information that are never queried. By definition, a data cube displays information of interest to users. Thus, there is no need for generating data cubes with information that are not retrieved by the benchmark OLAP queries. Data is distributed uniformly among tables. The OBAS dimensional data schema contains a fact table, five numeric measures and four dimension tables, each of them holding three attribute levels. For this schema, a data catalog definition is shown in Figure \ref{fig2}, representing the dimensions with their corresponding hierarchies that compose the data cube over which the OBAS OLAP queries are processed. The star schema was chosen because it is the most common logical data schema of DW applications \cite{kimball02} and provides better query processing performance than the corresponding normalized snowflake schemas because it avoids star join computations, which are costly. The cardinality of attributes in the dimension tables is fixed, e.g. there are 250 cities, 25 nations and 5 regions where suppliers are located.

\begin{figure*}[t]
 \begin{center}
    \includegraphics[height=3.5cm,width=15cm]{Fig2}
    \caption{The OBAS Data Catalog}\label{fig2}
  \end{center}
\end{figure*}

\subsection{Workload}
\label{sec:Workload}

By using MDX, users can perform queries over a multidimensional data cubes, making available configurable data viewed in different angles and aggregation levels by using multidimensional operators. Despite being similar to the traditional SQL, MDX is not an SQL extension. MDX has been optimized for querying multidimensional data. MDX allows the declaration and use of calculated members and sets that correspond to a reference alias that can be used in any part of the query. A MDX expression contains OLAP functions that are applied to calculated members and sets. Generally, the calculated sets are used to compose elements of an axis and may be obtained from the application of a MDX OLAP function to another calculated set. Currently, MDX is supported by most of the OLAP vendors. For these reasons, the MDX query language was chosen to define our workload.

\begin{center} 	% TABLES SHOULD BE CENTERED
\begin{table}[b]% TABLES SHOULD BE AT THE TOP [t] OR BOTTOM [b} OF PAGES
	\caption{The OBAS MDX Queries of Group I} % ALWAYS INCLUDE CAPTIONS IN YOUR TABLES
	\label{table:tab1} % USE THE CONTENT WITHIN label FOR REFERENCING IT (USE \ref WITHIN THE TEXT)
	\centering % used for centering table
%\ttfamily % CHANGES THE FONT WITHIN THE TABLE TO monotype
\begin{tabular}{l}
	\hline  % DRAWS HORIZONTAL LINE {\bf FROM} 2nd to 13th COLUMNS
          \multicolumn{1}{|l|}{  {\bf Q01 to Q07:} }\\
	\multicolumn{1}{|l|}{  {\bf SELECT} NON EMPTY \{ \{ \{[Measures].[Lo Revenue]\} \} \}  {\bf ON} COLUMNS, }\\
	\multicolumn{1}{|l|}{ NON EMPTY \{  \{[Dimension1].[Level1].Members\}  * \{...\} \} {\bf ON} {\bf ROWS} {\bf FROM} [LINEORDER] }\\
	\multicolumn{1}{|l|}{ {\bf WHERE}  [Dimension2].[Level2].[SomeMember] }\\
	\hline  % DRAWS HORIZONTAL LINE {\bf FROM} 2nd to 13th COLUMNS
          \multicolumn{1}{|l|}{  {\bf Q08 to Q10:} }\\
	\multicolumn{1}{|l|}{  {\bf WITH} MEMBER [Measures].[Profit] {\bf AS} 'SomeCalculation' }\\
	\multicolumn{1}{|l|}{ {\bf SELECT} NON EMPTY \{ \{ \{[Measures].[Profit]\} \} \} {\bf ON} COLUMNS,} \\
	\multicolumn{1}{|l|}{ NON EMPTY \{ \{[Dimension].[Level].Members\}  *  \{...\} \} {\bf ON} {\bf ROWS} {\bf FROM} [LINEORDER] }\\
	\multicolumn{1}{|l|}{ {\bf WHERE} [CUSTOMER].[C Region].[AMERICA] }\\
	\hline
\end{tabular}
%\rmfamily % RESTORES TO TIMES, SO THAT CAPTION IS WRITTEN PROPERLY
\end{table}
\end{center}

The queries that compose the workload of OBAS are divided into two groups. {\bf Group I} contains queries that were translated into MDX from the SSB workload, and aims at evaluating aspects of functionality, selectivity and interference of the sequential execution of queries. The template of Group I is listed in Table \ref{table:tab1} and focus on the performance evaluation of the chosen dataset, since queries vary the dimensionality (number of dimensions accessed) and the selectivity (ratio between the number of tuples accessed and the total number of tuples in the dataset), as shown in Table \ref{table:tab2}. To evaluate OLAP engines and the processing of OLAP functions, the MDX functions were classified according to the data types to which these functions are applied: hierarchy, level, logical, member, numeric, set and string. Thus, by using this classification, the query types of {\bf Group II} were designed and their templates are listed in Table \ref{table:tab3}. The query Q11 is shown completely in Table \ref{table:tab3}, whereas for the other queries from this group, only their templates are given in Table \ref{table:tab3}. Each template includes the set of OLAP functions and the type of data handled by the corresponding query type. The number of OLAP functions associated with each query type corresponds to the amount of members and sets calculated during query processing. However, implementations of OLAP functions of analysis services being tested may not be similar, and then, the OLAP functions should not be used in the experiments, in order to evaluate the services uniformly.

\begin{table}[t]
\caption{The Selectivity of OBAS MDX Queries of Group I} % title of Table
\sffamily % CHANGES THE FONT WITHIN THE TABLE TO sans serif
\centering % used for centering table
\begin{tabular}{c c c c r} % columns (5 columns)
\hline %\hline %inserts double horizontal lines
\multicolumn{1}{|l|}{\bf{Query Name}} & \bf{Number of Dimensions} & \multicolumn{1}{|c|}{\bf{Member}} & \bf{Set} & \multicolumn{1}{|r|}{\bf{Selectivity}} \\ [0.5ex] % inserts table 
%heading
\hline % inserts single horizontal line
\multicolumn{1}{|c|}{Q01} & 2 & \multicolumn{1}{|c|}{0} & 0 & \multicolumn{1}{|r|}{0.008} \\ \hline
\multicolumn{1}{|c|}{Q02}& 2 & \multicolumn{1}{|c|}{0} & 0 & \multicolumn{1}{|r|}{0.0016} \\ \hline
\multicolumn{1}{|c|}{Q03} & 2 & \multicolumn{1}{|c|}{0} & 0 & \multicolumn{1}{|r|}{0.0002} \\ \hline
\multicolumn{1}{|c|}{Q04} & 3 & \multicolumn{1}{|c|}{0} & 0 & \multicolumn{1}{|r|}{0.034} \\  \hline
\multicolumn{1}{|c|}{Q05} & 3 & \multicolumn{1}{|c|}{0} & 0 & \multicolumn{1}{|r|}{0.0014} \\ \hline
\multicolumn{1}{|c|}{Q06} & 3 & \multicolumn{1}{|c|}{0} & 0 & \multicolumn{1}{|r|}{0.000055} \\ \hline
\multicolumn{1}{|c|}{Q07} & 3 & \multicolumn{1}{|c|}{0} & 0 & \multicolumn{1}{|r|}{0.000000762} \\ \hline
\multicolumn{1}{|c|}{Q08} & 3 & \multicolumn{1}{|c|}{1} & 0 & \multicolumn{1}{|r|}{0.016} \\ \hline
\multicolumn{1}{|c|}{Q09} & 4 & \multicolumn{1}{|c|}{1} & 0 & \multicolumn{1}{|r|}{0.0046} \\ \hline
\multicolumn{1}{|c|}{Q10} & 3 & \multicolumn{1}{|c|}{1} & 0 & \multicolumn{1}{|r|}{0.000091} \\  \hline % [1ex] adds vertical space
%\hline %inserts single line
\end{tabular}
\label{table:tab2} % is used to refer this table in the text
\rmfamily % RESTORES TO TIMES, SO THAT CAPTION IS WRITTEN PROPERLY
\end{table}

The characteristics of queries from {\bf Group II} are described as follows. These queries are aimed at evaluating OLAP functions that were grouped by type of data handled by each of them. They were designed to investigate the processing costs related to the amount of sets and members calculated by each of them. Q11 assesses the impact of performing searches over hierarchic data types, while query Q12 evaluates the processing costs for navigating over levels. Queries ranging from Q13 to Q17 are associated with the application of OLAP functions to calculated members and sets and each of them assess the handling of a particular type of data: logical, member, numeric, set and string, respectively. The MDX functions\footnote{MDX Function Reference (MDX): \url{http://msdn.microsoft.com/en-us/library/ms145970.aspx}} were classified by the type of data handled and assigned to the query type, and each OLAP function applied to calculated members and sets is executed in an ad hoc order in the referred query type. Also, the number of members and sets of each query type may vary according to the number of OLAP functions of each type of data handled by the query type, as shown in Table \ref{table:tab4}. Each query has a number of dimension tables accessed, and a number of calculated members and sets. Queries whose selectivity rates are equal to zero are executed only over metadata. The queries that are listed in Table \ref{table:tab1} and \ref{table:tab3} were parameterized as follows. The terms listed in bold denote keywords, functions or variables depending on the MDX function to be processed. Also, depending on each query type of this group, indexes I and J may be used, varying from 1 to N or from 1 to M, respectively. For example, if there are 15 OLAP functions (N = 15) for a particular data type, then the notation \emph{DataTypeFunctionI} corresponds to the I-th OLAP function of the referred data type. For example, \emph{LogicalFunction1} denotes the first function from a list of 15 functions that are available for the logical type of data handled by the query type and where I varies from 1 to 15. The full workload specification for all queries is available under the URL: <\url{http://www.cin.ufpe.br/~bemaf/obas/OBAS-MDX-QUERIES.pdf}>.
% EXAMPLE OF HOW TO WRITE A TABLE IN LATEX
% DO CHECK http://en.wikibooks.org/wiki/LaTeX/Tables FOR MORE TIPS & TRICKS {\bf ON} TABLES
\begin{center} 	% TABLES SHOULD BE CENTERED
\begin{table}[t]% TABLES SHOULD BE AT THE TOP [t] OR BOTTOM [b} OF PAGES
	\caption{The OBAS MDX Queries from Group II} % ALWAYS INCLUDE CAPTIONS IN YOUR TABLES
	\label{table:tab3} % USE THE CONTENT WITHIN label FOR REFERENCING IT (USE \ref WITHIN THE TEXT)
	\centering % used for centering table
%\ttfamily % CHANGES THE FONT WITHIN THE TABLE TO monotype
\begin{tabular}{| p{15cm} |}
	\hline  % DRAWS HORIZONTAL LINE
          \multicolumn{1}{| p{15cm} |}{  {\bf Q11 (Hierarchy):} }\\
	\multicolumn{1}{| p{15cm} |}{ {\bf WITH} MEMBER Measures.[Dimension] {\bf AS} 'Dimensions( [CUSTOMER].CurrentMember.Level.Ordinal ).DefaultMember.Dimension.Name' MEMBER Measures.[Hierarchy] {\bf AS} 'Dimensions( [CUSTOMER].CurrentMember.Level.Ordinal ).DefaultMember.Hierarchy.Name' {\bf SELECT} \{ Measures.[Dimension] , Measures.[Hierarchy] \} {\bf ON} COLUMNS, \{ [CUSTOMER].Members \} {\bf ON} {\bf ROWS} {\bf FROM} LINEORDER  }\\
	\hline  % DRAWS HORIZONTAL LINE
          \multicolumn{1}{| p{15cm} |}{  {\bf Q12 (Level):} }\\
	\multicolumn{1}{| p{15cm} |}{ {\bf WITH} [ MEMBER Measures.[LevelI] {\bf AS} '[{\bf DimensionI}].CurrentMember.Level.Name' ...] {\bf SELECT} \{ Measures.[{\bf LevelI}] , ... \} {\bf ON} COLUMNS, \{ \{ [[{\bf DimensionI}].Levels(J).Members.Item(0), ...]\} * \{...\} \} {\bf ON} {\bf ROWS} {\bf FROM} LINEORDER (I = 1 to 4 and J = 0 to 3)  }\\
	\hline  % DRAWS HORIZONTAL LINE
          \multicolumn{1}{| p{15cm} |}{  {\bf Q13 (Logical):} }\\
	\multicolumn{1}{| p{15cm} |}{ {\bf WITH} [MEMBER Measures.[{\bf LogicalFunctionI}] {\bf AS} '{\bf LogicalFunctionI}({\bf Argument1}, ...)' ...] {\bf SELECT} \{ [Measures.[{\bf LogicalFunctionI}] , ...] \} {\bf ON} COLUMNS, \{ \{SUPPLIER.Members\} \} {\bf ON} {\bf ROWS} {\bf FROM} [LINEORDER] (I = 1 to N)  }\\
	\hline  % DRAWS HORIZONTAL LINE
          \multicolumn{1}{| p{15cm} |}{  {\bf Q14 (Member):} }\\
	\multicolumn{1}{| p{15cm} |}{ {\bf WITH} [MEMBER Measures.[{\bf MemberFunctionI}] {\bf AS} '( Measures.[SomeMeasure], {\bf MemberFunctionI}({\bf Argument1},...))' ...] {\bf SELECT} \{ [Measures.[{\bf MemberFunctionI}] , ... ]\} {\bf ON} COLUMNS, \{ [CUSTOMER].Members \} {\bf ON} {\bf ROWS} {\bf FROM} [LINEORDER] (I = 1 to N)  }\\
	\hline  % DRAWS HORIZONTAL LINE
          \multicolumn{1}{| p{15cm} |}{  {\bf Q15 (Numeric):} }\\
	\multicolumn{1}{| p{15cm} |}{ {\bf WITH} [MEMBER Measures.[{\bf NumericFunctionI}] {\bf AS} ' CoalesceEmpty ( {\bf NumericFunctionI}({\bf Argument1}, ...) , Measures.[SomeMeasure] )' ... ] {\bf SELECT} \{ [Measures.[{\bf NumericFunctionI}] , ...] \} {\bf ON} COLUMNS, [SUPPLIER].Members {\bf ON} {\bf ROWS} {\bf FROM} LINEORDER (I = 1 to N)  }\\
	\hline  % DRAWS HORIZONTAL LINE
          \multicolumn{1}{| p{15cm} |}{  {\bf Q16 (Set):} }\\
	\multicolumn{1}{| p{15cm} |}{ {\bf WITH} MEMBER Measures.[NewMeasure] {\bf AS} '1' [{\bf SET} [{\bf SetFunctionI}] {\bf AS} ' \{ {\bf SetFunctionI}({\bf Argument1}, [{\bf SetFunctionJ}]...) \}' ...] {\bf SET} [AddCalculatedMembers] {\bf AS} 'AddCalculatedMembers(Measures.Members)' {\bf SELECT} \{ [{\bf SetFunctionN}] \} {\bf ON} {\bf ROWS}, \{ [AddCalculatedMembers] \} {\bf ON} COLUMNS {\bf FROM} LINEORDER (I = 1 to N and J < I)  }\\
	\hline  % DRAWS HORIZONTAL LINE
          \multicolumn{1}{| p{15cm} |}{  {\bf Q17 (String):} }\\
	\multicolumn{1}{| p{15cm} |}{ {\bf WITH} [MEMBER Measures.[{\bf StringFunctionI}] {\bf AS} '{\bf StringFunctionI}({\bf Argument1}, ...)' ...] {\bf SELECT} \{ Measures.[{\bf StringFunctionI}], ... \} {\bf ON} COLUMNS, CUSTOMER.Members {\bf ON} {\bf ROWS} {\bf FROM} LINEORDER (I = 1 to N)  }\\
	\hline  % DRAWS HORIZONTAL LINE 
\end{tabular}
%\rmfamily % RESTORES TO TIMES, SO THAT CAPTION IS WRITTEN PROPERLY
\end{table}
\end{center}

\begin{table}[b]
\caption{The Selectivity of OBAS MDX Queries from Group II} % title of Table
\centering % used for centering table
\sffamily % CHANGES THE FONT WITHIN THE TABLE TO sans serif
\begin{tabular}{|c|c|c|c|r|l|} % columns (6 columns)
\hline %\hline %inserts double horizontal lines
\bf{Query Name} & \bf{Number of Dimensions} & \bf{Member} & \bf{Set} & \bf{Selectivity} & \bf{Main Functionality} \\ \hline
Q11 & 0 & 2 & 0 & 0 & Hierarchy  \\ \hline
Q12 & 4 & 4 & 0 & 0 & Level  \\ \hline
Q13 & 0 & 5 & 0 & 0 & Logical Functions \\ \hline
Q14 & 0 & 18 & 0 & 1 & Member Functions \\ \hline
Q15 & 0 & 20 & 0 & 1 & Numeric Functions \\ \hline
Q16 & 3 & 1 & 31 & 0.429 & Set Functions \\ \hline
Q17 & 0 & 10 & 0 & 0 & String Functions \\ \hline
\end{tabular}
\label{table:tab4} % is used to refer this table in the text
\rmfamily % RESTORES TO TIMES, SO THAT CAPTION IS WRITTEN PROPERLY
\end{table}

The execution of queries from Group II implies in greater OLAP processing than those of Group I because an increase in the number of calculated members and sets determined by queries from Group II produces a greater impact on the OLAP processing than on the DBMS processing, in contrast with queries from Group I. The reason is that OLAP functions use metadata, computations based on positioning among members and process information of the type of data handled. Thus, the goal is to compare the types of data handled by the MDX queries from Group II and the number of functions used in these queries, while the queries from Group I vary the number of dimensions used to filter data and focus on selectivity of queries. As a result, Group I is more concerned with measuring the performance of the DBMS, whereas Group II is more related to the performance evaluation of analysis services. The main motivations behind the OBAS queries are to: (1) extend the SSB workload, which is based on a star schema and was designed specifically for DW applications; (2) increase the number of members and calculated sets to determine how this affects query processing performance; and (3) assess OLAP services according to the execution of different categories of MDX functions.

\subsection{Running Settings}
\label{sec:runningSettings}

OBAS was designed to evaluate different services and data catalogs through the use of a common communication interface, in order to ensure portability of the benchmark. The assessment of services with different communication interfaces would compromise the evaluation results obtained due to the diversified processing found in the specific programming logic of each service. The OBAS run settings correspond to the execution modes of OBAS, which can be location (local or remote) and if cache is used or not (cache or clear). Other settings include the service identifier and the identifier of the data catalog to be assessed, and the initial and final number of threads executed in the experiment. Our benchmark evaluates the {\bf location} between client and server machines, considering the performance achieved by sharing resources of the machine in the local mode of execution (called Local), and by taking into account the network traffic in the remote mode of execution (called Remote). To assess the service degradation and especially, to investigate the impacts of using {\bf cache}, there are two execution configurations, in which one considers the cache (called Cache) and the another one discards the cache (called Clear), by restarting the service. With the re-initialization, the services return to their initial states preventing degradation, but data recently used and kept in the cache are lost. A good way to clear the cache of a service is restarting it. Furthermore, when services are restarted, software rejuvenation, which is a proactive fault management technique that works against performance degradation, is often applied. In the local execution, performance is impaired by the sharing of resources between the services being tested and the OBAS benchmark application that can be seen as a parallel service, while in the remote execution, performance is affected by the network traffic. Other running configurations consist in determining: the URL of the {\bf service} to be evaluated, the data catalog name or the order in which the data {\bf catalog} is found in the catalog list provided by the analysis service, and the initial and final numbers of concurrent processes ({\bf threads}). The proposed benchmark enables the execution of 100 concurrent processes.

Cubes with different data volumes can be created by the OBAS data generator in order to evaluate data cube scalability as shown as follows. The OBAS benchmark can be used for generating and loading data into a DBMS, and is flexible regarding the database scalability, which can have an arithmetic growth rather than just having geometric increases of data volume. This enables a comparative analysis related to different types of increases of data volumes, in which different performance results are obtained and this allows an analysis of these differences. Moreover, the scalability initial value of the size of the fact table used in the experiment is determined as the lowest available value and not as a fixed value. For comparing the performance results obtained, the metric values are multiplied by the database scale factor (SF). In OBAS, the scale factor calculation is given by the ratio between the number of records of the fact table and the value of six million, so that the results of OBAS are compatible with those generated by the TPC-H. 

The relationship between the scale factor and the minimum number of threads is used to determine the minimum load for each database size used in the computation of the peak execution rate. In this case, since fixed scale values are not considered, the minimum number of parallel processes is determined depending on the size of the database, as shown in Table \ref{table:tab5}. Thus, OBAS remains compatible with the TPC-H metrics, and at the same time, provides a greater flexibility for scalability.

\begin{table}[b]
\caption{Minimal Number of Threads by Fact Table Rows} % title of Table
\centering % used for centering table
\sffamily
\begin{tabular}{|r|r|r|} % columns (3 columns)
\hline %\hline %inserts double horizontal lines
\bf{Fact Table Rows} & \bf{Scale Factor} & \bf{Threads}  \\ \hline
6,000,000 & 1 & 2  \\ \hline
60,000,000 & 10 & 3  \\ \hline
180,000,000 & 30 & 4  \\ \hline
600,000,000 & 100 & 5  \\ \hline
1,800,000,000 & 300 & 6  \\ \hline
6,000,000,000 & 1,000 & 7  \\ \hline
18,000,000,000 & 3,000 & 8  \\ \hline
60,000,000,000 & 10,000 & 9  \\ \hline
180,000,000,000 & 30,000 & 10  \\ \hline
600,000,000,000 & 100,000 & 11  \\ \hline
\end{tabular}
\label{table:tab5} % is used to refer this table in the text
\rmfamily
\end{table}

Each service has limitations that are crucial in evaluating its performance. For the scalability analysis, the goal is to evaluate if only the size of the database directly affects the performance obtained. In this analysis, we consider that more than a kind of service is provided (e.g. DBMS service, analysis service and communication interface), the OLAP processing costs and, depending on the running configuration chosen, the impacts related to the share of resources, network traffic and to the use of cache memory of the analysis services tested. Also, the increase in the number of concurrent processes (threads) allows the execution of increasing workloads and helps in defining how the running configuration chosen affects the performance results achieved.

\subsection{Performance Metrics}
\label{sec:performanceMetrics}

The OBAS benchmark has three performance metrics: response time, rate of execution and reliability. Response time is the interval between the submission of the query request and the obtainment of query results. The rate of execution is the amount of executions per time unit, such that OBAS uses the calculation described in \cite{tpc09}, which evaluates Power as the running potential of a single thread and Throughput as the rate of execution of concurrent processes. In addition, another compound metric is Composite that consists of the geometric average of the Power and Throughput measurements. To compute the overall average, the response time of each query is considered as the result of Power, since the execution of concurrent processes compromise the response time of each individual query and therefore does represent isolated results of only the query execution. We also assume that queries and updates do not occur simultaneously.

\def\X#1{$#1$}
The Reliability value is a percentage of the number of valid executions in relation to the total number of executions. A differential of OBAS, compared to other benchmarks, is the evaluation of service reliability whose calculation is based on the presence of errors in query execution. These errors occur after the increase in the number of concurrent processes whose values are higher than those considered by previous benchmarks. Furthermore, these errors are not related to the size of the database tested. Thus, the presence of these errors allows the computation of the reliability criterion of the service tested. Furthermore, a new performance metric of reliability was created, the OBAS-QPH (OBAS Query-Per-Hour), such that OBAS-QPH = Composite\X\times Reliability. As a result, a single number is obtained and corresponds to the actual rate of execution, since no errors occured during such execution time.

\subsection{The OBAS System Prototype}
\label{sec:theOBASSystemPrototype}

The main components of OBAS System prototype are described as follows and their interactions are organized in an architecture illustrated in Figure \ref{fig3}. The data generator (a) generates data to be tested and creates a text data file (b) containing synthetic data according to the dimensional schema described in Section \ref{sec:DimensionalDataSchema}, while the data loader (c) runs SQL scripts to load the generated data into the DBMS (d). The data loader imports databases or reads a data file and inserts generated data into the DBMS. The data generator and data loader are both used in the data preparation phase of OBAS (first phase).

In the execution phase of OBAS (second phase), a test component (e) uses a running script file (f) and a workload file (g) to define the configuration parameters to be chosen and the MDX queries to be executed, respectively. For each execution, a number of configuration parameters that directly affect the performance results are set in the running script. The configuration parameters of OBAS are the URL of the analysis service being studied, the number of parallel executions (i.e. threads) used in the experiments, the execution plan of MDX queries (i.e. the execution order of the MDX queries listed in the workload file), the data catalog name or the order in which the data catalog is found in the catalog list provided by the analysis service, if the cache content of the analysis service should be erased during its initialization, and the paths of output files reporting test results and errors.

The test component (e) is detailed as follows. First, the execution plan of the MDX queries are sent to the evaluated analysis service (h) that is reinitialized together with the DBMS and communication interface services (i). The communication interface sends the MDX queries of the execution plan to the analysis service, and this can be done in parallel or in sequence. Then, the analysis service gets the data resulted from the execution of SQL queries submitted to the DBMS. In the clear execution mode, the services are restarted after each sequence of queries. Such sequence is called iteration. The restart is done after each iteration and automatically, or locally and utilizing the services management component (j) of the operating system. For remote re-initialization, this services management component is accessed indirectly through the OBAS restarter component (k) that was developed for this purpose. This component is activated remotely and runs on the server waiting for a request for re-initialization of services. At the end of execution, the files with test results (l) and errors (m) are generated to be used in the performance comparative analysis and in the erros resolution, respectively. The current implementation of the OBAS test component reuses both the SSB data generator (DBGEN), which allows the generation of data files whose number of records is given by a scale factor, and the Oracle SQL Loader, which produce standard SQL files used to load data into the DBMS. The implementation of our proposed OBAS benchmark is available at <\url{http://www.cin.ufpe.br/~bemaf/obas/OBAS.zip}>.

\begin{figure*}[t]
 \begin{center}
    \includegraphics[height=6.5cm,width=12cm]{Fig3}
    \caption{The OBAS System Architecture}\label{fig3}
  \end{center}
\end{figure*}