\section{Introduction}
\label{sec:Introduction}

The benchmark technique is a model for experimental analysis to assess the performance of a computer system by running a fixed set of tests \cite{ciferri95}. In database area, the application of such technique involves the specification of a database logical schema for defining how data are organized, the definition of a set of database transactions for investigating query response times and the execution of data collection for obtaining values of performance metrics. This article proposes a benchmark to help in the performance evaluation of analysis services. These services execute online analytical processing (OLAP) queries over Data Warehouses (DW). OLAP tools are software aimed at multidimensional processing of data extracted from DW and allow the analysis of data in different perspectives and levels of aggregation \cite{chaudhuri97}, while a DW is a subject-oriented, integrated, time-variant, non-volatile and multidimensional database often modeled as star schemas composed of fact and dimension tables \cite{kimball02}.

Some benchmarks have been developed to enable the performance evaluation of DW systems, such as the TPC-H \cite{tpc09} and the Star Schema Benchmark (SSB) \cite{oneil09}. The TPC-H benchmark illustrates a data warehousing application that represents historical data relating to orders and sales of a corporation. Its schema is composed of two fact tables and several dimension tables. The SSB extends the TPC-H by adapting its schema to use one fact table only and, therefore, structuring data according to the star schema. However, these benchmarks \emph{are not concerned with the performance evaluation of analysis services}, and they are related specifically to the experimental assessment of DBMS because their workloads, which are represented as SQL queries, \emph{do not address the performance of OLAP}. Also, these benchmarks \emph{do not provide service performance metrics}, such as service reliability and throughput, which are seen as essential for conducting performance evaluations of analysis services. In this article, we present OBAS, an \emph{OLAP Benchmark for Analysis Services}, which extends the SSB specifications to enable the performance evaluation of a set of services, including the communication interface service, the analytical service and the database management service. The differentials introduced by OBAS are as follows: (1) it assists in the performance assessment of analysis services, which are responsible for the analytical processing of queries (OLAP); (2) it incorporates reliability metrics of services to allow the experimental evaluation of a set of services that may exhibit errors; (3) it enables investigations about the processing performance of OLAP functions provided by analysis services by varying the OLAP processing costs instead of considering the query selectivity only; (4) it works with the metaphor of data cubes with increasing volumes of data; and (5) it employs throughput metrics defined according to the number of threads. We emphasize that the analysis services studied here adopt Multidimensional Expressions (MDX) as query language, since MDX has become a de facto standard for many OLAP products \cite{whitehorn05}. MDX is used in this article to measure the functionalities of the evaluated analysis services.

The remainder of this article is organized as follows. Section \ref{sec:RelatedWork} surveys related work. The contributions of the article are described in Section \ref{sec:theOBASBenchmark}, which presents the proposed OBAS benchmark by defining the multidimensional data schema used in the construction of data cubes, the workload composed of MDX queries, the configuration parameters of executions of the proposed benchmark and performance metrics, and the architecture of the OBAS system prototype. Section \ref{sec:tests} contains evaluation results from our experiments conducted to assess two existing analytical services. Finally, Section \ref{sec:Conclusion} concludes the article and highlights future work.