Skip to main content

Benchmarks

The XQuery development community has adopted the XMark benchmark to assess functionality and performance maturity of prototype implementations. The benchmark has been developed by a team of researchers worldwide under the guidance of CWI in 2002. It measures performance using a mixture of XQuery statements. Some of which are notoriously difficult to implement, because they explore the breath of the XQuery language features.

Try the benchmark queries that are ready-to-run as-is on MonetDB/XQuery.

The XML-document size to query over is scalable from 100KB to 10GB. Most XQuery implementations have reported results for documents up to 1GB. MonetDB/XQuery has been successfully used to process all the way up to 10GB, as illustrated below and explained in more details on the Pathfinder project website.

MonetDB/XQuery performance on XMark Benchmark
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
110KB 0.01 0.01 0.02 0.01 0.01 0.00 0.00 0.01 0.02 0.13 0.01 0.04 0.02 0.01 0.02 0.02 0.02 0.01 0.03 0.07
1MB 0.02 0.01 0.03 0.01 0.01 0.00 0.00 0.01 0.02 0.16 0.02 0.04 0.02 0.03 0.03 0.03 0.02 0.01 0.03 0.07
11MB 0.01 0.02 0.14 0.03 0.01 0.00 0.00 0.04 0.05 0.54 0.11 0.09 0.03 0.12 0.03 0.03 0.03 0.02 0.06 0.11
110MB 0.12 0.19 1.20 0.42 0.08 0.00 0.01 0.47 0.52 5.18 3.62 2.11 0.10 0.93 0.07 0.08 0.15 0.05 0.38 0.62
1GB 1.30 1.80 11.50 4.50 0.80 0.00 0.10 9.60 11.80 62.80 367.70 121.10 0.90 7.50 0.40 0.50 1.40 0.50 7.00 7.00
11GB 14.00 19.00 176.00 44.00 10.00 0.10 0.60 223.00 460.00 2413.00 DNF DNF 8.00 452.00 3.00 4.00 31.00 7.00 128.00 70.00
 
  < 0.01s
  < 1s
  < 5s
  < 10s
  < 100s
  > 100s
  Did Not Finish in 1 hour

The table shows the execution times (in seconds) for MonetDB/XQuery 0.10 on various sizes of the XMark benchmark (20 queries) on XML documents ranging from 110KB to 11GB on a 1.6GHz machine. MonetDB/XQuery 0.10 was compiled in 64-bits mode using 32-bits object-identifiers, and ran out-of-the-box (i.e. without any manual configuration or tuning). We have also compared our system with all important other XQuery systems, for which we did try some tuning (e.g. index creation).

Comparison of MonetDB with other XQuery systems and previous MonetDB versions

This shows that since the first release, response times improved by roughly a factor two. MonetDB/XQuery outperforms all XQuery systems currently available on XMark, especially on large datasets (10MB or larger) and when queries involve join conditions (Q8-Q12).

Without query optimization built-in that recognizes the opportunity to use a special join-operations, the performance of other XQuery systems deteriorates quadratically with respect to document size (i.e. becomes 100 times slower on a 10 times larger document). Note that XMark Q11 and Q12 contain non-selective joins that produce a quadratically increasing intermediate result, hence scale quadratic on any system (thus, MonetDB/XQuery also scales quadratically on Q11-Q12, resulting in a DNF result on the huge 11GB document).

While MonetDB/XQuery is especially fast on large and complex queries, in this release we also paid attention to query response times on small data sizes. Few-millisecond latency is especially important when MonetDB/XQuery is used in interactive systems (like a web server). If queries are packaged in XQuery functions, defined in an XQuery Module, the new automatic prepared query mechanism removes most run-time query translation and execution overhead and further improve performance on small (non-huge) document sizes, i.e. by a factor of 10 (116KB document) to 3 (10MB document), while for larger document sizes, performance remains equal.

We also provide ready-to-run modularized versions of the XMark benchmark queries. Each query consists of a call to a function defined in the xmark module.

Note: As the XMark Benchmark has been developed at the CWI as well, some might even think that XMark has been created to promote MonetDB/XQuery. The contrary is true. The benchmark was developed long before the Pathfinder project was born.