Skip to main content

XMark Benchmark

We measured XQuery performance on the XMark benchmark. This page provides the query execution time results in seconds for all the systems we tested, for all XMark queries, for all sizes of the XMark document (from 116KB to 11GB). The graphs below in this page show the execution speed of the competitors relative to MonetDB/XQuery (on a logarithmic scale). Thus, a bar at height X in the graphs means that MonetDB/XQuery is X times faster than the competitor (thus, if X < 1, MonetDB/XQuery is slower).

We ran these experiments on a (64-bits) Linux 2.6.1 server that has two single-core 1.6GHz Opteron processors, 8GB RAM, and a RAID system consisting of 8 ATA drives. MonetDB/XQuery only used one of these CPUs, though, and ran CPU-bound in almost all queries. We report the best of 5 runs without XML serialization (DNF means "Did Not Finish within an hour"). On MonetDB/XQuery, results tend not to vary on a running server (after the first query on a document). MonetDB/XQuery 0.10 was compiled in 64-bits mode using 32-bits object-identifiers, and ran out-of-the-box (i.e. without any manual configuration or tuning).

Other Systems

Our goal is to compare MonetDB/XQuery with the main alternative XQuery database systems available now (February 2006):

EXT

eXist is currently the most popular XQuery database system. We used the latest version available (the snapshot from January 24, 2006). We set the buffer cache (cacheSize) to 6GB, to make use of all our RAM. However, eXist, already failed to load the XMark documents of 100MB and beyond, stating these are too large or irregular for its indexing scheme. We extracted the precise query exectuin timings from the eXist execution log, because the interactive java client of eXist has a large startup overhead. The results show that eXist is only quick on the 116KB document. When processing a 1MB database, eXist already takes multiple seconds to process quite a few XMark queries (not only join queries Q8-12). On the 10MB dataset the expensive queries already take eXist half an hour. In contrast, MonetDB/XQuery executes all XMark queries in only (tens of) milliseconds on such small databases.
GLX

Galax. We tested the pre-compiled 0.50 release (as compilation of Galax on our system failed). Our timings are based on the pure query time as reported by the performance-monitoring features of Galax (excluding serialization and compilation). While designed and engineered as a true database system (inclusive attention for query optimization) the weakness of Galax is still scalability. Galax cannot process the XMark documents of 1GB and 11GB (it fails with memory allocation errors). Also, not all XMark join patterns are detected by it.
BDB

Berkeley DB XML. We compiled version 2.2 ourselves with 64-bits and all optimization flags enabled. Our timings are based on the timing feature of the execution shell provided by BDB XML (and do not include serialization). BDB XML 2.2 combines the well-known BDB B-trees with the Xerces (XML parsing) and Pathan (XPath processor) libraries. Until now, this combination of off-the-shelve components seems to produce only a rather naive XQuery interpreter. The joins in Q8-12 are not recognized (thus very slow), and we even had some trouble putting its indexing features to good use (we tried string- and decimal-value indices on the attributes used by XMark, as well as path indices -- not to much effect). In contrast, note that MonetDB/XQuery will reproduce our XMark results out-of-the-box (no database tuning needed).
XHV

X-Hive is a native XQuery database engine. We tested version 6.0 (version 7.0 is out now). Beware, this is a commercial product with significant list price (around $40.000). Our tests used a simple Java program that executes the XMark queries, derived from the example queries that come with X-Hive; timing was done inside this Java program and excludes printing of the query results. As in BDB XML, hand-tuning of your database by creating indices by hand (and giving X-Hive specific hints in queries) is very important to improve X-Hive performance. In general, query optimization in this product is rudimentary or absent. By creating a value index, we were able to make one XMark join query fast (Q8). We tried other indices; sometimes these accelerate one query, but then other queries become slower. X-Hive seems to be rather similar to DBD XML in terms of how it performs, and its manual index tuning needs. As the below results show, MonetDB/XQuery is significantly faster than both, without the need to tune indices to queries.

Note that our comparison only includes systems that were available to us on this platform. We also tested two other commercial XQuery databases: Tamino (4.2) and MarkLogic (3.0) on Windows. Due to the platform difference (Linux vs. Windows; different hardware), a direct comparison was not possible, but we can assure that both these products could not handle large documents (100MB and beyond) well, especially on the join queries (Q8-12), where both systems produced DNF (Did Not Finish) results. Another popular XQuery system is Saxon, however this is not a database system but a file processor: it reads in the entire XML file for each query. Therefore, even on small documents, a performance comparison is quite pointless.

If you think we left out or misrepresented relevant XQuery database systems, please do provide us with the software and instructions how to run the benchmark. Although MonetDB/XQuery does not need to create any special indices, we would be more than welcome to tweak all systems, so that they perform at their best.

Conclusion

While the performance of MonetDB/XQuery is comparable to (are even slightly surpassed by) some of the competitors when querying very small documents (110KB XMark), none of the other XQuery systems comes close to the performance on larger XMark documents (>110MB XMark), especially on the XMark queries that contain joins (Q8-Q12).

Comparison With Other Systems

XMark 110KB

System / QueryNo. (s)

Elapsed seconds on XMark Benchmark
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
MXQ 0.01 0.01 0.02 0.01 0.01 0.00 0.00 0.01 0.02 0.13 0.01 0.04 0.02 0.01 0.02 0.02 0.02 0.01 0.03 0.07
GLX 0.00 0.00 0.00 0.00 0.00 0.01 0.03 0.00 0.01 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.01 0.00
XHV 0.14 0.04 0.04 0.04 0.01 0.07 0.02 0.03 0.06 0.08 0.08 0.03 0.00 0.01 0.00 0.00 0.01 0.01 0.02 0.03
BDB 0.00 0.00 0.01 0.01 0.00 0.01 0.01 0.01 0.01 0.01 0.02 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.01 0.01
EXT 0.00 0.01 0.02 0.03 0.03 0.00 0.00 0.05 0.06 0.15 0.04 0.05 0.00 0.01 0.01 0.05 0.00 0.01 0.01 0.01
 
  < 0.01s
  < 1s
  < 5s
  < 10s
  < 100s
  > 100s
  Did Not Finish in 1 hour

Comparison between MonetDB/XQuery (MXQ) and ...

Calculated as: log10(other / MonetDB) (higher = other system is slower)

Galax (GLX)
X-Hive (XHV)
Berkeley DB/XML (BDB)
eXist (EXT)

XMark 1.1MB

System / QueryNo. (s)

Elapsed seconds on XMark Benchmark
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
MXQ 0.02 0.01 0.03 0.01 0.01 0.00 0.00 0.01 0.02 0.16 0.02 0.04 0.02 0.03 0.03 0.03 0.02 0.01 0.03 0.07
GLX 0.00 0.00 0.01 0.03 0.01 0.12 0.28 0.01 0.11 0.14 0.04 0.03 0.00 0.11 0.00 0.00 0.01 0.01 0.09 0.03
XHV 0.17 0.09 0.12 0.07 0.04 0.10 0.11 0.22 0.58 0.50 0.16 0.31 0.01 0.06 0.01 0.01 0.01 0.01 0.07 0.02
BDB 0.01 0.01 0.04 0.04 0.01 0.11 0.12 0.45 0.41 0.15 1.26 0.49 0.01 0.11 0.01 0.02 0.02 0.02 0.06 0.04
EXT 0.01 0.14 0.18 0.38 2.42 0.00 0.01 0.66 0.78 16.53 2.06 3.07 0.01 0.23 0.01 0.60 0.02 0.01 0.04 0.06
 
  < 0.01s
  < 1s
  < 5s
  < 10s
  < 100s
  > 100s
  Did Not Finish in 1 hour

Comparison between MonetDB/XQuery (MXQ) and ...

Calculated as: log10(other / MonetDB) (higher = other system is slower)

Galax (GLX)
X-Hive (XHV)
Berkeley DB/XML (BDB)
eXist (EXT)

XMark 11MB

System / QueryNo. (s)

Elapsed seconds on XMark Benchmark
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
MXQ 0.01 0.02 0.14 0.03 0.01 0.00 0.00 0.04 0.05 0.54 0.11 0.09 0.03 0.12 0.03 0.03 0.03 0.02 0.06 0.11
GLX 0.06 0.03 0.14 0.22 0.05 1.30 2.68 0.16 113.23 1.74 2.62 1.44 0.03 1.92 0.02 0.03 0.06 0.07 1.17 0.28
XHV 0.37 0.45 0.65 0.10 0.13 1.07 1.57 0.85 32.25 5.28 98.91 23.39 0.10 0.72 0.03 0.03 0.09 0.08 0.67 0.11
BDB 0.05 0.13 0.34 0.39 0.10 1.14 1.31 51.21 47.03 5.15 121.75 118.70 0.08 1.07 0.13 0.14 0.20 0.19 0.57 0.34
EXT 0.01 5.67 6.61 15.40 185.47 0.01 0.01 429.89 333.47 1559.17 374.46 1584.91 0.03 0.44 0.05 22.21 0.18 0.12 0.51 0.98
 
  < 0.01s
  < 1s
  < 5s
  < 10s
  < 100s
  > 100s
  Did Not Finish in 1 hour

Comparison between MonetDB/XQuery (MXQ) and ...

Calculated as: log10(other / MonetDB) (higher = other system is slower)

Galax (GLX)
X-Hive (XHV)
Berkeley DB/XML (BDB)
eXist (EXT)

XMark 110MB

System / QueryNo. (s)

Elapsed seconds on XMark Benchmark
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
MXQ 0.12 0.19 1.20 0.42 0.08 0.00 0.01 0.47 0.52 5.18 3.62 2.11 0.10 0.93 0.07 0.08 0.15 0.05 0.38 0.62
GLX 0.72 0.31 1.76 2.91 0.63 13.29 30.01 2.12 DNF 18.61 DNF DNF 0.66 99.53 0.20 0.46 0.82 0.73 14.73 2.98
XHV 1.29 1.75 5.66 1.00 0.90 10.17 24.84 3.51 12280.66 442.37 19927.29 5100.19 1.03 11.16 0.49 0.52 0.85 0.64 12.15 1.40
BDB 0.51 1.38 3.55 4.07 1.05 13.23 14.70 9316.72 DNF DNF DNF DNF 0.79 14.18 1.37 1.52 2.08 2.09 6.74 3.42
 
  < 0.01s
  < 1s
  < 5s
  < 10s
  < 100s
  > 100s
  Did Not Finish in 1 hour

Comparison between MonetDB/XQuery (MXQ) and ...

Calculated as: log10(other / MonetDB) (higher = other system is slower)

Galax (GLX)
X-Hive (XHV)
Berkeley DB/XML (BDB)

XMark 1.1GB

Elapsed seconds performance on XMark Benchmark
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
MXQ 1.30 1.80 11.50 4.50 0.80 0.01 0.10 9.60 11.80 62.80 367.70 121.10 0.90 7.50 0.40 0.50 1.40 0.50 7.00 7.00
XHV 9.90 33.00 25.10 18.10 20.70 178.10 278.40 49.10 DNF DNF DNF DNF 12.90 110.20 10.60 10.90 11.80 14.80 254.50 24.60
BDB 5.90 43.10 37.10 43.30 11.40 DNF DNF DNF DNF DNF DNF DNF 8.10 DNF 28.50 17.60 34.10 21.70 135.60 37.40
 
  < 0.01s
  < 1s
  < 5s
  < 10s
  < 100s
  > 100s
  Did Not Finish in 1 hour

Comparison between MonetDB/XQuery (MXQ) and ...

Calculated as: log10(other / MonetDB) (higher = other system is slower)

X-Hive (XHV)
Berkeley DB/XML (BDB)

Comparison With Previous Versions

XMark 110KB

Elapsed seconds on XMark Benchmark
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
MXQ 0.10 0.01 0.01 0.02 0.01 0.01 0.00 0.00 0.01 0.02 0.13 0.01 0.04 0.02 0.01 0.02 0.02 0.02 0.01 0.03 0.07
MXQ 0.10 adhoc 0.06 0.06 0.14 0.09 0.06 0.04 0.04 0.09 0.17 0.33 0.10 0.10 0.06 0.05 0.07 0.07 0.06 0.05 0.08 0.18
MXQ 0.8 0.09 0.10 0.28 0.15 0.10 0.06 0.07 0.17 0.29 0.55 0.18 0.14 0.11 0.08 0.12 0.13 0.10 0.08 0.14 0.34
 
  < 0.01s
  < 1s
  < 5s
  < 10s
  < 100s
  > 100s
  Did Not Finish in 1 hour

Comparison between MonetDB/XQuery 0.10 and ...

MonetDB/XQuery 0.10 adhoc (without modules)
MonetDB/XQuery 0.8

XMark 1MB

Elapsed seconds on XMark Benchmark
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
MXQ 0.10 0.02 0.01 0.03 0.01 0.01 0.00 0.00 0.01 0.02 0.16 0.02 0.04 0.02 0.03 0.03 0.03 0.02 0.01 0.03 0.07
MXQ 0.10 adhoc 0.06 0.06 0.15 0.10 0.06 0.04 0.04 0.09 0.17 0.36 0.10 0.11 0.06 0.06 0.07 0.08 0.06 0.05 0.09 0.19
MXQ 0.8 0.10 0.10 0.25 0.14 0.08 0.05 0.06 0.16 0.28 0.65 0.17 0.19 0.10 0.10 0.12 0.14 0.10 0.07 0.14 0.33
 
  < 0.01s
  < 1s
  < 5s
  < 10s
  < 100s
  > 100s
  Did Not Finish in 1 hour

Comparison between MonetDB/XQuery 0.10 and ...

MonetDB/XQuery 0.10 adhoc (without modules)
MonetDB/XQuery 0.8

XMark 11MB

Elapsed seconds on XMark Benchmark
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
MXQ 0.10 0.01 0.02 0.14 0.03 0.01 0.00 0.00 0.04 0.05 0.54 0.11 0.09 0.03 0.12 0.03 0.03 0.03 0.02 0.06 0.11
MXQ 0.10 adhoc 0.07 0.08 0.26 0.10 0.06 0.04 0.05 0.12 0.20 0.70 0.17 0.15 0.07 0.16 0.08 0.08 0.07 0.06 0.11 0.21
MXQ 0.8 0.10 0.12 0.37 0.15 0.09 0.05 0.07 0.20 0.32 1.01 0.24 0.21 0.11 0.30 0.14 0.16 0.11 0.08 0.17 0.36
 
  < 0.01s
  < 1s
  < 5s
  < 10s
  < 100s
  > 100s
  Did Not Finish in 1 hour

Comparison between MonetDB/XQuery 0.10 and ...

MonetDB/XQuery 0.10 adhoc (without modules)
MonetDB/XQuery 0.8

XMark 110MB

Elapsed seconds on XMark Benchmark
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
MXQ 0.10 0.12 0.19 1.20 0.42 0.08 0.00 0.01 0.47 0.52 5.18 3.62 2.11 0.10 0.93 0.07 0.08 0.15 0.05 0.38 0.62
MXQ 0.10 adhoc 0.15 0.26 1.32 0.39 0.10 0.04 0.05 0.55 0.65 4.90 3.72 2.12 0.14 0.94 0.11 0.12 0.18 0.09 0.42 0.49
MXQ 0.8 0.22 0.35 1.60 0.52 0.20 0.08 0.11 0.81 0.99 5.52 3.54 1.73 0.26 2.24 0.33 0.31 0.34 0.17 0.61 0.74
 
  < 0.01s
  < 1s
  < 5s
  < 10s
  < 100s
  > 100s
  Did Not Finish in 1 hour

Comparison between MonetDB/XQuery 0.10 and ...

MonetDB/XQuery 0.10 adhoc (without modules)
MonetDB/XQuery 0.8

XMark 1GB

Elapsed seconds on XMark Benchmark
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
MXQ 0.10 1.30 1.80 11.50 4.50 0.80 0.01 0.10 9.60 11.80 62.80 367.70 121.10 0.90 7.50 0.40 0.50 1.40 0.50 7.00 7.00
MXQ 0.10 adhoc 1.05 1.85 11.79 3.47 0.64 0.03 0.14 9.66 11.22 57.81 383.56 125.97 0.94 7.24 0.45 0.55 1.34 0.54 7.06 4.72
MXQ 0.8 1.25 2.45 12.59 3.87 1.24 0.33 0.44 10.46 13.02 55.21 960.96 431.37 1.34 21.34 1.75 1.85 2.84 0.94 5.36 5.02
 
  < 0.01s
  < 1s
  < 5s
  < 10s
  < 100s
  > 100s
  Did Not Finish in 1 hour

Comparison between MonetDB/XQuery 0.10 and ...

MonetDB/XQuery 0.10 adhoc (without modules)
MonetDB/XQuery 0.8

XMark 11GB

Elapsed seconds on XMark Benchmark
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
MXQ 0.10 14.00 19.00 176.00 44.00 10.00 0.10 0.60 223.00 460.00 2413.00 DNF DNF 8.00 452.00 3.00 4.00 31.00 7.00 128.00 70.00
MXQ 0.10 adhoc 11.05 19.05 233.09 33.07 8.04 0.13 0.64 220.06 417.12 2318.21 DNF DNF 8.04 356.04 3.05 4.05 29.04 7.04 123.06 51.12
MXQ 0.8 13.05 25.05 126.09 36.07 11.04 3.03 4.04 208.06 289.12 1882.21 DNF DNF 13.04 959.04 16.05 18.05 26.04 9.04 88.06 50.12
 
  < 0.01s
  < 1s
  < 5s
  < 10s
  < 100s
  > 100s
  Did Not Finish in 1 hour

Comparison between MonetDB/XQuery 0.10 and ...

MonetDB/XQuery 0.10 adhoc (without modules)
MonetDB/XQuery 0.8