Hi Daniel,
IMHO, for benchmarking, there are only three "rules of thumb" --- well, in fact, I consider them "real" rules:
1) Be aware of all parameters that (potentially) influence your metric. 2) Document & describe in detail which value( range)s you use for all these parameters. 3) Do not draw any conclusions that go beyond the parameter value( range)s you did measure with.
Obviously, these hold in general, not only for MonetDB.
Having said that, for the parameters you mention, there is IMHO no such thing as "rule of thumb"; neither for MonetDB nor for any other system.
If you want to know how the system performs under different number of simultaneous users, you need to vary that number accordingly. If you want to know how the system performs with different database sizes, you need to vary that number accordingly. If you want to know how the system performs with certain ratios of database size vs. RAM, you need to vary that ratio accordingly.
If you are not interested in the impact of these parameters, then you set them to a value of your choice, and report that value (and your rational for choosing it) in your benchmark description / documentation. Of course, if you run only single-user scenarios, you cannot conclude from the about multi-user behavior and vice versa. Likewise, if you only experiment with database sizes that fit well in RAM, you cannot conclude from that on the systems behavior with database sizes that significantly exceed RAM, and v.v.
If you want to analyze the performance of each query in isolation, this sounds likes a single-user scenario to me.
Best, Stefan
ps: I'd also be curious to learn more about what kind of intra-query checkpointing capabilities you added to MonetDB, and how, and how these are related to (read-only) TPC-H queries ...
----- On Apr 12, 2017, at 1:13 PM, Playfair, Daniel daniel.playfair@sap.com wrote:
Hi,
I was wondering is there any guidance or rules of thumb for setting the parameters (database size, number of simultaneous users) of benchmarks in MonetDB in relation to system properties (RAM, CPU, Disk, Network)?
Specifically I’m looking to run microbenchmarks with TPC-H, looking at performance of each query in isolation, so for my setup a RAM vs database size relationship is probably most important.
(This is to evaluate the impact of intra-query checkpointing capabilities I’ve added to a modified version of MonetDB as part of my PhD work.)
Thanks,
~ Daniel.
developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list