Jul2015-SP4 vs. Dec2016-SP5
All,
I know I've skipped multiple releases so this is a difficult question to answer, but should I expect the Dec2016-SP5 version of MonetDB to be faster than/equal to the Jul2015-SP4 version? I have a parallel set of servers (on the same box, 24 cores 192GB ram, listening to different ports) operating on identical data sets, and the Dec2016 version is slower than the Jul2015 version (~15-20%). My queries are basic joins between two tables with grouping. I have top running on the same box, and I can see that the Jul2015 server uses all 24 cores, but the Dec2016 server barely cracks 2-3.
The MAL plans obtained by explain are totally different (which partly explains the cores being utilized because I can see blocks of 24 commands) and the length of the Dec2016 one is a fraction of the Jul2015 version.
I guess the older version was inefficiently using all 24 cores, but the new version efficiently uses a smaller set of cores??
Thanks, Dave
Hai Dave,
On 15 Jun 2017, at 23:27, Anderson, David B david.b.anderson@citi.com wrote:
All,
I know I've skipped multiple releases so this is a difficult question to answer, but should I expect the Dec2016-SP5 version of MonetDB to be faster than/equal to the Jul2015-SP4 version?
Yeah, that’s the general idea, a new version shouldn’t be worse than a previous version.
Functional regression is easier to detect. I guess you already know our nightly testing system. So for a new release, the basic rule is that all tests that work in version X must also work in version X+1.
Performance regression is much harder to detect. Doing it automatically is still on our wish list. Currently, the best we can do is regularly run the several well known benchmarks to detect possible performance regression.
If you have a specific reproducible performance regression case, we’d be very interested to know about.
I have a parallel set of servers (on the same box, 24 cores 192GB ram, listening to different ports) operating on identical data sets,
What do you meant with “a parallel set of servers”? Have you replicated the data sets? How many servers do you use? How have you divided the system resources to each server? If you don’t restrict them (e.g. using VMs, containers), each MonetDB server will try to take all available resources causing resource contention.
and the Dec2016 version is slower than the Jul2015 version (~15-20%). My queries are basic joins between two tables with grouping. I have top running on the same box, and I can see that the Jul2015 server uses all 24 cores, but the Dec2016 server barely cracks 2-3.
The MAL plans obtained by explain are totally different (which partly explains the cores being utilized because I can see blocks of 24 commands) and the length of the Dec2016 one is a fraction of the Jul2015 version.
I guess the older version was inefficiently using all 24 cores, but the new version efficiently uses a smaller set of cores??
There are indeed significant changes from Jul2015 to Dec2016. You can get some information from the release notes since Jul2015: https://www.monetdb.org/OldReleaseNotes/Jun2016, and https://www.monetdb.org/Downloads/ReleaseNotes
That the MAL plans generated by Dec2016 are (much) shorter than Jul2015 can mostly be explained by that MonetDB has gone completely headless in Dec2016. For more information, please see this blog: https://www.monetdb.org/blog/monetdb-goes-headless. Moreover, we have done some clean up in the MAL functions, which is mentioned in the release notes.
You can prefix your queries with the keyword TRACE to get the execution time of each MAL operator. In this way, you can look deeper into where the time has been spent.
If the MonetDB version is the only factor that has been changed, I’m curious about why Dec2016 doesn’t use all 24 cores. Please send us the MAL plans of Dec2016 and Jul2015 of one such query (no guarantee though that we’ll then be able to find an explanation.).
Regards, Jennie
Thanks, Dave
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Jennie,
Yes, the servers are using the same data-set. I run them on the same box (real iron, non-virtual; unloaded; with the other instance of MonetDB shut down), and I run the queries multiple times so I know I'm working with hot servers.
I've looked at the release notes, but nothing has jumped out at me which would explain the behavior I am seeing (going from 24 cores down to 2+). I will setup an aggregation using many distinct columns which should force usage of multiple cores.
Will forward the MAL plans separately.
Thanks! Dave
-----Original Message----- From: users-list [mailto:users-list-bounces+david.b.anderson=citi.com@monetdb.org] On Behalf Of Ying Zhang Sent: Thursday, July 13, 2017 11:33 AM To: Communication channel for MonetDB users Subject: Re: Jul2015-SP4 vs. Dec2016-SP5
Hai Dave,
On 15 Jun 2017, at 23:27, Anderson, David B david.b.anderson@citi.com wrote:
All,
I know I've skipped multiple releases so this is a difficult question to answer, but should I expect the Dec2016-SP5 version of MonetDB to be faster than/equal to the Jul2015-SP4 version?
Yeah, that’s the general idea, a new version shouldn’t be worse than a previous version.
Functional regression is easier to detect. I guess you already know our nightly testing system. So for a new release, the basic rule is that all tests that work in version X must also work in version X+1.
Performance regression is much harder to detect. Doing it automatically is still on our wish list. Currently, the best we can do is regularly run the several well known benchmarks to detect possible performance regression.
If you have a specific reproducible performance regression case, we’d be very interested to know about.
I have a parallel set of servers (on the same box, 24 cores 192GB ram, listening to different ports) operating on identical data sets,
What do you meant with “a parallel set of servers”? Have you replicated the data sets? How many servers do you use? How have you divided the system resources to each server? If you don’t restrict them (e.g. using VMs, containers), each MonetDB server will try to take all available resources causing resource contention.
and the Dec2016 version is slower than the Jul2015 version (~15-20%). My queries are basic joins between two tables with grouping. I have top running on the same box, and I can see that the Jul2015 server uses all 24 cores, but the Dec2016 server barely cracks 2-3.
The MAL plans obtained by explain are totally different (which partly explains the cores being utilized because I can see blocks of 24 commands) and the length of the Dec2016 one is a fraction of the Jul2015 version.
I guess the older version was inefficiently using all 24 cores, but the new version efficiently uses a smaller set of cores??
There are indeed significant changes from Jul2015 to Dec2016. You can get some information from the release notes since Jul2015: https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_OldRele... , and https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_Downloa...
That the MAL plans generated by Dec2016 are (much) shorter than Jul2015 can mostly be explained by that MonetDB has gone completely headless in Dec2016. For more information, please see this blog: https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_blog_mo... . Moreover, we have done some clean up in the MAL functions, which is mentioned in the release notes.
You can prefix your queries with the keyword TRACE to get the execution time of each MAL operator. In this way, you can look deeper into where the time has been spent.
If the MonetDB version is the only factor that has been changed, I’m curious about why Dec2016 doesn’t use all 24 cores. Please send us the MAL plans of Dec2016 and Jul2015 of one such query (no guarantee though that we’ll then be able to find an explanation.).
Regards, Jennie
Thanks, Dave
users-list mailing list users-list@monetdb.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_m ailman_listinfo_users-2Dlist&d=DwIGaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=8_GYj k1edsyLJlNaxMxYxBJsviF3JXYvwDK42uy5KWU&m=r7_nmAeF6CUGqykE0jKYD49LB1gDD OT8fy9G_Hv34_o&s=yGd2m0ogMQ8SxdosziWbEyDhSYUdJ7XLd7ttl-B66_4&e=
_______________________________________________ users-list mailing list users-list@monetdb.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_mailman...
participants (2)
-
Anderson, David B -
Ying Zhang