[MonetDB-users] Problem with optimizer ?
Hi,
Please find attached a query (and corresponding schema) that results in mserver5 using more than 1GB of memory and terminating after 6 mins with this output:
MAPI = (monetdb) /tmp/.s.monetdb.50000 ACTION= read_line QUERY = PLAN SELECT DISTINCT h3.obj, CASE WHEN u_ENDPOINT.value IS NOT NULL THEN u_ENDPOINT.value WHEN l_ENDPOINT.value IS NOT NULL THEN l_ENDPOINT.value WHEN ll_ENDPOINT.value IS NOT NULL THEN ll_ENDPOINT.value WHEN lu_ENDPOINT.value IS NOT NULL THEN lu_ENDPOINT.value END, NULL FROM triples t0 INNER JOIN triples h1 ON (h1.pred = 21 AND h1.obj = 42 AND h1.subj = t0.subj) INNER JOIN triples h2 ON (h2.pred = 5 AND h2.subj = t0.subj) INNER JOIN triples h3 ON (h3.pred = 20 AND h3.subj = t0.s ERROR = !Connection terminated
Is it a problem of optimizer? Any guidelines to deal with it will be appreciated.
I also attached merovigian.log.
On 12-05-2011 15:37:30 +0300, Georgios Garmpis wrote:
Hi,
Please find attached a query (and corresponding schema) that results in mserver5 using more than 1GB of memory and terminating after 6 mins with this output:
2011-05-12 15:28:14 MSG merovingian[4114]: database 'testQuery' (4121) was killed by signal SIGKILL
I think your OS' OOM killer killed your database here. Can you check your dmesg output?
This is right (confirmed by dmesg), but why does mserver overuse memory for a query like that?
On Thu, May 12, 2011 at 3:44 PM, Fabian Groffen Fabian.Groffen@cwi.nlwrote:
On 12-05-2011 15:37:30 +0300, Georgios Garmpis wrote:
Hi,
Please find attached a query (and corresponding schema) that results in mserver5 using more than 1GB of memory and terminating after 6 mins with this output:
2011-05-12 15:28:14 MSG merovingian[4114]: database 'testQuery' (4121)
was killed by signal SIGKILL
I think your OS' OOM killer killed your database here. Can you check your dmesg output?
Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
To be more clear,
I cannot figure out the reason why mserver acts like that. Query doesn't seem very complicated to me. Should i refine my query or is it a problem of mserver (e.g. configuration issue, bug etc)?
On Thu, May 12, 2011 at 4:09 PM, George Garbis ggarbis@di.uoa.gr wrote:
This is right (confirmed by dmesg), but why does mserver overuse memory for a query like that?
On Thu, May 12, 2011 at 3:44 PM, Fabian Groffen Fabian.Groffen@cwi.nlwrote:
On 12-05-2011 15:37:30 +0300, Georgios Garmpis wrote:
Hi,
Please find attached a query (and corresponding schema) that results in mserver5 using more than 1GB of memory and terminating after 6 mins with this output:
2011-05-12 15:28:14 MSG merovingian[4114]: database 'testQuery' (4121)
was killed by signal SIGKILL
I think your OS' OOM killer killed your database here. Can you check your dmesg output?
Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- George Garbis
On 13-05-2011 12:54:05 +0300, Georgios Garmpis wrote:
To be more clear,
I cannot figure out the reason why mserver acts like that.
Query doesn't seem very complicated to me. Should i refine my query or is it a problem of mserver (e.g. configuration issue, bug etc)?
Try prefixing EXPLAIN to your "simple" query, and you'll get an idea what mserver is doing for you to perform the query.
On Fri, May 13, 2011 at 01:27:13PM +0200, Fabian Groffen wrote:
On 13-05-2011 12:54:05 +0300, Georgios Garmpis wrote:
To be more clear,
I cannot figure out the reason why mserver acts like that.
Query doesn't seem very complicated to me. Should i refine my query or is it a problem of mserver (e.g. configuration issue, bug etc)?
Try prefixing EXPLAIN to your "simple" query, and you'll get an idea what mserver is doing for you to perform the query.
Indeed.
Maybe you could prefix your query with both PLAN and EXPLAIN (one at a time) and share both results with us to see how MonetDB translates your query.
Also: how large is your data, i.e., how many records are there in the tables involved, and how large (number of records) is the expected result of your query?
Stefan
Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Both PLAN and EXPLAIN have the same result (being killed by OS' OOM killer). So, does size of data mater? Actually my tables are emtpy.
On Fri, May 13, 2011 at 2:57 PM, Stefan Manegold Stefan.Manegold@cwi.nlwrote:
On Fri, May 13, 2011 at 01:27:13PM +0200, Fabian Groffen wrote:
On 13-05-2011 12:54:05 +0300, Georgios Garmpis wrote:
To be more clear,
I cannot figure out the reason why mserver acts like that.
Query doesn't seem very complicated to me. Should i refine my query or is it a problem of mserver (e.g.
configuration
issue, bug etc)?
Try prefixing EXPLAIN to your "simple" query, and you'll get an idea what mserver is doing for you to perform the query.
Indeed.
Maybe you could prefix your query with both PLAN and EXPLAIN (one at a time) and share both results with us to see how MonetDB translates your query.
Also: how large is your data, i.e., how many records are there in the tables involved, and how large (number of records) is the expected result of your query?
Stefan
Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Fri, May 13, 2011 at 06:15:02PM +0300, Georgios Garmpis wrote:
Both PLAN and EXPLAIN have the same result (being killed by OS' OOM killer). So, does size of data mater?
In general it does for the resource consumption during query execution.
Actually my tables are emtpy.
This informtation, and fact that your query appears to make the make the server grows to an extend that it get's killed by the OS' OOM killer are curcial information for us. As far as I can tell now, this suggests that already the query transtlation or optimization failes, i.e., is does not even get to executing the generated plan.
Could you please file a bug report via http://bugs.monetdb.org/ including all informtation required to reporduce the crash?
Thanks!
Stefan
On Fri, May 13, 2011 at 2:57 PM, Stefan Manegold Stefan.Manegold@cwi.nlwrote:
On Fri, May 13, 2011 at 01:27:13PM +0200, Fabian Groffen wrote:
On 13-05-2011 12:54:05 +0300, Georgios Garmpis wrote:
To be more clear,
I cannot figure out the reason why mserver acts like that.
Query doesn't seem very complicated to me. Should i refine my query or is it a problem of mserver (e.g.
configuration
issue, bug etc)?
Try prefixing EXPLAIN to your "simple" query, and you'll get an idea what mserver is doing for you to perform the query.
Indeed.
Maybe you could prefix your query with both PLAN and EXPLAIN (one at a time) and share both results with us to see how MonetDB translates your query.
Also: how large is your data, i.e., how many records are there in the tables involved, and how large (number of records) is the expected result of your query?
Stefan
Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- George Garbis
Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Hi Jennie,
hm, I received it well via the list, and also this looks fine to me: http://sourceforge.net/mailarchive/message.php?msg_id=27498452
Maybe a (temp.?) problem with your email (-program)?
Stefan
On Sun, May 15, 2011 at 01:45:07PM +0200, Ying Zhang wrote:
Hi Stefan, it looks like that something has gone wrong with your reply.
On May 13, 2011, at 19:06 , Stefan Manegold wrote:
<<< No Message Collected >>>
Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Hi Stefan,
Thanks. This is strange... It could be a one-time problem somewhere at my side, as it has never happened before. It also explains why no one has said anything for about two days:) Sorry for disturbing you :P
Jennie
On May 15, 2011, at 17:48 , Stefan Manegold wrote:
Hi Jennie,
hm, I received it well via the list, and also this looks fine to me: http://sourceforge.net/mailarchive/message.php?msg_id=27498452
Maybe a (temp.?) problem with your email (-program)?
Stefan
On Sun, May 15, 2011 at 01:45:07PM +0200, Ying Zhang wrote:
Hi Stefan, it looks like that something has gone wrong with your reply.
On May 13, 2011, at 19:06 , Stefan Manegold wrote:
<<< No Message Collected >>>
Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
participants (4)
-
Fabian Groffen -
Georgios Garmpis -
Stefan Manegold -
Ying Zhang