Is minimal_pipe enforced on long MAL plans?
Hi there,
Is minimal_pipe enforced under any condition? For example when the the MAL plan generated by rel2bin is longer than a given threshold?
Context:
I have a pretty long query which results in a MAL plan of 33K lines.
Though the query is a bit long, I still would expect a 2-3K lines MAL plan. The plan indeed contains the same pieces of code re-evaluated over and over.
It first, I simply blamed commonTerms not doing its job. The thing is, the MAL plans generated with minimal_pipe and default_pipe are absolutely identical for this query. Hence my question above.
Best, Roberto
Hi Roberto,
On 22/06/15 13:13, Roberto Cornacchia wrote:
Hi there,
Is minimal_pipe enforced under any condition?
It actually should to ensure all SQL functionality to work properly. (I'll patch the code to enforce it for missing ones)
For example when the the MAL plan generated by rel2bin is longer than a given threshold?
No thresholds are involved.
Context:
I have a pretty long query which results in a MAL plan of 33K lines.
Would be nice to see the SQL part for this one.
Though the query is a bit long, I still would expect a 2-3K lines MAL plan. The plan indeed contains the same pieces of code re-evaluated over and over.
But over different horizontal slices (?) and differently fused into a parallel schedule.
It first, I simply blamed commonTerms not doing its job. The thing is, the MAL plans generated with minimal_pipe and default_pipe are absolutely identical for this query. Hence my question above.
Could well be possible. If there are too many side-effects to deal with, or if the data flow optimizer only sees an option to run in sequential mode, it leave the plan intact.
Best, Martin
Best, Roberto
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Is minimal_pipe enforced under any condition?
It actually should to ensure all SQL functionality to work properly. (I'll patch the code to enforce it for missing ones)
But please only as a last resort to ensure correctness. Not as an assumption on the efficiency of optimizers.
For example when the the MAL plan generated by rel2bin is longer than a
given threshold?
No thresholds are involved.
Good to know.
Context:
I have a pretty long query which results in a MAL plan of 33K lines.
Would be nice to see the SQL part for this one.
No problem. I can send a file with schema, required views, and query. Is it better to do it in a bug report? (I do think it's a possible bug, also because Jul2015 seems to cope much better with it). I forgot to mention that the version I'm having this problem with is Oct2014-SP4.
Though the query is a bit long, I still would expect a 2-3K lines MAL plan.
The plan indeed contains the same pieces of code re-evaluated over and over.
But over different horizontal slices (?) and differently fused into a parallel schedule.
No. No slices at all. Simply, the same tables are defined and read over and over. No UDFs, no side-effects. The MAL plan isn't just sub-optimized, it really is not optimized at all.
It first, I simply blamed commonTerms not doing its job.
The thing is, the MAL plans generated with minimal_pipe and default_pipe are absolutely identical for this query. Hence my question above.
Could well be possible. If there are too many side-effects to deal with, or if the data flow optimizer only sees an option to run in sequential mode, it leave the plan intact.
Yes, for some reason it doesn't even try to optimize. I'd love to know the reason.
I can send the SQL (about 80Kb in total) here, or in a bug report, or in personal message, what do you prefer? Roberto
On Mon, Jun 22, 2015 at 09:38:16PM +0200, Roberto Cornacchia wrote:
Is minimal_pipe enforced under any condition?
no, only the other way around (as martin just mentioned)
niels
It actually should to ensure all SQL functionality to work properly. (I'll patch the code to enforce it for missing ones)But please only as a last resort to ensure correctness. Not as an assumption on the efficiency of optimizers.
For example when the the MAL plan generated by rel2bin is longer than a given threshold? No thresholds are involved.Good to know.
Context: I have a pretty long query which results in a MAL plan of 33K lines. Would be nice to see the SQL part for this one.No problem. I can send a file with schema, required views, and query. Is it better to do it in a bug report? (I do think it's a possible bug, also because Jul2015 seems to cope much better with it). I forgot to mention that the version I'm having this problem with is Oct2014-SP4.
Though the query is a bit long, I still would expect a 2-3K lines MAL plan. The plan indeed contains the same pieces of code re-evaluated over and over. But over different horizontal slices (?) and differently fused into a parallel schedule.No. No slices at all. Simply, the same tables are defined and read over and over. No UDFs, no side-effects. The MAL plan isn't just sub-optimized, it really is not optimized at all.
It first, I simply blamed commonTerms not doing its job. The thing is, the MAL plans generated with minimal_pipe and default_pipe are absolutely identical for this query. Hence my question above. Could well be possible. If there are too many side-effects to deal with, or if the data flow optimizer only sees an option to run in sequential mode, it leave the plan intact.Yes, for some reason it doesn't even try to optimize. I'd love to know the reason.
I can send the SQL (about 80Kb in total) here, or in a bug report, or in personal message, what do you prefer? Roberto
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
participants (3)
-
Martin Kersten -
Niels Nes -
Roberto Cornacchia