[MonetDB-users] Recycler Optimizer
Hello:
I understand some of the optimizers/pipelines are experimental as indicated in the monetdb5.conf file. Has anyone used the Recycle pipeline for Skyserver? Would you mind sharing your experience, in terms of performance improvement and whether it can bet used in a production environment? We are using the Oct2010-SP1 release for a db that is read-only. I would guess if there are issues, I would think the problems would likely show up in read/write systems. Any feedback would be appreciated....
It looks like the Recycle pipeline for Skyserver pipeline is based on the Nov-2009 pipeline. Has anyone tried the recycler optimizer with the Feb2010 default pipeline? I was going to add "recycle" between "deadcode" and "reduce" in the default pipeline and see if it would work. Thanks.
Henry
Hi Henry
We would love to here experiences in using the recycler.
regards, Martin Kersten On 3/3/11 11:00 PM, Henry Addington wrote:
Hello: I understand some of the optimizers/pipelines are experimental as indicated in the monetdb5.conf file. Has anyone used the Recycle pipeline for Skyserver? Would you mind sharing your experience, in terms of performance improvement and whether it can bet used in a production environment? We are using the Oct2010-SP1 release for a db that is read-only. I would guess if there are issues, I would think the problems would likely show up in read/write systems. Any feedback would be appreciated.... It looks like the Recycle pipeline for Skyserver pipeline is based on the Nov-2009 pipeline. Has anyone tried the recycler optimizer with the Feb2010 default pipeline? I was going to add "recycle" between "deadcode" and "reduce" in the default pipeline and see if it would work. Thanks. Henry
What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Hi Martin:
We tried the Recycle pipeline for Skyserver on the Oct2010-SP1 release. And, we didn't see any problems, though the testing was fairly short..... One thing we did notice is that queries against defined views do not seem to benefit from the recycler pipeline at all. Running a simple query, select distinct x from y, back to back seem to take just as long each time. So, I don't think the Recycler pipeline for Skyserver is effective on views.
We set up a new Recycler pipeline using the Feb-2010 default pipeline by adding "recycle" between "deadcode" and "reduce". Unfotunately, we were getting erratic results after running a few queries. For example, we ran some simple queries like select distinct x from y which returned couple dozen rows. After running a few other queries, the same select distinct x from y returned empty result set. We restarted the MonetDB server and got expected results for a while. But, after running some more quries, the simple select distinct x from y started to return null set. So, I think there are some issues with using recycle optimizer with the Feb-2010 default pipeline, at least the way I have used it. Or, does the recycler pipeline need to be set up with a different set of Feb-2010 optimizers than what I have included? One of the reasons that we are looking at using the recyler option is because we saw performance degradation going from the Nov-2009SP2 to Oct-2010SP1 for queries on views though we saw many improvements, in terms of performance and resource use, from upgrading. Is this a bug or are queries on views treated differently by design? Thanks!
Henry On Thu, Mar 3, 2011 at 2:35 PM, Martin Kersten Martin.Kersten@cwi.nlwrote:
Hi Henry
We would love to here experiences in using the recycler.
regards, Martin Kersten On 3/3/11 11:00 PM, Henry Addington wrote:
Hello: I understand some of the optimizers/pipelines are experimental as
indicated in the monetdb5.conf file. Has anyone used the Recycle pipeline for Skyserver? Would you mind sharing your experience, in terms of performance improvement and whether it can bet
used in a production environment? We are using the Oct2010-SP1 release
for a db that is read-only. I would guess if there are issues, I would think the problems would likely show up in read/write systems. Any feedback would be appreciated....
It looks like the Recycle pipeline for Skyserver pipeline is based on the
Nov-2009 pipeline. Has anyone tried the recycler optimizer with the Feb2010 default pipeline? I was going to add "recycle" between "deadcode" and "reduce" in the default pipeline and
see if it would work. Thanks. Henry
What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Hi Henry,
Thanks for your interest in using MonetDB and the recycler module. The recycler is an experimental code, that was exhaustively tested with both TPCH and SkyServer databases during its development. However, we haven't used it ourselves in a production environment. Therefore, your experiences are very valuable for us.
I need to check your recycler problems with Oct2010-SP1 version. Our experiments were done with november 2009 pipeline extended with recycle. It is possible that some interplay of recycler with new/modified optimisers (e.g. mitosis,mergetable) in the pipeline causes the problems you encountered. I am surprised about your observation of lack of effect on views, since in the SkyServer tests there were a number of queries over views where the recycler was efficient.
I will look at the problem using our databases. But it will be usefull to have some more details about your database. Would it be possible to provide me with table and view definitions, and the problematic queries. Ideally, some mini-subset of your database will also be useful.
With respect to the performance of queries over views in Oct2010, Martin and Stefan have more experiences and observations.
Regards, Milena
----- Original Message ----- From: "Henry Addington" hs.addington@gmail.com To: "Martin Kersten" Kersten@cwi.nl>, "Communication channel for MonetDB users" sourceforge.net> Sent: Monday, March 7, 2011 3:41:23 AM Subject: Re: [MonetDB-users] Recycler Optimizer
Hi Martin:
We tried the Recycle pipeline for Skyserver on the Oct2010-SP1 release. And, we didn't see any problems, though the testing was fairly short..... One thing we did notice is that queries against defined views do not seem to benefit from the recycler pipeline at all. Running a simple query, select distinct x from y, back to back seem to take just as long each time. So, I don't think the Recycler pipeline for Skyserver is effective on views.
We set up a new Recycler pipeline using the Feb-2010 default pipeline by adding "recycle" between "deadcode" and "reduce". Unfotunately, we were getting erratic results after running a few queries. For example, we ran some simple queries like select distinct x from y which returned couple dozen rows. After running a few other queries, the same select distinct x from y returned empty result set. We restarted the MonetDB server and got expected results for a while. But, after running some more quries, the simple select distinct x from y started to return null set. So, I think there are some issues with using recycle optimizer with the Feb-2010 default pipeline, at least the way I have used it. Or, does the recycler pipeline need to be set up with a different set of Feb-2010 optimizers than what I have included? One of the reasons that we are looking at using the recyler option is because we saw performance degradation going from the Nov-2009SP2 to Oct-2010SP1 for queries on views though we saw many improvements, in terms of performance and resource use, from upgrading. Is this a bug or are queries on views treated differently by design? Thanks!
Henry On Thu, Mar 3, 2011 at 2:35 PM, Martin Kersten Kersten@cwi.nl>wrote:
Hi Henry
We would love to here experiences in using the recycler.
regards, Martin Kersten On 3/3/11 11:00 PM, Henry Addington wrote:
Hello: I understand some of the optimizers/pipelines are experimental as
indicated in the monetdb5.conf file. Has anyone used the Recycle pipeline for Skyserver? Would you mind sharing your experience, in terms of performance improvement and whether it can bet
used in a production environment? We are using the Oct2010-SP1 release
for a db that is read-only. I would guess if there are issues, I would think the problems would likely show up in read/write systems. Any feedback would be appreciated....
It looks like the Recycle pipeline for Skyserver pipeline is based on the
Nov-2009 pipeline. Has anyone tried the recycler optimizer with the Feb2010 default pipeline? I was going to add "recycle" between "deadcode" and "reduce" in the default pipeline and
see if it would work. Thanks. Henry
What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Hi Milena:
I apologize for the slow response.
I think you certainly are right. We tried the recycler pipeline in Oct-2010 release and that worked just fine. So, I do suspect that there may be issues in recycler interacting with the new optimizers in the Oct-2010 release.
One of the things that we did since my last email was to rewrite the view that was the source of very slow queries. And, what a difference that has made. We are getting over 90% reduction in query time. So, we stopped looking at using the recycler optimizer and implemented the updated view when we upgraded our MonetDB to 2010Oct-SP1 release.
Let me see if I can duplicate the issue that we saw in our test environment. We will need to put the old view back to check. And, I will get back to you. I am not sure if we can get the schema and the data to you easily though ..... We will figure something out to get you what you need. Thanks!
Henry
On Wed, Mar 16, 2011 at 7:40 AM, Milena Ivanova Milena.Ivanova@cwi.nlwrote:
Hi Henry,
Thanks for your interest in using MonetDB and the recycler module. The recycler is an experimental code, that was exhaustively tested with both TPCH and SkyServer databases during its development. However, we haven't used it ourselves in a production environment. Therefore, your experiences are very valuable for us.
I need to check your recycler problems with Oct2010-SP1 version. Our experiments were done with november 2009 pipeline extended with recycle. It is possible that some interplay of recycler with new/modified optimisers (e.g. mitosis,mergetable) in the pipeline causes the problems you encountered. I am surprised about your observation of lack of effect on views, since in the SkyServer tests there were a number of queries over views where the recycler was efficient.
I will look at the problem using our databases. But it will be usefull to have some more details about your database. Would it be possible to provide me with table and view definitions, and the problematic queries. Ideally, some mini-subset of your database will also be useful.
With respect to the performance of queries over views in Oct2010, Martin and Stefan have more experiences and observations.
Regards, Milena
----- Original Message ----- From: "Henry Addington" hs.addington@gmail.com To: "Martin Kersten" Kersten@cwi.nl>, "Communication channel for MonetDB users" sourceforge.net> Sent: Monday, March 7, 2011 3:41:23 AM Subject: Re: [MonetDB-users] Recycler Optimizer
Hi Martin:
We tried the Recycle pipeline for Skyserver on the Oct2010-SP1 release. And, we didn't see any problems, though the testing was fairly short..... One thing we did notice is that queries against defined views do not seem to benefit from the recycler pipeline at all. Running a simple query, select distinct x from y, back to back seem to take just as long each time. So, I don't think the Recycler pipeline for Skyserver is effective on views.
We set up a new Recycler pipeline using the Feb-2010 default pipeline by adding "recycle" between "deadcode" and "reduce". Unfotunately, we were getting erratic results after running a few queries. For example, we ran some simple queries like select distinct x from y which returned couple dozen rows. After running a few other queries, the same select distinct x from y returned empty result set. We restarted the MonetDB server and got expected results for a while. But, after running some more quries, the simple select distinct x from y started to return null set. So, I think there are some issues with using recycle optimizer with the Feb-2010 default pipeline, at least the way I have used it. Or, does the recycler pipeline need to be set up with a different set of Feb-2010 optimizers than what I have included? One of the reasons that we are looking at using the recyler option is because we saw performance degradation going from the Nov-2009SP2 to Oct-2010SP1 for queries on views though we saw many improvements, in terms of performance and resource use, from upgrading. Is this a bug or are queries on views treated differently by design? Thanks!
Henry On Thu, Mar 3, 2011 at 2:35 PM, Martin Kersten Kersten@cwi.nl>wrote:
Hi Henry
We would love to here experiences in using the recycler.
regards, Martin Kersten On 3/3/11 11:00 PM, Henry Addington wrote:
Hello: I understand some of the optimizers/pipelines are experimental as
indicated in the monetdb5.conf file. Has anyone used the Recycle pipeline for Skyserver? Would you mind sharing your experience, in terms of performance improvement and whether it can bet
used in a production environment? We are using the Oct2010-SP1 release
for a db that is read-only. I would guess if there are issues, I would
think
the problems would likely show up in read/write systems. Any feedback
would
be appreciated....
It looks like the Recycle pipeline for Skyserver pipeline is based on
the
Nov-2009 pipeline. Has anyone tried the recycler optimizer with the
Feb2010
default pipeline? I was going to add "recycle" between "deadcode" and "reduce" in the default pipeline and
see if it would work. Thanks. Henry
What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
participants (3)
-
Henry Addington -
Martin Kersten -
Milena Ivanova