[Monetdb-developers] BAT-leak in gdk_logger.mx ?
Niels, Sjoerd, Peter,
after (quite) some time of debugging for [ 1976341 ] XQ: leftovers after deleting document http://sourceforge.net/tracker/index.php?func=detail&aid=1976341&gro...
I found the --- or at least one possible --- cause for this bug.
In pathfinder, newly shredded read-only documents are added to the logger using logger_add_bat() --- let's ignore for the time being, that I am a bit puzzled by this, since I though / was told, pathfinder uses check-pointing instead of the logger (WAL) for shredding documents.
Obviously, logger_add_bat() increments the ref-count of the logged BATs (BBPincref()).
However, logger_del_bat() does not seem to decrements the ref-count, again, once a BAT is removed from the logger. This way, document BATs keep the logger_add_bat()-increased ref-count even after a pf:del-doc() and stay around until the Mserver is shutdown.
The below patch seem to "fix" this problem. However, since I'm everything but familiar with the logger or the usage of it in either XQuery or SQL, I'd rather have you double-check the patch before I'd check it in. (Of course I'll also run "private" testing of the whole suite overnight to see whether my patch causes any (obvious) harm.
Thank you very much in advance!
Stefan
======== Index: MonetDB/src/gdk/gdk_logger.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_logger.mx,v retrieving revision 1.17.2.1 diff -u -r1.17.2.1 gdk_logger.mx --- MonetDB/src/gdk/gdk_logger.mx 27 May 2008 20:33:39 -0000 1.17.2.1 +++ MonetDB/src/gdk/gdk_logger.mx 3 Jun 2008 23:27:15 -0000 @@ -1721,6 +1721,7 @@ void logger_del_bat(logger *lg, log_bid bid) { + BBPdecref(bid, TRUE); BUNdelHead(lg->catalog, &bid, FALSE); }
========
Gents,
while working fine for pathfinder, my patch seems to harm some SQL tests:
======== @@ -722,2 +722,2 @@ -03:19:02> src/test/marcin1.sql (<=60,60,180) ... 0.111s -marcin1.stable.out.FILTERED and marcin1.test.out.FILTERED are equal. +01:52:39> src/test/marcin1.sql (<=60,60,180) ... 0.136s +marcin1.stable.out.FILTERED and marcin1.test.out.FILTERED differ SIGNIFICANTLY! @@ -1295,2 +1295,2 @@ -03:23:31> src/test/BugTracker/case_in_aggr_bug.SF-1506545.sql (<=60,60,180) ... 0.156s -case_in_aggr_bug.SF-1506545.stable.out.FILTERED and case_in_aggr_bug.SF-1506545.test.out.FILTERED are equal. +01:57:09> src/test/BugTracker/case_in_aggr_bug.SF-1506545.sql (<=60,60,180) ... 0.160s +case_in_aggr_bug.SF-1506545.stable.out.FILTERED and case_in_aggr_bug.SF-1506545.test.out.FILTERED differ SIGNIFICANTLY! @@ -1313,2 +1313,2 @@ -03:23:32> src/test/BugTracker/groupby_with_alias.SF-1520575.sql (<=60,60,180) ... 0.105s -groupby_with_alias.SF-1520575.stable.out.FILTERED and groupby_with_alias.SF-1520575.test.out.FILTERED are equal. +01:57:10> src/test/BugTracker/groupby_with_alias.SF-1520575.sql (<=60,60,180) ... 0.118s +groupby_with_alias.SF-1520575.stable.out.FILTERED and groupby_with_alias.SF-1520575.test.out.FILTERED differ SIGNIFICANTLY! @@ -1400,2 +1400,2 @@ -03:23:37> src/test/BugTracker/view_creation_using_func.SF-1714814.sql (<=60,60,180) ... 0.115s -view_creation_using_func.SF-1714814.stable.out.FILTERED and view_creation_using_func.SF-1714814.test.out.FILTERED are equal. +01:57:15> src/test/BugTracker/view_creation_using_func.SF-1714814.sql (<=60,60,180) ... 0.115s +view_creation_using_func.SF-1714814.stable.out.FILTERED and view_creation_using_func.SF-1714814.test.out.FILTERED differ SIGNIFICANTLY! @@ -1472,2 +1472,2 @@ -03:23:41> src/test/BugTracker/number_null_first_row.SF-1833286.sql (<=60,60,180) ... 0.118s -number_null_first_row.SF-1833286.stable.out.FILTERED and number_null_first_row.SF-1833286.test.out.FILTERED are equal. +01:57:20> src/test/BugTracker/number_null_first_row.SF-1833286.sql (<=60,60,180) ... 0.130s +number_null_first_row.SF-1833286.stable.out.FILTERED and number_null_first_row.SF-1833286.test.out.FILTERED differ SIGNIFICANTLY! @@ -2063,2 +2063,2 @@ -03:32:00> src/test/leaks/temp2.sql (<=60,60,180) ... 0.105s -temp2.stable.out.Five.FILTERED and temp2.test.out.FILTERED are equal. +02:05:39> src/test/leaks/temp2.sql (<=60,60,180) ... 0.105s +temp2.stable.out.Five.FILTERED and temp2.test.out.FILTERED differ SIGNIFICANTLY! @@ -2066,2 +2066,2 @@ -03:32:00> src/test/leaks/check3.sql (<=60,60,180) ... 0.104s -check3.stable.out.Five.FILTERED and check3.test.out.FILTERED are equal. +02:05:39> src/test/leaks/check3.sql (<=60,60,180) ... 0.105s +check3.stable.out.Five.FILTERED and check3.test.out.FILTERED differ SIGNIFICANTLY! @@ -2075,2 +2075,2 @@ -03:32:01> src/test/leaks/check4.sql (<=60,60,180) ... 0.104s -check4.stable.out.Five.FILTERED and check4.test.out.FILTERED are equal. +02:05:39> src/test/leaks/check4.sql (<=60,60,180) ... 0.105s +check4.stable.out.Five.FILTERED and check4.test.out.FILTERED differ SIGNIFICANTLY! @@ -2165 +2165 @@ - 23 out of 650 tests produced SIGNIFICANTLY different output + 31 out of 650 tests produced SIGNIFICANTLY different output @@ -2175,0 +2176,417 @@ ========
I still need to check wha the actual cause/problem is.
Stefan
On Wed, Jun 04, 2008 at 01:32:42AM +0200, Stefan Manegold wrote:
Niels, Sjoerd, Peter,
after (quite) some time of debugging for [ 1976341 ] XQ: leftovers after deleting document http://sourceforge.net/tracker/index.php?func=detail&aid=1976341&gro...
I found the --- or at least one possible --- cause for this bug.
In pathfinder, newly shredded read-only documents are added to the logger using logger_add_bat() --- let's ignore for the time being, that I am a bit puzzled by this, since I though / was told, pathfinder uses check-pointing instead of the logger (WAL) for shredding documents.
Obviously, logger_add_bat() increments the ref-count of the logged BATs (BBPincref()).
However, logger_del_bat() does not seem to decrements the ref-count, again, once a BAT is removed from the logger. This way, document BATs keep the logger_add_bat()-increased ref-count even after a pf:del-doc() and stay around until the Mserver is shutdown.
The below patch seem to "fix" this problem. However, since I'm everything but familiar with the logger or the usage of it in either XQuery or SQL, I'd rather have you double-check the patch before I'd check it in. (Of course I'll also run "private" testing of the whole suite overnight to see whether my patch causes any (obvious) harm.
Thank you very much in advance!
Stefan
======== Index: MonetDB/src/gdk/gdk_logger.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_logger.mx,v retrieving revision 1.17.2.1 diff -u -r1.17.2.1 gdk_logger.mx --- MonetDB/src/gdk/gdk_logger.mx 27 May 2008 20:33:39 -0000 1.17.2.1 +++ MonetDB/src/gdk/gdk_logger.mx 3 Jun 2008 23:27:15 -0000 @@ -1721,6 +1721,7 @@ void logger_del_bat(logger *lg, log_bid bid) {
- BBPdecref(bid, TRUE); BUNdelHead(lg->catalog, &bid, FALSE);
}
========
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
Gents,
with my logger patch, the 8 tests fail at server start-up with multiple " !ERROR: BBPdecref: tmp_XXX does not have logical references. "
Apparently, with SQL the extra ref-count is taken care of elsewhere!??
Hence, we need to look in more detail into the logger and its use in both SQL & XQuery ...
Any ideas?
Stefan
On Wed, Jun 04, 2008 at 07:32:55AM +0200, Stefan Manegold wrote:
Gents,
while working fine for pathfinder, my patch seems to harm some SQL tests:
======== @@ -722,2 +722,2 @@ -03:19:02> src/test/marcin1.sql (<=60,60,180) ... 0.111s -marcin1.stable.out.FILTERED and marcin1.test.out.FILTERED are equal. +01:52:39> src/test/marcin1.sql (<=60,60,180) ... 0.136s +marcin1.stable.out.FILTERED and marcin1.test.out.FILTERED differ SIGNIFICANTLY! @@ -1295,2 +1295,2 @@ -03:23:31> src/test/BugTracker/case_in_aggr_bug.SF-1506545.sql (<=60,60,180) ... 0.156s -case_in_aggr_bug.SF-1506545.stable.out.FILTERED and case_in_aggr_bug.SF-1506545.test.out.FILTERED are equal. +01:57:09> src/test/BugTracker/case_in_aggr_bug.SF-1506545.sql (<=60,60,180) ... 0.160s +case_in_aggr_bug.SF-1506545.stable.out.FILTERED and case_in_aggr_bug.SF-1506545.test.out.FILTERED differ SIGNIFICANTLY! @@ -1313,2 +1313,2 @@ -03:23:32> src/test/BugTracker/groupby_with_alias.SF-1520575.sql (<=60,60,180) ... 0.105s -groupby_with_alias.SF-1520575.stable.out.FILTERED and groupby_with_alias.SF-1520575.test.out.FILTERED are equal. +01:57:10> src/test/BugTracker/groupby_with_alias.SF-1520575.sql (<=60,60,180) ... 0.118s +groupby_with_alias.SF-1520575.stable.out.FILTERED and groupby_with_alias.SF-1520575.test.out.FILTERED differ SIGNIFICANTLY! @@ -1400,2 +1400,2 @@ -03:23:37> src/test/BugTracker/view_creation_using_func.SF-1714814.sql (<=60,60,180) ... 0.115s -view_creation_using_func.SF-1714814.stable.out.FILTERED and view_creation_using_func.SF-1714814.test.out.FILTERED are equal. +01:57:15> src/test/BugTracker/view_creation_using_func.SF-1714814.sql (<=60,60,180) ... 0.115s +view_creation_using_func.SF-1714814.stable.out.FILTERED and view_creation_using_func.SF-1714814.test.out.FILTERED differ SIGNIFICANTLY! @@ -1472,2 +1472,2 @@ -03:23:41> src/test/BugTracker/number_null_first_row.SF-1833286.sql (<=60,60,180) ... 0.118s -number_null_first_row.SF-1833286.stable.out.FILTERED and number_null_first_row.SF-1833286.test.out.FILTERED are equal. +01:57:20> src/test/BugTracker/number_null_first_row.SF-1833286.sql (<=60,60,180) ... 0.130s +number_null_first_row.SF-1833286.stable.out.FILTERED and number_null_first_row.SF-1833286.test.out.FILTERED differ SIGNIFICANTLY! @@ -2063,2 +2063,2 @@ -03:32:00> src/test/leaks/temp2.sql (<=60,60,180) ... 0.105s -temp2.stable.out.Five.FILTERED and temp2.test.out.FILTERED are equal. +02:05:39> src/test/leaks/temp2.sql (<=60,60,180) ... 0.105s +temp2.stable.out.Five.FILTERED and temp2.test.out.FILTERED differ SIGNIFICANTLY! @@ -2066,2 +2066,2 @@ -03:32:00> src/test/leaks/check3.sql (<=60,60,180) ... 0.104s -check3.stable.out.Five.FILTERED and check3.test.out.FILTERED are equal. +02:05:39> src/test/leaks/check3.sql (<=60,60,180) ... 0.105s +check3.stable.out.Five.FILTERED and check3.test.out.FILTERED differ SIGNIFICANTLY! @@ -2075,2 +2075,2 @@ -03:32:01> src/test/leaks/check4.sql (<=60,60,180) ... 0.104s -check4.stable.out.Five.FILTERED and check4.test.out.FILTERED are equal. +02:05:39> src/test/leaks/check4.sql (<=60,60,180) ... 0.105s +check4.stable.out.Five.FILTERED and check4.test.out.FILTERED differ SIGNIFICANTLY! @@ -2165 +2165 @@
- 23 out of 650 tests produced SIGNIFICANTLY different output
- 31 out of 650 tests produced SIGNIFICANTLY different output
@@ -2175,0 +2176,417 @@
I still need to check wha the actual cause/problem is.
Stefan
On Wed, Jun 04, 2008 at 01:32:42AM +0200, Stefan Manegold wrote:
Niels, Sjoerd, Peter,
after (quite) some time of debugging for [ 1976341 ] XQ: leftovers after deleting document http://sourceforge.net/tracker/index.php?func=detail&aid=1976341&gro...
I found the --- or at least one possible --- cause for this bug.
In pathfinder, newly shredded read-only documents are added to the logger using logger_add_bat() --- let's ignore for the time being, that I am a bit puzzled by this, since I though / was told, pathfinder uses check-pointing instead of the logger (WAL) for shredding documents.
Obviously, logger_add_bat() increments the ref-count of the logged BATs (BBPincref()).
However, logger_del_bat() does not seem to decrements the ref-count, again, once a BAT is removed from the logger. This way, document BATs keep the logger_add_bat()-increased ref-count even after a pf:del-doc() and stay around until the Mserver is shutdown.
The below patch seem to "fix" this problem. However, since I'm everything but familiar with the logger or the usage of it in either XQuery or SQL, I'd rather have you double-check the patch before I'd check it in. (Of course I'll also run "private" testing of the whole suite overnight to see whether my patch causes any (obvious) harm.
Thank you very much in advance!
Stefan
======== Index: MonetDB/src/gdk/gdk_logger.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_logger.mx,v retrieving revision 1.17.2.1 diff -u -r1.17.2.1 gdk_logger.mx --- MonetDB/src/gdk/gdk_logger.mx 27 May 2008 20:33:39 -0000 1.17.2.1 +++ MonetDB/src/gdk/gdk_logger.mx 3 Jun 2008 23:27:15 -0000 @@ -1721,6 +1721,7 @@ void logger_del_bat(logger *lg, log_bid bid) {
- BBPdecref(bid, TRUE); BUNdelHead(lg->catalog, &bid, FALSE);
}
========
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
LS,
a possible "HACK" to cure the sympthoms (not necessarily to heal the actual problem) would be to add a bit/boolean parameter to logger_del_bat() to control the use of BBPdecref() in logger_del_bat() from outside.
======== Index: MonetDB/src/gdk/gdk_logger.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_logger.mx,v retrieving revision 1.17.2.1 diff -u -r1.17.2.1 gdk_logger.mx --- MonetDB/src/gdk/gdk_logger.mx 27 May 2008 20:33:39 -0000 1.17.2.1 +++ MonetDB/src/gdk/gdk_logger.mx 4 Jun 2008 08:14:05 -0000 @@ -158,7 +158,7 @@ gdk_export int log_sequence(logger *lg, int seq, lng id);
gdk_export log_bid logger_add_bat(logger *lg, BAT *b, char *name); -gdk_export void logger_del_bat(logger *lg, log_bid bid); +gdk_export void logger_del_bat(logger *lg, log_bid bid, bit decref); gdk_export log_bid logger_find_bat(logger *lg, char *name);
#endif /*_LOGGER_H_*/ @@ -517,7 +517,7 @@ log_bid bid = logger_find_bat(lg, la->name);
if (bid) - logger_del_bat(lg, bid); + logger_del_bat(lg, bid, FALSE); }
static int @@ -1719,8 +1719,10 @@ }
void -logger_del_bat(logger *lg, log_bid bid) +logger_del_bat(logger *lg, log_bid bid, bit decref) { + if (decref == TRUE) + BBPdecref(bid, TRUE); BUNdelHead(lg->catalog, &bid, FALSE); }
========
SQL would call loger_del_bat with FALSE to maintain the current behavior, while pathfinder would call it with TRUE to enforce the BBPdecref().
Stefan
On Wed, Jun 04, 2008 at 07:58:50AM +0200, Stefan Manegold wrote:
Gents,
with my logger patch, the 8 tests fail at server start-up with multiple " !ERROR: BBPdecref: tmp_XXX does not have logical references. "
Apparently, with SQL the extra ref-count is taken care of elsewhere!??
Hence, we need to look in more detail into the logger and its use in both SQL & XQuery ...
Any ideas?
Stefan
On Wed, Jun 04, 2008 at 07:32:55AM +0200, Stefan Manegold wrote:
Gents,
while working fine for pathfinder, my patch seems to harm some SQL tests:
======== @@ -722,2 +722,2 @@ -03:19:02> src/test/marcin1.sql (<=60,60,180) ... 0.111s -marcin1.stable.out.FILTERED and marcin1.test.out.FILTERED are equal. +01:52:39> src/test/marcin1.sql (<=60,60,180) ... 0.136s +marcin1.stable.out.FILTERED and marcin1.test.out.FILTERED differ SIGNIFICANTLY! @@ -1295,2 +1295,2 @@ -03:23:31> src/test/BugTracker/case_in_aggr_bug.SF-1506545.sql (<=60,60,180) ... 0.156s -case_in_aggr_bug.SF-1506545.stable.out.FILTERED and case_in_aggr_bug.SF-1506545.test.out.FILTERED are equal. +01:57:09> src/test/BugTracker/case_in_aggr_bug.SF-1506545.sql (<=60,60,180) ... 0.160s +case_in_aggr_bug.SF-1506545.stable.out.FILTERED and case_in_aggr_bug.SF-1506545.test.out.FILTERED differ SIGNIFICANTLY! @@ -1313,2 +1313,2 @@ -03:23:32> src/test/BugTracker/groupby_with_alias.SF-1520575.sql (<=60,60,180) ... 0.105s -groupby_with_alias.SF-1520575.stable.out.FILTERED and groupby_with_alias.SF-1520575.test.out.FILTERED are equal. +01:57:10> src/test/BugTracker/groupby_with_alias.SF-1520575.sql (<=60,60,180) ... 0.118s +groupby_with_alias.SF-1520575.stable.out.FILTERED and groupby_with_alias.SF-1520575.test.out.FILTERED differ SIGNIFICANTLY! @@ -1400,2 +1400,2 @@ -03:23:37> src/test/BugTracker/view_creation_using_func.SF-1714814.sql (<=60,60,180) ... 0.115s -view_creation_using_func.SF-1714814.stable.out.FILTERED and view_creation_using_func.SF-1714814.test.out.FILTERED are equal. +01:57:15> src/test/BugTracker/view_creation_using_func.SF-1714814.sql (<=60,60,180) ... 0.115s +view_creation_using_func.SF-1714814.stable.out.FILTERED and view_creation_using_func.SF-1714814.test.out.FILTERED differ SIGNIFICANTLY! @@ -1472,2 +1472,2 @@ -03:23:41> src/test/BugTracker/number_null_first_row.SF-1833286.sql (<=60,60,180) ... 0.118s -number_null_first_row.SF-1833286.stable.out.FILTERED and number_null_first_row.SF-1833286.test.out.FILTERED are equal. +01:57:20> src/test/BugTracker/number_null_first_row.SF-1833286.sql (<=60,60,180) ... 0.130s +number_null_first_row.SF-1833286.stable.out.FILTERED and number_null_first_row.SF-1833286.test.out.FILTERED differ SIGNIFICANTLY! @@ -2063,2 +2063,2 @@ -03:32:00> src/test/leaks/temp2.sql (<=60,60,180) ... 0.105s -temp2.stable.out.Five.FILTERED and temp2.test.out.FILTERED are equal. +02:05:39> src/test/leaks/temp2.sql (<=60,60,180) ... 0.105s +temp2.stable.out.Five.FILTERED and temp2.test.out.FILTERED differ SIGNIFICANTLY! @@ -2066,2 +2066,2 @@ -03:32:00> src/test/leaks/check3.sql (<=60,60,180) ... 0.104s -check3.stable.out.Five.FILTERED and check3.test.out.FILTERED are equal. +02:05:39> src/test/leaks/check3.sql (<=60,60,180) ... 0.105s +check3.stable.out.Five.FILTERED and check3.test.out.FILTERED differ SIGNIFICANTLY! @@ -2075,2 +2075,2 @@ -03:32:01> src/test/leaks/check4.sql (<=60,60,180) ... 0.104s -check4.stable.out.Five.FILTERED and check4.test.out.FILTERED are equal. +02:05:39> src/test/leaks/check4.sql (<=60,60,180) ... 0.105s +check4.stable.out.Five.FILTERED and check4.test.out.FILTERED differ SIGNIFICANTLY! @@ -2165 +2165 @@
- 23 out of 650 tests produced SIGNIFICANTLY different output
- 31 out of 650 tests produced SIGNIFICANTLY different output
@@ -2175,0 +2176,417 @@
I still need to check wha the actual cause/problem is.
Stefan
On Wed, Jun 04, 2008 at 01:32:42AM +0200, Stefan Manegold wrote:
Niels, Sjoerd, Peter,
after (quite) some time of debugging for [ 1976341 ] XQ: leftovers after deleting document http://sourceforge.net/tracker/index.php?func=detail&aid=1976341&gro...
I found the --- or at least one possible --- cause for this bug.
In pathfinder, newly shredded read-only documents are added to the logger using logger_add_bat() --- let's ignore for the time being, that I am a bit puzzled by this, since I though / was told, pathfinder uses check-pointing instead of the logger (WAL) for shredding documents.
Obviously, logger_add_bat() increments the ref-count of the logged BATs (BBPincref()).
However, logger_del_bat() does not seem to decrements the ref-count, again, once a BAT is removed from the logger. This way, document BATs keep the logger_add_bat()-increased ref-count even after a pf:del-doc() and stay around until the Mserver is shutdown.
The below patch seem to "fix" this problem. However, since I'm everything but familiar with the logger or the usage of it in either XQuery or SQL, I'd rather have you double-check the patch before I'd check it in. (Of course I'll also run "private" testing of the whole suite overnight to see whether my patch causes any (obvious) harm.
Thank you very much in advance!
Stefan
======== Index: MonetDB/src/gdk/gdk_logger.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_logger.mx,v retrieving revision 1.17.2.1 diff -u -r1.17.2.1 gdk_logger.mx --- MonetDB/src/gdk/gdk_logger.mx 27 May 2008 20:33:39 -0000 1.17.2.1 +++ MonetDB/src/gdk/gdk_logger.mx 3 Jun 2008 23:27:15 -0000 @@ -1721,6 +1721,7 @@ void logger_del_bat(logger *lg, log_bid bid) {
- BBPdecref(bid, TRUE); BUNdelHead(lg->catalog, &bid, FALSE);
}
========
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
On Wed, Jun 04, 2008 at 10:18:53AM +0200, Stefan Manegold wrote:
LS,
a possible "HACK" to cure the sympthoms (not necessarily to heal the actual problem) would be to add a bit/boolean parameter to logger_del_bat() to control the use of BBPdecref() in logger_del_bat() from outside.
Thats not correct. The query isn't committed yet, ie you still need to be able to abort (aren't transactions lovely ;-).
Niels
======== Index: MonetDB/src/gdk/gdk_logger.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_logger.mx,v retrieving revision 1.17.2.1 diff -u -r1.17.2.1 gdk_logger.mx --- MonetDB/src/gdk/gdk_logger.mx 27 May 2008 20:33:39 -0000 1.17.2.1 +++ MonetDB/src/gdk/gdk_logger.mx 4 Jun 2008 08:14:05 -0000 @@ -158,7 +158,7 @@ gdk_export int log_sequence(logger *lg, int seq, lng id);
gdk_export log_bid logger_add_bat(logger *lg, BAT *b, char *name); -gdk_export void logger_del_bat(logger *lg, log_bid bid); +gdk_export void logger_del_bat(logger *lg, log_bid bid, bit decref); gdk_export log_bid logger_find_bat(logger *lg, char *name);
#endif /*_LOGGER_H_*/ @@ -517,7 +517,7 @@ log_bid bid = logger_find_bat(lg, la->name);
if (bid)
logger_del_bat(lg, bid);
logger_del_bat(lg, bid, FALSE);}
static int @@ -1719,8 +1719,10 @@ }
void -logger_del_bat(logger *lg, log_bid bid) +logger_del_bat(logger *lg, log_bid bid, bit decref) {
- if (decref == TRUE)
BBPdecref(bid, TRUE);}
========
SQL would call loger_del_bat with FALSE to maintain the current behavior, while pathfinder would call it with TRUE to enforce the BBPdecref().
Stefan
On Wed, Jun 04, 2008 at 07:58:50AM +0200, Stefan Manegold wrote:
Gents,
with my logger patch, the 8 tests fail at server start-up with multiple " !ERROR: BBPdecref: tmp_XXX does not have logical references. "
Apparently, with SQL the extra ref-count is taken care of elsewhere!??
Hence, we need to look in more detail into the logger and its use in both SQL & XQuery ...
Any ideas?
Stefan
On Wed, Jun 04, 2008 at 07:32:55AM +0200, Stefan Manegold wrote:
Gents,
while working fine for pathfinder, my patch seems to harm some SQL tests:
======== @@ -722,2 +722,2 @@ -03:19:02> src/test/marcin1.sql (<=60,60,180) ... 0.111s -marcin1.stable.out.FILTERED and marcin1.test.out.FILTERED are equal. +01:52:39> src/test/marcin1.sql (<=60,60,180) ... 0.136s +marcin1.stable.out.FILTERED and marcin1.test.out.FILTERED differ SIGNIFICANTLY! @@ -1295,2 +1295,2 @@ -03:23:31> src/test/BugTracker/case_in_aggr_bug.SF-1506545.sql (<=60,60,180) ... 0.156s -case_in_aggr_bug.SF-1506545.stable.out.FILTERED and case_in_aggr_bug.SF-1506545.test.out.FILTERED are equal. +01:57:09> src/test/BugTracker/case_in_aggr_bug.SF-1506545.sql (<=60,60,180) ... 0.160s +case_in_aggr_bug.SF-1506545.stable.out.FILTERED and case_in_aggr_bug.SF-1506545.test.out.FILTERED differ SIGNIFICANTLY! @@ -1313,2 +1313,2 @@ -03:23:32> src/test/BugTracker/groupby_with_alias.SF-1520575.sql (<=60,60,180) ... 0.105s -groupby_with_alias.SF-1520575.stable.out.FILTERED and groupby_with_alias.SF-1520575.test.out.FILTERED are equal. +01:57:10> src/test/BugTracker/groupby_with_alias.SF-1520575.sql (<=60,60,180) ... 0.118s +groupby_with_alias.SF-1520575.stable.out.FILTERED and groupby_with_alias.SF-1520575.test.out.FILTERED differ SIGNIFICANTLY! @@ -1400,2 +1400,2 @@ -03:23:37> src/test/BugTracker/view_creation_using_func.SF-1714814.sql (<=60,60,180) ... 0.115s -view_creation_using_func.SF-1714814.stable.out.FILTERED and view_creation_using_func.SF-1714814.test.out.FILTERED are equal. +01:57:15> src/test/BugTracker/view_creation_using_func.SF-1714814.sql (<=60,60,180) ... 0.115s +view_creation_using_func.SF-1714814.stable.out.FILTERED and view_creation_using_func.SF-1714814.test.out.FILTERED differ SIGNIFICANTLY! @@ -1472,2 +1472,2 @@ -03:23:41> src/test/BugTracker/number_null_first_row.SF-1833286.sql (<=60,60,180) ... 0.118s -number_null_first_row.SF-1833286.stable.out.FILTERED and number_null_first_row.SF-1833286.test.out.FILTERED are equal. +01:57:20> src/test/BugTracker/number_null_first_row.SF-1833286.sql (<=60,60,180) ... 0.130s +number_null_first_row.SF-1833286.stable.out.FILTERED and number_null_first_row.SF-1833286.test.out.FILTERED differ SIGNIFICANTLY! @@ -2063,2 +2063,2 @@ -03:32:00> src/test/leaks/temp2.sql (<=60,60,180) ... 0.105s -temp2.stable.out.Five.FILTERED and temp2.test.out.FILTERED are equal. +02:05:39> src/test/leaks/temp2.sql (<=60,60,180) ... 0.105s +temp2.stable.out.Five.FILTERED and temp2.test.out.FILTERED differ SIGNIFICANTLY! @@ -2066,2 +2066,2 @@ -03:32:00> src/test/leaks/check3.sql (<=60,60,180) ... 0.104s -check3.stable.out.Five.FILTERED and check3.test.out.FILTERED are equal. +02:05:39> src/test/leaks/check3.sql (<=60,60,180) ... 0.105s +check3.stable.out.Five.FILTERED and check3.test.out.FILTERED differ SIGNIFICANTLY! @@ -2075,2 +2075,2 @@ -03:32:01> src/test/leaks/check4.sql (<=60,60,180) ... 0.104s -check4.stable.out.Five.FILTERED and check4.test.out.FILTERED are equal. +02:05:39> src/test/leaks/check4.sql (<=60,60,180) ... 0.105s +check4.stable.out.Five.FILTERED and check4.test.out.FILTERED differ SIGNIFICANTLY! @@ -2165 +2165 @@
- 23 out of 650 tests produced SIGNIFICANTLY different output
- 31 out of 650 tests produced SIGNIFICANTLY different output
@@ -2175,0 +2176,417 @@
I still need to check wha the actual cause/problem is.
Stefan
On Wed, Jun 04, 2008 at 01:32:42AM +0200, Stefan Manegold wrote:
Niels, Sjoerd, Peter,
after (quite) some time of debugging for [ 1976341 ] XQ: leftovers after deleting document http://sourceforge.net/tracker/index.php?func=detail&aid=1976341&gro...
I found the --- or at least one possible --- cause for this bug.
In pathfinder, newly shredded read-only documents are added to the logger using logger_add_bat() --- let's ignore for the time being, that I am a bit puzzled by this, since I though / was told, pathfinder uses check-pointing instead of the logger (WAL) for shredding documents.
Obviously, logger_add_bat() increments the ref-count of the logged BATs (BBPincref()).
However, logger_del_bat() does not seem to decrements the ref-count, again, once a BAT is removed from the logger. This way, document BATs keep the logger_add_bat()-increased ref-count even after a pf:del-doc() and stay around until the Mserver is shutdown.
The below patch seem to "fix" this problem. However, since I'm everything but familiar with the logger or the usage of it in either XQuery or SQL, I'd rather have you double-check the patch before I'd check it in. (Of course I'll also run "private" testing of the whole suite overnight to see whether my patch causes any (obvious) harm.
Thank you very much in advance!
Stefan
======== Index: MonetDB/src/gdk/gdk_logger.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_logger.mx,v retrieving revision 1.17.2.1 diff -u -r1.17.2.1 gdk_logger.mx --- MonetDB/src/gdk/gdk_logger.mx 27 May 2008 20:33:39 -0000 1.17.2.1 +++ MonetDB/src/gdk/gdk_logger.mx 3 Jun 2008 23:27:15 -0000 @@ -1721,6 +1721,7 @@ void logger_del_bat(logger *lg, log_bid bid) {
- BBPdecref(bid, TRUE); BUNdelHead(lg->catalog, &bid, FALSE);
}
========
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
participants (2)
-
Niels Nes -
Stefan Manegold