August 2007 - developers-list

[Monetdb-developers] glib - double free or corruption
by Agustin Schapira 08 Aug '07

08 Aug '07

Dear Monet developers, We're having trouble with a script that uses the {} operator to compute the avg of a large (4M rows) table: we get a 'double free' error from glib, and Monet crashes. This is using Monet 4.16.2 compiled for 64-bits and 32-bit oids on Linux. (BTW, it also happens if we use {sum}, but not if we use {count}) The code takes two tables, links and attr, and does the equivalent of SELECT avg(attr.value) FROM links, attr WHERE links.id = attr.id GROUP BY links.from Here's the MIL code: # Get the BATs var var_attr:=bat(bat("prox_link_attr").fetch(2)).find(oid(0)); var var_attr_id:=bat(bat(var_attr).fetch(0)); var var_attr_val:=bat(bat(var_attr).fetch(1)); var var_link_id:=bat(bat("prox_link").fetch(0)); var var_link_from:=bat(bat("prox_link").fetch(1)); # Join ATTR x LINK, keep LINK.from, ATTR.val var var_1:=var_link_id.join(var_attr_id.reverse()); var var_2:=var_1.mark(0@0); var var_3:=var_1.reverse().mark(0@0); var var_5:=var_2.reverse().join(var_link_from); var var_8:=var_3.reverse().join(var_attr_val); # GROUP BY LINK.from and compute AVG(ATTR.val) var var_9:=var_5.reverse().join(var_8); var_5.info().print(); var_8.info().print(); var_9.info().print(); var var_10:={avg}(var_9); At the end of the last statement, Monet crashes with the error: *** glibc detected *** double free or corruption (!prev): 0x0000000000632080 *** I've run this using gdb; below is the trace, including the information printed by the .info() calls at the end of the script, and some extra info that is printed with debugmask(32 and 131072). Any thoughts? Any more information that you would need to debug this problem? Thanks a lot, -- Agustin ------------ $ gdb --args /usr/local/Monet-venus-debug/bin/Mserver --dbname xxx ~/ test.mil GNU gdb Red Hat Linux (6.3.0.0-1.143.el4rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1". (gdb) run Starting program: /usr/local/Monet-venus-debug/bin/Mserver --dbname xxx /home/aschapira/test.mil [Thread debugging using libthread_db enabled] [New Thread 182918188384 (LWP 14037)] [New Thread 1082132832 (LWP 14040)] # Monet Database Server V4.16.2 # Copyright (c) 1993-2007, CWI. All rights reserved. # Compiled for x86_64-redhat-linux-gnu/64bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. #-----------------------------------------# # h t # name # str str # type #-----------------------------------------# [ "version", "25105" ] [ "batId", "tmp_41" ] [ "batCacheid", "-33" ] [ "batParentid", "0" ] [ "batSharecnt", "0" ] [ "head", "oid" ] [ "tail", "oid" ] [ "batPersistence", "transient" ] [ "batRestricted", "read-only" ] [ "batRefcnt", "1" ] [ "batLRefcnt", "1" ] [ "batDirty", "dirty" ] [ "batSet", "0" ] [ "void_tid", "0" ] [ "void_cnt", "0" ] [ "hsorted", "65" ] [ "hident", "t" ] [ "hdense", "1" ] [ "hseqbase", "0@0" ] [ "hkey", "1" ] [ "hloc", "4" ] [ "hvarsized", "0" ] [ "halign", "1008981" ] [ "hnosorted", "0" ] [ "hnosorted_rev", "0" ] [ "hnodense", "0" ] [ "hnokey[0]", "0" ] [ "hnokey[1]", "0" ] [ "tident", "h" ] [ "tdense", "0" ] [ "tseqbase", "nil" ] [ "tsorted", "0" ] [ "tkey", "0" ] [ "tloc", "0" ] [ "tvarsized", "0" ] [ "talign", "1005793" ] [ "tnosorted", "58570" ] [ "tnosorted_rev", "0" ] [ "tnodense", "0" ] [ "tnokey[0]", "0" ] [ "tnokey[1]", "1" ] [ "batInserted", "0" ] [ "batDeleted", "0" ] [ "batFirst", "0" ] [ "top", "4180239" ] [ "batStamp", "-73" ] [ "lastUsed", "24012" ] [ "curStamp", "86" ] [ "batCopiedtodisk", "0" ] [ "batDirtydesc", "dirty" ] [ "batDirtybuns", "clean" ] [ "batBuns.free", "33441912" ] [ "batBuns.size", "33441912" ] [ "batBuns.maxsize", "40173552" ] [ "batBuns.storage", "malloced" ] [ "batBuns.filename", "41.buns" ] [ "hheapdirty", "clean" ] [ "theapdirty", "clean" ] #-----------------------------------------# # h t # name # str str # type #-----------------------------------------# [ "version", "25105" ] [ "batId", "tmp_42" ] [ "batCacheid", "34" ] [ "batParentid", "0" ] [ "batSharecnt", "0" ] [ "head", "oid" ] [ "tail", "int" ] [ "batPersistence", "transient" ] [ "batRestricted", "read-only" ] [ "batRefcnt", "1" ] [ "batLRefcnt", "1" ] [ "batDirty", "dirty" ] [ "batSet", "0" ] [ "void_tid", "-1" ] [ "void_cnt", "0" ] [ "hsorted", "65" ] [ "hident", "h" ] [ "hdense", "1" ] [ "hseqbase", "0@0" ] [ "hkey", "1" ] [ "hloc", "0" ] [ "hvarsized", "0" ] [ "halign", "1009002" ] [ "hnosorted", "0" ] [ "hnosorted_rev", "0" ] [ "hnodense", "0" ] [ "hnokey[0]", "0" ] [ "hnokey[1]", "0" ] [ "tident", "t" ] [ "tdense", "0" ] [ "tseqbase", "0@0" ] [ "tsorted", "0" ] [ "tkey", "0" ] [ "tloc", "4" ] [ "tvarsized", "0" ] [ "talign", "1009003" ] [ "tnosorted", "0" ] [ "tnosorted_rev", "0" ] [ "tnodense", "0" ] [ "tnokey[0]", "0" ] [ "tnokey[1]", "0" ] [ "batInserted", "0" ] [ "batDeleted", "0" ] [ "batFirst", "0" ] [ "top", "4180239" ] [ "batStamp", "-84" ] [ "lastUsed", "24028" ] [ "curStamp", "86" ] [ "batCopiedtodisk", "0" ] [ "batDirtydesc", "dirty" ] [ "batDirtybuns", "clean" ] [ "batBuns.free", "33441912" ] [ "batBuns.size", "35202008" ] [ "batBuns.maxsize", "42270704" ] [ "batBuns.storage", "malloced" ] [ "batBuns.filename", "42.buns" ] [ "hheapdirty", "clean" ] [ "theapdirty", "clean" ] #-----------------------------------------# # h t # name # str str # type #-----------------------------------------# [ "version", "25105" ] [ "batId", "tmp_43" ] [ "batCacheid", "-35" ] [ "batParentid", "0" ] [ "batSharecnt", "0" ] [ "head", "oid" ] [ "tail", "int" ] [ "batPersistence", "transient" ] [ "batRestricted", "read-only" ] [ "batRefcnt", "1" ] [ "batLRefcnt", "1" ] [ "batDirty", "dirty" ] [ "batSet", "0" ] [ "void_tid", "6122832" ] [ "void_cnt", "0" ] [ "hsorted", "0" ] [ "hident", "t" ] [ "hdense", "0" ] [ "hseqbase", "nil" ] [ "hkey", "0" ] [ "hloc", "4" ] [ "hvarsized", "0" ] [ "halign", "1005793" ] [ "hnosorted", "58570" ] [ "hnosorted_rev", "0" ] [ "hnodense", "0" ] [ "hnokey[0]", "0" ] [ "hnokey[1]", "1" ] [ "tident", "h" ] [ "tdense", "0" ] [ "tseqbase", "0@0" ] [ "tsorted", "0" ] [ "tkey", "0" ] [ "tloc", "0" ] [ "tvarsized", "0" ] [ "talign", "1009003" ] [ "tnosorted", "0" ] [ "tnosorted_rev", "0" ] [ "tnodense", "0" ] [ "tnokey[0]", "0" ] [ "tnokey[1]", "0" ] [ "batInserted", "0" ] [ "batDeleted", "0" ] [ "batFirst", "0" ] [ "top", "4180239" ] [ "batStamp", "-85" ] [ "lastUsed", "24043" ] [ "curStamp", "86" ] [ "batCopiedtodisk", "0" ] [ "batDirtydesc", "dirty" ] [ "batDirtybuns", "clean" ] [ "batBuns.free", "33441912" ] [ "batBuns.size", "33441912" ] [ "batBuns.maxsize", "40173552" ] [ "batBuns.storage", "malloced" ] [ "batBuns.filename", "43.buns" ] [ "hheapdirty", "clean" ] [ "theapdirty", "clean" ] *** double free or corruption (!prev): 0x0000000000632080 *** Program received signal SIGABRT, Aborted. [Switching to Thread 182918188384 (LWP 14037)] 0x0000003a7b72e21d in raise () from /lib64/tls/libc.so.6 (gdb) bt #0 0x0000003a7b72e21d in raise () from /lib64/tls/libc.so.6 #1 0x0000003a7b72fa1e in abort () from /lib64/tls/libc.so.6 #2 0x0000003a7b763451 in __libc_message () from /lib64/tls/libc.so.6 #3 0x0000003a7b76906e in _int_free () from /lib64/tls/libc.so.6 #4 0x0000003a7b7693b6 in free () from /lib64/tls/libc.so.6 #5 0x0000002a96665c43 in GDKfree (blk=0x632088) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB/src/ gdk/gdk_utils.mx:1121 #6 0x0000002a965ea1cf in HEAPfree (h=0x631940) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB/src/ gdk/gdk_heap.mx:264 #7 0x0000002a966dc693 in BATdelete (b=0x631828) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB/src/ gdk/gdk_storage.mx:716 #8 0x0000002a965e5e6d in BBPaddtobin (b=0x631828) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB/src/ gdk/gdk_bbp.mx:2366 #9 0x0000002a965e252a in BBPdestroy (b=0x631828) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB/src/ gdk/gdk_bbp.mx:1665 #10 0x0000002a965e12b1 in BBPreclaim (b=0x631828) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB/src/ gdk/gdk_bbp.mx:1514 #11 0x0000002a95ef5e0f in interpret_setaggr (name=0x62a038 "{avg}", argc=2, argv=0x620c18, res=0x7fbffff790, tt=0x631078, stk=1) at /export/scratch0/monet/monet.GNU.64.64.d.14791/ MonetDB4/src/monet/monet_multiplex.mx:1819 #12 0x0000002a95e95c57 in interpret (stk=1, lt=0x625090, res=0x7fbffff790) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB4/src/ monet/monet_interpreter.mx:1202 #13 0x0000002a95e9da04 in interpret_assignment (stk=1, lt=0x625040, res=0x7fbffff790) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB4/src/ monet/monet_interpreter.mx:1842 #14 0x0000002a95e97349 in interpret_var (stk=1, lt=0x625018, res=0x7fbffff790) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB4/src/ monet/monet_interpreter.mx:1329 #15 0x0000002a95e93e30 in interpret (stk=1, lt=0x625018, res=0x7fbffff790) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB4/src/ monet/monet_interpreter.mx:832 #16 0x0000002a95e9dca3 in interpret_seqblock (stk=1, lt=0x5ca8b8, res=0x7fbffff790, scope=0) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB4/src/ monet/monet_interpreter.mx:1892 #17 0x0000002a95e93606 in interpret (stk=1, lt=0x629478, res=0x7fbffff790) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB4/src/ monet/monet_interpreter.mx:770 #18 0x0000002a95e9173d in interpret_str (stk=0, buf=0x629908 "# Get the BATs\nvar var_attr:=bat(bat (\"prox_link_attr\").fetch(2)).find(oid(0));\nvar var_attr_id:=bat(bat (var_attr).fetch(0));\nvar var_attr_val:=bat(bat(var_attr).fetch(1)); \n\nvar var_link_id:=bat(bat(\"p"..., res=0x7fbffff790) at /export/scratch0/monet/monet.GNU.64.64.d. 14791/MonetDB4/src/monet/monet_interpreter.mx:246 #19 0x0000002a95e91bf0 in interpret_file (stk=0, lt=0x5411c8, res=0x7fbffff790) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB4/src/ monet/monet_interpreter.mx:291 #20 0x0000002a95e93f39 in interpret (stk=0, lt=0x5411c8, res=0x7fbffff790) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB4/src/ monet/monet_interpreter.mx:858 #21 0x0000002a95ebf65e in handleRequest (t=0x2a96a46840, q=0x64d5a8, res=0x7fbffff790) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB4/src/ monet/monet_queue.mx:537 ---Type <return> to continue, or q <return> to quit--- #22 0x0000002a95ebfadb in doRequest (t=0x2a96a46840, preference=0x0) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB4/src/ monet/monet_queue.mx:563 #23 0x0000002a95f1108c in monetInterpreter (status=0x7fbffff7f8) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB4/src/ monet/monet_process.mx:112 #24 0x0000000000402674 in main (argc=4, av=0x7fbffff908) at /export/scratch0/monet/monet.GNU.64.64.d.14791/MonetDB4/src/ tools/Mserver.mx:403 When I run this with debugmask(32 + 131072), I get the following debug output as well. This is right after the calls to .info(), during the processing of {avg}: #interpret_unpin(print) on bat(59) refcnt = 0 ##60 = new tmp_74(int,int) ##BBPreclaim: bat(60) view=0 lrefs=0 ref=1 stat=1 #interpret_pin({avg}) on bat(-35) refcnt = 1 ##61 = new tmp_75(oid,int) ##62 = new tmp_76(oid,void) ##65 = new tmp_101(oid,void) ##66 = new tmp_102(oid,void) ##67 = new tmp_103(oid,void) ##BBPreclaim: bat(67) view=0 lrefs=0 ref=1 stat=1 ##clear 67 (tmp_103) ##uncache 67 (tmp_103) ##BBPreclaim: bat(66) view=0 lrefs=0 ref=1 stat=1 ##clear 66 (tmp_102) ##uncache 66 (tmp_102) ##BBPreclaim: bat(65) view=0 lrefs=0 ref=1 stat=1 ##clear 65 (tmp_101) ##uncache 65 (tmp_101) ##BBPreclaim: bat(62) view=0 lrefs=0 ref=1 stat=1 ##clear 62 (tmp_76) ##uncache 62 (tmp_76) ##BBPreclaim: bat(61) view=0 lrefs=0 ref=1 stat=1 ##clear 61 (tmp_75) ##uncache 61 (tmp_75) #setaggr impl: non-optimized hash #interpret_pin(count) on bat(61) refcnt = 2 #interpret_unpin(count) on bat(61) refcnt = 1 #interpret_pin(sum_lng) on bat(61) refcnt = 2 #interpret_unpin(sum_lng) on bat(61) refcnt = 1 ... [these 4 lines repeat 20,000 times, which corresponds to the number of unique values in the HEAD of var_9] ... ##BBPreclaim: bat(65) view=0 lrefs=0 ref=1 stat=1 ##clear 65 (tmp_101) ##uncache 65 (tmp_101) ##BBPreclaim: bat(61) view=0 lrefs=0 ref=1 stat=1 I hope this helps. Thanks again.

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_bbp.mx, , 1.201, 1.202
by Sjoerd Mullender 06 Aug '07

06 Aug '07

Should this be back-ported to the stable branch? Peter Boncz wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv30287/src/gdk > > Modified Files: > gdk_bbp.mx > Log Message: > - race condition in BBOclear > > > > Index: gdk_bbp.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_bbp.mx,v > retrieving revision 1.201 > retrieving revision 1.202 > diff -u -d -r1.201 -r1.202 > --- gdk_bbp.mx 22 Jun 2007 06:46:53 -0000 1.201 > +++ gdk_bbp.mx 5 Aug 2007 23:00:50 -0000 1.202 > @@ -1116,7 +1116,7 @@ > THRprintf(GDKout, "#clear %d (%s)\n", (int) i, BBPname(i)); > } > BBPuncacheit_(i, TRUE); > - BBP_status_set(i, 0, "BBPclear"); > + BBP_status_set(i, BBPUNLOADING, "BBPclear"); > BBP_refs(i) = 0; > BBP_lrefs(i) = 0; > if (lock) > @@ -1137,6 +1137,7 @@ > } > if (BBP_logical(i) != BBP[ABS(i)].bak[(i) < 0]) > GDKfree(BBP_logical(i)); > + BBP_status_set(i, 0, "BBPclear"); > BBP_logical(i) = NULL; > BBP_next(i) = BBP_free(idx); > BBP_free(idx) = i; > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

1 0

[Monetdb-developers] Unimplemented XQuery standard functions
by Djoerd Hiemstra 03 Aug '07

03 Aug '07

Dear all, For demo purposes we are in need for implementations of some XQuery standard functions, especially fn:tokenize() and fn:base-uri(). Are there plans for implementing more (or all) standard functions? Best, Djoerd.

2 1

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/tests/BugsViaSourgeforce/Tests ID.1766414.stable.err, , NONE, 1.1 ID.1766414.stable.out, , NONE, 1.1 ID.1766414.xml, , NONE, 1.1 ID.1766414.xq, , NONE, 1.1 All, , 1.34, 1.35
by Stefan Manegold 02 Aug '07

02 Aug '07

On Thu, Aug 02, 2007 at 08:42:23PM +0000, Peter Boncz wrote: > Update of /cvsroot/monetdb/pathfinder/tests/BugsViaSourgeforce/Tests > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv13362 > > Modified Files: > All > Added Files: > ID.1766414.stable.err ID.1766414.stable.out ID.1766414.xml > ID.1766414.xq > Log Message: > adding stable output, produced on branch -- will still fail in head until Why hasn't this test been added to the release branch? (Apparently, the bug occured and was fixed there...) > I propagate the fix Is there an urgend need for individual "by hand" propagation (with all possible conflicts it might cause for the ustandard (semi-)automa[tgn]ic propagation by Sjoerd and/or me or does the head-fix require more than the usual standard (semi-)automa[tgn]ic propagation by Sjoerd and/or me? If 2x no, I'd prefer to have this done during / via the usual standard (semi-)automa[tgn]ic propagation by Sjoerd and/or me. (Please feel free to request a usual standard (semi-)automa[tgn]ic propagation by Sjoerd and/or me whenever you "need" to have changes propagated from the latest release branch to the development head "urgendly".) Thanks. Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] Any efforts on using bitmap indices?
by Gabriel Welsche 02 Aug '07

02 Aug '07

Hello, does anybody consider supporting bitmap indices? An indexing software called FastBit was integrated to MonetDB (http://crd.lbl.gov/~kewu/ps/LBNL-61768.pdf) and some benchmarks show that there were serious improvements concerning combined numerical and text queries (offers up to 10 times better performance then MonetDB without FastBit). Gabriel

5 6