On Jan 22, 2008 8:48 AM, Bob Ippolito <bob@redivi.com> wrote:
So here's the core dump for our ETL test with --nightly=stable. We
have not yet tried to restart mserver5. Here's some stats on the data
dirs:

$ du -sh *
10G     bat
4.0K    box
5.6G    core.6950

$ find bat|wc -l
777933


We've run into another error during ETL processing.  I've posted details similar to those above.

$ du -sh *
4.2G bat
4.0K box

$ find bat|wc -l
3553

The screen output from the mserver5 process is below, thanks for any help you can provide:

Travis

!WARNING: GDKlockHome: created directory /data/db/monetdb/mochiads_stats/
!WARNING: GDKlockHome: ignoring empty or invalid .gdk_lock.
!WARNING: BBPdir: initializing BBP.
# MonetDB server v5.2.3, based on kernel v1.20.1
# Serving database 'mochiads_stats'
# Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked
# Copyright (c) 1993-2007 CWI, all rights reserved
# Visit http://monetdb.cwi.nl/ for further information
#warning: please don't forget to set your vault key!
#(see /mochi/etc/monetdb5.conf)
# Listening for connection requests on mapi:monetdb://127.0.0.1:50000/
# MonetDB/SQL module v2.20.3 loaded
>
>\q
# MonetDB server v5.2.3, based on kernel v1.20.1
# Serving database 'mochiads_stats'
# Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked
# Copyright (c) 1993-2007 CWI, all rights reserved
# Visit http://monetdb.cwi.nl/ for further information
#warning: please don't forget to set your vault key!
#(see /mochi/etc/monetdb5.conf)
# Listening for connection requests on mapi:monetdb://127.0.0.1:50000/
# MonetDB/SQL module v2.20.3 loaded
>#MALexit: clients still active
#MALexit: clients still active
#MALexit: server forced exit 0 finishing 0 claimed 1 waiting
# MonetDB server v5.2.3, based on kernel v1.20.1
# Serving database 'mochiads_stats'
# Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked
# Copyright (c) 1993-2007 CWI, all rights reserved
# Visit http://monetdb.cwi.nl/ for further information
#warning: please don't forget to set your vault key!
#(see /mochi/etc/monetdb5.conf)
# Listening for connection requests on mapi:monetdb://127.0.0.1:50000/
# MonetDB/SQL module v2.20.3 loaded
># MonetDB server v5.2.3, based on kernel v1.20.1
# Serving database 'mochiads_stats'
# Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked
# Copyright (c) 1993-2007 CWI, all rights reserved
# Visit http://monetdb.cwi.nl/ for further information
#warning: please don't forget to set your vault key!
#(see /mochi/etc/monetdb5.conf)
# Listening for connection requests on mapi:monetdb://127.0.0.1:50000/
# MonetDB/SQL module v2.20.3 loaded
>*** glibc detected *** //mochi/bin/mserver5: free(): invalid pointer: 0x00002aaaf9969008 ***
======= Backtrace: =========
/lib/libc.so.6[0x2aaaff2f8b23]
/lib/libc.so.6(cfree+0x8c)[0x2aaaff2fc26c]
/mochi/lib/libbat.so.0(GDKfree+0x2e)[0x2aaafd8d163e]
/mochi/lib/libbat.so.0(HEAPfree+0xad)[0x2aaafd86dd2d]
/mochi/lib/libbat.so.0(BATdelete+0x24b)[0x2aaafd9134ab]
/mochi/lib/libbat.so.0[0x2aaafd86a52f]
/mochi/lib/libbat.so.0[0x2aaafd86b5f4]
/mochi/lib/libmal.so.0[0x2aaafcfc2ae0]
/mochi/lib/libmal.so.0(callMAL+0x25b)[0x2aaafcfc41cb]
/mochi/lib/MonetDB5/lib/lib_sql.so(SQLexecutePrepared+0x25c)[0x2aaab266b19c]
/mochi/lib/MonetDB5/lib/lib_sql.so(SQLengineIntern+0x21d)[0x2aaab266cd0d]
/mochi/lib/libmal.so.0(runScenarioBody+0x14d)[0x2aaafcfe305d]
/mochi/lib/libmal.so.0(runScenario+0x3a)[0x2aaafcfe313a]
/mochi/lib/libmal.so.0(MSserveClient+0x58)[0x2aaafcfbe1d8]
/lib/libpthread.so.0[0x2aaafe6782a5]
/lib/libc.so.6(clone+0x6d)[0x2aaaff35861d]
======= Memory map: ========
00400000-00404000 r-xp 00000000 fe:00 2115695                            /mochi/bin/mserver5
00604000-00605000 rw-p 00004000 fe:00 2115695                            /mochi/bin/mserver5
00605000-3fff6000 rw-p 00605000 00:00 0                                  [heap]
40000000-40001000 ---p 40000000 00:00 0 
40001000-40801000 rw-p 40001000 00:00 0 
40801000-40802000 ---p 40801000 00:00 0 
40802000-41002000 rw-p 40802000 00:00 0 
41002000-41003000 ---p 41002000 00:00 0 
41003000-41803000 rw-p 41003000 00:00 0 
41803000-41804000 ---p 41803000 00:00 0 
41804000-42004000 rw-p 41804000 00:00 0 
42004000-42005000 ---p 42004000 00:00 0 
42005000-42805000 rw-p 42005000 00:00 0 
2aaaaaaab000-2aaaaaaad000 rw-p 2aaaaaaab000 00:00 0 
2aaaaabad000-2aaaaabb1000 r-xp 00000000 fe:00 2085181                    /mochi/lib/MonetDB5/lib/lib_mdb.so.0.0.0
2aaaaabb1000-2aaaaadb1000 ---p 00004000 fe:00 2085181                    /mochi/lib/MonetDB5/lib/lib_mdb.so.0.0.0
2aaaaadb1000-2aaaaadb2000 rw-p 00004000 fe:00 2085181                    /mochi/lib/MonetDB5/lib/lib_mdb.so.0.0.0
2aaaaadb2000-2aaaaadb4000 r-xp 00000000 fe:00 2084480                    /mochi/lib/MonetDB5/lib/lib_xml.so.0.0.0
2aaaaadb4000-2aaaaafb3000 ---p 00002000 fe:00 2084480                    /mochi/lib/MonetDB5/lib/lib_xml.so.0.0.0
2aaaaafb3000-2aaaaafb4000 rw-p 00001000 fe:00 2084480                    /mochi/lib/MonetDB5/lib/lib_xml.so.0.0.0
2aaaaafbb000-2aaaab03c000 r-xp 00000000 fe:00 540697                     /lib/libm- 2.5.so
2aaaab03c000-2aaaab23b000 ---p 00081000 fe:00 540697                     /lib/libm-2.5.so
2aaaab23b000-2aaaab23d000 rw-p 00080000 fe:00 540697                     /lib/libm- 2.5.so
2aaaab23d000-2aaaab24b000 r-xp 00000000 fe:00 2016109                    /mochi/lib/libMapi.so.0.0.0
2aaaab24b000-2aaaab44a000 ---p 0000e000 fe:00 2016109                    /mochi/lib/libMapi.so.0.0.0
2aaaab44a000-2aaaab44b000 rw-p 0000d000 fe:00 2016109                    /mochi/lib/libMapi.so.0.0.0
2aaaab44b000-2aaaab44c000 r-xp 00000000 fe:00 2085120                    /mochi/lib/MonetDB5/lib/lib_unix.so.0.0.0
2aaaab44c000-2aaaab64b000 ---p 00001000 fe:00 2085120                    /mochi/lib/MonetDB5/lib/lib_unix.so.0.0.0
2aaaab64b000-2aaaab64c000 rw-p 00000000 fe:00 2085120                    /mochi/lib/MonetDB5/lib/lib_unix.so.0.0.0
2aaaab64c000-2aaaab64e000 r-xp 00000000 fe:00 2085099                    /mochi/lib/MonetDB5/lib/lib_lock.so.0.0.0
2aaaab64e000-2aaaab84d000 ---p 00002000 fe:00 2085099                    /mochi/lib/MonetDB5/lib/lib_lock.so.0.0.0
2aaaab84d000-2aaaab84e000 rw-p 00001000 fe:00 2085099                    /mochi/lib/MonetDB5/lib/lib_lock.so.0.0.0
2aaaab84e000-2aaaab84f000 r-xp 00000000 fe:00 2085118                    /mochi/lib/MonetDB5/lib/lib_sema.so.0.0.0
2aaaab84f000-2aaaaba4f000 ---p 00001000 fe:00 2085118                    /mochi/lib/MonetDB5/lib/lib_sema.so.0.0.0
2aaaaba4f000-2aaaaba50000 rw-p 00001000 fe:00 2085118                    /mochi/lib/MonetDB5/lib/lib_sema.so.0.0.0
2aaaaba50000-2aaaaba55000 r-xp 00000000 fe:00 2085116                    /mochi/lib/MonetDB5/lib/lib_mmath.so.0.0.0
2aaaaba55000-2aaaabc54000 ---p 00005000 fe:00 2085116                    /mochi/lib/MonetDB5/lib/lib_mmath.so.0.0.0
2aaaabc54000-2aaaabc55000 rw-p 00004000 fe:00 2085116                    /mochi/lib/MonetDB5/lib/lib_mmath.so.0.0.0
2aaaabc55000-2aaaabc58000 r-xp 00000000 fe:00 2084470                    /mochi/lib/MonetDB5/lib/lib_streams.so.0.0.0
2aaaabc58000-2aaaabe58000 ---p 00003000 fe:00 2084470                    /mochi/lib/MonetDB5/lib/lib_streams.so.0.0.0
2aaaabe58000-2aaaabe59000 rw-p 00003000 fe:00 2084470                    /mochi/lib/MonetDB5/lib/lib_streams.so.0.0.0
2aaaabe59000-2aaaabe69000 r-xp 00000000 fe:00 2085061                    /mochi/lib/MonetDB5/lib/lib_bat5.so.0.0.0
2aaaabe69000-2aaaac069000 ---p 00010000 fe:00 2085061                    /mochi/lib/MonetDB5/lib/lib_bat5.so.0.0.0
2aaaac069000-2aaaac06a000 rw-p 00010000 fe:00 2085061                    /mochi/lib/MonetDB5/lib/lib_bat5.so.0.0.0
2aaaac06a000-2aaaac06d000 r-xp 00000000 fe:00 2085124                    /mochi/lib/MonetDB5/lib/lib_batExtensions.so.0.0.0
2aaaac06d000-2aaaac26c000 ---p 00003000 fe:00 2085124                    /mochi/lib/MonetDB5/lib/lib_batExtensions.so.0.0.0
2aaaac26c000-2aaaac26d000 rw-p 00002000 fe:00 2085124                    /mochi/lib/MonetDB5/lib/lib_batExtensions.so.0.0.0
2aaaac26d000-2aaaac270000 r-xp 00000000 fe:00 2085202                    /mochi/lib/MonetDB5/lib/lib_pcre.so.0.0.0
2aaaac270000-2aaaac46f000 ---p 00003000 fe:00 2085202                    /mochi/lib/MonetDB5/lib/lib_pcre.so.0.0.0
2aaaac46f000-2aaaac470000 rw-p 00002000 fe:00 2085202                    /mochi/lib/MonetDB5/lib/lib_pcre.so.0.0.0
2aaaac470000-2aaaac49e000 r-xp 00000000 fe:00 2084507                    /mochi/lib/MonetDB5/lib/lib_algebra.so.0.0.0
2aaaac49e000-2aaaac69d000 ---p 0002e000 fe:00 2084507                    /mochi/lib/MonetDB5/lib/lib_algebra.so.0.0.0
2aaaac69d000-2aaaac69e000 rw-p 0002d000 fe:00 2084507                    /mochi/lib/MonetDB5/lib/lib_algebra.so.0.0.0
2aaaac69e000-2aaaac69f000 r-xp 00000000 fe:00 2085122                    /mochi/lib/MonetDB5/lib/lib_algebraExtensions.so.0.0.0
2aaaac69f000-2aaaac89f000 ---p 00001000 fe:00 2085122                    /mochi/lib/MonetDB5/lib/lib_algebraExtensions.so.0.0.0
2aaaac89f000-2aaaac8a0000 rw-p 00001000 fe:00 2085122                    /mochi/lib/MonetDB5/lib/lib_algebraExtensions.so.0.0.0
2aaaac8a0000-2aaaac8c3000 r-xp 00000000 fe:00 2085097                    /mochi/lib/MonetDB5/lib/lib_calc.so.0.0.0
2aaaac8c3000-2aaaacac2000 ---p 00023000 fe:00 2085097                    /mochi/lib/MonetDB5/lib/lib_calc.so.0.0.0
2aaaacac2000-2aaaacac3000 rw-p 00022000 fe:00 2085097                    /mochi/lib/MonetDB5/lib/lib_calc.so.0.0.0
2aaaacac3000-2aaaacac7000 r-xp 00000000 fe:00 2085119                    /mochi/lib/MonetDB5/lib/lib_status.so.0.0.0
2aaaacac7000-2aaaaccc7000 ---p 00004000 fe:00 2085119                    /mochi/lib/MonetDB5/lib/lib_status.so.0.0.0
2aaaaccc7000-2aaaaccc8000 rw-p 00004000 fe:00 2085119                    /mochi/lib/MonetDB5/lib/lib_status.so.0.0.0
2aaaaccc8000-2aaaacd00000 r-xp 00000000 fe:00 2085098                    /mochi/lib/MonetDB5/lib/lib_group.so.0.0.0
2aaaacd00000-2aaaacf00000 ---p 00038000 fe:00 2085098                    /mochi/lib/MonetDB5/lib/lib_group.so.0.0.0
2aaaacf00000-2aaaacf01000 rw-p 00038000 fe:00 2085098                    /mochi/lib/MonetDB5/lib/lib_group.so.0.0.0
2aaaacf01000-2aaaad049000 r-xp 00000000 fe:00 2084493                    /mochi/lib/MonetDB5/lib/lib_aggr.so.0.0.0
2aaaad049000-2aaaad248000 ---p 00148000 fe:00 2084493                    /mochi/lib/MonetDB5/lib/lib_aggr.so.0.0.0
2aaaad248000-2aaaad249000 rw-p 00147000 fe:00 2084493                    /mochi/lib/MonetDB5/lib/lib_aggr.so.0.0.0
2aaaad249000-2aaaad24f000 r-xp 00000000 fe:00 2084562                    /mochi/lib/MonetDB5/lib/lib_array.so.0.0.0
2aaaad24f000-2aaaad44e000 ---p 00006000 fe:00 2084562                    /mochi/lib/MonetDB5/lib/lib_arra