So we've been testing out a big ETL and it crashed again, and now every time we start mserver5 it will seemingly start up ok but as soon as mclient connects it will crash. I've generated a core dump.
The db farm is 3.6GB and the core dump is 15MB. I can send either to someone if they need it to look further into the issue. The platform is 2.6.20-15-generic (root@yellow) (gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)) #2 SMP on amd64. It's running an unmodified SR5 superball that we compiled ourselves.
I tried looking at strace and the --debug option with all of the flags on but it didn't produce anything that looked very interesting.
The backtrace is here:
Core was generated by `//mochi/bin/mserver5 --dbname=mochiads_stats --dbinit=include sql; --dbfarm=/da'. Program terminated with signal 11, Segmentation fault. #0 0x00002b88a35bf285 in BUNinplace (b=0x12458e8, p=0x0, h=<value optimized out>, t=0x1246eb0, force=1 '\001') at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB/src/gdk/gdk_bat.mx:1555 1555 ATOMreplace(b->ttype, b->theap, BUNtloc(b, p), t); (gdb) bt #0 0x00002b88a35bf285 in BUNinplace (b=0x12458e8, p=0x0, h=<value optimized out>, t=0x1246eb0, force=1 '\001') at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB/src/gdk/gdk_bat.mx:1555 #1 0x00002b88a35bfb75 in void_inplace (b=0x12458e8, id=1, val=0x1245a20, force=1 '\001') at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB/src/gdk/gdk_bat.mx:1671 #2 0x00002b88a35bfc33 in void_replace_bat (b=0x12458e8, u=0x1246d08, force=<value optimized out>) at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB/src/gdk/gdk_bat.mx:1688 #3 0x00002aaab26c9a5a in load_bat (sname=<value optimized out>, tname=<value optimized out>, bname=<value optimized out>, type=<value optimized out>, sz=<value optimized out>, cnt=0x1214400) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/storage/bat/bat_storage.mx:364 #4 0x00002aaab26c9b83 in create_col (tr=<value optimized out>, c=0x1245878) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/storage/bat/bat_storage.mx:456 #5 0x00002aaab26c44a9 in load_table (tr=0xd3dec8, s=0xd3df38, rid=149) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/storage/store.mx:511 #6 0x00002aaab26c5523 in load_trans (tr=0xd3dec8, id=1128) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/storage/store.mx:845 #7 0x00002aaab26c6ff4 in store_init (debug=<value optimized out>, store=<value optimized out>, logdir=<value optimized out>, dbname=<value optimized out>, stk=0) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/storage/store.mx:1318 #8 0x00002aaab26aab79 in mvc_init (debug=0, store=store_bat, stk=0) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/server/sql_mvc.mx:246 #9 0x00002aaab2665c97 in SQLinit (c=<value optimized out>) at /ufs/sjoerd/src/MonetDB/virgin-stable/sql/src/backends/monet5/sql_scenario.mx:225 #10 0x00002b88a2c7d42a in initScenario (c=0x604980, s=0x2b88a2e934d0) at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB5/src/mal/mal_scenario.mx:264 #11 0x00002b88a2c58d82 in MSscheduleClient (command=0xb0a1b8 "LIT", challenge=0x41000fd0 "KdhtXMmr29n", fin=0xb10270, fout=0xb03030) at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB5/src/mal/mal_session.mx:342 #12 0x00002aaab0e78247 in doChallenge (cmd=<value optimized out>, in=<value optimized out>, out=<value optimized out>) at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB5/src/modules/mal/mserver.mx:482 #13 0x00002aaab0e79036 in SERVERlistenThread (Sock=<value optimized out>) at /ufs/sjoerd/src/MonetDB/virgin-stable/MonetDB5/src/modules/mal/mserver.mx:564 #14 0x00002b88a43122a5 in start_thread () from /lib/libpthread.so.0 #15 0x00002b88a4ff261d in clone () from /lib/libc.so.6 #16 0x0000000000000000 in ?? ()
-bob