September 2008 - developers-list

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/runtime pathfinder.mx, XQuery_0-24, 1.416.2.12, 1.416.2.13
by Ying Zhang 16 Sep '08

16 Sep '08

On Sep 15, 2008, at 13:33 , Ying Zhang wrote: > Update of /cvsroot/monetdb/pathfinder/runtime > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv13268 > > Modified Files: > Tag: XQuery_0-24 > pathfinder.mx > Log Message: > backported (JanR's) bugfix from HEAD: Jan, as I could understand that this bug is not an important item in your agenda, I felt free to backport your fix, so that this bug could be closed. I hope this is OK. Regards, Jennie > > > <original log message> > -- Try to fix bug 2015219 by changing the mark offset. > > identical checkin, expecting no conflicts during propagation > > > > U pathfinder.mx > Index: pathfinder.mx > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/runtime/pathfinder.mx,v > retrieving revision 1.416.2.12 > retrieving revision 1.416.2.13 > diff -u -d -r1.416.2.12 -r1.416.2.13 > --- pathfinder.mx 28 Jun 2008 10:19:09 -0000 1.416.2.12 > +++ pathfinder.mx 15 Sep 2008 11:33:45 -0000 1.416.2.13 > @@ -3226,7 +3226,7 @@ > var attrid := new(void,oid, > 2 > *count > (attr1 > )).seqbase > (0(a)0).append(attr1.project(oid_nil)).append(attr1.project(oid_nil)); > attrid.replace(reverse([oid]([lng] > (reverse(attr_constr(attr1.project(at1qn),attr1,ws).fetch(1))).[*] > (2LL)))); > attrid.replace(reverse([oid]([lng] > (reverse(attr_constr(attr1.project(at2qn),attr2,ws).fetch(1))).[*] > (2LL).[+](1LL)))); > - return elem_constr(attr1.mark(1(a)0),attr1.project(eltqn), > attrid.mirror().[lng]().[/](2LL).[oid](), > + return elem_constr(attr1.mark(0(a)0),attr1.project(eltqn), > attrid.mirror().[lng]().[/](2LL).[oid](), > > attrid.project(oid_nil), attrid.project(oid_nil), > attrid, > attrid.project(WS), ws).fetch(1); > } > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's > challenge > Build the coolest Linux based applications with Moblin SDK & win > great prizes > Grand prize is a trip for two to an Open Source event anywhere in > the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins

2 1

[Monetdb-developers] MoneDB/XQuery problem (importing a module, http access)
by Clarisse Kagoyire 15 Sep '08

15 Sep '08

Hello, I am a student in GeoInformatics at ITC(International Institute for Geo-Information Science and Earth Observation) and I have just started using MonetDB/XQuery for my MSc Research project. But I am facing some problems and your help will be really appreciated: 1. Whenever I try to use any of the options available in the Administrative GUI(like adding, deleting documents, list collections,...), I get an error message. Here is an example of the error returned if I try to list collections: !ERROR: "collections execution failed at the remote side" !ERROR: HTTP/1.1 404 !ERROR: HTTP Response: <?xml version="1.0" encoding="utf-8"?> <env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope"><env:Body><env:Fault ><env:Code><env:Value>env:Sender</env:Value></env:Code><env:Reason><env: Text <http://www.w3.org/2003/05/soap-envelope"><env:Body><env:Fault><env:Code ><env:Value>env:Sender</env:Value></env:Code><env:Reason><env:Text> xml:lang="en">xquery_module_load: could not load module. Due to earlier error: parse error: syntax error, unexpected $end on line 1, column 0 parse error: XQuery parsing failed </env:Text></env:Reason></env:Fault></env:Body></env:Envelope> Note that I can do this (add documents, delete,list...) without any problem using the mclient XQuery.(I think that it's a problem of HTTP access, but I couldn't fix it) 2. Another problem is that I cannot make use of XQuery module, because whenever I import a module i get an error. Here is an example of a simple xquery module from the tutorial that I was tried to import: -the XQuery module saved in E:\Program Files\CWI\MonetDB4\share\MonetDB\xrpc\export\xrpc-mod.xq module namespace foo = "xrpc-test-function"; declare function foo:add($v1 as xs:integer, $v2 as xs:integer) as xs:integer { $v1 + $v2 }; -on mclient, I typed the following to import the module: > import module namespace test = "xrpc-test-function" at "http://127.0.0.1:50001/export/xrpc-mod.xq <http://127.0.0.1:50001/export/xrpc-mod.xq> "; >test:add(100, 200) Which returned also an error: !parse eroor: error parsing module.... I'm looking forward to hearing from you. Thank you , Clarisse K.

2 1

[Monetdb-developers] COPY, terminating connection
by sylver_b 14 Sep '08

14 Sep '08

Hi, When i run the COPY command from a batch script and after few minutes of processing (2,3minutes), and few thousand lines inserted in the DB, im getting : Connection terminated merovingian log is showing: MSG demo[5084]: # MonetDB server v5.6.0, based on kernel v1.24.0 MSG demo[5084]: # Serving database 'demo' MSG demo[5084]: # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs dynamically linked MSG demo[5084]: # Copyright (c) 1993-2008 CWI, all rights reserved MSG demo[5084]: # Visit http://monetdb.cwi.nl/ for further information MSG demo[5084]: # Listening for connection requests on mapi:monetdb://ulys:50001/ MSG merovingian[3981]: proxying client 127.0.0.1:41420 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41420 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41422 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41422 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41424 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41424 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41426 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41426 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41428 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41428 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41430 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41430 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41432 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41434 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41432 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41436 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41434 disconnected from proxy MSG merovingian[3981]: client has 127.0.0.1:41436 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41438 for database 'demo' to mapi:monetdb://ulys:50001/ MSG merovingian[3981]: client has 127.0.0.1:41438 disconnected from proxy MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) MSG merovingian[3981]: proxying client 127.0.0.1:41440 for database 'demo' to mapi:monetdb://ulys:50001/ TME 2008-09-12 11:56:46 MSG merovingian[3981]: client has 127.0.0.1:41440 disconnected from proxy This is what my script does : #!/bin/sh FILE=$1 awk -F "\"*;\"*" '{print ";",$1,";",$2,";",$3,";",$4,";",$5,";",$6,";",$8,";",$9,";",$10,";",$13,";",$15,";",$18,";",$22,";",$24,";",$25,";",$26,";",$27,";",$28,";",$29,";",$30,";",$31,";",$32,";",$33,";",$35,";",$36,";",$38,";",$39,";",$42,";",$47,";",$48,";",$49,";",$53,";",$54,";",$55,";","FILE"}' $1 | sed -e "s/ ; /;/g; s/; /;/g" | mclient -lsql -d demo -i -s "COPY INTO cdrwholesaler FROM STDIN USING DELIMITERS ';' NULL AS ''" and this is how i call it : find /usr/ftp -name '*.CDR' -exec /var/www/consolecdr/batch/test_monetdb/awk_csv_generator.sh {} \; Is there anything im doing wrong ? I have the feeling there must be too many connections crashing the system .. coz after that i can't use mclient anymore, i have to restart the DB. Please advise, Thanks SB

4 3

[Monetdb-developers] Re : Re : COPY, terminating connection
by sylver_b 13 Sep '08

13 Sep '08

Hi, Yes it's a 32bit system. 4G of Ram, Quad Xeon. OS: Ubuntu Server 6.10 - Raid10(4x HDs) Total data size > 4G (but could be a lot more, sometimes i need to restream 1month worth of CDRs). I was using a postgres/bizgres DB, and loading data with psql -c, + the COPY command - but didnt like the fact that, if a line was rejected the whole process was stopped, so that's why i've created my own perl ETL script able to ignore rejected entries - which might be the best option finally ? MonetDB's COPY command is doing the same. Also can the mclient re-use an active connection ? Let me know what do you think. Thanks ----- Message d'origine ---- De : Stefan Manegold <Stefan.Manegold(a)cwi.nl> À : sylver_b <sylver_b(a)yahoo.com> Cc : monetdb-developers(a)lists.sourceforge.net Envoyé le : Samedi, 13 Septembre 2008, 9h12mn 53s Objet : Re: [Monetdb-developers] Re : COPY, terminating connection On Sat, Sep 13, 2008 at 12:43:32AM -0700, sylver_b wrote: > Hi Stephan > > >How many (sequential, not concurrent, right?) connections are we talking > >about, i.e., how many *.CDR files do you have? > I wanted to stream about 60 to 100 CDR files > > >How many rows do these *.CDR files hold in total? > some files can contain over half a million records - average is 100~200K lines > > >Are all attributes of the cdrwholesaler table string attributes? > some of them are date/time, timestamp, int, double, but majority is varchar How large is your total data volumne? Does it fit into a 32-bit address space? What kind of machine are you running on? CPU, memory, OS? Given that you're using a 32-bit MonetDB installation, I suppose you hardware and OS is 32-bit, too, isn't it? > It looks like for each call of mclient (probably for each cdr line) a connection is opened. Obviously, you call mclient once for each of you CBR files, and yes, each of these mclient calls make a new connection to the server. For each CBR file (not line!) there should be only one mclient call, and hence, only one connection. Stefan ps: please also note that the vrey fast bulk loading I mentioned only holds for loading from files with the number of rows to be loaded specified in the COPY INTO statement. > SB > > > ----- Message d'origine ---- > De : Stefan Manegold <Stefan.Manegold(a)cwi.nl> > À : sylver_b <sylver_b(a)yahoo.com> > Cc : monetdb-developers(a)lists.sourceforge.net > Envoyé le : Samedi, 13 Septembre 2008, 6h36mn 19s > Objet : Re: [Monetdb-developers] COPY, terminating connection > > On Fri, Sep 12, 2008 at 04:48:43AM -0700, sylver_b wrote: > > Hi, > > > > When i run the COPY command from a batch script and after few minutes of processing (2,3minutes), and few thousand lines inserted in the DB, im getting : > > > > Connection terminated > > > > > > merovingian log is showing: > > > > MSG demo[5084]: # MonetDB server v5.6.0, based on kernel v1.24.0 > > MSG demo[5084]: # Serving database 'demo' > > MSG demo[5084]: # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs dynamically linked > > MSG demo[5084]: # Copyright (c) 1993-2008 CWI, all rights reserved > > MSG demo[5084]: # Visit http://monetdb.cwi.nl/ for further information > > MSG demo[5084]: # Listening for connection requests on mapi:monetdb://ulys:50001/ > > MSG merovingian[3981]: proxying client 127.0.0.1:41420 for database 'demo' to mapi:monetdb://ulys:50001/ > > MSG merovingian[3981]: client has 127.0.0.1:41420 disconnected from proxy > > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > > MSG merovingian[3981]: proxying client 127.0.0.1:41422 for database 'demo' to mapi:monetdb://ulys:50001/ > > MSG merovingian[3981]: client has 127.0.0.1:41422 disconnected from proxy > > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > > MSG merovingian[3981]: proxying client 127.0.0.1:41424 for database 'demo' to mapi:monetdb://ulys:50001/ > > MSG merovingian[3981]: client has 127.0.0.1:41424 disconnected from proxy > > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > > MSG merovingian[3981]: proxying client 127.0.0.1:41426 for database 'demo' to mapi:monetdb://ulys:50001/ > > MSG merovingian[3981]: client has 127.0.0.1:41426 disconnected from proxy > > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > > MSG merovingian[3981]: proxying client 127.0.0.1:41428 for database 'demo' to mapi:monetdb://ulys:50001/ > > MSG merovingian[3981]: client has 127.0.0.1:41428 disconnected from proxy > > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > > MSG merovingian[3981]: proxying client 127.0.0.1:41430 for database 'demo' to mapi:monetdb://ulys:50001/ > > MSG merovingian[3981]: client has 127.0.0.1:41430 disconnected from proxy > > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > > MSG merovingian[3981]: proxying client 127.0.0.1:41432 for database 'demo' to mapi:monetdb://ulys:50001/ > > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > > MSG merovingian[3981]: proxying client 127.0.0.1:41434 for database 'demo' to mapi:monetdb://ulys:50001/ > > MSG merovingian[3981]: client has 127.0.0.1:41432 disconnected from proxy > > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > > MSG merovingian[3981]: proxying client 127.0.0.1:41436 for database 'demo' to mapi:monetdb://ulys:50001/ > > MSG merovingian[3981]: client has 127.0.0.1:41434 disconnected from proxy > > MSG merovingian[3981]: client has 127.0.0.1:41436 disconnected from proxy > > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > > MSG merovingian[3981]: proxying client 127.0.0.1:41438 for database 'demo' to mapi:monetdb://ulys:50001/ > > MSG merovingian[3981]: client has 127.0.0.1:41438 disconnected from proxy > > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > > MSG merovingian[3981]: proxying client 127.0.0.1:41440 for database 'demo' to mapi:monetdb://ulys:50001/ > > TME 2008-09-12 11:56:46 > > MSG merovingian[3981]: client has 127.0.0.1:41440 disconnected from proxy > > > > > > This is what my script does : > > > > #!/bin/sh > > > > FILE=$1 > > awk -F "\"*;\"*" '{print ";",$1,";",$2,";",$3,";",$4,";",$5,";",$6,";",$8,";",$9,";",$10,";",$13,";",$15,";",$18,";",$22,";",$24,";",$25,";",$26,";",$27,";",$28,";",$29,";",$30,";",$31,";",$32,";",$33,";",$35,";",$36,";",$38,";",$39,";",$42,";",$47,";",$48,";",$49,";",$53,";",$54,";",$55,";","FILE"}' $1 | sed -e "s/ ; /;/g; s/; /;/g" | mclient -lsql -d demo -i -s "COPY INTO cdrwholesaler FROM STDIN USING DELIMITERS ';' NULL AS ''" > > > > and this is how i call it : > > > > find /usr/ftp -name '*.CDR' -exec /var/www/consolecdr/batch/test_monetdb/awk_csv_generator.sh {} \; > > > > > > Is there anything im doing wrong ? I have the feeling there must be too many connections crashing the system .. coz after that i can't use mclient anymore, i have to restart the DB. > > How many (sequential, not concurrent, right?) connections are we talking > about, i.e., how many *.CDR files do you have? > > How many rows do these *.CDR files hold in total? > > Are all attributes of the cdrwholesaler table string attributes? > > ... "too many connections" *might* be the problem (though it should of > course not) --- strange, though, that the merovingian log do not give any > indication of whether, and if so why the server might have crashed ... > > Stefan > > > Please advise, > > Thanks > > SB > > > > > > > > ------------------------------------------------------------------------- > > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > > Build the coolest Linux based applications with Moblin SDK & win great prizes > > Grand prize is a trip for two to an Open Source event anywhere in the world > > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > > _______________________________________________ > > Monetdb-developers mailing list > > Monetdb-developers(a)lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/monetdb-developers > > > -- > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > | The Netherlands | Fax : +31 (20) 592-4312 | > > > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-developers mailing list > Monetdb-developers(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-developers -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] Re : COPY, terminating connection
by sylver_b 13 Sep '08

13 Sep '08

Hi Stephan >How many (sequential, not concurrent, right?) connections are we talking >about, i.e., how many *.CDR files do you have? I wanted to stream about 60 to 100 CDR files >How many rows do these *.CDR files hold in total? some files can contain over half a million records - average is 100~200K lines >Are all attributes of the cdrwholesaler table string attributes? some of them are date/time, timestamp, int, double, but majority is varchar It looks like for each call of mclient (probably for each cdr line) a connection is opened. SB ----- Message d'origine ---- De : Stefan Manegold <Stefan.Manegold(a)cwi.nl> À : sylver_b <sylver_b(a)yahoo.com> Cc : monetdb-developers(a)lists.sourceforge.net Envoyé le : Samedi, 13 Septembre 2008, 6h36mn 19s Objet : Re: [Monetdb-developers] COPY, terminating connection On Fri, Sep 12, 2008 at 04:48:43AM -0700, sylver_b wrote: > Hi, > > When i run the COPY command from a batch script and after few minutes of processing (2,3minutes), and few thousand lines inserted in the DB, im getting : > > Connection terminated > > > merovingian log is showing: > > MSG demo[5084]: # MonetDB server v5.6.0, based on kernel v1.24.0 > MSG demo[5084]: # Serving database 'demo' > MSG demo[5084]: # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs dynamically linked > MSG demo[5084]: # Copyright (c) 1993-2008 CWI, all rights reserved > MSG demo[5084]: # Visit http://monetdb.cwi.nl/ for further information > MSG demo[5084]: # Listening for connection requests on mapi:monetdb://ulys:50001/ > MSG merovingian[3981]: proxying client 127.0.0.1:41420 for database 'demo' to mapi:monetdb://ulys:50001/ > MSG merovingian[3981]: client has 127.0.0.1:41420 disconnected from proxy > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > MSG merovingian[3981]: proxying client 127.0.0.1:41422 for database 'demo' to mapi:monetdb://ulys:50001/ > MSG merovingian[3981]: client has 127.0.0.1:41422 disconnected from proxy > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > MSG merovingian[3981]: proxying client 127.0.0.1:41424 for database 'demo' to mapi:monetdb://ulys:50001/ > MSG merovingian[3981]: client has 127.0.0.1:41424 disconnected from proxy > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > MSG merovingian[3981]: proxying client 127.0.0.1:41426 for database 'demo' to mapi:monetdb://ulys:50001/ > MSG merovingian[3981]: client has 127.0.0.1:41426 disconnected from proxy > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > MSG merovingian[3981]: proxying client 127.0.0.1:41428 for database 'demo' to mapi:monetdb://ulys:50001/ > MSG merovingian[3981]: client has 127.0.0.1:41428 disconnected from proxy > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > MSG merovingian[3981]: proxying client 127.0.0.1:41430 for database 'demo' to mapi:monetdb://ulys:50001/ > MSG merovingian[3981]: client has 127.0.0.1:41430 disconnected from proxy > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > MSG merovingian[3981]: proxying client 127.0.0.1:41432 for database 'demo' to mapi:monetdb://ulys:50001/ > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > MSG merovingian[3981]: proxying client 127.0.0.1:41434 for database 'demo' to mapi:monetdb://ulys:50001/ > MSG merovingian[3981]: client has 127.0.0.1:41432 disconnected from proxy > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > MSG merovingian[3981]: proxying client 127.0.0.1:41436 for database 'demo' to mapi:monetdb://ulys:50001/ > MSG merovingian[3981]: client has 127.0.0.1:41434 disconnected from proxy > MSG merovingian[3981]: client has 127.0.0.1:41436 disconnected from proxy > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > MSG merovingian[3981]: proxying client 127.0.0.1:41438 for database 'demo' to mapi:monetdb://ulys:50001/ > MSG merovingian[3981]: client has 127.0.0.1:41438 disconnected from proxy > MSG merovingian[3981]: database 'demo' already running since 2008-09-12 11:44:44, up min/avg/max: 30/37441/213742, crash average: 0.00 0.10 0.03 (8-6=1) > MSG merovingian[3981]: proxying client 127.0.0.1:41440 for database 'demo' to mapi:monetdb://ulys:50001/ > TME 2008-09-12 11:56:46 > MSG merovingian[3981]: client has 127.0.0.1:41440 disconnected from proxy > > > This is what my script does : > > #!/bin/sh > > FILE=$1 > awk -F "\"*;\"*" '{print ";",$1,";",$2,";",$3,";",$4,";",$5,";",$6,";",$8,";",$9,";",$10,";",$13,";",$15,";",$18,";",$22,";",$24,";",$25,";",$26,";",$27,";",$28,";",$29,";",$30,";",$31,";",$32,";",$33,";",$35,";",$36,";",$38,";",$39,";",$42,";",$47,";",$48,";",$49,";",$53,";",$54,";",$55,";","FILE"}' $1 | sed -e "s/ ; /;/g; s/; /;/g" | mclient -lsql -d demo -i -s "COPY INTO cdrwholesaler FROM STDIN USING DELIMITERS ';' NULL AS ''" > > and this is how i call it : > > find /usr/ftp -name '*.CDR' -exec /var/www/consolecdr/batch/test_monetdb/awk_csv_generator.sh {} \; > > > Is there anything im doing wrong ? I have the feeling there must be too many connections crashing the system .. coz after that i can't use mclient anymore, i have to restart the DB. How many (sequential, not concurrent, right?) connections are we talking about, i.e., how many *.CDR files do you have? How many rows do these *.CDR files hold in total? Are all attributes of the cdrwholesaler table string attributes? ... "too many connections" *might* be the problem (though it should of course not) --- strange, though, that the merovingian log do not give any indication of whether, and if so why the server might have crashed ... Stefan > Please advise, > Thanks > SB > > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-developers mailing list > Monetdb-developers(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-developers -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

[Monetdb-developers] Re : COPY, terminating connection
by sylver_b 13 Sep '08

13 Sep '08

Hi I can send you a CDR file sample if needed.. please let me know. Thanks ----- Message d'origine ---- De : Fabian Groffen <Fabian.Groffen(a)cwi.nl> À : monetdb-developers(a)lists.sourceforge.net Envoyé le : Samedi, 13 Septembre 2008, 8h33mn 00s Objet : Re: [Monetdb-developers] COPY, terminating connection On 13-09-2008 07:36:19 +0200, Stefan Manegold wrote: > > Is there anything im doing wrong ? I have the feeling there must be too many connections crashing the system .. coz after that i can't use mclient anymore, i have to restart the DB. > > How many (sequential, not concurrent, right?) connections are we talking > about, i.e., how many *.CDR files do you have? > > How many rows do these *.CDR files hold in total? > > Are all attributes of the cdrwholesaler table string attributes? > > ... "too many connections" *might* be the problem (though it should of > course not) --- strange, though, that the merovingian log do not give any > indication of whether, and if so why the server might have crashed ... It can't, because the server didn't crash. It indicates it is still alive. Most probably the server is confused by the amount of inserts, I've seen this a few times before. Somewhere something gets stuck, making the mserver refuse to operate. A restart of the server solves it, as the original reporter also indicates. Note from the logs that the server "demo" had one crash in the past, though. ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

1 0

Re: [Monetdb-developers] [Monetdb-checkins] clients/src/perl Mapi.pm, , 1.5, 1.6
by Sjoerd Mullender 12 Sep '08

12 Sep '08

Sjoerd Mullender wrote: > Update of /cvsroot/monetdb/clients/src/perl > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv29671/src/perl > > Modified Files: > Mapi.pm > Log Message: > propagated changes of Wednesday Sep 10 2008 - Friday Sep 12 2008 > from the Clients_1-24 branch to the development trunk > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > 2008/09/10 - mr-meltdown: src/perl/Mapi.pm,1.5.4.1 > Implemented redirect support in Mapi.pm. While testing I hit the > following error which I don't know what it means (let alone how to fix > it): > > Can't use string ("Xexport 1 100 100") as a HASH ref while "strict refs" > in use at Mapi.pm line 352, <> line 2. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ I just noticed this log message. Could the cause be that in Mapi.pm there is one place where the function putblock is called *without* $self-> in front of it, and that is exactly the only occurrence of the string "Xexport" in the complete source? My guess is that unpacking the arguments in the putblock function causes the error message. Can you test this? -- Sjoerd Mullender

2 3

[Monetdb-developers] Re : Re : Re : ODBC, no database selected
by sylver_b 11 Sep '08

11 Sep '08

Ok great!! i'm looking forward to some updates when Sjoerd has a chance. Thanks SB ----- Message d'origine ---- De : Stefan Manegold <Stefan.Manegold(a)cwi.nl> À : sylver_b <sylver_b(a)yahoo.com> Cc : monetdb-developers(a)lists.sourceforge.net Envoyé le : Mercredi, 10 Septembre 2008, 9h24mn 06s Objet : Re: [Monetdb-developers] Re : Re : ODBC, no database selected On Wed, Sep 10, 2008 at 01:17:49AM -0700, sylver_b wrote: > That will do fine and i can scrap the perl tool all together if this can do the job - I will try that . Good. --- In fact bulk-loading via COPY INTO is by far the fasted way to load data into MonetDB. > any idea for the ODBC issue ? is it maintained by you guys (cwi ) ? ODBC is maintained at CWI by Sjoerd. I guess, Sjoerd will have a look at it as soon as his time allows. Stefan > Thanks > > > > > ----- Message d'origine ---- > De : Stefan Manegold <Stefan.Manegold(a)cwi.nl> > À : sylver_b <sylver_b(a)yahoo.com> > Cc : monetdb-developers(a)lists.sourceforge.net > Envoyé le : Mercredi, 10 Septembre 2008, 9h06mn 57s > Objet : Re : [Monetdb-developers] ODBC, no database selected > > On Wed, Sep 10, 2008 at 12:21:46AM -0700, sylver_b wrote: > > Hi Stefan, > > > > I've started the DB with the following commands : > > 1. merovingian > > 2. monetdb create demo > > 3. monetdb start demo > > 3. mclient -lsql --time -d demo > > > > I get the prompt straight away, but i have to specify the db - mclient > > -lsql , says: !please specify a database - i don't have any other DB at > > the moment, just evaluating monetdb to migrate a call detail records > > application ; for that purpose i've developed a simple ETL tool in perl > > that extract specific fields from a CSV file and insert them in the DB. > > Maybe there is a better way to do that ? > > In case you need the perl part only for extracting and loading, a simple > work around to get a quick start, in case you/we do not manage to get the > perl interface working for you quickly, would be to either load all fields > from your CSV file using sql "COPY ... INTO .. FROM ..." command (cf., > http://monetdb.cwi.nl/SQL/Documentation/Data-Manipulation.html), or extract > the desired fileds into a separate CSV file, first, and then load using > "COPY ... INTO .. FROM ..." (e.g., via mclient) ... > > Stefan > > > by the way , ps faux gives me : > > root 11224 0.0 0.0 63160 1816 pts/2 Sl 00:17 0:00 merovingian > > root 11229 0.0 1.7 107680 69816 ? Ssl 00:17 0:00 \_ /root/MonetDB/bin/mserver5 --config=/root/MonetD > > B/etc/monetdb5.conf --dbname=demo --dbinit=include sql; --set monet_daemon yes --set mapi_open true --set mapi_port 0 > > --set monet_vault_key /root/MonetDB/var/MonetDB5/dbfarm/demo/.vaultkey > > > > Please advise, > > Thanks > > SB > > > > > > > > > > ----- Message d'origine ---- > > De : Stefan Manegold <Stefan.Manegold(a)cwi.nl> > > À : sylver_b <sylver_b(a)yahoo.com> > > Cc : monetdb-developers(a)lists.sf.net > > Envoyé le : Mercredi, 10 Septembre 2008, 6h19mn 03s > > Objet : Re: [Monetdb-developers] ODBC, no database selected > > > > Hi "sylver_b", > > > > just a simple & quick saniy check: > > > > How did you start your MonetDB server? > > > > Can you connect with `mclient -lsql`? > > > > Stefan > > > > On Tue, Sep 09, 2008 at 04:56:47PM -0700, sylver_b wrote: > > > Hi Eveyone, > > > > > > I'm trying to connect to monetdb with this time, ODBC - but still no luck. > > > > > > Here is the output of my script : > > > > > > ./monetdb2.pl > > > DBI connect('DSN_MonetDB','monetdb',...) failed: [unixODBC][MonetDB][ODBC Driver 1.0]Client unable to establish connection (SQL-08001)(DBD: db_login/SQLConnect err=-1) at ./monetdb2.pl line 6 > > > [unixODBC][MonetDB][ODBC Driver 1.0]Client unable to establish connection (SQL-08001)(DBD: db_login/SQLConnect err=-1) > > > > > > cat monetdb2.pl > > > #!/usr/bin/perl > > > > > > use DBI(); > > > > > > > > > my $dbh = DBI-> connect('dbi:ODBC:DSN_MonetDB', 'monetdb', 'monetdb', {database => demo}) or die ($DBI::errstr . "\n"); > > > > > > if (!$dbh) > > > { > > > print "$DBI::err\n$DBI::errstr\n$DBI::state"; > > > } > > > else > > > { > > > $dbh->disconnect if ($dbh); > > > } > > > > > > my $sth = $dbh->prepare('SELECT * FROM env() env'); > > > $sth->execute; > > > $sth->dump_results; > > > > > > This the merovingian output : > > > tail -f /root/MonetDB/var/log/merovingian.log > > > MSG demo[11229]: # MonetDB server v5.6.0, based on kernel v1.24.0 > > > MSG demo[11229]: # Serving database 'demo' > > > MSG demo[11229]: # Compiled for i686-pc-linux-gnu/32bit with 32bit OIDs dynamically linked > > > MSG demo[11229]: # Copyright (c) 1993-2008 CWI, all rights reserved > > > MSG demo[11229]: # Visit http://monetdb.cwi.nl/ for further information > > > MSG demo[11229]: # Listening for connection requests on mapi:monetdb://ulys:50001/ > > > ERR merovingian[11224]: client error: no database specified > > > MSG merovingian[11224]: database 'demo' already running since 2008-09-10 00:17:32, up min/avg/max: 30/2181/7036, crash average: 0.00 0.00 0.00 (6-5=0) > > > MSG merovingian[11224]: proxying client 127.0.0.1:59244 for database 'demo' to mapi:monetdb://ulys:50001/ > > > MSG merovingian[11224]: client has 127.0.0.1:59244 disconnected from proxy > > > > > > ERR merovingian[11224]: client error: no database specified > > > > > > This is the content of my odbc.ini & odbcinst.ini : > > > > > > [DSN_MonetDB] > > > Description = ODBC Driver for MonetDB SQL Server > > > Driver = /root/MonetDB/lib/libMonetODBC.so > > > Port = 50000 > > > Host = localhost > > > Database = demo > > > User = monetdb > > > Password = monetdb > > > Debug = 1 > > > Trace = 1 > > > DebugFile = /var/log/odbc-debugfile.log > > > TraceFile = /var/log/odbc-tracefile.log > > > > > > odbcinst.ini > > > [DSN_MonetDB] > > > Description = ODBC Driver for MonetDB SQL Server > > > Driver = /root/MonetDB/lib/libMonetODBC.so > > > Setup = /root/MonetDB/lib/libMonetODBCs.so > > > FileUsage = 1 > > > CommLog = 1 > > > Debug = 1 > > > > > > I also tried with PHP odbc, same result - how to specify the DB right from the script ? > > > > > > > > > > > > ------------------------------------------------------------------------- > > > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > > > Build the coolest Linux based applications with Moblin SDK & win great prizes > > > Grand prize is a trip for two to an Open Source event anywhere in the world > > > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > > > _______________________________________________ > > > Monetdb-developers mailing list > > > Monetdb-developers(a)lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/monetdb-developers > > > > > > -- > > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > > | The Netherlands | Fax : +31 (20) 592-4312 | > > > > > > > > > -- > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > | The Netherlands | Fax : +31 (20) 592-4312 | > > > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-developers mailing list > Monetdb-developers(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-developers -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal bpm.mx, , 1.137, 1.138 mal_mapi.mx, , 1.16, 1.17
by Stefan Manegold 11 Sep '08

11 Sep '08

On Wed, Sep 10, 2008 at 07:06:16PM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv20655 > > Modified Files: > bpm.mx mal_mapi.mx > Log Message: > bpm- defense against failed BATextend As mentioned before, proper error handling would be the better solution than an assert in such cases --- recall and be aware the assertions are disabled in released versions; hence, case of a failing BATextend the released version will react with a segfault! Stefan > > > U bpm.mx > Index: bpm.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/bpm.mx,v > retrieving revision 1.137 > retrieving revision 1.138 > diff -u -d -r1.137 -r1.138 > --- bpm.mx 25 Aug 2008 16:12:42 -0000 1.137 > +++ bpm.mx 10 Sep 2008 19:06:14 -0000 1.138 > @@ -1535,7 +1535,8 @@ > bn= BATdescriptor(px->bid); > if( bn== NULL) > throw(MAL,"bpm.addPartition","Partition BAT not found"); > - BATappend(bn,b,TRUE); > + bn= BATappend(bn,b,TRUE); > + assert(bn!=NULL); ^^^^^^^^^^^^^^^^^ > px ->cnt += BATcount(b); > BBPunfix(bn->batCacheid); > #ifdef _DEBUG_BPM_ > @@ -2266,7 +2267,8 @@ > b= BATdescriptor(i); > if( b== NULL) > throw(MAL,"bpm.fold","Partitioned BAT not found"); > - BATappend(bn,b,FALSE); > + bn= BATappend(bn,b,FALSE); > + assert(bn!=NULL); ^^^^^^^^^^^^^^^^^ > /* remove its persistency counter */ > BBPdecref(b->batCacheid,TRUE); > BBPunfix(b->batCacheid); > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 2

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal tablet.mx, , 1.170, 1.171
by Stefan Manegold 10 Sep '08

10 Sep '08

On Wed, Sep 10, 2008 at 10:36:27AM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv17819 > > Modified Files: > tablet.mx > Log Message: > More defenses. > > > U tablet.mx > Index: tablet.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/tablet.mx,v > retrieving revision 1.170 > retrieving revision 1.171 > diff -u -d -r1.170 -r1.171 > --- tablet.mx 10 Sep 2008 09:30:16 -0000 1.170 > +++ tablet.mx 10 Sep 2008 10:36:19 -0000 1.171 > @@ -1964,10 +1964,12 @@ > for( i=0; i< (int)as->nr_attrs; i++) { > if (as->format[i].c) { > /* prepare enough space */ > - BATextend( as->format[i].c, cnt ); > + as->format[i].c = BATextend( as->format[i].c, cnt ); > + assert(as->format[i].c != NULL); Assertions are (usually) ment for ensuring consistency of the code and its usage and (hence) to detect situations that may/must ("can") not occur under "normal" circumstances; A BATextend (i.e., a malloc) can very well fail in normal circumstance (e.g., when running out of memory); hence, such a situation should (must?) better be detected and handled by some error handling code, not by an assertion; Recall and be aware that assertions are disabled in the release production version, i.e., in this case a failing BATextend (e.g., due to shortage of memory or address space) will lead to undefined behavor of the code, potentially a segmentation fault. This is IMHO not desired ... Stefan > for (j=0; j<nway; j++) { > assert (partition[j]->format[i].c); > - BATappend(as->format[i].c, partition[j]->format[i].c,TRUE); > + as->format[i].c= BATappend(as->format[i].c, partition[j]->format[i].c,TRUE); > + assert(as->format[i].c!= NULL); > BATsave(as->format[i].c); > /* Throw away the temporary bats a.s.a.p. without saveing */ > BBPunfix(partition[j]->format[i].c->batCacheid); > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0