January 2009 - users-list

[MonetDB-users] Optimizer exceptions when working with Mapi Interface
by David Svenson 08 Jan '09

08 Jan '09

Hi all, I am encountering strange exceptions that are related to the optimizer when working with the MAPI interface. I seem to be getting them when I use a few parallel threads to query the database (Always with the same query). However, I don't get them in a consistent way - I can't reproduce an exact scenario. The exception messages I get also vary - but they are always related to the optimizer. Here are a few of the exception messages I got: 'optimizer.rsColumn' undefined in: _50:any := optimizer.rsColumn() 'calc.opt_remoteQueries' undefined in: _16:any := calc.opt_remoteQueries(_17:oid) 'optimizer.inc_parts' undefined in: _39:any := optimizer.inc_parts() Sometimes, once I get one of these, any following query will give me an exception too, unless I restart MonetDB. In these cases I tried to run the query through mclient, and got the exception too, but followed by the expected query result. Any idea where this is coming from and what I can do to prevent it? Thanks, David. -- View this message in context: http://www.nabble.com/Optimizer-exceptions-when-working-with-Mapi-Interface… Sent from the monetdb-users mailing list archive at Nabble.com.

2 3

[MonetDB-users] large COPY INTO jobs vs. INSERT
by David Svenson 08 Jan '09

08 Jan '09

Hi, I am working with MonetDB v 5.8.2 on a 32 bit machine with 2GB of RAM. I have a 20 CSV files, each containing 200,000 rows and 59 columns. I want to import all of them into one table. When I start running 'COPY INTO's to do that, everything goes well untill the 15th file (which should bring the table up from 2.8 million rows up to 3 million). The 15th file fails. When working with the MAPI interface, the database ends up being corrupt - but works again if I remove the logs. When trying the same directly through mclient.exe, it also fails at around the 15th file, and gives this error: !SQLException:sql.bind:Cannot access descriptor !ERROR: GDKload: cannot read: name=11\1100, ext=tail, 400000 bytes missing. !OS: The operation completed successfully. !ERROR: GDKload: failed name=11\1100, ext=tail 0 tuples I tried two things: 1. I ran 16 'COPY INTO', but this time into 16 different tables. Then I ran INSERT sqls on all of them into one target table. That worked fine, and I passed the 3 Million barrier I ran into with COPY INTO. 2. I tried the same on a 64 bit machine - Everything worked just fine with 'COPY INTO' So what does all of this mean? How do I interpret the error message? Can I / Should I expect from a 32 bit machine to be able to handle what I wanted it to? And if it's a memory issue - then how come everything worked with INSERTs but not with COPY INTOs. Thanks, David. -- View this message in context: http://www.nabble.com/large-COPY-INTO-jobs-vs.-INSERT-tp21351353p21351353.h… Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

[MonetDB-users] copy data from postgresql to monetdb?
by Chris Curvey 08 Jan '09

08 Jan '09

Hi! Newbie here! Before I go about trying to re-invent the wheel, has anyone written scripts to reverse-engineer a PostgreSQL database, convert the SQL to MonetDB's dialect and copy in the data? (I've Google and I can't seem to find such an animal.) -Chris -- Corruptissima republica plurimae leges

2 1

[MonetDB-users] Function errors
by David Svenson 08 Jan '09

08 Jan '09

Hi there... I've been trying to export the mtime. str_to_date function to the sql interface by calling the following script from the sql client: create function str_to_date(s string, format string) returns date external name mtime."str_to_date"; After the function was created, I am trying to use it in the following way: select str_to_date('6 Dec 2001 12:33:45', '%d %b %Y %H:%M:%S'); But, I always get an error: "mtime.str_to_date:strptime support missing" Am I supposed to load some extra module into the server that is not loaded into the monet server on startup? Thanks for everthing… -- View this message in context: http://www.nabble.com/Function-errors-tp21336686p21336686.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

Re: [MonetDB-users] Hello. some questions about XRPC...
by Ying Zhang 07 Jan '09

07 Jan '09

Hello Nicolas, I need some more information about your problems to be able to help. First of all, some general questions: - which version of MonetDB/XQuery do you use? - which OS? - some hardware specifications On Jan 06, 2009, at 14:45 , Nicolas James wrote: > Hello Ying, > > Thanks for all these responses, they have been very helpful to me > (the real problem I had, and now resolved, was about the call to the > functions of pathfinder). > > I've coded a Java client for indexing my data (image descriptions > and annotations in XML files) in MonetDB/XQuery, 46,557 XML > documents / 181.1Mo and I use XRPC. Is 181.1MB the size of one document (then the total size is > 8TB!) or the size of all 46557 documents? Do I understand it correctly that you have a Java client which send XRPC request messages to MonetDB/XQuery server to add documents? How does your Java client exactly work, i.e.: - does it add one document at a time, or all 46557 documents in one request? - is your Java client and MonetDB/XQuery server on the same machine? - do you use pf:add-doc() or fn:put() to add the documents? > My client works well but I don't understand some MonetDB behaviors. > My first tests created collections like my dataset and his > directories structure, I don't understand what you mean here. As far as I know, it is not possible to create nested collections in MonetDB/XQuery, and it is not possible to create empty collections (without adding a document). So, how did you create the collections? > but I've some directories with over that 900 documents, and I think > that MonetDB doesn't like that (i.e. 900 documents in one > collections), because the indexation failed. I don't know the exact max nr. of documents a collection may have, I think it is limited by if you are using a 32bits system or a 64bits system. Using mlcient, adding 46,557 documents of 11k each on a Fedora machine was no problem; adding > 900 documents of 200MB each is still processing (disk usage is still growing) when I'm writing this e-mail :) Did you add your documents using your Java client? Have you tried to add your documents without using XRPC, e.g., using mclient? What do you mean with "indexation failed"? Is there any error message? > Then, I try to split my dataset, I create collections of 128 > documents, but my problem is that in MonetDB all my collections does > not contain 128 doc, some 128 others a number between 5 and 128... > It's a little strange because I don't think the reason is my client... For this problem, I have similar questions, i.e.: - how large are the documents? - how are they added _exactly_? - Did you get any error message? Kind regards, Jennie > > Have you ever heard about a similar problem in MonetDB? > > Thanks previous responses, > > best regards, > > Nicolas James > > > Ying Zhang wrote: >> Hello Nicolas, >> >> First of all, thanks for trying MonetDB. I will try to answer your >> XRPC >> questions. >> >> >> On Tue, Dec 16, 2008 at 04:35:33PM +0100, Nicolas James wrote: >> >>> Hello MonetDB, >>> >>> I have some questions about the writing of a client for MonetDB in >>> Java using the XRPC, and It could be very very helpful for my >>> research if you could answer to some questions. >>> >>> I'm a PhD student in the MAS laboratory, Ecole Centrale Paris, >>> France, and I work on image automatic annotation. >>> I think MonetDB is extremely powerful and I want to use it in my >>> research, I would to store some images annotations under the form >>> of XML file in MonetDB since my application (a Java application). >>> >>> So, with your documentation I coded a light client using the JDBC >>> driver for adding my documents in MonetDB, but this way don't >>> satisfy me, the XRPC method is very attractive, but I've some >>> problems. >>> >>> 1) how add a document using the XRPCWrapper (using the pf:add-doc, >>> it's considered like an updating function, I've read the document, >>> but I don't understand...). >>> >> >> XRPC has several components. >> The XRPC protocol is a way of communicating with MonetDB/XQuery to >> run >> XQuery queries. An XRPC server is integrated in MonetDB/XQuery to >> handle requests speaking the XRPC protocol. >> >> Hence, to communicate with MonetDB/XQuery using XRPC, you only need >> to >> send requests according to the XRPC protocol. For Java, there are >> several API functions for creating request messages, sending and >> receiving messages, and handling response messages. The Java doc of >> these APIs could be found here: >> http://homepages.cwi.nl/~fabian/MonetDB/Java%20Package.html/index.html?nl/c… >> The Java package of the APIs could be found under the directory where >> you have installed MonetDB: share/MonetDB/lib/xrpcapi.jar >> In the same directory there is an xrpcclient.jar package, which gives >> an example of how to use the APIs. It's source could found in: >> java/src/nl/cwi/monetdb/xquery/xrpc/client >> >> You don't need the XRPCWrapper, in your case. XRPCWrapper is >> needed if >> you want to handle (simple) XRPC requests using another XQuery engine >> than MonetDB/XQuery. >> >> >> >>> 2) Is-it possible to create collections using the XRPC (and add >>> files in this collection) ? >>> >> >> I'm not sure if I understand you correctly. >> >> With XRPC, you can execute the pf:add-doc() function (you need to >> put it >> into an user-defined-function). with pf:add-doc(), you could specify >> the name of the collection to which you want to add your document. >> If >> such collection doesn't exist, it is created. Otherwise, your >> document >> will be added to the existing collection. >> >> >>> 3) and with the JDBC driver ? >>> 4) using the JDBC driver, is it possible to specify the reference >>> id for the file (for instance indexing the file "foobar.xml" and >>> referenced "foobar.xml" in the database ? >>> >> >> What do you mean with 'indexing the file "foobar.xml"'? >> (Disclaimer: I don't know much about the JDBC driver) >> >> >> Hope this helps a bit. If you have any problem with using the XRPC >> API, >> please don't hesitate to ask. >> >> >> >> Regards, >> >> Jennie >> >> >>> These answers could help me, hope you could have time to answer! >>> >>> best regards, >>> >>> -- >>> Nicolas James >>> PhD student MAS / ECP >>> Phone: +33.1.41.13.17.37 >>> Office: Lab, C 441A >>> Laboratoire MAS - Ecole Centrale Paris >>> Grande Voie des Vignes >>> 92295 CHATENAY MALABRY Cedex >>> France >>> >>> >> >> > > > -- > Nicolas James > PhD student MAS / ECP > Phone: +33.1.41.13.17.37 > Office: Lab, C 441A > Laboratoire MAS - Ecole Centrale Paris > Grande Voie des Vignes > 92295 CHATENAY MALABRY Cedex > France >

1 0

[MonetDB-users] what is monet_vault_key used for?
by 小波顾 05 Jan '09

05 Jan '09

_________________________________________________________________ More than messages–check out the rest of the Windows Live™. http://www.microsoft.com/windows/windowslive/

3 2

[MonetDB-users] Is there any way to run MonetDB as a Windows Service?
by 小波顾 05 Jan '09

05 Jan '09

Thanks. _________________________________________________________________ More than messages–check out the rest of the Windows Live™. http://www.microsoft.com/windows/windowslive/

3 2

[MonetDB-users] COPY INTO on aggregate queries crashes MonetDB
by Peter Farkash 04 Jan '09

04 Jan '09

Hi, When I use COPY INTO with a count query, monetDB crashes. For example: copy select count (*) into 'C:\\copyinto.txt' using delimiters ','; Gives me the following result and kills the server: MAPI = monetdb@localhost:50000 ACTION= read_line QUERY = copy select count (*) into 'C:\\copyinto.txt' using delimiters ','; ERROR = Connection terminated The same happens if I use sum, average, etc. On normal queries the copy into works just fine. I am using v5.8.2, and am getting the crash both when using mclient and the MAPI programming interface. Any ideas? -- View this message in context: http://www.nabble.com/COPY-INTO-on-aggregate-queries-crashes-MonetDB-tp2127… Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] What code page does Monetdb currentlly use?
by 小波顾 03 Jan '09

03 Jan '09

We are Chinese users, and hoping Monetdb will support multi-byte character encodings like UTF-8 and GBK. _________________________________________________________________ More than messages–check out the rest of the Windows Live™. http://www.microsoft.com/windows/windowslive/

5 6

[MonetDB-users] Monetdb5 threading question.
by 小波顾 02 Jan '09

02 Jan '09

When I start the default demo database (or a new created one) process on Windows XP, the following messaged appears in the command window : # Serving databse 'demo', using 1 thread Does it mean the whole database process is single-threaded? The following section in the monetdb5.config file shows we can config multiple threads # Mserver 5 recognizes a series of command line names instead:# --threads ( 1 | PARMASK)# --memory (MEMMASK )# --properties (CHECKMASK | PROPMASK | BATMASK )# --io (IOMASK | PERFMASK )# --transactions ( TMMASK | DELTAMASK | TEMMASK)# --modules (DLMASK | LOADMASK)# --algorithms (ALGOMASK | ESTIMASK)# --xproperties (XPROPMASK )# --performance (JOINPROPMASK | DEADBEEFMASK) What does 'parmsk' mean, and how to config monetdb to use multiple threads? _________________________________________________________________ More than messages–check out the rest of the Windows Live™. http://www.microsoft.com/windows/windowslive/

3 4