users-list June 2010

users-list@monetdb.org

23 participants
24 discussions

[MonetDB-users] Bulk Data Extraction & Embedded mserver5

by Hering Cheng

Hi, I was wondering if you can give me some ideas on how to retrieve data from MonetDB at very high throughput. The query will be very simple, issued against a single table. This table contains billions of records and I'd like to retrieve the data at a rate of at least one million records per second. From my naive tests with mclient running on the same 16-core machine as mserver5, I am only able to extract the data at about 20,000 records per second with the Feb2010 release. As a baseline case, with the same data in compressed (gzip'd) ASCII form stored in a regular file on the same (SAN) file system as MonetDB, I am able to read at the desired speed of one million records per second. I understand that there is communication overhead between mserver5 and mclient (or whatever client I use). Therefore, one possibility is to embed my application within mserver5. The embedded application basically just needs to be able to issue a SQL (or even MAL) query against the enclosing mserver5 and process the result set. If this is a viable approach, I'd like some guidance on where the hooks are. Thanks. Hering Cheng

11 years, 11 months

[MonetDB-users] hash accelerator

by Miguel Ángel Águila Lorente

Hello, I'm working with mal and I've seen in the web documentation, in the part of the inner core that there are built-in accelerator functions. As I can see, there is one search accelerator: hashing. ¿Could you tell me how to use this accelerator using mal language? Looking forward to hearing from you. Thank you. Mike

12 years

[MonetDB-users] full text index?

by Patrick Schäfer

Hi, is there any support for a full text index? In the documentation it says: "port PF/Tijah to support the XQuery Full-Text specification." Thank you Patrick

12 years

[MonetDB-users] How to recover data from a crash?

by 李坤

Hi all. I plan to deploy monetdb into production environment, but i don't know how to recover from a crash. I find some sql log files in var\sql_logs\$dbname\sql\, can i use these log files to recover, and how? Or are there any other methods to recover from crashes?

12 years

[MonetDB-users] problem with monetdb

by Andrea Panetta

Hi at All. I've a problem with monetdb... I need to use it with jdbcDriver eith iSQL viewer but, when I user mi driver, and i load it with my credential, I've the following error Error: ResultSet is closed. It's strange because, when I try the driver with eclipse, my programs work well but, when I use this driver with an external software, like isql viewer, it don't work. How I can do to fix this error? Thank you so much. (PS: I've the monetdb jdbc driver version 1.15, and it should be the laste updated driver) -- Panetta Andrea Via Pertini 22 87055 - San Giovanni in Fiore (CS) tf - 329 / 3554684

12 years, 1 month

[MonetDB-users] MonetDB aborts when importing over 10 million rows

by Luis Fernando Kauer

Hi, I'm still trying to evaluate MonetDB but I keep getting errors when trying to import over 10 million rows to a table. The server simply aborts and closes. I'm using MonetDB on Windows XP. Whenener I restart after the server had aborted it shows messages like this: #GDKmmap(71827456) fails, try to free up space [memory in use=18552560,virtual m emory in use=1837957120] #GDKmmap(71827456) result [mem=18552560,vm=1837957120] #GDKmmap(36175872) fails, try to free up space [memory in use=16499872,virtual m emory in use=1844903936] #GDKmmap(36175872) result [mem=16499872,vm=1844903936] #GDKmmap(143130624) fails, try to free up space [memory in use=15697728,virtual memory in use=1744764928] #GDKmmap(143130624) result [mem=15697728,vm=1744764928] #GDKmmap(143130624) fails, try to free up space [memory in use=13554656,virtual memory in use=1654456320] #GDKmmap(143130624) result [mem=13554656,vm=1654456320] #GDKmmap(143130624) fails, try to free up space [memory in use=8747960,virtual m emory in use=1650917376] #GDKmmap(143130624) result [mem=8747960,vm=1650917376] #GDKmmap(143130624) fails, try to free up space [memory in use=7946840,virtual m emory in use=1531052032] #GDKmmap(143130624) result [mem=7946840,vm=1531052032] !mvc_init: unable to create system tables !SQLException:SQLinit:Catalogue initialization failed !ERROR: HEAPextend: failed to extend to 71722596 for 14\1475tail !ERROR: HEAPextend: failed to extend to 35861298 for 15\1537tail !ERROR: HEAPextend: failed to extend to 142816048 for 15\1543tail !ERROR: HEAPextend: failed to extend to 142816048 for 15\1550tail !ERROR: HEAPextend: failed to extend to 142816048 for 15\1556tail !ERROR: HEAPextend: failed to extend to 142816048 for 15\1557tail #function user.main():void; # sql.prelude(); #end main; First I was trying to import files with about 3 million rows each but it started giving errors after the 3rd file. Then I tried to import through files with about 1 million rows each and I was able to import about 14 million rows, but after about 10 million rows I was only able to import one or two files in a row without MonetDB resetting. Is this a bug or a size limit for the available memory. Anything I could try? With regards, Luis Fernando

12 years, 1 month

[MonetDB-users] "Parent Directory" links in download area give 404.

by Mark Bucciarelli

(1) open http://monetdb.cwi.nl/downloads/Ubuntu/ (2) click "Parent Directory" link (3) 404 I saw same behavior after opening http://monetdb.cwi.nl/downloads/sources/Feb2010-SP2 so it may be a global issue. Thanks, m

12 years, 1 month

[MonetDB-users] Copy Into and .new files

by Alex Bo.

Hi, I am doing a lot of COPY INTOs, and I have noticed that sometimes while doing this MonetDB seems to hang while its process memory is slowly rising. I then noticed that while this is happening, files with a '.new' extension are added to the dbfarm folders. Now suddenly I reached a state in which no '.new' files are created at all, and the COPY INTOs work much faster because no 'hanging' occurs in which something is loaded into memory. What's the explanation for this? What can I do in order to always have COPY INTOs write directly to the final files without the '.new' intermediates? Thanks. -- View this message in context: http://old.nabble.com/Copy-Into-and-.new-files-tp28973215p28973215.html Sent from the monetdb-users mailing list archive at Nabble.com.

12 years, 1 month

Re: [MonetDB-users] Embedded server in Java with JDBC

by Luis Fernando Kauer

Hi, Thanks for the quick reply. When I say embedded I mean like in HSQLDB, which allows to embed the db server in the application instead of starting the db server in a separete process accepting connections from all clients. My requirements are that the end user should not have to deal with the database directly, like start and stop it, not even need to install/configure it separetely. But I would like to access the data through JDBC driver because the libraries I'm working with require it. I also would like a small footprint. If that's not possible, then I would like to be able to start and stop the server through the application, even if it runs in a separete process. The application allows the user to analyse and audit a large volume of data on different tables and MonetDB seems to be really fast for this. Each user project will have its own set of tables so it would be nice to be able to set the database files where the user chooses for each project. If that's not possible I'll have to stick with different schemas for each project. I'm still having some problems importing large tables (> 10 million rows divided in 10 files with 1 million rows each) when MonetDB is reporting the following error SQLException:sql:Failed to extend the BAT, perhaps disk full SQLException:importTable:failed to import table and on the console appears many times the follwing: #GDKmmap(111673344) fails, try to free up space [memory in use=56663400,virtual memory in use=1521352704] #GDKmmap(111673344) result [mem=56663160,vm=1127612416] #GDKmmap: recovery ok. Continuing.. Is there any configuration or a way to make MonetDB work with large tables without having more memory available? Thanks for your help! Regards, Luis Fernando

12 years, 1 month

[MonetDB-users] Embedded server in Java with JDBC

by Luis Fernando Kauer

Hi, I would like to use the embedded server within a Java application connecting through JDBC. Is it possible? I would probably need to start the embedded server before trying to connect through JDBC, right? Or is there a connection string that makes the jdbc driver start and use an embedded server? That would be great! If not, what's the best way to start the embedded server in Java? Thanks for this great and fast dabatase. Regards, Luis Fernando

12 years, 1 month

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

users-list June 2010