December 2009 - users-list

Re: [MonetDB-users] mserver5 crashed after 10, 000 times sql queries through JDBC interface
by xulizhong 26 Dec '09

26 Dec '09

> Could you provide the query and schema? CREATE TABLE STOCKINFO ( STOCK_ID INTEGER NOT NULL, STOCK_CODE VARCHAR(30), EX_CODE VARCHAR(30), SH_CODE VARCHAR(30), LONGNAME_CN VARCHAR(50), SHORTNAME_CN VARCHAR(30), EX_ID INTEGER NOT NULL, CSRC_CAT INTEGER DEFAULT NULL, SW_CAT INTEGER DEFAULT NULL, SH_CAT INTEGER DEFAULT NULL, HAS_DETAIL BOOLEAN DEFAULT NULL, BG_CODE VARCHAR(20), IPO_TIME DATE DEFAULT NULL, BOARD_CAT INTEGER DEFAULT NULL, IS_ST BOOLEAN DEFAULT NULL, HAS_OPTION BOOLEAN DEFAULT NULL, CURRENCY INTEGER DEFAULT NULL, CARRY INTEGER DEFAULT NULL, DECIMALS INTEGER DEFAULT NULL, MIN_PRICE_CHANGE DOUBLE DEFAULT NULL, STOCK_CAT INTEGER DEFAULT NULL, SURGED_LIMIT DOUBLE DEFAULT NULL, DECLINE_LIMIT DOUBLE DEFAULT NULL, HIGH_52WK INTEGER DEFAULT NULL, LOW_52WK INTEGER DEFAULT NULL ); query: SELECT STOCK_ID FROM STOCKINFO WHERE LOW_52WK<85 and HIGH_52WK>110 and EX_ID=15 AND IPO_TIME>'2000-1-1' AND STOCK_CODE LIKE '1%'; I write a simple java program using 4 threads to execute this query with a connection pool of 4 max active connections. The mserver5 process' memory increase steadily upto 2.5G, then the process crashes with segmentation fault:( I have noticed something special, when I run the query using : ResultSet result = conn.prepareStatement(sql).executeQuery(); the monetdb5 server will consume large amount of memory and crash definitely. However, when I query using: ResultSet result = conn.createStatement().executeQuery(sql); the server's memory usage is very stable. This may indicate there's something wrong with the prepared statement of the sql module of MonetDB5. Below is my test program: ////////////////////////////////////////////////////////// import java.sql.Connection; import java.sql.ResultSet; import java.sql.SQLException; import java.util.Date; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.TimeUnit; import org.apache.commons.dbcp.BasicDataSource; public class MonetBenchmark { /** * @param args */ public static void main(String[] args) { ExecutorService executor = Executors.newFixedThreadPool(4); final BasicDataSource dataSource = new BasicDataSource(); dataSource.setMaxActive(4); dataSource.setInitialSize(4); dataSource.setDriverClassName("nl.cwi.monetdb.jdbc.MonetDriver"); dataSource.setUrl("jdbc:monetdb://localhost/demo"); dataSource.setUsername("monetdb"); dataSource.setPassword("monetdb"); final String sql = "SELECT STOCK_ID FROM STOCKINFO WHERE LOW_52WK<85 and HIGH_52WK>110 and EX_ID=15 AND IPO_TIME>'2000-1-1' AND STOCK_CODE LIKE '1%';"; System.out.println("running ... " + new Date());; long start = System.currentTimeMillis(); int loop = 10000; for (int x = 0; x < loop; x++) { executor.submit(new Runnable() { public void run() { Connection conn = null; try { conn = dataSource.getConnection(); ResultSet rs = conn.createStatement().executeQuery(sql); //using prepared statement will cause the server crashed with segmentation fault and very big memory usage. // ResultSet rs = conn.prepareStatement(sql).executeQuery(); while(rs.next()) { //... } } catch (SQLException e) { e.printStackTrace(); } finally { try { conn.close(); } catch (Exception e) { } } } }); } try { executor.shutdown(); executor.awaitTermination(5, TimeUnit.MINUTES); } catch (InterruptedException e) { e.printStackTrace(); } long time = System.currentTimeMillis() - start; System.out.println(time + " ms for " + loop + " times query! " + (time / loop) + " ms/q"); } } And my last question, is this newest version of MonetDB suitable for production use if I always use normal jdbc statement to execute my query? Thanks alot.

3 3

[MonetDB-users] OCaml binding for embedded MonetDB5.
by Guillaume Yziquel 24 Dec '09

24 Dec '09

Hello. I would like to create a binding for the Objective Caml language of an embedded MonetDB5. My personal focus would be on embedded_mal, but I'm perfectly willing to get started on embedded_sql. I've been looking at some documentation, which I unfortunately find rather terse. For instance: http://monetdb.cwi.nl/MonetDB/Documentation/A-Simple-Example.html I'm, for example, wondering about the semantics of the embedded_sql function whose code from embeddedclient.c.in is reproduced below. > Mapi > embedded_sql(opt *set, int len) > { > Mapi mid; > pthread_t sqlthread; > stream **server; > > if (set) { > embedded_set = set; > embedded_len = len; > } > server = mapi_embedded_init(&mid,"sql"); > > pthread_create(&sqlthread, NULL, start_sql_server, (void *) server); > > mapi_start_talking(mid); > > return mid; > } I'd therefore like to know if there is some piece of documentation (aside the code itself) where I could gain a deeper understanding of what I want to wrap up and bind. Links appreciated. -- Guillaume Yziquel http://yziquel.homelinux.org/

1 1

[MonetDB-users] monetdb program in debian packages?
by Guillaume Yziquel 24 Dec '09

24 Dec '09

Hello. I've been trying to follow section 1.78.1 Start and Stop the Server of the manual for MonetDB 5, and I fail to find the monetdb program in the available debian packages. Which package should I install to be able to launch the monetdb program as in section 1.78.1? All the best, -- Guillaume Yziquel http://yziquel.homelinux.org/

3 5

[MonetDB-users] mserver5 crashed after 10, 000 times sql queries through JDBC interface
by xulizhong 24 Dec '09

24 Dec '09

I'm using the latest MonetDB5: MonetDB-Nov2009-SuperBall-SP1 on Red Hat Enterprise Linux AS release 4 (Nahant Update 4) x86_64. dependencies: openssl-0.9.8l, pcre-8.00. The box has 8G physical memory, about half of them is available. building options: ./monetdb-install.sh --prefix=/opt/monetdb5 --enable-sql --enable-optimise After installation, I started the mserver5 process: /opt/monetdb5/bin/mserver5 --dbinit "include sql;", then I load one table exported from MySQL with about 1M rows and 20 columns. I noticed small memory increase when I run a simple query on the mclient console for some times. So I run this query through the JDBC interface for 10000, I can see a big memory consumption rise when the test program is running. and finally the mserver5 process crashed with segmentation fault. I can always reproduce this problem. What's wrong? Have I used some wrong build options? Help! Frank

2 1

[MonetDB-users] Fatal error: aborted too deep recursion
by Harm van Beek(DT) 24 Dec '09

24 Dec '09

During the development of our tool, using Xquery on MonetDB, we ran onto the following error: ERROR = !fatal error: aborted too deep recursion >From previous posts (I guess) I can conclude that this relates to an Xquery that simply is too big (too many operators) to create a good plan for it. I tried several MonetDB releases (May2009-SP2, Aug2009-SP2, Nov2009-DB0) on Red Hat Enterprise Linux Server release 5.4 (Tikanga), all showing the same result. Does anyone have any directions on how to solve this? Can we reconfigure the compiler to fix this (or recompile it with options)? Thanks in advance! Harm van Beek Netherlands Forensic Institute

6 11

[MonetDB-users] man page for debian package.
by Guillaume Yziquel 23 Dec '09

23 Dec '09

Hi. Just a simple suggestion: I'm using the Debian package for MonetDB v5, and have just started tinkering with it. One thing I really like is a man page. It would be nice to have one. Not necessarily an exhaustive one, but one that would comment the mserver5 options a bit more clearly than mserver5 --help, and which would provide pointers to documentation. Finding documentation on monetdb wasn't easy. Writing good documentation is a painful task, but providing pointers to documentation within a man page is much easier... All the best, -- Guillaume Yziquel http://yziquel.homelinux.org/

1 0

[MonetDB-users] Memory Usage
by Rui Anastácio 23 Dec '09

23 Dec '09

Hi ! I'm runnig MonetDB under Linux Ububtu with 4Gb of RAM. I need to benchmark it running 1Gb, 2Gb, 3Gb and 4Gb. Todo so I need to limit the memory usage. How can I configure MonetDB to limit it's memory usage? and doing so am I able to run complex queries (with short memory) ? Regards Rui Anastácio

4 4

[MonetDB-users] MonetDB scalability
by Bojan Šmid 22 Dec '09

22 Dec '09

Hi, I am thinking of using MonetDB as a data storage for my application. The application will create lots of data (few GB of new data each day) while there will not be many queries (probably few thousands/day). Queries will be complex, but they should execute really fast. Since data will grow pretty fast, I must prepare for scaling, so I need to know is there a way to distribute MonetDB over multiple machines? Thanks and regards

4 5

Re: [MonetDB-users] MonetDB scalability
by OG 22 Dec '09

22 Dec '09

Hello, Somebody on this list asked about scalability the other day: http://markmail.org/message/2unmuwq2cfz7mrmm I was wondering the same thing. Is there a way to partition/shard MonetDB and still have joins work across partitions living on separate servers? If not, is there a recommended practice for what to do with MonetDB once the limit of a single server has been reached? Thanks, Otis -- Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch

2 1

[MonetDB-users] UTF-8 for table/column names
by Alex Bo. 22 Dec '09

22 Dec '09

Hi, Is there support for UTF8/Unicode for field/table names? It seems like i can only use ASCII characters (for example, a field name with specific Spanish letters wouldn't work) inside field/table names. I'm using the Nov09 on Windows. Thanks. -- View this message in context: http://old.nabble.com/UTF-8-for-table-column-names-tp26888868p26888868.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 3