January 2008 - users-list

Re: [MonetDB-users] tuning advice for TPC-H
by Rahul Chopra 18 Jan '08

18 Jan '08

I thought I would try and ask these questions again, since there was no response. - How does how MonetDB handles its virtual memory? Does it swap out entire BATs at a time? Or just smaller blocks? What type of caching algorithms does it use? - If I configure a large swap, MonetDB is not using it. I guess it allocates based on actual physical memory. If I want to run a test to let the OS handle virtual memory and have MonetDB try and use more of system virtual memory, how do I do that? I am interested in seeing how this affects performance when the database is too large to fit in memory. - Are there any other tuning settings I should set when using a database that is much larger than memory? Thanks in advance, Rahul > ----- Original Message ----- > From: "Rahul Chopra" <rchopra(a)linuxmail.org> > To: "Communication channel for MonetDB users" <monetdb-users(a)lists.sourceforge.net> > Subject: Re: [MonetDB-users] tuning advice for TPC-H > Date: Thu, 10 Jan 2008 03:14:54 +0800 > > > > I am still trying to get better performance when doing TPC-H tests > for a database more than twice the size of physical memory. > > I am not sure how MonetDB handles its virtual memory, but it sure > seems to be going to disk an awful lot. Does it swap out entire > BATs? Or just smaller blocks? What type of caching algorithms does > it use? > > Also, one of the things I wanted to try is to setup a really big > swap partition on a separate device with write-back cache enabled > and have MonetDB use it (Basically, see if Linux will virtual > memory better than MonetDB does, and take advantage of putting swap > on a separate device for speed). When I try and do that though, > MonetDB seems to just be trying to stay within the confines of > physical memory available. Is there something I can configure to > try and get it to use system virtual memory? Has anyone tried out > anything like that? > > -Rahul > > > > > > > > > > > > > > I am trying to run some TPC-H queries to get a sense for how > > > > large of databases MonetDB can handle. I am currently trying > > a > > factor 20 database. I notice that on query 9, it is taking > > over > > two hours (still running). In "top" it claims that > > virtual > memory > for the process started at 27 GB or so, and is > > now down > to 10 GB. > What is noteworthy is that CPU usage is > > pretty small, > at 2%. I > previously observed 80-100% with a > > factor 1 database. > > > > > > Does this mean you got the sf-20 loaded? What fixed your problem with > > > loading? > > > > Yes, I got it loaded. Since I had broken up the file into smaller > > chunks, I just restarted on the last chunk that it failed on > > before the crash. Eventually, it went through all of the tables. > > > > > > > > The low cpu usage indicates that MonetDB is waiting for IO. I'll later > > > this week try to repeat your experiment on a similar system. Then I'll > > > have more input on how to improve things. > > > > After writing the email I noticed that system CPU wait time was > > at 25%, with MonetDB at 2% and idle at 73% or so. So, yes, it is > > waiting on IO. Perhaps it is just my hardware. > > > > I did disable the 2 GB swap partition and rerun. After doing so, > > execution time did improve somewhat. I am not sure if this means > > one should always disable swap on a MonetDB system, since the > > system drive is slower than the data drives. I will try another > > experiment and let you know the results. > > > > > = > College For Financial Planning > Nation's most successful CFP® Program with 35+ years of experience. > http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=0badaaf35b41b0f3ca7504b… > > > -- > Powered by Outblaze > > ------------------------------------------------------------------------- > Check out the new SourceForge.net Marketplace. > It's the best place to buy or sell services for > just about anything Open Source. > http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace > _______________________________________________ > MonetDB-users mailing list > MonetDB-users(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users > = Las Vegas, Nevada Home for Sale Dave Radcliffe will help you find your Las Vegas new home for sale, dream condo, townhome, or real estate investment property. Search here for all the homes for sale in Las Vegas, Nevada. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=95c6cb18131e4c352f8c920… -- Powered by Outblaze

2 1

[MonetDB-users] Distributed database
by Michaela Buergle 17 Jan '08

17 Jan '08

Hi there, I'm new to MonetDB and trying to figure out if it can be used for a db distributed over several computers. Have any of you tried to do that? Is there any documentation available? Thanks a lot micha

4 7

[MonetDB-users] create database on different drive
by Jos van Dongen 16 Jan '08

16 Jan '08

Hi all, I’m having a few difficulties on a Windows server: - I want to move the dbfarm to another drive and have altered the gdk_farm setting in the etc\monetdb5.conf file: gdk_dbfarm=G:\MonetDB5\dbfarm. MonetDB server however keeps creating a new MonetDB dir in the ‘documents and settings’ directory, the last place where I want to store a database. Do I need to alter other settings, or is there another config file? - How do you create a new database (on the different drive) and connect to that one instead of the standard ‘demo’ db? Met vriendelijke groet/kind regards, Jos van Dongen Tholis Consulting T: 06-51169606 F: 0842-231136 M: jvdongen(a)tholis.com www.tholis.com No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 269.19.2/1224 - Release Date: 14-1-2008 17:39

3 2

[MonetDB-users] make failed for buildtools/mel
by xxiao 15 Jan '08

15 Jan '08

Hi, I am trying to build MonetDB on SPARC/Solaris 10 platform from scratch. I started from buildtools. bootstrap and configure succeed. make of Mx succeed. make of mel failed. error messages: make[3]: Leaving directory `/tmp/MonetDB-CVS/buildtools/Mx' make[2]: Leaving directory `/tmp/MonetDB-CVS/buildtools/Mx' Making all in mel make[2]: Entering directory `/tmp/MonetDB-CVS/buildtools/mel' make[2]: *** No rule to make target `all'. Stop. make[2]: Leaving directory `/tmp/MonetDB-CVS/buildtools/mel' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/tmp/MonetDB-CVS/buildtools' any help is greatly appreciated! -- View this message in context: http://www.nabble.com/make-failed-for-buildtools-mel-tp14841558p14841558.ht… Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] Incorrect version (SR4 instead of SR5) linked to from download page
by Bob Ippolito 10 Jan '08

10 Jan '08

The Super Source Tarball referenced on http://monetdb.cwi.nl/projects/monetdb/Download/index.html is SR4, when it should be SR5. I wasted a couple hours trying to make SR4 build correctly on Mac OS X 10.5 when I finally found out that I was building an old version. I re-built with SR5 and everything seems to be working fine so far. -bob

2 1

[MonetDB-users] Sqlalchemy support to be in trunk soon
by m h 10 Jan '08

10 Jan '08

Just thought I'd give any python developers here a heads up. I've recently gotten commit access to sqlalchemy and am planning on pushing Monetdb support in. Hopefully the 0.4.3 release will include it as "experimental". For motivated python users who want to try it know, the source is found at bug 874[0] cheers, matt 0 - http://www.sqlalchemy.org/trac/ticket/874

1 0

Re: [MonetDB-users] tuning advice for TPC-H
by Rahul Chopra 09 Jan '08

09 Jan '08

I am still trying to get better performance when doing TPC-H tests for a database more than twice the size of physical memory. I am not sure how MonetDB handles its virtual memory, but it sure seems to be going to disk an awful lot. Does it swap out entire BATs? Or just smaller blocks? What type of caching algorithms does it use? Also, one of the things I wanted to try is to setup a really big swap partition on a separate device with write-back cache enabled and have MonetDB use it (Basically, see if Linux will virtual memory better than MonetDB does, and take advantage of putting swap on a separate device for speed). When I try and do that though, MonetDB seems to just be trying to stay within the confines of physical memory available. Is there something I can configure to try and get it to use system virtual memory? Has anyone tried out anything like that? -Rahul > > > > > > > > I am trying to run some TPC-H queries to get a sense for how > > > large of databases MonetDB can handle. I am currently trying a > > > factor 20 database. I notice that on query 9, it is taking over > > > two hours (still running). In "top" it claims that virtual > > memory > for the process started at 27 GB or so, and is now down > > to 10 GB. > What is noteworthy is that CPU usage is pretty small, > > at 2%. I > previously observed 80-100% with a factor 1 database. > > > > Does this mean you got the sf-20 loaded? What fixed your problem with > > loading? > > Yes, I got it loaded. Since I had broken up the file into smaller > chunks, I just restarted on the last chunk that it failed on before > the crash. Eventually, it went through all of the tables. > > > > > The low cpu usage indicates that MonetDB is waiting for IO. I'll later > > this week try to repeat your experiment on a similar system. Then I'll > > have more input on how to improve things. > > After writing the email I noticed that system CPU wait time was at > 25%, with MonetDB at 2% and idle at 73% or so. So, yes, it is > waiting on IO. Perhaps it is just my hardware. > > I did disable the 2 GB swap partition and rerun. After doing so, > execution time did improve somewhat. I am not sure if this means > one should always disable swap on a MonetDB system, since the > system drive is slower than the data drives. I will try another > experiment and let you know the results. > = College For Financial Planning Nation's most successful CFP® Program with 35+ years of experience. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=0badaaf35b41b0f3ca7504b… -- Powered by Outblaze

1 0

[MonetDB-users] Info reg the Physical Tables..
by sateesh 08 Jan '08

08 Jan '08

Hi All, If i create two logical tables which has the same column then when MonetDB actually creates the Physical Tables does it create two tables for this Column.. For Ex: I created these two Tables at the Logical Level.. TAB_1 col_1 varchar(100), col_2 int TAB_2 col_3 varchar(256), col_2 int Now I home MonetDB will create these physical tables.. For TAB_1 PHYSICAL_TAB_1 PHYSICAL_TAB_2 OID , COL_1 OID , COL_2 How abt for TAB_2 Does it create PHYSICAL_TAB_3 PHYSICAL_TAB_4 OID , COL_3 OID , COL_2 or Just PHYSICAL_TAB_3 OID , COL_3 Thanks Sateesh -- View this message in context: http://www.nabble.com/Info-reg-the-Physical-Tables..-tp14692698p14692698.ht… Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

[MonetDB-users] Noob question: Why can I only initiate mserver5 as root?
by Claude Monet 08 Jan '08

08 Jan '08

Logged as root mserver5 works perfectly. Yet when I login as a user and call mserver5: >mserver5 --dbinit="include sql;" I get the error: !FATAL: GDKlockHome: Database lock '.gdk_lock' denied Any help is greatly appreciated

3 2

Re: [MonetDB-users] tuning advice for TPC-H
by Rahul Chopra 08 Jan '08

08 Jan '08

> > > > I am trying to run some TPC-H queries to get a sense for how > > large of databases MonetDB can handle. I am currently trying a > > factor 20 database. I notice that on query 9, it is taking over > > two hours (still running). In "top" it claims that virtual memory > > for the process started at 27 GB or so, and is now down to 10 GB. > > What is noteworthy is that CPU usage is pretty small, at 2%. I > > previously observed 80-100% with a factor 1 database. > > Does this mean you got the sf-20 loaded? What fixed your problem with > loading? Yes, I got it loaded. Since I had broken up the file into smaller chunks, I just restarted on the last chunk that it failed on before the crash. Eventually, it went through all of the tables. > > The low cpu usage indicates that MonetDB is waiting for IO. I'll later > this week try to repeat your experiment on a similar system. Then I'll > have more input on how to improve things. After writing the email I noticed that system CPU wait time was at 25%, with MonetDB at 2% and idle at 73% or so. So, yes, it is waiting on IO. Perhaps it is just my hardware. I did disable the 2 GB swap partition and rerun. After doing so, execution time did improve somewhat. I am not sure if this means one should always disable swap on a MonetDB system, since the system drive is slower than the data drives. I will try another experiment and let you know the results. = Series 7 NASD Exam Training Series 7 and related NASD Exam Books, software and web based training. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=16e765a220119782741a407… -- Powered by Outblaze

1 0