January 2008 - users-list

[MonetDB-users] MonetDB scalability
by Djordje Djokovic 08 Jan '08

08 Jan '08

2 1

[MonetDB-users] tuning advice for TPC-H
by Rahul Chopra 08 Jan '08

08 Jan '08

I am trying to run some TPC-H queries to get a sense for how large of databases MonetDB can handle. I am currently trying a factor 20 database. I notice that on query 9, it is taking over two hours (still running). In "top" it claims that virtual memory for the process started at 27 GB or so, and is now down to 10 GB. What is noteworthy is that CPU usage is pretty small, at 2%. I previously observed 80-100% with a factor 1 database. I have 8 GB RAM, with 2 GB RAM swap. System swap is on a slower partition than the data drive, which is a relatively fast RAID 5 disk array. Might I be better off disabling system swap altogether when using MonetDB? Are there any other configuration changes I should make or MonetDB best practices I should follow? Thanks, Rahul = Mercedes Homes- San Antonio real estate Mercedes Homes builds homes and communities in the San Antonio area. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=9c24f83a902bfb70ef8d7bc… -- Powered by Outblaze

2 1

[MonetDB-users] indexes
by Rahul Chopra 07 Jan '08

07 Jan '08

Hello, I was just curious, what happens behind the scenes when an index is created in MonetDB? I noticed it is instantaneous... just wondering why. Is the BAT file reordered (for a single column index) kind of like for a clustered index, or is a separate index created? Thanks, Rahul = Truelife Financial Get the loan you deserve today regardless your credit history. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=cc5aa117b4038bf76c9b23d… -- Powered by Outblaze

2 1

[MonetDB-users] loading data
by Rahul Chopra 07 Jan '08

07 Jan '08

I ran into some problems loading data for doing some TPC-H tests. After a successful test for a factor 1 database, I moved on to test a factor 20 database. mserver5 appeared to crash when loading partsupp, orders, or lineitem. On a hunch, I ran split on these to ensure that each file was less than 2 GB each, splitting partsupp and orders into two files, and orders into 10. After doing this, partsupp and orders loaded without any problems. However, lineitem would cause mserver5 to crash after loading 2 or 3 of the 1.5 GB chunks. Also, in restarting mserver5, I noticed that CPU usage was sometimes heavy for a while, before starting loading. I am guessing it is doing some recovery or cleanup? Is loading an issue? Since it was busy upon restart, do I need to wait a while before loading a new file to give the system some time to do any housekeeping or something? I am using the latest released sources (not from HEAD), compiled on CentOS 5.1 (64 bit). Thanks, Rahul = Live Answering Service Answering. Live operators 24/7. Rates start at $39/100 calls. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=202d1706cc4cee71bfb381f… -- Powered by Outblaze

2 1

Re: [MonetDB-users] gdk_mem_bigsize and gdk_vm_minsize
by Rahul Chopra 03 Jan '08

03 Jan '08

> > On Fri, Jan 04, 2008 at 12:09:05AM +0800, Rahul Chopra wrote: > > Can you please explain gdk_mem_bigsize and gdk_vm_minsize? I > > searched around on the site and I am not really clear on what the > > optimal settings for these are. > > Memory blocks allocated via GDKmalloc and friends larger then > gdk_mem_bigsize will be mmaped using virtual swap space (vmalloc). This > is done to avoid fragmentation. So this is just for an individual block, and not for total allocated memory? Is a "block" allocated via entire BATs? Or are blocks allocated in segments when read from a single BAT? > > BATs have heaps, these can be malloced (GDKmalloc), virtual mmapped > using swap space (GDKvmalloc) or mmapped (needs a filename). > Blocks (heaps) larger then vm_minsize will be vmalloced. > The backends (sql/xquery) control when we switch to mmapped with files. > > > > > I am currently using the defaults, on a system with 8 GB of RAM. > It depends on your os if thats okay. Thanks. I am using Linux (64 bit), CentOS 5.1. My main concern was if setting these too low somehow restricts the amount of memory used and leads to more swapping. I am not familiar enough with the system to understand all of this, but with 8 GB of ram, should I be using a larger gdk_mem_bigsize than the config default of 262144? The config file mentions these being set to 1/2 of available memory, but default values appear commented out. Are those the values that are actually used? Or, if these are commented out, are these dynamically set? -Rahul > > Niels > > > > Thanks, > > > > Rahul > > > > > > = > > Limousine and Luxury Bus Wine Tours > > Private Napa & Sonoma wine-tasting tours via luxury limousine or bus. > > http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=2c5439db6084d9969b4f63e… > > > > > > -- Powered by Outblaze > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Microsoft > > Defy all challenges. Microsoft(R) Visual Studio 2005. > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > _______________________________________________ > > MonetDB-users mailing list > > MonetDB-users(a)lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/monetdb-users > > -- > > Niels Nes, Centre for Mathematics and Computer Science (CWI) > Kruislaan 413, 1098 SJ Amsterdam, The Netherlands > room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 > url: http://www.cwi.nl/~niels e-mail: Niels.Nes(a)cwi.nl > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2005. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > MonetDB-users mailing list > MonetDB-users(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users > = Mercedes Homes - In FL, TX, NC, & SC Mercedes Homes builds homes in Florida, Texas, and the Carolinas. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=e0639e1f90664efd0041110… -- Powered by Outblaze

2 1

Re: [MonetDB-users] killing queries
by Rahul Chopra 03 Jan '08

03 Jan '08

Thanks for the information, I understand. Other than killing the client application, is there a way to separately connect to the server and gracefully kill the query or connection? > ----- Original Message ----- > From: "Niels Nes" <Niels.Nes(a)cwi.nl> > To: "Communication channel for MonetDB users" <monetdb-users(a)lists.sourceforge.net> > Subject: Re: [MonetDB-users] killing queries > Date: Thu, 3 Jan 2008 17:47:45 +0100 > > > On Fri, Jan 04, 2008 at 12:06:37AM +0800, Rahul Chopra wrote: > > How can I safely kill queries? > > > > I notice if I ctrl-c out of mclient when executing a long run > > query, mserver5 still seems to be using a lot of memory, maybe it > > has not deallocated it all? Also, if I start a new mclient > > session queries now run slower. Of course, one should always > > avoid ctrl-c, but in case we do have multiple queries running and > > we wish to kill a long running one for some reason, is there a > > way to safely do so? (Like, start a new mclient session, and > > execute some kind of admin commands). > Resources taken because of killed clients should disappear. This can > however take some time as the cleanup is only done within the > interpreter loop. Also we keep the used persistent tables in memory. > They only get freed when running out of memory or when idle for some > time. > > I'm not sure which version your using, but also the current released > version has some memory leaks which maybe the reason for your memory > growth and slowdown. > > Niels > > > > TIA > > > > > > = > > Espresso Machines - Free Shipping > > Espresso Machines, coffee beans and pods, cappuccino cups and > > spoons. Free shipping. No sales tax. > > http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=8427826bf39d56eb7c331a1… > > > > > > -- Powered by Outblaze > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Microsoft > > Defy all challenges. Microsoft(R) Visual Studio 2005. > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > _______________________________________________ > > MonetDB-users mailing list > > MonetDB-users(a)lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/monetdb-users > > -- > > Niels Nes, Centre for Mathematics and Computer Science (CWI) > Kruislaan 413, 1098 SJ Amsterdam, The Netherlands > room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 > url: http://www.cwi.nl/~niels e-mail: Niels.Nes(a)cwi.nl > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2005. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > MonetDB-users mailing list > MonetDB-users(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users > = Area Homes Realty Online Home Search Area Homes Realty is a full service residential real estate company servicing Eastern Massachusetts. Our website is a valuable resource of information whether you are buying or selling. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=6d65a7e04b5c96a53badb07… -- Powered by Outblaze

2 1

[MonetDB-users] gdk_mem_bigsize and gdk_vm_minsize
by Rahul Chopra 03 Jan '08

03 Jan '08

Can you please explain gdk_mem_bigsize and gdk_vm_minsize? I searched around on the site and I am not really clear on what the optimal settings for these are. I am currently using the defaults, on a system with 8 GB of RAM. Thanks, Rahul = Limousine and Luxury Bus Wine Tours Private Napa & Sonoma wine-tasting tours via luxury limousine or bus. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=2c5439db6084d9969b4f63e… -- Powered by Outblaze

2 1

[MonetDB-users] killing queries
by Rahul Chopra 03 Jan '08

03 Jan '08

How can I safely kill queries? I notice if I ctrl-c out of mclient when executing a long run query, mserver5 still seems to be using a lot of memory, maybe it has not deallocated it all? Also, if I start a new mclient session queries now run slower. Of course, one should always avoid ctrl-c, but in case we do have multiple queries running and we wish to kill a long running one for some reason, is there a way to safely do so? (Like, start a new mclient session, and execute some kind of admin commands). TIA = Espresso Machines - Free Shipping Espresso Machines, coffee beans and pods, cappuccino cups and spoons. Free shipping. No sales tax. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=8427826bf39d56eb7c331a1… -- Powered by Outblaze

2 1

[MonetDB-users] mserver5 daemon
by Rahul Chopra 03 Jan '08

03 Jan '08

I had a question about using the daemon option. If I start mserver5 with daemon=yes, it seems to stop: ./mserver5 --dbinit="include sql;" --dbname=db1 --daemon=yes & [1] 10038 [monetdb@localhost bin]$ # MonetDB server v5.2.0, based on kernel v1.20.0 # Serving database 'db1' # Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs dynamically linked # Copyright (c) 1993-2007 CWI, all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see /home/monetdb/MonetDB/etc/monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module v2.20.0 loaded [1]+ Stopped ./mserver5 --dbinit="include sql;" --dbname=db1 --daemon=yes If I leave off &, I still get a > prompt and can interact with it. I also tried nohup. I am not sure what I need to do to get the server running in the background and keep it running without it being in the foreground.... = nantucket vacation in Nantucket, MA Sanford Real Estate provides professional real estate services in Nantucket, Massachusetts. Visit us today for home buying, selling, rentals and more. Customer satisfaction is our priority. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=751b4f3b7386a2440fa9cc8… -- Powered by Outblaze

2 1