January 2015 - users-list

MonetDB Sharding
by gms_xy＠gmx.de 21 Jan '15

21 Jan '15

As mentioned in previous posts, I have some use cases for MonetDB, which act on tables with a billion rows. MonetDB works fast and stable on straight SELECT statements or aggregates on single tables Problems occured when doing big joins, which should produce a result set of 30 rows only, but running hours before failing. SELECT tbl1.*, tbl2.* FROM tbl1, tbl2 WHERE tbl1.xid = tbl2.xid AND tbl1.pk = 12345678 (same behavior, if written with JOIN explicitly) So I am wondering, if I can help MonetDB with partitioning or sharding. Asked already here: https://www.monetdb.org/pipermail/developers-list/2014-October/004325.html What is the state regarding manual partitioning? During last weeks I saw significant acceleration with sharding on other DBMS. Is there anybody, who can give some introduction or share experiences with sharding on MonetDB? 3rd party tools? What ist the maximum number of tables within a MonetDB database? What is the estimated overhead of a single table, multiplied a million times? Thank you Robert

3 2

GDK
by Brian Hood 20 Jan '15

20 Jan '15

Hi Guys, I have seen it around so much in Monetdb what does the acronym gdk actually stand for ? Regards, Brian Hood

2 1

Re: users-list Digest, Vol 29, Issue 14
by bhuvan Mitra 20 Jan '15

20 Jan '15

Thank you stefan. Is there any way through which I can get to know about those implementations or buying it from actian is the only way? Actually I want to contribute to c-store, open source column oriented database in this compression/decompression area. This answer from you will be very worthy to me. Thanks in advance. On Mon, Jan 19, 2015 at 9:29 PM, <users-list-request(a)monetdb.org> wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: users-list Digest, Vol 29, Issue 12 (Stefan Manegold) > 2. Next release? (g4(a)novadsp.com) > 3. Re: Not utilising Mutithreads (shamsul hassan) > 4. Re: Next release? (Sjoerd Mullender) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 19 Jan 2015 14:58:13 +0100 (CET) > From: Stefan Manegold <Stefan.Manegold(a)cwi.nl> > To: Communication channel for MonetDB users <users-list(a)monetdb.org> > Subject: Re: users-list Digest, Vol 29, Issue 12 > Message-ID: <895297092.253122.1421675893625.JavaMail.zimbra(a)cwi.nl> > Content-Type: text/plain; charset=utf-8 > > > Hi, > > MonetDB and Actian/VectorWise are two different systems that do not share > any code. > > MonetDB does not use any compression in the way Actian/VectorWise does. > > Best, > Stefan > > > ----- Original Message ----- > > Thank you Martin. > > So, did monetdb bought the codes from actian code base and included in > it's > > source code? > > Also kindly tell me if monetdb is following any other compression > techniques > > apart from pfor, pfor-delta and pdict. > > Thanks in advance. > > > > On Mon, Jan 19, 2015 at 4:30 PM, < users-list-request(a)monetdb.org > > wrote: > > > > > > Send users-list mailing list submissions to > > users-list(a)monetdb.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://www.monetdb.org/mailman/listinfo/users-list > > or, via email, send a message with subject or body 'help' to > > users-list-request(a)monetdb.org > > > > You can reach the person managing the list at > > users-list-owner(a)monetdb.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of users-list digest..." > > > > > > Today's Topics: > > > > 1. Re: Not utilising Mutithreads (Martin Kersten) > > 2. regarding source code (bhuvan Mitra) > > 3. Re: regarding source code (Martin Kersten) > > 4. information about compression/decompression techniques of > > monetdb (bhuvan Mitra) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Sun, 18 Jan 2015 12:20:15 +0100 > > From: Martin Kersten < martin(a)monetdb.org > > > To: users-list(a)monetdb.org > > Subject: Re: Not utilising Mutithreads > > Message-ID: < 54BB96EF.3030906(a)monetdb.org > > > Content-Type: text/plain; charset=windows-1252; format=flowed > > > > Hi > > > > Each MAL instruction is executed by a single thread. > > Some operators require the complete input to be available, > > such as in your case the group.subgroup(). > > > > martin > > > > On 18/01/15 01:27, shamsul hassan wrote: > > > Hi All, > > > > > > I have recently install monetdb "OCT 2014-SP1" release on my Oracle > > > Enterprise 6 Linux server. > > > > > > Server configuration - > > > > > > 8 CPU Cores > > > 64 GB RAM > > > > > > I have loaded a data set with 70 columns having 15 Millions Rows. > > > > > > I have started mserver5 as below -- > > > > > > mserver5 --debug=10 --set *gdk_nr_threads=8* > --dbpath=/data/monetdb/testdb > > > --set mapi_open=true --set mapi_port=54321 --set > > > merovingian_uri=mapi:monetdb://localhost:54321/testdb --set > > > mapi_usock=/data/monetdb/testdb/.mapi.sock --set > > > monet_vault_key=/data/monetdb/testdb/.vaultkey --set > > > sql_optimizer=default_pipe --set max_clients=64 > > > > > > Now I have ran a query as below -- > > > > > > select product_type,COUNT(DISTINCT sales_document),sum(ord_qty) FROM > > > bookings where ord_entry_date between '2014-04-01' and '2014-10-01' > group > > > by product_type; > > > > > > This query took around 90 Secs .. > > > So I started tomograph and checked plotted graph ... which shows the > > > parallelism usage as 39.9 % only and its utilizing only 1 thread with > > > maximum time went in the step - group.subgroupdone almost 98%. > > > > > > My Question is that even I have started the server with > gdk_nr_threads=8 , > > > then why it is just using single thread. > > > > > > More info -- > > > > > > mserver5 --version > > > > > > MonetDB 5 server v11.19.7 "Oct2014-SP1" (64-bit, 64-bit oids) > > > > > > Copyright (c) 1993-July 2008 CWI > > > > > > Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved > > > > > > Visit http://www.monetdb.org/ for further information > > > > > > Found 63.0GiB available memory, 8 available cpu cores > > > > > > Libraries: > > > > > > libpcre: 7.8 2008-09-05 (compiled with 7.8) > > > > > > openssl: OpenSSL 1.0.1e 11 Feb 2013 (compiled with OpenSSL 1.0.1e-fips > 11 > > > Feb 2013) > > > > > > libxml2: 2.7.6 (compiled with 2.7.6) > > > > > > Compiled by: root@localhost (x86_64-unknown-linux-gnu) > > > > > > Compilation: gcc -O3 -fomit-frame-pointer -pipe -D_FORTIFY_SOURCE=2 > > > > > > Linking : /usr/bin/ld -m elf_x86_64 > > > > > > Any pointers ?? > > > > > > Thanks > > > > > > > > > > > > _______________________________________________ > > > users-list mailing list > > > users-list(a)monetdb.org > > > https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > > > > ------------------------------ > > > > Message: 2 > > Date: Mon, 19 Jan 2015 12:28:05 +0530 > > From: bhuvan Mitra < bhuvanrjb(a)gmail.com > > > To: users-list(a)monetdb.org > > Subject: regarding source code > > Message-ID: > > < CAMmryP6eg5AAMY1x70KJZMdc3k7P8aqjnHuS5xN-fNSJ8mN_sQ(a)mail.gmail.com > > > Content-Type: text/plain; charset="utf-8" > > > > Hello, > > > > I am very interested to know and learn about implementations of > > compression/decompression techniques that monetdb is following. I have > seen > > the monetdb's light weight compression algorithms(pfor, pfor-delta, > pdict), > > cache involved decompression techniques and excited to know how actually > > are they implemented in form of codes. Kindly tell me in which > folder/file > > is the source code of this compression/decompression algorithms present. > I > > have downloaded the source tarball from the website and could not find > > these code snippets. Plese help me to identify the path. > > > > Regards, > > mg > >

2 1

Next release?
by g4＠novadsp.com 19 Jan '15

19 Jan '15

Hello - is there any timeframe for a next release? There are a couple of post Oct2014-SP1 bug-fixes which would be really good to have included. Many thanks.

2 1

Re: users-list Digest, Vol 29, Issue 12
by bhuvan Mitra 19 Jan '15

19 Jan '15

Thank you Martin. So, did monetdb bought the codes from actian code base and included in it's source code? Also kindly tell me if monetdb is following any other compression techniques apart from pfor, pfor-delta and pdict. Thanks in advance. On Mon, Jan 19, 2015 at 4:30 PM, <users-list-request(a)monetdb.org> wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: Not utilising Mutithreads (Martin Kersten) > 2. regarding source code (bhuvan Mitra) > 3. Re: regarding source code (Martin Kersten) > 4. information about compression/decompression techniques of > monetdb (bhuvan Mitra) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 18 Jan 2015 12:20:15 +0100 > From: Martin Kersten <martin(a)monetdb.org> > To: users-list(a)monetdb.org > Subject: Re: Not utilising Mutithreads > Message-ID: <54BB96EF.3030906(a)monetdb.org> > Content-Type: text/plain; charset=windows-1252; format=flowed > > Hi > > Each MAL instruction is executed by a single thread. > Some operators require the complete input to be available, > such as in your case the group.subgroup(). > > martin > > On 18/01/15 01:27, shamsul hassan wrote: > > Hi All, > > > > I have recently install monetdb "OCT 2014-SP1" release on my Oracle > Enterprise 6 Linux server. > > > > Server configuration - > > > > 8 CPU Cores > > 64 GB RAM > > > > I have loaded a data set with 70 columns having 15 Millions Rows. > > > > I have started mserver5 as below -- > > > > mserver5 --debug=10 --set *gdk_nr_threads=8* > --dbpath=/data/monetdb/testdb --set mapi_open=true --set mapi_port=54321 > --set merovingian_uri=mapi:monetdb://localhost:54321/testdb --set > mapi_usock=/data/monetdb/testdb/.mapi.sock --set > > monet_vault_key=/data/monetdb/testdb/.vaultkey --set > sql_optimizer=default_pipe --set max_clients=64 > > > > Now I have ran a query as below -- > > > > select product_type,COUNT(DISTINCT sales_document),sum(ord_qty) FROM > bookings where ord_entry_date between '2014-04-01' and '2014-10-01' group > by product_type; > > > > This query took around 90 Secs .. > > So I started tomograph and checked plotted graph ... which shows the > parallelism usage as 39.9 % only and its utilizing only 1 thread with > maximum time went in the step - group.subgroupdone almost 98%. > > > > My Question is that even I have started the server with gdk_nr_threads=8 > , then why it is just using single thread. > > > > More info -- > > > > mserver5 --version > > > > MonetDB 5 server v11.19.7 "Oct2014-SP1" (64-bit, 64-bit oids) > > > > Copyright (c) 1993-July 2008 CWI > > > > Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved > > > > Visit http://www.monetdb.org/ for further information > > > > Found 63.0GiB available memory, 8 available cpu cores > > > > Libraries: > > > > libpcre: 7.8 2008-09-05 (compiled with 7.8) > > > > openssl: OpenSSL 1.0.1e 11 Feb 2013 (compiled with OpenSSL > 1.0.1e-fips 11 Feb 2013) > > > > libxml2: 2.7.6 (compiled with 2.7.6) > > > > Compiled by: root@localhost (x86_64-unknown-linux-gnu) > > > > Compilation: gcc -O3 -fomit-frame-pointer -pipe -D_FORTIFY_SOURCE=2 > > > > Linking : /usr/bin/ld -m elf_x86_64 > > > > Any pointers ?? > > > > Thanks > > > > > > > > _______________________________________________ > > users-list mailing list > > users-list(a)monetdb.org > > https://www.monetdb.org/mailman/listinfo/users-list > > > > > > ------------------------------ > > Message: 2 > Date: Mon, 19 Jan 2015 12:28:05 +0530 > From: bhuvan Mitra <bhuvanrjb(a)gmail.com> > To: users-list(a)monetdb.org > Subject: regarding source code > Message-ID: > < > CAMmryP6eg5AAMY1x70KJZMdc3k7P8aqjnHuS5xN-fNSJ8mN_sQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > I am very interested to know and learn about implementations of > compression/decompression techniques that monetdb is following. I have seen > the monetdb's light weight compression algorithms(pfor, pfor-delta, pdict), > cache involved decompression techniques and excited to know how actually > are they implemented in form of codes. Kindly tell me in which folder/file > is the source code of this compression/decompression algorithms present. I > have downloaded the source tarball from the website and could not find > these code snippets. Plese help me to identify the path. > > Regards, > mg >

2 1

information about compression/decompression techniques of monetdb
by bhuvan Mitra 19 Jan '15

19 Jan '15

I have been studying about various column-oriented databases and found monetdb to be very interesting as it is efficient and open source. Please tell me in which file/folder does source code of compression/decompression techniques written in monetdb. I am interested to learn the implementations of light weight compression algorithms and cache-decompression. I have source code of monetdb downloaded, just let me know the path of requested codes.

1 0

regarding source code
by bhuvan Mitra 19 Jan '15

19 Jan '15

Hello, I am very interested to know and learn about implementations of compression/decompression techniques that monetdb is following. I have seen the monetdb's light weight compression algorithms(pfor, pfor-delta, pdict), cache involved decompression techniques and excited to know how actually are they implemented in form of codes. Kindly tell me in which folder/file is the source code of this compression/decompression algorithms present. I have downloaded the source tarball from the website and could not find these code snippets. Plese help me to identify the path. Regards, mg

2 1

analyse performance slow query
by Pierre-Adrien Coustillas 16 Jan '15

16 Jan '15

Hi, I try to analyze the slowness of a query with EXPLAIN and stethoscope The query is : SELECT count (*) FROM ( SELECT id_unique, siteniveau1, siteniveau2, '0000-00-00', 0, 0, MAX( debutvisite ), 0, 0 FROM tmp __ at __ lm __visites_ agregees _id_unique GROUP BY id_unique,siteniveau1,siteniveau2 ) tbl ; the beginning of explain is ( there is 272 tuples ) +============================================================================================================================================================================================+ | function user.s2_5{ autoCommit = true }(A0: str ,A1: bte ,A2: bte ,A3: bte ,A4: bte ): void ; | | barrier X_429 := language . dataflow (); | | X_7 := sql . mvc (); | | X_55:bat[: oid ,: oid ] := sql . tid (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique",0,8); | | X_70:bat[: oid ,: int ] := sql . bind (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique","id_unique",0,0,8); | | (X_78:bat[: oid ,: oid ] ,X_79:bat[: oid ,: int ] ) := sql . bind (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique","id_unique",2,0,8); | | X_161 := sql .delta(X_70,X_78,X_79); | | X_169 := algebra . leftfetchjoin (X_55,X_161); | | X_96:bat[: oid ,: int ] := sql . bind (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique","siteniveau1",0,0,8); | | (X_111:bat[: oid ,: oid ] ,X_112:bat[: oid ,: int ] ) := sql . bind (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique","siteniveau1",2,0,8); | | X_184 := sql .delta(X_96,X_111,X_112); | | X_193 := algebra . leftfetchjoin (X_55,X_184); | | X_129:bat[: oid ,: int ] := sql . bind (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique","siteniveau2",0,0,8); | | (X_144:bat[: oid ,: oid ] ,X_145:bat[: oid ,: int ] ) := sql . bind (X_7," sys "," tmp __ at __ lm __visites_ agregees _id_unique","siteniveau2",2,0,8); | | X_201 := sql .delta(X_129,X_144,X_145); | | X_209 := algebra . leftfetchjoin (X_55,X_201); | | (X_224,X_225,X_226) := group. subgroup (X_209); | | (X_248,X_249,X_250) := group. subgroup (X_193,X_224); | | (X_272,X_273,X_274) := group. subgroupdone (X_169,X_248); | | X_275 := algebra . leftfetchjoin (X_273,X_169); | | X_304 := algebra . leftfetchjoin (X_273,X_248); | | X_331 := algebra . leftfetchjoin (X_304,X_249); group. subgroupdone (X_169,X_248); set 10 minutes to run . Why ? how to understand this function ? Thank Pierre -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http ://1g6. biz Tél : 06 64 63 70 35

3 2

MSVCR80.dll Error Question
by Zola Javzmaa 16 Jan '15

16 Jan '15

Hello All, I am having trouble installing Monetdb on to my computer. Would anybody have any suggestions to fixing it? I'm on a Window 7, 64 bit operating system and I get the following error message when I try to load the server program: "Error message: This application has failed to start because MSVCR80.dll was not found. Re-installing the application might fix the problem." Thanks, Zola

3 2

rename database
by Pierre-Adrien Coustillas 15 Jan '15

15 Jan '15

Hi, Is it possible to rename a database ? I thought to do like this: monetdb stop database mv /var/lib/monetdb/database /var/lib/monetdb/new_database monetdb start new_database it's okay ? Thank Pierre -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35

4 3