[MonetDB-users] disk usage
Hi, I've loaded 60 GB of data file into a table. On Windows (NTFS) the size of dbfarm changed to 80GB but on Linux (ext4 or reiserfs) dbfarm changed to 180GB. Why is that? Thanks. Dariusz.
Hello Dariusz,
The first thing came to my mind is the bits system. I guess your windows is 32bits, while your linux is 64 bits. Since, on 32bits systems, 'long', 'long int', oids and pointers are 4 bytes; while on they are 8 bytes. Thus, you data will most probably take more storage on a 64bits system, than on a 32bits system. How many more it is, depends on the type of your columns.
Regards,
Jennie
On Mon, Jun 22, 2009 at 12:35:52PM -0400, dariuszs wrote:
Hi, I've loaded 60 GB of data file into a table. On Windows (NTFS) the size of dbfarm changed to 80GB but on Linux (ext4 or reiserfs) dbfarm changed to 180GB. Why is that? Thanks. Dariusz.
Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today! http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
since the last release, there is a storage/memory consumption optimization in place when using the configure option --enable-oid32, which causes tables to be restricted to 2Gtuples, but leads to smaller data storage for surrogate keys (oids). As a second effect, this configure setting also enables a more compact string heap organization (restricting the amount of string data in a single column to 32GB).
windows is configured by default with --enable-oid32
If you can live with these restrictions, you can also compile/get linux MonetDB binaries with --enable-oid32
-----Original Message----- From: Ying Zhang [mailto:Y.Zhang@cwi.nl] Sent: Monday, June 22, 2009 7:01 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hello Dariusz,
The first thing came to my mind is the bits system. I guess your windows is 32bits, while your linux is 64 bits. Since, on 32bits systems, 'long', 'long int', oids and pointers are 4 bytes; while on they are 8 bytes. Thus, you data will most probably take more storage on a 64bits system, than on a 32bits system. How many more it is, depends on the type of your columns.
Regards,
Jennie
On Mon, Jun 22, 2009 at 12:35:52PM -0400, dariuszs wrote:
Hi, I've loaded 60 GB of data file into a table. On Windows (NTFS) the size of dbfarm changed to 80GB but on Linux (ext4 or reiserfs) dbfarm changed to 180GB. Why is that? Thanks. Dariusz.
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge. org
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today! http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge. org _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date: 06/21/09 20:02:00
Hi, That makes sense, now does the performance suffer with 64bit object ids, when you have to go thru more storage? Dariusz.
Peter Boncz wrote:
since the last release, there is a storage/memory consumption optimization in place when using the configure option --enable-oid32, which causes tables to be restricted to 2Gtuples, but leads to smaller data storage for surrogate keys (oids). As a second effect, this configure setting also enables a more compact string heap organization (restricting the amount of string data in a single column to 32GB).
windows is configured by default with --enable-oid32
If you can live with these restrictions, you can also compile/get linux MonetDB binaries with --enable-oid32
-----Original Message----- From: Ying Zhang [mailto:Y.Zhang@cwi.nl] Sent: Monday, June 22, 2009 7:01 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hello Dariusz,
The first thing came to my mind is the bits system. I guess your windows is 32bits, while your linux is 64 bits. Since, on 32bits systems, 'long', 'long int', oids and pointers are 4 bytes; while on they are 8 bytes. Thus, you data will most probably take more storage on a 64bits system, than on a 32bits system. How many more it is, depends on the type of your columns.
Regards,
Jennie
On Mon, Jun 22, 2009 at 12:35:52PM -0400, dariuszs wrote:
Hi, I've loaded 60 GB of data file into a table. On Windows (NTFS) the size of dbfarm changed to 80GB but on Linux (ext4 or reiserfs) dbfarm changed to 180GB. Why is that? Thanks. Dariusz.
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge. org
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today! http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge. org _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date: 06/21/09 20:02:00
Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today! http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
The effect of the more compact storage only marginally affects direct performance (for the better). The big win is that it reduces RAM consumption when query processing. This means that where MonetDB performance suffers with 64-bits oids due to swapping when you use datasets that just do not fit your memory, the reduced storage of 32-bits oids may avoid avoid this problem. You can also say that you need less RAM to make MonetDB fast. Memory consumption may be reduced to 70% or even 50% in (rare) extreme cases.
However, it will not work with columns with many tuples (2*1024*1024*1024 being the hard limit) or huge string columns (where all string data of a single column exceeds 32GB).
-----Original Message----- From: dariuszs [mailto:dariuszs@svnp.com] Sent: Monday, June 22, 2009 7:52 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hi, That makes sense, now does the performance suffer with 64bit object ids, when you have to go thru more storage? Dariusz.
Peter Boncz wrote:
since the last release, there is a storage/memory consumption optimization
in
place when using the configure option --enable-oid32, which causes tables
to be
restricted to 2Gtuples, but leads to smaller data storage for surrogate
keys
(oids). As a second effect, this configure setting also enables a more
compact
string heap organization (restricting the amount of string data in a
single
column to 32GB).
windows is configured by default with --enable-oid32
If you can live with these restrictions, you can also compile/get linux
MonetDB
binaries with --enable-oid32
-----Original Message----- From: Ying Zhang [mailto:Y.Zhang@cwi.nl] Sent: Monday, June 22, 2009 7:01 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hello Dariusz,
The first thing came to my mind is the bits system. I guess your windows is 32bits, while your linux is 64 bits. Since, on 32bits systems, 'long', 'long int', oids and pointers are 4 bytes; while on they are 8 bytes. Thus, you data will most probably take more storage on a 64bits system, than on a 32bits system. How many more it is, depends on the type of your columns.
Regards,
Jennie
On Mon, Jun 22, 2009 at 12:35:52PM -0400, dariuszs wrote:
Hi, I've loaded 60 GB of data file into a table. On Windows (NTFS) the size of dbfarm changed to 80GB but on Linux (ext4 or reiserfs) dbfarm changed to 180GB. Why is that? Thanks. Dariusz.
-------------------------------------------------------------------------
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.
org
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
---------------------------------------------------------------------------
Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.
org _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date:
06/21/09
20:02:00
---------------------------------------------------------------------------- --
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o rg
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
---------------------------------------------------------------------------- -- Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today! http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o rg _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date: 06/21/09 20:02:00
Hi, Is the 2 billion records also a limit with 64bit oids? Dariusz.
Peter Boncz wrote:
The effect of the more compact storage only marginally affects direct performance (for the better). The big win is that it reduces RAM consumption when query processing. This means that where MonetDB performance suffers with 64-bits oids due to swapping when you use datasets that just do not fit your memory, the reduced storage of 32-bits oids may avoid avoid this problem. You can also say that you need less RAM to make MonetDB fast. Memory consumption may be reduced to 70% or even 50% in (rare) extreme cases.
However, it will not work with columns with many tuples (2*1024*1024*1024 being the hard limit) or huge string columns (where all string data of a single column exceeds 32GB).
-----Original Message----- From: dariuszs [mailto:dariuszs@svnp.com] Sent: Monday, June 22, 2009 7:52 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hi, That makes sense, now does the performance suffer with 64bit object ids, when you have to go thru more storage? Dariusz.
Peter Boncz wrote:
since the last release, there is a storage/memory consumption optimization
in
place when using the configure option --enable-oid32, which causes tables
to be
restricted to 2Gtuples, but leads to smaller data storage for surrogate
keys
(oids). As a second effect, this configure setting also enables a more
compact
string heap organization (restricting the amount of string data in a
single
column to 32GB).
windows is configured by default with --enable-oid32
If you can live with these restrictions, you can also compile/get linux
MonetDB
binaries with --enable-oid32
-----Original Message----- From: Ying Zhang [mailto:Y.Zhang@cwi.nl] Sent: Monday, June 22, 2009 7:01 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hello Dariusz,
The first thing came to my mind is the bits system. I guess your windows is 32bits, while your linux is 64 bits. Since, on 32bits systems, 'long', 'long int', oids and pointers are 4 bytes; while on they are 8 bytes. Thus, you data will most probably take more storage on a 64bits system, than on a 32bits system. How many more it is, depends on the type of your columns.
Regards,
Jennie
On Mon, Jun 22, 2009 at 12:35:52PM -0400, dariuszs wrote:
Hi, I've loaded 60 GB of data file into a table. On Windows (NTFS) the size of dbfarm changed to 80GB but on Linux (ext4 or reiserfs) dbfarm changed to 180GB. Why is that? Thanks. Dariusz.
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.
org
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.
org _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date:
06/21/09
20:02:00
--
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o rg
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today! http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o rg _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date: 06/21/09 20:02:00
No. 2 billion is related to 32-bits OIDs only.
-----Original Message----- From: dariuszs [mailto:dariuszs@svnp.com] Sent: Monday, June 22, 2009 11:25 PM To: P.Boncz@cwi.nl Cc: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hi, Is the 2 billion records also a limit with 64bit oids? Dariusz.
Peter Boncz wrote:
The effect of the more compact storage only marginally affects direct performance (for the better). The big win is that it reduces RAM
consumption
when query processing. This means that where MonetDB performance suffers with 64-bits oids due to swapping when you use datasets that just do not
fit
your memory, the reduced storage of 32-bits oids may avoid avoid this problem. You can also say that you need less RAM to make MonetDB fast. Memory consumption may be reduced to 70% or even 50% in (rare) extreme cases.
However, it will not work with columns with many tuples (2*1024*1024*1024 being the hard limit) or huge string columns (where all string data of a single column exceeds 32GB).
-----Original Message----- From: dariuszs [mailto:dariuszs@svnp.com] Sent: Monday, June 22, 2009 7:52 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hi, That makes sense, now does the performance suffer with 64bit object ids, when you have to go thru more storage? Dariusz.
Peter Boncz wrote:
since the last release, there is a storage/memory consumption
optimization
in
place when using the configure option --enable-oid32, which causes tables
to be
restricted to 2Gtuples, but leads to smaller data storage for surrogate
keys
(oids). As a second effect, this configure setting also enables a more
compact
string heap organization (restricting the amount of string data in a
single
column to 32GB).
windows is configured by default with --enable-oid32
If you can live with these restrictions, you can also compile/get linux
MonetDB
binaries with --enable-oid32
-----Original Message----- From: Ying Zhang [mailto:Y.Zhang@cwi.nl] Sent: Monday, June 22, 2009 7:01 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hello Dariusz,
The first thing came to my mind is the bits system. I guess your windows is 32bits, while your linux is 64 bits. Since, on 32bits systems, 'long', 'long int', oids and pointers are 4 bytes; while on they are 8 bytes. Thus, you data will most probably take more storage on a 64bits system, than on a 32bits system. How many more it is, depends on the type of your columns.
Regards,
Jennie
On Mon, Jun 22, 2009 at 12:35:52PM -0400, dariuszs wrote:
Hi, I've loaded 60 GB of data file into a table. On Windows (NTFS) the size of dbfarm changed to 80GB but on Linux (ext4 or reiserfs) dbfarm changed to 180GB. Why is that? Thanks. Dariusz.
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of
unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.
org
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
---------------------------------------------------------------------------
Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.
org _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date:
06/21/09
20:02:00
----------------------------------------------------------------------------
--
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o
rg
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
----------------------------------------------------------------------------
-- Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o
rg _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date: 06/21/09 20:02:00
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.374 / Virus Database: 270.12.70/2177 - Release Date: 06/22/09 06:54:00
Hi, Thanks. By the way, speaking of memory, maybe not a big problem in the near future, at least 2 solutions, check it out.
http://www.fusionio.com/Products.aspx http://www.engadget.com/2009/01/15/laser-hard-drives-boast-1tbits-s-access-t...
Dariusz.
Peter Boncz wrote:
No. 2 billion is related to 32-bits OIDs only.
-----Original Message----- From: dariuszs [mailto:dariuszs@svnp.com] Sent: Monday, June 22, 2009 11:25 PM To: P.Boncz@cwi.nl Cc: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hi, Is the 2 billion records also a limit with 64bit oids? Dariusz.
Peter Boncz wrote:
The effect of the more compact storage only marginally affects direct performance (for the better). The big win is that it reduces RAM
consumption
when query processing. This means that where MonetDB performance suffers with 64-bits oids due to swapping when you use datasets that just do not
fit
your memory, the reduced storage of 32-bits oids may avoid avoid this problem. You can also say that you need less RAM to make MonetDB fast. Memory consumption may be reduced to 70% or even 50% in (rare) extreme cases.
However, it will not work with columns with many tuples (2*1024*1024*1024 being the hard limit) or huge string columns (where all string data of a single column exceeds 32GB).
-----Original Message----- From: dariuszs [mailto:dariuszs@svnp.com] Sent: Monday, June 22, 2009 7:52 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hi, That makes sense, now does the performance suffer with 64bit object ids, when you have to go thru more storage? Dariusz.
Peter Boncz wrote:
since the last release, there is a storage/memory consumption
optimization
in
place when using the configure option --enable-oid32, which causes tables
to be
restricted to 2Gtuples, but leads to smaller data storage for surrogate
keys
(oids). As a second effect, this configure setting also enables a more
compact
string heap organization (restricting the amount of string data in a
single
column to 32GB).
windows is configured by default with --enable-oid32
If you can live with these restrictions, you can also compile/get linux
MonetDB
binaries with --enable-oid32
-----Original Message----- From: Ying Zhang [mailto:Y.Zhang@cwi.nl] Sent: Monday, June 22, 2009 7:01 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hello Dariusz,
The first thing came to my mind is the bits system. I guess your windows is 32bits, while your linux is 64 bits. Since, on 32bits systems, 'long', 'long int', oids and pointers are 4 bytes; while on they are 8 bytes. Thus, you data will most probably take more storage on a 64bits system, than on a 32bits system. How many more it is, depends on the type of your columns.
Regards,
Jennie
On Mon, Jun 22, 2009 at 12:35:52PM -0400, dariuszs wrote:
Hi, I've loaded 60 GB of data file into a table. On Windows (NTFS) the size of dbfarm changed to 80GB but on Linux (ext4 or reiserfs) dbfarm changed to 180GB. Why is that? Thanks. Dariusz.
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of
unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.
org
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.
org _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date:
06/21/09
20:02:00
--
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o
rg
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o
rg _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date: 06/21/09 20:02:00
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.374 / Virus Database: 270.12.70/2177 - Release Date: 06/22/09 06:54:00
Hi, Once more http://thefutureofthings.com/news/43/laser-hard-drives-on-the-horizon.html Dariusz.
dariuszs wrote:
Hi, Thanks. By the way, speaking of memory, maybe not a big problem in the near future, at least 2 solutions, check it out.
http://www.fusionio.com/Products.aspx http://www.engadget.com/2009/01/15/laser-hard-drives-boast-1tbits-s-access-t...
Dariusz.
Peter Boncz wrote:
No. 2 billion is related to 32-bits OIDs only.
-----Original Message----- From: dariuszs [mailto:dariuszs@svnp.com] Sent: Monday, June 22, 2009 11:25 PM To: P.Boncz@cwi.nl Cc: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hi, Is the 2 billion records also a limit with 64bit oids? Dariusz.
Peter Boncz wrote:
The effect of the more compact storage only marginally affects direct performance (for the better). The big win is that it reduces RAM
consumption
when query processing. This means that where MonetDB performance suffers with 64-bits oids due to swapping when you use datasets that just do not
fit
your memory, the reduced storage of 32-bits oids may avoid avoid this problem. You can also say that you need less RAM to make MonetDB fast. Memory consumption may be reduced to 70% or even 50% in (rare) extreme cases.
However, it will not work with columns with many tuples (2*1024*1024*1024 being the hard limit) or huge string columns (where all string data of a single column exceeds 32GB).
-----Original Message----- From: dariuszs [mailto:dariuszs@svnp.com] Sent: Monday, June 22, 2009 7:52 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hi, That makes sense, now does the performance suffer with 64bit object ids, when you have to go thru more storage? Dariusz.
Peter Boncz wrote:
since the last release, there is a storage/memory consumption
optimization
in
place when using the configure option --enable-oid32, which causes tables
to be
restricted to 2Gtuples, but leads to smaller data storage for surrogate
keys
(oids). As a second effect, this configure setting also enables a more
compact
string heap organization (restricting the amount of string data in a
single
column to 32GB).
windows is configured by default with --enable-oid32
If you can live with these restrictions, you can also compile/get linux
MonetDB
binaries with --enable-oid32
-----Original Message----- From: Ying Zhang [mailto:Y.Zhang@cwi.nl] Sent: Monday, June 22, 2009 7:01 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hello Dariusz,
The first thing came to my mind is the bits system. I guess your windows is 32bits, while your linux is 64 bits. Since, on 32bits systems, 'long', 'long int', oids and pointers are 4 bytes; while on they are 8 bytes. Thus, you data will most probably take more storage on a 64bits system, than on a 32bits system. How many more it is, depends on the type of your columns.
Regards,
Jennie
On Mon, Jun 22, 2009 at 12:35:52PM -0400, dariuszs wrote:
Hi, I've loaded 60 GB of data file into a table. On Windows (NTFS) the size of dbfarm changed to 80GB but on Linux (ext4 or reiserfs) dbfarm changed to 180GB. Why is that? Thanks. Dariusz.
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of
unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.
org
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.
org _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date:
06/21/09
20:02:00
--
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o
rg
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o
rg _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date: 06/21/09 20:02:00
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.374 / Virus Database: 270.12.70/2177 - Release Date: 06/22/09 06:54:00
Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today! http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Hi, Where do you exactly pass the option of --enable-oid32? Dariusz.
Peter Boncz wrote:
since the last release, there is a storage/memory consumption optimization in place when using the configure option --enable-oid32, which causes tables to be restricted to 2Gtuples, but leads to smaller data storage for surrogate keys (oids). As a second effect, this configure setting also enables a more compact string heap organization (restricting the amount of string data in a single column to 32GB).
windows is configured by default with --enable-oid32
If you can live with these restrictions, you can also compile/get linux MonetDB binaries with --enable-oid32
-----Original Message----- From: Ying Zhang [mailto:Y.Zhang@cwi.nl] Sent: Monday, June 22, 2009 7:01 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hello Dariusz,
The first thing came to my mind is the bits system. I guess your windows is 32bits, while your linux is 64 bits. Since, on 32bits systems, 'long', 'long int', oids and pointers are 4 bytes; while on they are 8 bytes. Thus, you data will most probably take more storage on a 64bits system, than on a 32bits system. How many more it is, depends on the type of your columns.
Regards,
Jennie
On Mon, Jun 22, 2009 at 12:35:52PM -0400, dariuszs wrote:
Hi, I've loaded 60 GB of data file into a table. On Windows (NTFS) the size of dbfarm changed to 80GB but on Linux (ext4 or reiserfs) dbfarm changed to 180GB. Why is that? Thanks. Dariusz.
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge. org
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today! http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge. org _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date: 06/21/09 20:02:00
Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today! http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
dariuszs wrote:
Where do you exactly pass the option of --enable-oid32? Dariusz.
./configure --enable-oid32
for example.
Stefan
Hi, I've downloaded the SuperBall and changed directory to MonetDB5, and then I ran ./configure - got the message MonetDB was not found? What am I missing? So far I've used install script for the sources and that compiled without any problems but the script does not take the --enable-oid32 option. Dariusz.
Stefan de Konink wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
dariuszs wrote:
Where do you exactly pass the option of --enable-oid32? Dariusz.
./configure --enable-oid32
for example.
Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEAREKAAYFAkpCeM4ACgkQYH1+F2Rqwn2dpQCfRUcT4QOxoHIvAoSC4I8npdW1 ci4AnjrHD2JvMZ6WjdCztP2pN/vRjWRR =DRoA -----END PGP SIGNATURE-----
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Hi,
you can simply use the monetdb-install.sh script as follows:
export EXTRA_ECONF=--enable-oid32 ; ./monetdb-install.sh ...
see below hlep/usage messgaes for details.
Stefan
======== $ ./monetdb-install.sh --help usage: ./monetdb-install.sh < OPTS ... >
where OPTS are: --prefix=path install into location path, defaults to /ufs/manegold/MonetDB --build=path use path as (temporary) build directory, defaults to /var/tmp/MonetDB-XXXXXXXXX --enable-sql build the MonetDB/SQL server --enable-xquery build the MonetDB/XQuery server --enable-geom build geom extension for MonetDB/SQL, this option implies --enable-sql --enable-template build template module for MonetDB extensions, this option requires nightly or cvs sources --enable-testing build the MonetDB testing suite necessary for running 'make check', Mtest.py or RunMtest --nightly=target download and install a nightly snapshot of the stable or current branch, target must be 'stable' or 'current' --cvs[=tag] checkout a CVS snapshot of the current branch or optionally given tag --enable-debug compile with debugging support via e.g. gdb --enable-optimise compile with high optimisation flags, enabling this option increases compilation time considerably but often yields in a faster MonetDB server --enable-optimize alias for --enable-optimise --quiet suppress output going to stdout --help this message --devhelp special help for developers ^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^ --version show revision number and quit ======== $ ./monetdb-install.sh --devhelp There are a few hooks in this script that may be interesting to developers or experienced users.
************************************************************************ If you know what you're doing, by setting EXTRA_ECONF in the environment you can pass extra arguments to configure invocations. Use with care. ************************************************************************
The variable MAKEOPTS is passed onto each make invocation, and can be used to e.g. set "-j2" to speed up compilation times. By setting MAKE, a make with a special name or location can be used.
When using the --cvs option, the variable CVSROOTM controls the used root (-d option to cvs) which defaults to: :pserver:anonymous@monetdb.cvs.sourceforge.net:/cvsroot/monetdb
Two targets exist which control what the script is doing: unpack and install. They can be given as argument to the script. unpack only runs the fetch and unpacking phase, while install skips the fetching/unpacking of the sources, and starts configure/make/make install. By using --build=/path/to with the install target it is possible to resume/retry a (failed) compilation. ========
On Wed, Jun 24, 2009 at 03:32:33PM -0400, dariuszs wrote:
Hi, I've downloaded the SuperBall and changed directory to MonetDB5, and then I ran ./configure - got the message MonetDB was not found? What am I missing? So far I've used install script for the sources and that compiled without any problems but the script does not take the --enable-oid32 option. Dariusz.
Stefan de Konink wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
dariuszs wrote:
Where do you exactly pass the option of --enable-oid32? Dariusz.
./configure --enable-oid32
for example.
Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEAREKAAYFAkpCeM4ACgkQYH1+F2Rqwn2dpQCfRUcT4QOxoHIvAoSC4I8npdW1 ci4AnjrHD2JvMZ6WjdCztP2pN/vRjWRR =DRoA -----END PGP SIGNATURE-----
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Hi, Thank you. Dariusz.
Stefan Manegold wrote:
Hi,
you can simply use the monetdb-install.sh script as follows:
export EXTRA_ECONF=--enable-oid32 ; ./monetdb-install.sh ...
see below hlep/usage messgaes for details.
Stefan
======== $ ./monetdb-install.sh --help usage: ./monetdb-install.sh < OPTS ... >
where OPTS are: --prefix=path install into location path, defaults to /ufs/manegold/MonetDB --build=path use path as (temporary) build directory, defaults to /var/tmp/MonetDB-XXXXXXXXX --enable-sql build the MonetDB/SQL server --enable-xquery build the MonetDB/XQuery server --enable-geom build geom extension for MonetDB/SQL, this option implies --enable-sql --enable-template build template module for MonetDB extensions, this option requires nightly or cvs sources --enable-testing build the MonetDB testing suite necessary for running 'make check', Mtest.py or RunMtest --nightly=target download and install a nightly snapshot of the stable or current branch, target must be 'stable' or 'current' --cvs[=tag] checkout a CVS snapshot of the current branch or optionally given tag --enable-debug compile with debugging support via e.g. gdb --enable-optimise compile with high optimisation flags, enabling this option increases compilation time considerably but often yields in a faster MonetDB server --enable-optimize alias for --enable-optimise --quiet suppress output going to stdout --help this message --devhelp special help for developers ^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^
--version show revision number and quit
$ ./monetdb-install.sh --devhelp There are a few hooks in this script that may be interesting to developers or experienced users.
If you know what you're doing, by setting EXTRA_ECONF in the environment you can pass extra arguments to configure invocations. Use with care.
The variable MAKEOPTS is passed onto each make invocation, and can be used to e.g. set "-j2" to speed up compilation times. By setting MAKE, a make with a special name or location can be used.
When using the --cvs option, the variable CVSROOTM controls the used root (-d option to cvs) which defaults to: :pserver:anonymous@monetdb.cvs.sourceforge.net:/cvsroot/monetdb
Two targets exist which control what the script is doing: unpack and install. They can be given as argument to the script. unpack only runs the fetch and unpacking phase, while install skips the fetching/unpacking of the sources, and starts configure/make/make install. By using --build=/path/to with the install target it is possible to resume/retry a (failed) compilation. ========
On Wed, Jun 24, 2009 at 03:32:33PM -0400, dariuszs wrote:
Hi, I've downloaded the SuperBall and changed directory to MonetDB5, and then I ran ./configure - got the message MonetDB was not found? What am I missing? So far I've used install script for the sources and that compiled without any problems but the script does not take the --enable-oid32 option. Dariusz.
Stefan de Konink wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
dariuszs wrote:
Where do you exactly pass the option of --enable-oid32? Dariusz.
./configure --enable-oid32
for example.
Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEAREKAAYFAkpCeM4ACgkQYH1+F2Rqwn2dpQCfRUcT4QOxoHIvAoSC4I8npdW1 ci4AnjrHD2JvMZ6WjdCztP2pN/vRjWRR =DRoA -----END PGP SIGNATURE-----
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Hi, I've installed it successfully but importing data into table resulted in this
# MonetDB server v5.10.4, based on kernel v1.28.4 # Serving database 'demo', using 8 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 32bit OIDs dynamically linked # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see /vol1/MonetDB5/etc/monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module v2.28.4 loaded
Segmentation fault
monetdb kernel: [83847.135692] mserver5[3577]: segfault at 7f06a0bc4000 ip 00007f06a05a9a2b sp 00007f068f84f160 error 7 in libbat.so.1.28.4[7f069ffaa000+83e000]
Any ideas? Thanks. Dariusz.
Stefan Manegold wrote:
Hi,
you can simply use the monetdb-install.sh script as follows:
export EXTRA_ECONF=--enable-oid32 ; ./monetdb-install.sh ...
see below hlep/usage messgaes for details.
Stefan
======== $ ./monetdb-install.sh --help usage: ./monetdb-install.sh < OPTS ... >
where OPTS are: --prefix=path install into location path, defaults to /ufs/manegold/MonetDB --build=path use path as (temporary) build directory, defaults to /var/tmp/MonetDB-XXXXXXXXX --enable-sql build the MonetDB/SQL server --enable-xquery build the MonetDB/XQuery server --enable-geom build geom extension for MonetDB/SQL, this option implies --enable-sql --enable-template build template module for MonetDB extensions, this option requires nightly or cvs sources --enable-testing build the MonetDB testing suite necessary for running 'make check', Mtest.py or RunMtest --nightly=target download and install a nightly snapshot of the stable or current branch, target must be 'stable' or 'current' --cvs[=tag] checkout a CVS snapshot of the current branch or optionally given tag --enable-debug compile with debugging support via e.g. gdb --enable-optimise compile with high optimisation flags, enabling this option increases compilation time considerably but often yields in a faster MonetDB server --enable-optimize alias for --enable-optimise --quiet suppress output going to stdout --help this message --devhelp special help for developers ^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^
--version show revision number and quit
$ ./monetdb-install.sh --devhelp There are a few hooks in this script that may be interesting to developers or experienced users.
If you know what you're doing, by setting EXTRA_ECONF in the environment you can pass extra arguments to configure invocations. Use with care.
The variable MAKEOPTS is passed onto each make invocation, and can be used to e.g. set "-j2" to speed up compilation times. By setting MAKE, a make with a special name or location can be used.
When using the --cvs option, the variable CVSROOTM controls the used root (-d option to cvs) which defaults to: :pserver:anonymous@monetdb.cvs.sourceforge.net:/cvsroot/monetdb
Two targets exist which control what the script is doing: unpack and install. They can be given as argument to the script. unpack only runs the fetch and unpacking phase, while install skips the fetching/unpacking of the sources, and starts configure/make/make install. By using --build=/path/to with the install target it is possible to resume/retry a (failed) compilation. ========
On Wed, Jun 24, 2009 at 03:32:33PM -0400, dariuszs wrote:
Hi, I've downloaded the SuperBall and changed directory to MonetDB5, and then I ran ./configure - got the message MonetDB was not found? What am I missing? So far I've used install script for the sources and that compiled without any problems but the script does not take the --enable-oid32 option. Dariusz.
Stefan de Konink wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
dariuszs wrote:
Where do you exactly pass the option of --enable-oid32? Dariusz.
./configure --enable-oid32
for example.
Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEAREKAAYFAkpCeM4ACgkQYH1+F2Rqwn2dpQCfRUcT4QOxoHIvAoSC4I8npdW1 ci4AnjrHD2JvMZ6WjdCztP2pN/vRjWRR =DRoA -----END PGP SIGNATURE-----
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
dariuszs wrote:
I've loaded 60 GB of data file into a table. On Windows (NTFS) the size of dbfarm changed to 80GB but on Linux (ext4 or reiserfs) dbfarm changed to 180GB. Why is that? Thanks. Dariusz.
Your windows build is 32bit, while your Linux build is 64bit?
Stefan
participants (5)
-
dariuszs -
Peter Boncz -
Stefan de Konink -
Stefan Manegold -
Ying Zhang