[MonetDB-users] Memory Issues
Hi,
Is there any relation between CPU, RAM and the amount of data that can be loaded into a MonetDB database?
I am trying to load 13.4 million rows of data and I am getting "Segmentation Fault" on a Linux box running on 32 bit CPU with 4GB RAM.
I had similar issues on a Windows box too?
Thanks,
- Venkatesh
On Tue, Apr 24, 2007 at 09:27:13PM -0400, Venks wrote:
Hi,
Is there any relation between CPU, RAM and the amount of data that can be loaded into a MonetDB database?
I am trying to load 13.4 million rows of data and I am getting "Segmentation Fault" on a Linux box running on 32 bit CPU with 4GB RAM.
I had similar issues on a Windows box too?
Yes there are relations between the 'addressable' memory and the amount of data MonetDB can handle. With a 32 bit system your adress space is at best 4G. So if your records are larger than about 320 bytes 13.4 Million rows cannot fit.
Niels
Thanks,
- Venkatesh
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On 25-04-2007 08:31:13 +0200, Niels Nes wrote:
On Tue, Apr 24, 2007 at 09:27:13PM -0400, Venks wrote:
I am trying to load 13.4 million rows of data and I am getting "Segmentation Fault" on a Linux box running on 32 bit CPU with 4GB RAM.
Yes there are relations between the 'addressable' memory and the amount of data MonetDB can handle. With a 32 bit system your adress space is at best 4G. So if your records are larger than about 320 bytes 13.4 Million rows cannot fit.
Still I think "Segmentation Fault" is the wrong message to convey here.
Thanks very much for your reply confirming what I understood based on MonetDB architecture. Please allow me another follow-up question
Will there be an issue on a 64 bit system? Is there an algorithm/logic that I can use to identify the maximum number rows that I can load in one table based on the size of the row?
Regards,
- Venkatesh
On 4/25/07, Niels Nes Niels.Nes@cwi.nl wrote:
On Tue, Apr 24, 2007 at 09:27:13PM -0400, Venks wrote:
Hi,
Is there any relation between CPU, RAM and the amount of data that can be loaded into a MonetDB database?
I am trying to load 13.4 million rows of data and I am getting "Segmentation Fault" on a Linux box running on 32 bit CPU with 4GB RAM.
I had similar issues on a Windows box too?
Yes there are relations between the 'addressable' memory and the amount of data MonetDB can handle. With a 32 bit system your adress space is at best 4G. So if your records are larger than about 320 bytes 13.4 Million rows cannot fit.
Niels
Thanks,
- Venkatesh
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
--
Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
On 64bit I have two large tables with 49M and 250M rows respectively.
--- Venks venkatesh.mantha@gmail.com wrote:
Thanks very much for your reply confirming what I understood based on MonetDB architecture. Please allow me another follow-up question
Will there be an issue on a 64 bit system? Is there an algorithm/logic that I can use to identify the maximum number rows that I can load in one table based on the size of the row?
Regards,
- Venkatesh
On 4/25/07, Niels Nes Niels.Nes@cwi.nl wrote:
On Tue, Apr 24, 2007 at 09:27:13PM -0400, Venks
wrote:
Hi,
Is there any relation between CPU, RAM and
the amount of data that
can be loaded into a MonetDB database?
I am trying to load 13.4 million rows of data
and I am getting
"Segmentation Fault" on a Linux box running
on 32 bit CPU with 4GB
RAM.
I had similar issues on a Windows box too?
Yes there are relations between the 'addressable'
memory and the
amount of data MonetDB can handle. With a 32 bit
system your adress
space is at best 4G. So if your records are larger
than about 320 bytes
13.4 Million rows cannot fit.
Niels
Thanks,
- Venkatesh
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2
express and take
control of your XML. No limits. Just data. Click
to get it now.
http://sourceforge.net/powerbar/db2/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users
--
Niels Nes, Centre for Mathematics and Computer
Science (CWI)
Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20
592-4312
url: http://www.cwi.nl/~niels e-mail:
Niels.Nes@cwi.nl
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/%3E
_______________________________________________
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net
Colin,
Do you know the size of each row? I have a table with 50 columns and could grow to 100 columns for our testing. Does this matter?
On 4/25/07, Colin Foss cgfoss@yahoo.com wrote:
On 64bit I have two large tables with 49M and 250M rows respectively.
--- Venks venkatesh.mantha@gmail.com wrote:
Thanks very much for your reply confirming what I understood based on MonetDB architecture. Please allow me another follow-up question
Will there be an issue on a 64 bit system? Is there an algorithm/logic that I can use to identify the maximum number rows that I can load in one table based on the size of the row?
Regards,
- Venkatesh
On 4/25/07, Niels Nes Niels.Nes@cwi.nl wrote:
On Tue, Apr 24, 2007 at 09:27:13PM -0400, Venks
wrote:
Hi,
Is there any relation between CPU, RAM and
the amount of data that
can be loaded into a MonetDB database?
I am trying to load 13.4 million rows of data
and I am getting
"Segmentation Fault" on a Linux box running
on 32 bit CPU with 4GB
RAM.
I had similar issues on a Windows box too?
Yes there are relations between the 'addressable'
memory and the
amount of data MonetDB can handle. With a 32 bit
system your adress
space is at best 4G. So if your records are larger
than about 320 bytes
13.4 Million rows cannot fit.
Niels
Thanks,
- Venkatesh
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2
express and take
control of your XML. No limits. Just data. Click
to get it now.
http://sourceforge.net/powerbar/db2/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users
--
Niels Nes, Centre for Mathematics and Computer
Science (CWI)
Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20
592-4312
url: http://www.cwi.nl/~niels e-mail:
Niels.Nes@cwi.nl
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/%3E
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net
Venks,
Number of columns is limited only by disk space. I have columns which are approx 2GB on disk but I believe column size is limited to the maximum size of a file in your given file system, or maybe it is 2GB.
The developers are authority on max column size.
Colin
--- Venks venkatesh.mantha@gmail.com wrote:
Colin,
Do you know the size of each row? I have a table with 50 columns and could grow to 100 columns for our testing. Does this matter?
On 4/25/07, Colin Foss cgfoss@yahoo.com wrote:
On 64bit I have two large tables with 49M and 250M rows respectively.
--- Venks venkatesh.mantha@gmail.com wrote:
Thanks very much for your reply confirming what
I
understood based on MonetDB architecture. Please allow me another follow-up question
Will there be an issue on a 64 bit system? Is
there
an algorithm/logic that I can use to identify the maximum number rows
that I
can load in one table based on the size of the row?
Regards,
- Venkatesh
On 4/25/07, Niels Nes Niels.Nes@cwi.nl wrote:
On Tue, Apr 24, 2007 at 09:27:13PM -0400,
Venks
wrote:
Hi,
Is there any relation between CPU, RAM
and
the amount of data that
can be loaded into a MonetDB database?
I am trying to load 13.4 million rows of
data
and I am getting
"Segmentation Fault" on a Linux box
running
on 32 bit CPU with 4GB
RAM.
I had similar issues on a Windows box
too?
Yes there are relations between the
'addressable'
memory and the
amount of data MonetDB can handle. With a 32
bit
system your adress
space is at best 4G. So if your records are
larger
than about 320 bytes
13.4 Million rows cannot fit.
Niels
Thanks,
- Venkatesh
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2
Express
Download DB2 Express C - the FREE version of
DB2
express and take
control of your XML. No limits. Just data.
Click
to get it now.
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users
--
Niels Nes, Centre for Mathematics and Computer
Science (CWI)
Kruislaan 413, 1098 SJ Amsterdam, The
Netherlands
room C0.02, phone ++31 20 592-4098, fax ++31
20
592-4312
url: http://www.cwi.nl/~niels e-mail:
Niels.Nes@cwi.nl
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click
to
get it now. http://sourceforge.net/powerbar/db2/%3E
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users
Colin,
Thanks for your reply. My question was specific to your data based on "Neils" response regarding addressable space and the byte size of a record.
Anyhow, I think I need a 64 Bit machine for sure to load huge amounts of data for testing.
Thanks,
- Venkatesh
On 4/25/07, Colin Foss cgfoss@yahoo.com wrote:
Venks,
Number of columns is limited only by disk space. I have columns which are approx 2GB on disk but I believe column size is limited to the maximum size of a file in your given file system, or maybe it is 2GB.
The developers are authority on max column size.
Colin
--- Venks venkatesh.mantha@gmail.com wrote:
Colin,
Do you know the size of each row? I have a table with 50 columns and could grow to 100 columns for our testing. Does this matter?
On 4/25/07, Colin Foss cgfoss@yahoo.com wrote:
On 64bit I have two large tables with 49M and 250M rows respectively.
--- Venks venkatesh.mantha@gmail.com wrote:
Thanks very much for your reply confirming what
I
understood based on MonetDB architecture. Please allow me another follow-up question
Will there be an issue on a 64 bit system? Is
there
an algorithm/logic that I can use to identify the maximum number rows
that I
can load in one table based on the size of the row?
Regards,
- Venkatesh
On 4/25/07, Niels Nes Niels.Nes@cwi.nl wrote:
On Tue, Apr 24, 2007 at 09:27:13PM -0400,
Venks
wrote:
Hi,
Is there any relation between CPU, RAM
and
the amount of data that
can be loaded into a MonetDB database?
I am trying to load 13.4 million rows of
data
and I am getting
"Segmentation Fault" on a Linux box
running
on 32 bit CPU with 4GB
RAM.
I had similar issues on a Windows box
too?
Yes there are relations between the
'addressable'
memory and the
amount of data MonetDB can handle. With a 32
bit
system your adress
space is at best 4G. So if your records are
larger
than about 320 bytes
13.4 Million rows cannot fit.
Niels
Thanks,
- Venkatesh
This SF.net email is sponsored by DB2
Express
Download DB2 Express C - the FREE version of
DB2
express and take
control of your XML. No limits. Just data.
Click
to get it now.
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users
--
Niels Nes, Centre for Mathematics and Computer
Science (CWI)
Kruislaan 413, 1098 SJ Amsterdam, The
Netherlands
room C0.02, phone ++31 20 592-4098, fax ++31
20
592-4312
url: http://www.cwi.nl/~niels e-mail:
Niels.Nes@cwi.nl
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click
to
get it now. http://sourceforge.net/powerbar/db2/%3E
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Wed, Apr 25, 2007 at 01:08:28PM -0400, Venks wrote:
Colin,
Thanks for your reply. My question was specific to your data based on "Neils" response regarding addressable space and the byte size of a record.
Intels 64 bit machines have 40 bit adress space (ie Up to one terabyte (TB) of platform address space) Amd's 64 bit machines have 48 bit adress space (ie 256 TB).
A crude measure for the maximum number of rows is to simply divide thes numbers by your record width (even tough we store each column separately).
Anyhow, I think I need a 64 Bit machine for sure to load huge amounts of data for testing.
For huge tables 64 bit is indeed needed.
Niels
Thanks,
- Venkatesh
On 4/25/07, Colin Foss <[1]cgfoss@yahoo.com> wrote:
Venks, Number of columns is limited only by disk space. I have columns which are approx 2GB on disk but I believe column size is limited to the maximum size of a file in your given file system, or maybe it is 2GB. The developers are authority on max column size. Colin --- Venks <[2] venkatesh.mantha@gmail.com> wrote: > Colin, > > Do you know the size of each row? I have a table > with 50 columns and could > grow to 100 columns for our testing. Does this > matter? > > > On 4/25/07, Colin Foss <[3]cgfoss@yahoo.com> wrote: > > > > On 64bit I have two large tables with 49M and 250M > > rows respectively. > > > > > > --- Venks <[4]venkatesh.mantha@gmail.com> wrote: > > > > > Thanks very much for your reply confirming what > I > > > understood based on > > > MonetDB architecture. Please allow me another > > > follow-up question > > > > > > Will there be an issue on a 64 bit system? Is > there > > > an algorithm/logic that > > > I can use to identify the maximum number rows > that I > > > can load in one table > > > based on the size of the row? > > > > > > Regards, > > > > > > - Venkatesh > > > > > > > > > On 4/25/07, Niels Nes <[5]Niels.Nes@cwi.nl > wrote: > > > > > > > > On Tue, Apr 24, 2007 at 09:27:13PM -0400, > Venks > > > wrote: > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Is there any relation between CPU, RAM > and > > > the amount of data that > > > > > can be loaded into a MonetDB database? > > > > > > > > > > > > > > > > > > > > I am trying to load 13.4 million rows of > data > > > and I am getting > > > > > "Segmentation Fault" on a Linux box > running > > > on 32 bit CPU with 4GB > > > > > RAM. > > > > > > > > > > > > > > > > > > > > I had similar issues on a Windows box > too? > > > > > > > > > Yes there are relations between the > 'addressable' > > > memory and the > > > > amount of data MonetDB can handle. With a 32 > bit > > > system your adress > > > > space is at best 4G. So if your records are > larger > > > than about 320 bytes > > > > 13.4 Million rows cannot fit. > > > > > > > > Niels > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > - Venkatesh > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ ------- > > > > > This SF.net email is sponsored by DB2 > Express > > > > > Download DB2 Express C - the FREE version of > DB2 > > > express and take > > > > > control of your XML. No limits. Just data. > Click > > > to get it now. > > > > > [6]http://sourceforge.net/powerbar/db2/ > > > > > > _______________________________________________ > > > > > MonetDB-users mailing list > > > > > [7]MonetDB-users@lists.sourceforge.net > > > > > > > > > > > [8]https://lists.sourceforge.net/lists/listinfo/monetdb-users > > > > > > > > > > > > -- > > > > > > > > Niels Nes, Centre for Mathematics and Computer > > > Science (CWI) > > > > Kruislaan 413, 1098 SJ Amsterdam, The > Netherlands > > > > room C0.02, phone ++31 20 592-4098, fax ++31 > 20 > > > 592-4312 > > > > url: [9]http://www.cwi.nl/~niels e-mail: > > > [10]Niels.Nes@cwi.nl > > > > > > > > > > > ------------------------------------------------------------------ ------- > > > This SF.net email is sponsored by DB2 Express > > > Download DB2 Express C - the FREE version of DB2 > > > express and take > > > control of your XML. No limits. Just data. Click > to > > > get it now. > > > [11]http://sourceforge.net/powerbar/db2/> > > _______________________________________________ > > > MonetDB-users mailing list > > > [12]MonetDB-users@lists.sourceforge.net > > > > > > [13]https://lists.sourceforge.net/lists/listinfo/monetdb-users > > > > > > > >References
- mailto:cgfoss@yahoo.com
- mailto:venkatesh.mantha@gmail.com
- mailto:cgfoss@yahoo.com
- mailto:venkatesh.mantha@gmail.com
- mailto:Niels.Nes@cwi.nl
- http://sourceforge.net/powerbar/db2/
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://www.cwi.nl/~niels
- mailto:Niels.Nes@cwi.nl
- http://sourceforge.net/powerbar/db2/
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
participants (4)
-
Colin Foss -
Fabian Groffen -
Niels Nes -
Venks