25 Apr
2007
25 Apr
'07
8:12 p.m.
On Wed, Apr 25, 2007 at 01:08:28PM -0400, Venks wrote:
Colin,
Thanks for your reply. My question was specific to your data based on "Neils" response regarding addressable space and the byte size of a record.
Intels 64 bit machines have 40 bit adress space (ie Up to one terabyte (TB) of platform address space) Amd's 64 bit machines have 48 bit adress space (ie 256 TB).
A crude measure for the maximum number of rows is to simply divide thes numbers by your record width (even tough we store each column separately).
Anyhow, I think I need a 64 Bit machine for sure to load huge amounts of data for testing.
For huge tables 64 bit is indeed needed.
Niels
Thanks,
- Venkatesh
On 4/25/07, Colin Foss <[1]cgfoss@yahoo.com> wrote:
Venks, Number of columns is limited only by disk space. I have columns which are approx 2GB on disk but I believe column size is limited to the maximum size of a file in your given file system, or maybe it is 2GB. The developers are authority on max column size. Colin --- Venks <[2] venkatesh.mantha@gmail.com> wrote: > Colin, > > Do you know the size of each row? I have a table > with 50 columns and could > grow to 100 columns for our testing. Does this > matter? > > > On 4/25/07, Colin Foss <[3]cgfoss@yahoo.com> wrote: > > > > On 64bit I have two large tables with 49M and 250M > > rows respectively. > > > > > > --- Venks <[4]venkatesh.mantha@gmail.com> wrote: > > > > > Thanks very much for your reply confirming what > I > > > understood based on > > > MonetDB architecture. Please allow me another > > > follow-up question > > > > > > Will there be an issue on a 64 bit system? Is > there > > > an algorithm/logic that > > > I can use to identify the maximum number rows > that I > > > can load in one table > > > based on the size of the row? > > > > > > Regards, > > > > > > - Venkatesh > > > > > > > > > On 4/25/07, Niels Nes <[5]Niels.Nes@cwi.nl > wrote: > > > > > > > > On Tue, Apr 24, 2007 at 09:27:13PM -0400, > Venks > > > wrote: > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Is there any relation between CPU, RAM > and > > > the amount of data that > > > > > can be loaded into a MonetDB database? > > > > > > > > > > > > > > > > > > > > I am trying to load 13.4 million rows of > data > > > and I am getting > > > > > "Segmentation Fault" on a Linux box > running > > > on 32 bit CPU with 4GB > > > > > RAM. > > > > > > > > > > > > > > > > > > > > I had similar issues on a Windows box > too? > > > > > > > > > Yes there are relations between the > 'addressable' > > > memory and the > > > > amount of data MonetDB can handle. With a 32 > bit > > > system your adress > > > > space is at best 4G. So if your records are > larger > > > than about 320 bytes > > > > 13.4 Million rows cannot fit. > > > > > > > > Niels > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > - Venkatesh > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ ------- > > > > > This SF.net email is sponsored by DB2 > Express > > > > > Download DB2 Express C - the FREE version of > DB2 > > > express and take > > > > > control of your XML. No limits. Just data. > Click > > > to get it now. > > > > > [6]http://sourceforge.net/powerbar/db2/ > > > > > > _______________________________________________ > > > > > MonetDB-users mailing list > > > > > [7]MonetDB-users@lists.sourceforge.net > > > > > > > > > > > [8]https://lists.sourceforge.net/lists/listinfo/monetdb-users > > > > > > > > > > > > -- > > > > > > > > Niels Nes, Centre for Mathematics and Computer > > > Science (CWI) > > > > Kruislaan 413, 1098 SJ Amsterdam, The > Netherlands > > > > room C0.02, phone ++31 20 592-4098, fax ++31 > 20 > > > 592-4312 > > > > url: [9]http://www.cwi.nl/~niels e-mail: > > > [10]Niels.Nes@cwi.nl > > > > > > > > > > > ------------------------------------------------------------------ ------- > > > This SF.net email is sponsored by DB2 Express > > > Download DB2 Express C - the FREE version of DB2 > > > express and take > > > control of your XML. No limits. Just data. Click > to > > > get it now. > > > [11]http://sourceforge.net/powerbar/db2/> > > _______________________________________________ > > > MonetDB-users mailing list > > > [12]MonetDB-users@lists.sourceforge.net > > > > > > [13]https://lists.sourceforge.net/lists/listinfo/monetdb-users > > > > > > > >
References
- mailto:cgfoss@yahoo.com
- mailto:venkatesh.mantha@gmail.com
- mailto:cgfoss@yahoo.com
- mailto:venkatesh.mantha@gmail.com
- mailto:Niels.Nes@cwi.nl
- http://sourceforge.net/powerbar/db2/
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://www.cwi.nl/~niels
- mailto:Niels.Nes@cwi.nl
- http://sourceforge.net/powerbar/db2/
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
--
Niels Nes, Centre for Mathematics and Computer Science (CWI)
Kruislaan 413, 1098 SJ Amsterdam, The Netherlands
room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312
url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl