[MonetDB-users] Combined table is not really combined after all
two tables: table_1 and table_2 both have exactly the same schema. table_1 has 2.15MM rows of data, table_2 has 3.35MM row of data. No indexing created a new table: table_combined using same schema as table_1
insert into table_combined select * from table_1;
(30 minutes later)
insert into table_combined select * from table_2;
(40 minutes later)
select count(*) * 1 from table_combined;
this give ONLY 2.15MM rows with same rows as table_1
What happened to table_2?!?
Yes, I've check dfarm size. It equals 2*(table_1 + table_2 )
So it's there but "just not seeing it"
Hi Yue,
is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation.
lefteris
On Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng yuesheng8@gmail.com wrote:
two tables: table_1 and table_2 both have exactly the same schema. table_1 has 2.15MM rows of data, table_2 has 3.35MM row of data. No indexing created a new table: table_combined using same schema as table_1 insert into table_combined select * from table_1; (30 minutes later) insert into table_combined select * from table_2;
(40 minutes later) select count(*) * 1 from table_combined; this give ONLY 2.15MM rows with same rows as table_1 What happened to table_2?!? Yes, I've check dfarm size. It equals 2*(table_1 + table_2 ) So it's there but "just not seeing it"
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
(1) There's no index or key or anything on the table.(2) The combined table is the size of table_1 + table_2. But only table_1 shows up in query
On Sat, Mar 21, 2009 at 1:51 PM, Lefteris lsidir@gmail.com wrote:
Hi Yue,
is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation.
lefteris
On Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng yuesheng8@gmail.com wrote:
two tables: table_1 and table_2 both have exactly the same schema.
table_1
has 2.15MM rows of data, table_2 has 3.35MM row of data. No indexing created a new table: table_combined using same schema as table_1 insert into table_combined select * from table_1; (30 minutes later) insert into table_combined select * from table_2;
(40 minutes later) select count(*) * 1 from table_combined; this give ONLY 2.15MM rows with same rows as table_1 What happened to table_2?!? Yes, I've check dfarm size. It equals 2*(table_1 + table_2 ) So it's there but "just not seeing it"
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Tried the whole process. Still the same problem (table_combined has the same scheme as table_1): insert into table_combined select * from table_1; insert into table_combined select * from table_2;
select count(*) from table_combined;
this gives ONLY the number of records in table_1, but table_combined as roughly same size as the other two combined.
NONE OF THE TABLE HAS ANY INDEXING
Could someone please confirm if this is a bug? Both table_1 and table_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng yuesheng8@gmail.com wrote:
(1) There's no index or key or anything on the table.(2) The combined table is the size of table_1 + table_2. But only table_1 shows up in query
On Sat, Mar 21, 2009 at 1:51 PM, Lefteris lsidir@gmail.com wrote:
Hi Yue,
is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation.
lefteris
On Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng yuesheng8@gmail.com wrote:
two tables: table_1 and table_2 both have exactly the same schema.
table_1
has 2.15MM rows of data, table_2 has 3.35MM row of data. No indexing created a new table: table_combined using same schema as table_1 insert into table_combined select * from table_1; (30 minutes later) insert into table_combined select * from table_2;
(40 minutes later) select count(*) * 1 from table_combined; this give ONLY 2.15MM rows with same rows as table_1 What happened to table_2?!? Yes, I've check dfarm size. It equals 2*(table_1 + table_2 ) So it's there but "just not seeing it"
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng wrote:
Tried the whole process. Still the same problem (table_combined has the same scheme as table_1):
insert into table_combined select * from table_1; insert into table_combined select * from table_2; select count(*) from table_combined; this gives ONLY the number of records in table_1, but table_combined as roughly same size as the other two combined. NONE OF THE TABLE HAS ANY INDEXING Could someone please confirm if this is a bug? Both table_1 and
The way you explain it, it seems a bug.
Niels
table_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng <[1]yuesheng8@gmail.com> wrote:
(1) There's no index or key or anything on the table.(2) The combined table is the size of table_1 + table_2. But only table_1 shows up in query On Sat, Mar 21, 2009 at 1:51 PM, Lefteris <[2]lsidir@gmail.com> wrote:
Hi Yue, is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation. lefterisOn Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng <[3]yuesheng8@gmail.com> wrote:
two tables: table_1 and table_2 both have exactly the same
schema. table_1
has 2.15MM rows of data, table_2 has 3.35MM row of data. No
indexing
created a new table: table_combined using same schema as
table_1
insert into table_combined select * from table_1; (30 minutes later) insert into table_combined select * from table_2;
(40 minutes later) select count(*) * 1 from table_combined; this give ONLY 2.15MM rows with same rows as table_1 What happened to table_2?!? Yes, I've check dfarm size. It equals 2*(table_1 + table_2 ) So it's there but "just not seeing it"
> ------------------------------------------------------------ ------------------ > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. [4]http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > [5]MonetDB-users@lists.sourceforge.net > [6]https://lists.sourceforge.net/lists/listinfo/monetdb-user s > > ------------------------------------------------------------ ------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [7]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [8]MonetDB-users@lists.sourceforge.net [9]https://lists.sourceforge.net/lists/listinfo/monetdb-user sReferences
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Hi, then I'm totally lost. That was my last resort in creating a big table.
So: (1) I can't create a 600-million table using COPY (because it hangs) (2) I can't create two 300-million tables and combined them into one (bug)
Have I missed other ways of doing this?
Just wanted to make sure, the MonetDB can support >600-million-by-10 records in a SINGLE table?
Thanks.
On Mon, Mar 23, 2009 at 9:14 AM, Niels Nes Niels.Nes@cwi.nl wrote:
On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng wrote:
Tried the whole process. Still the same problem (table_combined has the same scheme as table_1):
insert into table_combined select * from table_1; insert into table_combined select * from table_2; select count(*) from table_combined; this gives ONLY the number of records in table_1, but table_combined as roughly same size as the other two combined. NONE OF THE TABLE HAS ANY INDEXING Could someone please confirm if this is a bug? Both table_1 and
The way you explain it, it seems a bug.
Niels
table_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng <[1]yuesheng8@gmail.com> wrote:
(1) There's no index or key or anything on the table.(2) The combined table is the size of table_1 + table_2. But only table_1 shows up in query On Sat, Mar 21, 2009 at 1:51 PM, Lefteris <[2]lsidir@gmail.com> wrote:
Hi Yue, is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation. lefterisOn Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng <[3]yuesheng8@gmail.com> wrote:
two tables: table_1 and table_2 both have exactly the same
schema. table_1
has 2.15MM rows of data, table_2 has 3.35MM row of data. No
indexing
created a new table: table_combined using same schema as
table_1
insert into table_combined select * from table_1; (30 minutes later) insert into table_combined select * from table_2;
(40 minutes later) select count(*) * 1 from table_combined; this give ONLY 2.15MM rows with same rows as table_1 What happened to table_2?!? Yes, I've check dfarm size. It equals 2*(table_1 + table_2 ) So it's there but "just not seeing it"
> ------------------------------------------------------------ ------------------ > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. [4]http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > [5]MonetDB-users@lists.sourceforge.net > [6]https://lists.sourceforge.net/lists/listinfo/monetdb-user s > > ------------------------------------------------------------ ------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [7]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [8]MonetDB-users@lists.sourceforge.net [9]https://lists.sourceforge.net/lists/listinfo/monetdb-user sReferences
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Mon, Mar 23, 2009 at 09:28:24AM -0700, Yue Sheng wrote:
Hi, then I'm totally lost.
That was my last resort in creating a big table.
So:
(1) I can't create a 600-million table using COPY (because it hangs)
(2) I can't create two 300-million tables and combined them into one (bug)
Since we don't have your data. And your probably not able to supply us with it, it is probably better that we look for some 'similar' generated data set which could be the basis to solve this bug(s). So could you make an anonimized schema (create table statement(s) etc) and possibly some data generator which give the before mentioned problems.
Niels
Have I missed other ways of doing this?
Just wanted to make sure, the MonetDB can support
600-million-by-10 records in a SINGLE table?
Thanks. On Mon, Mar 23, 2009 at 9:14 AM, Niels Nes <[1]Niels.Nes@cwi.nl> wrote:
On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng wrote:
Tried the whole process. Still the same problem
(table_combined
has the same scheme as table_1):
insert into table_combined select * from table_1; insert into table_combined select * from table_2; select count(*) from table_combined; this gives ONLY the number of records in table_1, but table_combined as roughly same size as the other two
combined.
NONE OF THE TABLE HAS ANY INDEXING Could someone please confirm if this is a bug? Both
table_1 and
The way you explain it, it seems a bug. Nielstable_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng
<[1][2]yuesheng8@gmail.com> wrote:
(1) There's no index or key or anything on the table.(2) The combined table is the size of table_1 + table_2.
But
only table_1 shows up in query
> On Sat, Mar 21, 2009 at 1:51 PM, Lefteris <[2][3]lsidir@gmail.com>wrote:
Hi Yue, is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation. lefterisOn Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng
<[3][4]yuesheng8@gmail.com> wrote:
two tables: table_1 and table_2 both have exactly the
same
schema. table_1
has 2.15MM rows of data, table_2 has 3.35MM row of data.
No
indexing
created a new table: table_combined using same schema as
table_1
insert into table_combined select * from table_1; (30 minutes later) insert into table_combined select * from table_2;
(40 minutes later) select count(*) * 1 from table_combined; this give ONLY 2.15MM rows with same rows as table_1 What happened to table_2?!? Yes, I've check dfarm size. It equals 2*(table_1 +
table_2 )
So it's there but "just not seeing it"
>
------------------ > Apps built with the Adobe(R) Flex(R) framework andFlex
Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding andstep-through
debugging. > Download the free 60 day trial.> [4][5]http://p.sf.net/sfu/www-adobe-com > > _______________________________________________ > > MonetDB-users mailing list > > [5][6]MonetDB-users@lists.sourceforge.net > > > [6][7]https://lists.sourceforge.net/lists/listinfo/monetdb-u ser > s > > > > > ------------------------------------------------------------ > ------------------Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platformcapabilities.
Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding andstep-through
debugging. Download the free 60 day trial.> [7][8]http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > [8][9]MonetDB-users@lists.sourceforge.net > [9][10]https://lists.sourceforge.net/lists/listinfo/monetdb- user > s > > References > > 1. mailto:[11]yuesheng8@gmail.com > 2. mailto:[12]lsidir@gmail.com > 3. mailto:[13]yuesheng8@gmail.com> 5. mailto:[15]MonetDB-users@lists.sourceforge.net > 6. [16]https://lists.sourceforge.net/lists/listinfo/monetdb-use rs> 8. mailto:[18]MonetDB-users@lists.sourceforge.net > 9. [19]https://lists.sourceforge.net/lists/listinfo/monetdb-use rs
Apps built with the Adobe(R) Flex(R) framework and Flex
Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities.
Quickly and
easily build your RIAs with Flex Builder, the
Eclipse(TM)based development
software that enables intelligent coding and step-through
debugging.
Download the free 60 day trial.
[20]http://p.sf.net/sfu/www-adobe-com
MonetDB-users mailing list [21]MonetDB-users@lists.sourceforge.net
[22]https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: [23]http://www.cwi.nl/~niels e-mail: [24]Niels.Nes@cwi.nl
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [25]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [26]MonetDB-users@lists.sourceforge.net [27]https://lists.sourceforge.net/lists/listinfo/monetdb-users
References
- mailto:Niels.Nes@cwi.nl
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-user
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-user
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://www.cwi.nl/~niels
- mailto:Niels.Nes@cwi.nl
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
(1) This problem pertains to the size of the data in question. I've tested insert 17K rows into table_1 and 13K rows into table_2 then the combined table gives the correct 30K rows
(2) The amount of data is over 45GB, but very simple 10 columns of doubles. Nothing fancy, no characters or funny code. Just plain numbers
(3) Is it necessary to have the exactly numbers?
(4) If not, then any 10 column deep data of doubles would suffice.
(5) How to replicate the problem
(i) generate 600 million rows of doubles, 10 column deep (ii) insert 300 million into one table (iii) insert 300 million into another (iv) combined the two tables (v) see if the combined table has all the data from the two smaller tables
I was wondering if someone has successfully created a SINGLE table with
600million-by-10; Maybe the bug has something to do with not able to
address all the data in a large table? We have 6 billion cells...
My MonetDB is complied in 64bit.
Thanks.
On Mon, Mar 23, 2009 at 10:26 AM, Niels Nes Niels.Nes@cwi.nl wrote:
On Mon, Mar 23, 2009 at 09:28:24AM -0700, Yue Sheng wrote:
Hi, then I'm totally lost.
That was my last resort in creating a big table.
So:
(1) I can't create a 600-million table using COPY (because it hangs)
(2) I can't create two 300-million tables and combined them into one (bug)
Since we don't have your data. And your probably not able to supply us with it, it is probably better that we look for some 'similar' generated data set which could be the basis to solve this bug(s). So could you make an anonimized schema (create table statement(s) etc) and possibly some data generator which give the before mentioned problems.
Niels
Have I missed other ways of doing this?
Just wanted to make sure, the MonetDB can support
600-million-by-10 records in a SINGLE table?
Thanks. On Mon, Mar 23, 2009 at 9:14 AM, Niels Nes <[1]Niels.Nes@cwi.nl> wrote:
On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng wrote:
Tried the whole process. Still the same problem
(table_combined
has the same scheme as table_1):
insert into table_combined select * from table_1; insert into table_combined select * from table_2; select count(*) from table_combined; this gives ONLY the number of records in table_1, but table_combined as roughly same size as the other two
combined.
NONE OF THE TABLE HAS ANY INDEXING Could someone please confirm if this is a bug? Both
table_1 and
The way you explain it, it seems a bug. Nielstable_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng
<[1][2]yuesheng8@gmail.com> wrote:
(1) There's no index or key or anything on the table.(2) The combined table is the size of table_1 + table_2.
But
only table_1 shows up in query
> On Sat, Mar 21, 2009 at 1:51 PM, Lefteris <[2][3]lsidir@gmail.com>wrote:
Hi Yue, is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation. lefterisOn Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng
<[3][4]yuesheng8@gmail.com> wrote:
two tables: table_1 and table_2 both have exactly the
same
schema. table_1
has 2.15MM rows of data, table_2 has 3.35MM row of data.
No
indexing
created a new table: table_combined using same schema as
table_1
insert into table_combined select * from table_1; (30 minutes later) insert into table_combined select * from table_2;
(40 minutes later) select count(*) * 1 from table_combined; this give ONLY 2.15MM rows with same rows as table_1 What happened to table_2?!? Yes, I've check dfarm size. It equals 2*(table_1 +
table_2 )
So it's there but "just not seeing it"
>
------------------ > Apps built with the Adobe(R) Flex(R) framework andFlex
Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding andstep-through
debugging. > Download the free 60 day trial.> [4][5]http://p.sf.net/sfu/www-adobe-com > > _______________________________________________ > > MonetDB-users mailing list > > [5][6]MonetDB-users@lists.sourceforge.net > > > [6][7]https://lists.sourceforge.net/lists/listinfo/monetdb-u ser > s > > > > > ------------------------------------------------------------ > ------------------Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platformcapabilities.
Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding andstep-through
debugging. Download the free 60 day trial.> [7][8]http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > [8][9]MonetDB-users@lists.sourceforge.net > [9][10]https://lists.sourceforge.net/lists/listinfo/monetdb- user > s > > References > > 1. mailto:[11]yuesheng8@gmail.com > 2. mailto:[12]lsidir@gmail.com > 3. mailto:[13]yuesheng8@gmail.com> 5. mailto:[15]MonetDB-users@lists.sourceforge.net > 6. [16]https://lists.sourceforge.net/lists/listinfo/monetdb-use rs> 8. mailto:[18]MonetDB-users@lists.sourceforge.net > 9. [19]https://lists.sourceforge.net/lists/listinfo/monetdb-use rs
Apps built with the Adobe(R) Flex(R) framework and Flex
Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities.
Quickly and
easily build your RIAs with Flex Builder, the
Eclipse(TM)based development
software that enables intelligent coding and step-through
debugging.
Download the free 60 day trial.
[20]http://p.sf.net/sfu/www-adobe-com
MonetDB-users mailing list [21]MonetDB-users@lists.sourceforge.net
[22]https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: [23]http://www.cwi.nl/~niels e-mail: [24]Niels.Nes@cwi.nl
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [25]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [26]MonetDB-users@lists.sourceforge.net [27]https://lists.sourceforge.net/lists/listinfo/monetdb-users
References
- mailto:Niels.Nes@cwi.nl
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-user
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-user
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://www.cwi.nl/~niels
- mailto:Niels.Nes@cwi.nl
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Hi, then I'm totally lost. That was my last resort in creating a big table.
So: (1) I can't create a 600-million table using COPY (because it hangs) (2) I can't create two 300-million tables and combined them into one (bug)
Have I missed other ways of doing this?
Just wanted to make sure, the MonetDB can support >600-million-by-10 records in a SINGLE table?
Hi, In SkyServer DR6 database of 2.5TB MonetDB supports tables longer than 600M rows, although with smaller number of columns. For instance: sql>select count(*) from photoprofile; +-------------+ | count_ | +=============+ | 13282921742 | +-------------+ 1 tuple sql>select * from photoprofile limit 1; +-----+------+---------------+---------------+--------------------+ | bin | band | profmean | proferr | objid | +=====+======+===============+===============+====================+ | 0 | 0 | 1.49587098e-0 | 6.95472624e-1 | 587722951693303809 | : : : 8 : 0 : : +-----+------+---------------+---------------+--------------------+
where 2 columns are tinyint, 2 - real and 1 -bigint. The table was loaded by 'copy .. into ..' statement. Milena
Thanks.
On Mon, Mar 23, 2009 at 9:14 AM, Niels Nes Niels.Nes@cwi.nl wrote:
On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng wrote:
Tried the whole process. Still the same problem (table_combined has the same scheme as table_1):
insert into table_combined select * from table_1; insert into table_combined select * from table_2; select count(*) from table_combined; this gives ONLY the number of records in table_1, but table_combined as roughly same size as the other two combined. NONE OF THE TABLE HAS ANY INDEXING Could someone please confirm if this is a bug? Both table_1 and
The way you explain it, it seems a bug.
Niels
table_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng <[1]yuesheng8@gmail.com> wrote:
(1) There's no index or key or anything on the table.(2) The combined table is the size of table_1 + table_2. But only table_1 shows up in query On Sat, Mar 21, 2009 at 1:51 PM, Lefteris <[2]lsidir@gmail.com> wrote:
Hi Yue, is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation. lefterisOn Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng <[3]yuesheng8@gmail.com> wrote:
two tables: table_1 and table_2 both have exactly the same
schema. table_1
has 2.15MM rows of data, table_2 has 3.35MM row of data. No
indexing
created a new table: table_combined using same schema as
table_1
insert into table_combined select * from table_1; (30 minutes later) insert into table_combined select * from table_2;
(40 minutes later) select count(*) * 1 from table_combined; this give ONLY 2.15MM rows with same rows as table_1 What happened to table_2?!? Yes, I've check dfarm size. It equals 2*(table_1 + table_2 ) So it's there but "just not seeing it"
> ------------------------------------------------------------ ------------------ > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. [4]http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > [5]MonetDB-users@lists.sourceforge.net > [6]https://lists.sourceforge.net/lists/listinfo/monetdb-user s > > ------------------------------------------------------------ ------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [7]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [8]MonetDB-users@lists.sourceforge.net [9]https://lists.sourceforge.net/lists/listinfo/monetdb-user sReferences
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM)
are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly
and
easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com___________________________________________... MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Melina, great to get confirmation that it is indeed possible to create large single table. I presume the DB6 is hosted on Linux?
May I ask a few questions: (1) did you create the table from multiple files? If so, what is the average and maximum size of each of these files? (2) did you try to create the table in one session? (i.e. loop loading all necessary files over several hour for example) (3) did you use the multi-thread version of bulk load COPY... INTO? (4) what version of DB and Kernel did you use? Is it compiled in 64bit? (5) how much memory does the server have? (6) finally, did you run into any problem in the process?
Many thanks!
On Tue, Mar 24, 2009 at 3:38 AM, Milena.Ivanova@cwi.nl wrote:
Hi, then I'm totally lost. That was my last resort in creating a big table.
So: (1) I can't create a 600-million table using COPY (because it hangs) (2) I can't create two 300-million tables and combined them into one
(bug)
Have I missed other ways of doing this?
Just wanted to make sure, the MonetDB can support >600-million-by-10 records in a SINGLE table?
Hi, In SkyServer DR6 database of 2.5TB MonetDB supports tables longer than 600M rows, although with smaller number of columns. For instance: sql>select count(*) from photoprofile; +-------------+ | count_ | +=============+ | 13282921742 | +-------------+ 1 tuple sql>select * from photoprofile limit 1; +-----+------+---------------+---------------+--------------------+ | bin | band | profmean | proferr | objid | +=====+======+===============+===============+====================+ | 0 | 0 | 1.49587098e-0 | 6.95472624e-1 | 587722951693303809 | : : : 8 : 0 : : +-----+------+---------------+---------------+--------------------+
where 2 columns are tinyint, 2 - real and 1 -bigint. The table was loaded by 'copy .. into ..' statement. Milena
Thanks.
On Mon, Mar 23, 2009 at 9:14 AM, Niels Nes Niels.Nes@cwi.nl wrote:
On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng wrote:
Tried the whole process. Still the same problem (table_combined has the same scheme as table_1):
insert into table_combined select * from table_1; insert into table_combined select * from table_2; select count(*) from table_combined; this gives ONLY the number of records in table_1, but table_combined as roughly same size as the other two combined. NONE OF THE TABLE HAS ANY INDEXING Could someone please confirm if this is a bug? Both table_1 and
The way you explain it, it seems a bug.
Niels
table_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng <[1]yuesheng8@gmail.com> wrote:
(1) There's no index or key or anything on the table.(2) The combined table is the size of table_1 + table_2. But only table_1 shows up in query On Sat, Mar 21, 2009 at 1:51 PM, Lefteris <[2]lsidir@gmail.com> wrote:
Hi Yue, is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation. lefterisOn Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng <[3]yuesheng8@gmail.com> wrote:
two tables: table_1 and table_2 both have exactly the same
schema. table_1
has 2.15MM rows of data, table_2 has 3.35MM row of data. No
indexing
created a new table: table_combined using same schema as
table_1
insert into table_combined select * from table_1; (30 minutes later) insert into table_combined select * from table_2;
(40 minutes later) select count(*) * 1 from table_combined; this give ONLY 2.15MM rows with same rows as table_1 What happened to table_2?!? Yes, I've check dfarm size. It equals 2*(table_1 + table_2 ) So it's there but "just not seeing it"
> ------------------------------------------------------------ ------------------ > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. [4]http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > [5]MonetDB-users@lists.sourceforge.net > [6]https://lists.sourceforge.net/lists/listinfo/monetdb-user s > > ------------------------------------------------------------ ------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [7]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [8]MonetDB-users@lists.sourceforge.net [9]https://lists.sourceforge.net/lists/listinfo/monetdb-user sReferences
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM)
are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly
and
easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial.
http://p.sf.net/sfu/www-adobe-com___________________________________________...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Melina, great to get confirmation that it is indeed possible to create large single table. I presume the DB6 is hosted on Linux?
Hi, I just wanted to point out that there are examples of MonetDB supporting large tables. I have to clarify that the DB was created in October 2008 using the current version of MonetDB at that time on 64bit Linux. Arjen de Rijke, who made the load, can give you the other details if needed. However, a lot of important features have been changed since then. As Niels suggested, the best would be to re-create your situation as close as possible to identify the cause of the problem. Milena
May I ask a few questions: (1) did you create the table from multiple files? If so, what is the average and maximum size of each of these files? (2) did you try to create the table in one session? (i.e. loop loading all necessary files over several hour for example) (3) did you use the multi-thread version of bulk load COPY... INTO? (4) what version of DB and Kernel did you use? Is it compiled in 64bit? (5) how much memory does the server have? (6) finally, did you run into any problem in the process?
Many thanks!
On Tue, Mar 24, 2009 at 3:38 AM, Milena.Ivanova@cwi.nl wrote:
Hi, then I'm totally lost. That was my last resort in creating a big table.
So: (1) I can't create a 600-million table using COPY (because it hangs) (2) I can't create two 300-million tables and combined them into one
(bug)
Have I missed other ways of doing this?
Just wanted to make sure, the MonetDB can support >600-million-by-10 records in a SINGLE table?
Hi, In SkyServer DR6 database of 2.5TB MonetDB supports tables longer than 600M rows, although with smaller number of columns. For instance: sql>select count(*) from photoprofile; +-------------+ | count_ | +=============+ | 13282921742 | +-------------+ 1 tuple sql>select * from photoprofile limit 1; +-----+------+---------------+---------------+--------------------+ | bin | band | profmean | proferr | objid | +=====+======+===============+===============+====================+ | 0 | 0 | 1.49587098e-0 | 6.95472624e-1 | 587722951693303809 | : : : 8 : 0 : : +-----+------+---------------+---------------+--------------------+
where 2 columns are tinyint, 2 - real and 1 -bigint. The table was loaded by 'copy .. into ..' statement. Milena
Thanks.
On Mon, Mar 23, 2009 at 9:14 AM, Niels Nes Niels.Nes@cwi.nl wrote:
On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng wrote:
Tried the whole process. Still the same problem (table_combined has the same scheme as table_1):
insert into table_combined select * from table_1; insert into table_combined select * from table_2; select count(*) from table_combined; this gives ONLY the number of records in table_1, but table_combined as roughly same size as the other two combined. NONE OF THE TABLE HAS ANY INDEXING Could someone please confirm if this is a bug? Both table_1 and
The way you explain it, it seems a bug.
Niels
table_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng <[1]yuesheng8@gmail.com> wrote:
(1) There's no index or key or anything on the table.(2) The combined table is the size of table_1 + table_2. But only table_1 shows up in query On Sat, Mar 21, 2009 at 1:51 PM, Lefteris <[2]lsidir@gmail.com> wrote:
Hi Yue, is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation. lefterisOn Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng <[3]yuesheng8@gmail.com> wrote:
two tables: table_1 and table_2 both have exactly the same
schema. table_1
has 2.15MM rows of data, table_2 has 3.35MM row of data. No
indexing
created a new table: table_combined using same schema as
table_1
insert into table_combined select * from table_1; (30 minutes later) insert into table_combined select * from table_2;
(40 minutes later) select count(*) * 1 from table_combined; this give ONLY 2.15MM rows with same rows as table_1 What happened to table_2?!? Yes, I've check dfarm size. It equals 2*(table_1 + table_2 ) So it's there but "just not seeing it"
> ------------------------------------------------------------ ------------------ > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. [4]http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > [5]MonetDB-users@lists.sourceforge.net > [6]https://lists.sourceforge.net/lists/listinfo/monetdb-user s > > ------------------------------------------------------------ ------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [7]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [8]MonetDB-users@lists.sourceforge.net [9]https://lists.sourceforge.net/lists/listinfo/monetdb-user sReferences
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM)
are
powering Web 2.0 with engaging, cross-platform capabilities.
Quickly
and
easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through
debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM)
are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly
and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM)
are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly
and
easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial.
http://p.sf.net/sfu/www-adobe-com___________________________________________...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com___________________________________________... MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Thanks for the that. Recreating would be the way to go to replicate the issues. The problem is that I'm on Mac Box and the data set is (which itself is quite simple; maybe we can use a few line as guide then replicate 600million times?) too big to send around (not to mention data licensing issues).
On Tue, Mar 24, 2009 at 10:46 AM, Milena.Ivanova@cwi.nl wrote:
Melina, great to get confirmation that it is indeed possible to create large single table. I presume the DB6 is hosted on Linux?
Hi, I just wanted to point out that there are examples of MonetDB supporting large tables. I have to clarify that the DB was created in October 2008 using the current version of MonetDB at that time on 64bit Linux. Arjen de Rijke, who made the load, can give you the other details if needed. However, a lot of important features have been changed since then. As Niels suggested, the best would be to re-create your situation as close as possible to identify the cause of the problem. Milena
May I ask a few questions: (1) did you create the table from multiple files? If so, what is the average and maximum size of each of these files? (2) did you try to create the table in one session? (i.e. loop loading
all
necessary files over several hour for example) (3) did you use the multi-thread version of bulk load COPY... INTO? (4) what version of DB and Kernel did you use? Is it compiled in 64bit? (5) how much memory does the server have? (6) finally, did you run into any problem in the process?
Many thanks!
On Tue, Mar 24, 2009 at 3:38 AM, Milena.Ivanova@cwi.nl wrote:
Hi, then I'm totally lost. That was my last resort in creating a big table.
So: (1) I can't create a 600-million table using COPY (because it hangs) (2) I can't create two 300-million tables and combined them into one
(bug)
Have I missed other ways of doing this?
Just wanted to make sure, the MonetDB can support >600-million-by-10 records in a SINGLE table?
Hi, In SkyServer DR6 database of 2.5TB MonetDB supports tables longer than 600M rows, although with smaller number of columns. For instance: sql>select count(*) from photoprofile; +-------------+ | count_ | +=============+ | 13282921742 | +-------------+ 1 tuple sql>select * from photoprofile limit 1; +-----+------+---------------+---------------+--------------------+ | bin | band | profmean | proferr | objid | +=====+======+===============+===============+====================+ | 0 | 0 | 1.49587098e-0 | 6.95472624e-1 | 587722951693303809 | : : : 8 : 0 : : +-----+------+---------------+---------------+--------------------+
where 2 columns are tinyint, 2 - real and 1 -bigint. The table was loaded by 'copy .. into ..' statement. Milena
Thanks.
On Mon, Mar 23, 2009 at 9:14 AM, Niels Nes Niels.Nes@cwi.nl wrote:
On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng wrote:
Tried the whole process. Still the same problem (table_combined has the same scheme as table_1):
insert into table_combined select * from table_1; insert into table_combined select * from table_2; select count(*) from table_combined; this gives ONLY the number of records in table_1, but table_combined as roughly same size as the other two combined. NONE OF THE TABLE HAS ANY INDEXING Could someone please confirm if this is a bug? Both table_1 and
The way you explain it, it seems a bug.
Niels
table_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng <[1]yuesheng8@gmail.com> wrote:
(1) There's no index or key or anything on the table.(2) The combined table is the size of table_1 + table_2. But only table_1 shows up in query On Sat, Mar 21, 2009 at 1:51 PM, Lefteris <[2]lsidir@gmail.com> wrote:
Hi Yue, is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation. lefterisOn Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng <[3]yuesheng8@gmail.com> wrote: > two tables: table_1 and table_2 both have exactly the same schema. table_1 > has 2.15MM rows of data, table_2 has 3.35MM row of data. No indexing > created a new table: table_combined using same schema as table_1 > insert into table_combined select * from table_1; > (30 minutes later) > insert into table_combined select * from table_2; > > (40 minutes later) > select count(*) * 1 from table_combined; > this give ONLY 2.15MM rows with same rows as table_1 > What happened to table_2?!? > Yes, I've check dfarm size. It equals 2*(table_1 + table_2 ) > So it's there but "just not seeing it" > > > > >
> ------------------------------------------------------------ ------------------ > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. [4]http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > [5]MonetDB-users@lists.sourceforge.net > [6]https://lists.sourceforge.net/lists/listinfo/monetdb-user s > > ------------------------------------------------------------ ------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [7]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [8]MonetDB-users@lists.sourceforge.net [9]https://lists.sourceforge.net/lists/listinfo/monetdb-user sReferences
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM)
are
powering Web 2.0 with engaging, cross-platform capabilities.
Quickly
and
easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through
debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM)
are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly
and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM)
are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly
and
easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial.
http://p.sf.net/sfu/www-adobe-com___________________________________________...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial.
http://p.sf.net/sfu/www-adobe-com___________________________________________...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Tue, Mar 24, 2009 at 11:04:29AM -0700, Yue Sheng wrote:
Thanks for the that. Recreating would be the way to go to replicate the issues. The problem is that I'm on Mac Box and the data set is (which itself is quite simple; maybe we can use a few line as guide then replicate 600million times?) too big to send around (not to mention data licensing issues).
I have generated data and I'm busy loading it. Ie later more
Niels
On Tue, Mar 24, 2009 at 10:46 AM, <[1]Milena.Ivanova@cwi.nl> wrote:
Melina, great to get confirmation that it is indeed possible
to create
large single table. I presume the DB6 is hosted on Linux?
Hi, I just wanted to point out that there are examples of MonetDB supporting large tables. I have to clarify that the DB was created in October 2008 using the current version of MonetDB at that time on 64bit Linux. Arjen de Rijke, who made the load, can give you the other details if needed. However, a lot of important features have been changed since then. As Niels suggested, the best would be to re-create your situation as close as possible to identify the cause of the problem. MilenaMay I ask a few questions: (1) did you create the table from multiple files? If so, what
is the
average and maximum size of each of these files? (2) did you try to create the table in one session? (i.e.
loop loading all
necessary files over several hour for example) (3) did you use the multi-thread version of bulk load COPY...
INTO?
(4) what version of DB and Kernel did you use? Is it compiled
in 64bit?
(5) how much memory does the server have? (6) finally, did you run into any problem in the process?
Many thanks!
On Tue, Mar 24, 2009 at 3:38 AM, <[2]Milena.Ivanova@cwi.nl>
wrote:
Hi, then I'm totally lost. That was my last resort in creating a big table.
So: (1) I can't create a 600-million table using COPY (because
it hangs)
(2) I can't create two 300-million tables and combined
them into one
(bug)
Have I missed other ways of doing this?
Just wanted to make sure, the MonetDB can support
600-million-by-10
records in a SINGLE table?
Hi, In SkyServer DR6 database of 2.5TB MonetDB supports tables
longer than
600M rows, although with smaller number of columns. For instance: sql>select count(*) from photoprofile; +-------------+ | count_ | +=============+ | 13282921742 | +-------------+ 1 tuple sql>select * from photoprofile limit 1;
+-----+------+---------------+---------------+----------------- ---+
| bin | band | profmean | proferr | objid
|+=====+======+===============+===============+================= ===+
| 0 | 0 | 1.49587098e-0 | 6.95472624e-1 |
587722951693303809 |
: : : 8 : 0 :
:+-----+------+---------------+---------------+----------------- ---+
where 2 columns are tinyint, 2 - real and 1 -bigint. The
table was
loaded by 'copy .. into ..' statement. Milena
Thanks.
On Mon, Mar 23, 2009 at 9:14 AM, Niels Nes
<[3]Niels.Nes@cwi.nl> wrote:
On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng
wrote:
Tried the whole process. Still the same problem
(table_combined
has the same scheme as table_1):
insert into table_combined select * from table_1; insert into table_combined select * from table_2; select count(*) from table_combined; this gives ONLY the number of records in table_1,
but
table_combined as roughly same size as the other two
combined.
NONE OF THE TABLE HAS ANY INDEXING Could someone please confirm if this is a bug? Both
table_1 and
The way you explain it, it seems a bug.
Niels
table_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng <[1][4]yuesheng8@gmail.com> wrote:
(1) There's no index or key or anything on thetable.
(2) The combined table is the size of table_1 +
table_2. But
only table_1 shows up in query On Sat, Mar 21, 2009 at 1:51 PM, Lefteris
<[2][5]lsidir@gmail.com>
wrote:
Hi Yue, is there any chance that you have set uniquenessin the
table specs and thus not inserting duplicate tuples fromtable_1 and
table_2? If not, then I would guess that this is a bug and itneeds
further investigation. lefterisOn Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng <[3][6]yuesheng8@gmail.com> wrote: > two tables: table_1 and table_2 both have exactly
the same
schema. table_1 > has 2.15MM rows of data, table_2 has 3.35MM row of
data. No
indexing > created a new table: table_combined using same
schema as
table_1 > insert into table_combined select * from table_1; > (30 minutes later) > insert into table_combined select * from table_2; > > (40 minutes later) > select count(*) * 1 from table_combined; > this give ONLY 2.15MM rows with same rows as
table_1
> What happened to table_2?!? > Yes, I've check dfarm size. It equals 2*(table_1 +
table_2 )
> So it's there but "just not seeing it" > > > > >
>
------------------ > Apps built with the Adobe(R) Flex(R) frameworkand Flex
Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding andstep-through
debugging. > Download the free 60 day trial. [4][7]http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > [5][8]MonetDB-users@lists.sourceforge.net >[6][9]https://lists.sourceforge.net/lists/listinfo/monetdb-user
s > >
------------------ Apps built with the Adobe(R) Flex(R) framework andFlex
Builder(TM) are powering Web 2.0 with engaging, cross-platformcapabilities.
Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding andstep-through
debugging. Download the free 60 day trial. [7][10]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [8][11]MonetDB-users@lists.sourceforge.net[9][12]https://lists.sourceforge.net/lists/listinfo/monetdb-use r
sReferences
- mailto:[13]yuesheng8@gmail.com
- mailto:[14]lsidir@gmail.com
- mailto:[15]yuesheng8@gmail.com
- [16]http://p.sf.net/sfu/www-adobe-com
- mailto:[17]MonetDB-users@lists.sourceforge.net
[18]https://lists.sourceforge.net/lists/listinfo/monetdb-users
- [19]http://p.sf.net/sfu/www-adobe-com
- mailto:[20]MonetDB-users@lists.sourceforge.net
[21]https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex
Builder(TM)
are
powering Web 2.0 with engaging, cross-platform
capabilities.
Quickly
and
easily build your RIAs with Flex Builder, the
Eclipse(TM)based
development
software that enables intelligent coding and
step-through
debugging.
Download the free 60 day trial.
[22]http://p.sf.net/sfu/www-adobe-com
MonetDB-users mailing list [23]MonetDB-users@lists.sourceforge.net
[24]https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science
(CWI)
Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: [25]http://www.cwi.nl/~niels e-mail:
[26]Niels.Nes@cwi.nl
Apps built with the Adobe(R) Flex(R) framework and Flex
Builder(TM)
are
powering Web 2.0 with engaging, cross-platform
capabilities. Quickly
and
easily build your RIAs with Flex Builder, the
Eclipse(TM)based
development software that enables intelligent coding and step-through
debugging.
Download the free 60 day trial.
[27]http://p.sf.net/sfu/www-adobe-com
MonetDB-users mailing list [28]MonetDB-users@lists.sourceforge.net
[29]https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex
Builder(TM)
are
powering Web 2.0 with engaging, cross-platform
capabilities. Quickly
and
easily build your RIAs with Flex Builder, the
Eclipse(TM)based
development
software that enables intelligent coding and step-through
debugging.
Download the free 60 day trial.
[30]http://p.sf.net/sfu/www-adobe-com__________________________ _____________________
MonetDB-users mailing list [31]MonetDB-users@lists.sourceforge.net
[32]https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex
Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities.
Quickly and
easily build your RIAs with Flex Builder, the
Eclipse(TM)based
development software that enables intelligent coding and step-through
debugging.
Download the free 60 day trial.
[33]http://p.sf.net/sfu/www-adobe-com
MonetDB-users mailing list [34]MonetDB-users@lists.sourceforge.net
[35]https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex
Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities.
Quickly and
easily build your RIAs with Flex Builder, the
Eclipse(TM)based development
software that enables intelligent coding and step-through
debugging.
Download the free 60 day trial.
[36]http://p.sf.net/sfu/www-adobe-com__________________________ _____________________
MonetDB-users mailing list [37]MonetDB-users@lists.sourceforge.net
[38]https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [39]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [40]MonetDB-users@lists.sourceforge.net [41]https://lists.sourceforge.net/lists/listinfo/monetdb-users
References
- mailto:Milena.Ivanova@cwi.nl
- mailto:Milena.Ivanova@cwi.nl
- mailto:Niels.Nes@cwi.nl
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-user
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-user
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://www.cwi.nl/~niels
- mailto:Niels.Nes@cwi.nl
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com___________________________________________...
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com___________________________________________...
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Tue, Mar 24, 2009 at 11:38:59AM +0100, Milena.Ivanova@cwi.nl wrote:
Hi, then I'm totally lost. That was my last resort in creating a big table.
So: (1) I can't create a 600-million table using COPY (because it hangs) (2) I can't create two 300-million tables and combined them into one (bug)
Have I missed other ways of doing this?
Just wanted to make sure, the MonetDB can support >600-million-by-10 records in a SINGLE table?
Hi, In SkyServer DR6 database of 2.5TB MonetDB supports tables longer than 600M rows, although with smaller number of columns. For instance: sql>select count(*) from photoprofile; +-------------+ | count_ | +=============+ | 13282921742 | +-------------+ 1 tuple sql>select * from photoprofile limit 1; +-----+------+---------------+---------------+--------------------+ | bin | band | profmean | proferr | objid | +=====+======+===============+===============+====================+ | 0 | 0 | 1.49587098e-0 | 6.95472624e-1 | 587722951693303809 | : : : 8 : 0 : : +-----+------+---------------+---------------+--------------------+
where 2 columns are tinyint, 2 - real and 1 -bigint. The table was loaded by 'copy .. into ..' statement. Milena
In this line, I also recall reports that the TPC-H dataset at scale factor 100 (i.e., ~100GB .cvs) has been successfully loaded into MonetDB/SQL (both Nov2008-SP2 & Feb2009) on a 64-bit 8 GB RAM Linux desktop. Among others, the SF-100 TPC-H dataset contains a 600037902 record "lineitem" table with the following schema:
CREATE TABLE lineitem ( l_orderkey INT NOT NULL, l_partkey INT NOT NULL, l_suppkey INT NOT NULL, l_linenumber INT NOT NULL, l_quantity FLOAT NOT NULL, l_extendedprice FLOAT NOT NULL, l_discount FLOAT NOT NULL, l_tax FLOAT NOT NULL, l_returnflag VARCHAR(1) NOT NULL, l_linestatus VARCHAR(1) NOT NULL, l_shipdate DATE NOT NULL, l_commitdate DATE NOT NULL, l_receiptdate DATE NOT NULL, l_shipinstruct VARCHAR(25) NOT NULL, l_shipmode VARCHAR(10) NOT NULL, l_comment VARCHAR(44) NOT NULL );
AFAIK, the table has been loaded using "COPY 600037902 RECORDS INTO ... FROM FILE ..." from a single 75 GB .csv file.
Stefan
Thanks.
On Mon, Mar 23, 2009 at 9:14 AM, Niels Nes Niels.Nes@cwi.nl wrote:
On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng wrote:
Tried the whole process. Still the same problem (table_combined has the same scheme as table_1):
insert into table_combined select * from table_1; insert into table_combined select * from table_2; select count(*) from table_combined; this gives ONLY the number of records in table_1, but table_combined as roughly same size as the other two combined. NONE OF THE TABLE HAS ANY INDEXING Could someone please confirm if this is a bug? Both table_1 and
The way you explain it, it seems a bug.
Niels
table_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng <[1]yuesheng8@gmail.com> wrote:
(1) There's no index or key or anything on the table.(2) The combined table is the size of table_1 + table_2. But only table_1 shows up in query On Sat, Mar 21, 2009 at 1:51 PM, Lefteris <[2]lsidir@gmail.com> wrote:
Hi Yue, is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation. lefterisOn Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng <[3]yuesheng8@gmail.com> wrote:
two tables: table_1 and table_2 both have exactly the same
schema. table_1
has 2.15MM rows of data, table_2 has 3.35MM row of data. No
indexing
created a new table: table_combined using same schema as
table_1
insert into table_combined select * from table_1; (30 minutes later) insert into table_combined select * from table_2;
(40 minutes later) select count(*) * 1 from table_combined; this give ONLY 2.15MM rows with same rows as table_1 What happened to table_2?!? Yes, I've check dfarm size. It equals 2*(table_1 + table_2 ) So it's there but "just not seeing it"
> ------------------------------------------------------------ ------------------ > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. [4]http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > [5]MonetDB-users@lists.sourceforge.net > [6]https://lists.sourceforge.net/lists/listinfo/monetdb-user s > > ------------------------------------------------------------ ------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [7]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [8]MonetDB-users@lists.sourceforge.net [9]https://lists.sourceforge.net/lists/listinfo/monetdb-user sReferences
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM)
are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly
and
easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com___________________________________________... MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Niels, let me give you my schema and a few lines of data:
# mar 23 test db stuff create table table_ combined( type_flag tinyint, ins_key mediumint, trade_price double, trade_size int, trade_time int, bid_price double, bid_size int, bid_time int, ask_price double, ask_size int, ask_time int, total_volume int, avg_bid_ask_spread double, nullcol varchar(5) );
Example data ("select * from table_combined;"): type_flag,ins_key,trade_price,trade_size,trade_time,bid_price,bid_size,bid_time,ask_price,ask_size,ask_time,total_volume,avg_bid_ask_spread,nullcol 1,151631,1258,22000,1199113221,1258,8,1199113221,1260,24,1199113221,0,1, 1,160133,1395,71100,1199113221,1390,116,1199113221,1395,113,1199113221,0,1, 1,154315,186000,130,1199113238,186000,737,1199113238,186100,151,1199113238,0,1,
Thanks.
On Wed, Mar 25, 2009 at 12:37 AM, Stefan Manegold Stefan.Manegold@cwi.nlwrote:
On Tue, Mar 24, 2009 at 11:38:59AM +0100, Milena.Ivanova@cwi.nl wrote:
Hi, then I'm totally lost. That was my last resort in creating a big table.
So: (1) I can't create a 600-million table using COPY (because it hangs) (2) I can't create two 300-million tables and combined them into one
(bug)
Have I missed other ways of doing this?
Just wanted to make sure, the MonetDB can support >600-million-by-10 records in a SINGLE table?
Hi, In SkyServer DR6 database of 2.5TB MonetDB supports tables longer than 600M rows, although with smaller number of columns. For instance: sql>select count(*) from photoprofile; +-------------+ | count_ | +=============+ | 13282921742 | +-------------+ 1 tuple sql>select * from photoprofile limit 1; +-----+------+---------------+---------------+--------------------+ | bin | band | profmean | proferr | objid | +=====+======+===============+===============+====================+ | 0 | 0 | 1.49587098e-0 | 6.95472624e-1 | 587722951693303809 | : : : 8 : 0 : : +-----+------+---------------+---------------+--------------------+
where 2 columns are tinyint, 2 - real and 1 -bigint. The table was loaded by 'copy .. into ..' statement. Milena
In this line, I also recall reports that the TPC-H dataset at scale factor 100 (i.e., ~100GB .cvs) has been successfully loaded into MonetDB/SQL (both Nov2008-SP2 & Feb2009) on a 64-bit 8 GB RAM Linux desktop. Among others, the SF-100 TPC-H dataset contains a 600037902 record "lineitem" table with the following schema:
CREATE TABLE lineitem ( l_orderkey INT NOT NULL, l_partkey INT NOT NULL, l_suppkey INT NOT NULL, l_linenumber INT NOT NULL, l_quantity FLOAT NOT NULL, l_extendedprice FLOAT NOT NULL, l_discount FLOAT NOT NULL, l_tax FLOAT NOT NULL, l_returnflag VARCHAR(1) NOT NULL, l_linestatus VARCHAR(1) NOT NULL, l_shipdate DATE NOT NULL, l_commitdate DATE NOT NULL, l_receiptdate DATE NOT NULL, l_shipinstruct VARCHAR(25) NOT NULL, l_shipmode VARCHAR(10) NOT NULL, l_comment VARCHAR(44) NOT NULL );
AFAIK, the table has been loaded using "COPY 600037902 RECORDS INTO ... FROM FILE ..." from a single 75 GB .csv file.
Stefan
Thanks.
On Mon, Mar 23, 2009 at 9:14 AM, Niels Nes Niels.Nes@cwi.nl wrote:
On Mon, Mar 23, 2009 at 09:03:09AM -0700, Yue Sheng wrote:
Tried the whole process. Still the same problem (table_combined has the same scheme as table_1):
insert into table_combined select * from table_1; insert into table_combined select * from table_2; select count(*) from table_combined; this gives ONLY the number of records in table_1, but table_combined as roughly same size as the other two combined. NONE OF THE TABLE HAS ANY INDEXING Could someone please confirm if this is a bug? Both table_1 and
The way you explain it, it seems a bug.
Niels
table_2 has over 200million rows of data.
On Sat, Mar 21, 2009 at 4:47 PM, Yue Sheng <[1]yuesheng8@gmail.com> wrote:
(1) There's no index or key or anything on the table.(2) The combined table is the size of table_1 + table_2. But only table_1 shows up in query On Sat, Mar 21, 2009 at 1:51 PM, Lefteris <[2]lsidir@gmail.com> wrote:
Hi Yue, is there any chance that you have set uniqueness in the table specs and thus not inserting duplicate tuples from table_1 and table_2? If not, then I would guess that this is a bug and it needs further investigation. lefterisOn Sat, Mar 21, 2009 at 3:12 AM, Yue Sheng <[3]yuesheng8@gmail.com> wrote:
two tables: table_1 and table_2 both have exactly the same
schema. table_1
has 2.15MM rows of data, table_2 has 3.35MM row of data. No
indexing
created a new table: table_combined using same schema as
table_1
insert into table_combined select * from table_1; (30 minutes later) insert into table_combined select * from table_2;
(40 minutes later) select count(*) * 1 from table_combined; this give ONLY 2.15MM rows with same rows as table_1 What happened to table_2?!? Yes, I've check dfarm size. It equals 2*(table_1 + table_2 ) So it's there but "just not seeing it"
> ------------------------------------------------------------ ------------------ > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. [4]http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > MonetDB-users mailing list > [5]MonetDB-users@lists.sourceforge.net > [6]https://lists.sourceforge.net/lists/listinfo/monetdb-user s > > ------------------------------------------------------------ ------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. [7]http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list [8]MonetDB-users@lists.sourceforge.net [9]https://lists.sourceforge.net/lists/listinfo/monetdb-user sReferences
- mailto:yuesheng8@gmail.com
- mailto:lsidir@gmail.com
- mailto:yuesheng8@gmail.com
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
- http://p.sf.net/sfu/www-adobe-com
- mailto:MonetDB-users@lists.sourceforge.net
- https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM)
are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly
and
easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM)
are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly
and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly
and
easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial.
http://p.sf.net/sfu/www-adobe-com___________________________________________...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based
development
software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
participants (5)
-
Lefteris -
Milena.Ivanova＠cwi.nl -
Niels Nes -
Stefan Manegold -
Yue Sheng