[MonetDB-users] Copy records into table failed

newer
[MonetDB-users] MonetDB/XQuery...

older
[MonetDB-users] Get MAL from SQL

Tibor Barna

10 Mar 2011 10 Mar '11

1:45 p.m.

Hello,

I am trying to import data from MySQL into MonetDB usig the bulk loader, the csv has ~11GB, 163Mil Rows. When running:

sql>copy 163715240 records into sys.clicks from '/data/csv_monet_dump/clicks.tbl' using delimiters '|',' more>', '"' null as '';

I always get this error message: SQLException:importTable:failed to import table ERROR: HEAPextend: failed to extend to 654860960 for 32/3231tail ERROR: TABLETcreate_bats: Failed to create bat of size 163715240

merovingian.log: 2011-03-10 14:12:31 MSG performance[6281]: #GDKmmap(2383544320) fails, try to free up space [memory in use=41411528,virtual memory in use=12310675456] 2011-03-10 14:12:31 MSG performance[6281]: #GDKmmap(2383544320) result [mem=32087208,vm=12310675456]

I am using 64bit openSUSE 11.3, 12GB RAM, 4GB Swap.

Does anybody know how can i overcome this error? Any help would be most welcome :)

Cherers, Tibor.

-- View this message in context: http://old.nabble.com/Copy-records-into-table-failed-tp31115783p31115783.htm... Sent from the monetdb-users mailing list archive at Nabble.com.

Show replies by date

dariuszs

10 Mar 10 Mar

2:38 p.m.

New subject: [MonetDB-users] Copy records into table failed

Hi, Split the data into several tables, load them sequentially.

Tibor Barna wrote:

...

Hello,

I am trying to import data from MySQL into MonetDB usig the bulk loader, the csv has ~11GB, 163Mil Rows. When running:

sql>copy 163715240 records into sys.clicks from '/data/csv_monet_dump/clicks.tbl' using delimiters '|',' more>', '"' null as '';

I always get this error message: SQLException:importTable:failed to import table ERROR: HEAPextend: failed to extend to 654860960 for 32/3231tail ERROR: TABLETcreate_bats: Failed to create bat of size 163715240

merovingian.log: 2011-03-10 14:12:31 MSG performance[6281]: #GDKmmap(2383544320) fails, try to free up space [memory in use=41411528,virtual memory in use=12310675456] 2011-03-10 14:12:31 MSG performance[6281]: #GDKmmap(2383544320) result [mem=32087208,vm=12310675456]

I am using 64bit openSUSE 11.3, 12GB RAM, 4GB Swap.

Does anybody know how can i overcome this error? Any help would be most welcome :)

Cherers, Tibor.

Tibor Barna

4 p.m.

New subject: [MonetDB-users] Copy records into table failed

Hi dariuszs,

This sounds like a good idea, but i am still interested what the cause of the problem is. I had run the same script again, not long ago, and it worked (for the 11GB CSV).

Than the next problem: sql>copy 163715212 records into sys.page_urls from '/data/csv_monet_dump/page_urls.tbl' using delimiters '|',' more>', '"' null as '';

MAPI = (monetdb) /home/wired/MonetDB/var/MonetDB5/dbfarm/mapi_socket ACTION= read_line QUERY = copy 163715212 records into sys.page_urls from '/data/csv_monet_dump/page_urls.tbl' using delimiters '|',' ', '"' null as ''; ERROR = !Connection terminated

merovingian.log: 2011-03-10 16:28:51 MSG merovingian[3266]: database 'performance' (17635) was killed by signal SIGSEGV

My database just crashed, I restarted it, run the command again, it worked. Strange :) The second CSV was only 2GB.

Cheers, Tibor.

dariuszs wrote:

...

Hi, Split the data into several tables, load them sequentially.

Tibor Barna wrote:

...
Hello,

I am trying to import data from MySQL into MonetDB usig the bulk loader, the csv has ~11GB, 163Mil Rows. When running:

sql>copy 163715240 records into sys.clicks from '/data/csv_monet_dump/clicks.tbl' using delimiters '|',' more>', '"' null as '';

I always get this error message: SQLException:importTable:failed to import table ERROR: HEAPextend: failed to extend to 654860960 for 32/3231tail ERROR: TABLETcreate_bats: Failed to create bat of size 163715240

merovingian.log: 2011-03-10 14:12:31 MSG performance[6281]: #GDKmmap(2383544320) fails, try to free up space [memory in use=41411528,virtual memory in use=12310675456] 2011-03-10 14:12:31 MSG performance[6281]: #GDKmmap(2383544320) result [mem=32087208,vm=12310675456]

I am using 64bit openSUSE 11.3, 12GB RAM, 4GB Swap.

Does anybody know how can i overcome this error? Any help would be most welcome :)

Cherers, Tibor.

Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- View this message in context: http://old.nabble.com/Copy-records-into-table-failed-tp31115783p31117004.htm... Sent from the monetdb-users mailing list archive at Nabble.com.

dariuszs

5:28 p.m.

New subject: [MonetDB-users] Copy records into table failed

Hi, When you run out of memory and swap space that's what happens.

Tibor Barna wrote:

...

Hi dariuszs,

This sounds like a good idea, but i am still interested what the cause of the problem is. I had run the same script again, not long ago, and it worked (for the 11GB CSV).

Than the next problem: sql>copy 163715212 records into sys.page_urls from '/data/csv_monet_dump/page_urls.tbl' using delimiters '|',' more>', '"' null as '';

MAPI = (monetdb) /home/wired/MonetDB/var/MonetDB5/dbfarm/mapi_socket ACTION= read_line QUERY = copy 163715212 records into sys.page_urls from '/data/csv_monet_dump/page_urls.tbl' using delimiters '|',' ', '"' null as ''; ERROR = !Connection terminated

merovingian.log: 2011-03-10 16:28:51 MSG merovingian[3266]: database 'performance' (17635) was killed by signal SIGSEGV

My database just crashed, I restarted it, run the command again, it worked. Strange :) The second CSV was only 2GB.

Cheers, Tibor.

dariuszs wrote:

...
Hi, Split the data into several tables, load them sequentially.

Tibor Barna wrote:

...
Hello,

I am trying to import data from MySQL into MonetDB usig the bulk loader, the csv has ~11GB, 163Mil Rows. When running:

sql>copy 163715240 records into sys.clicks from '/data/csv_monet_dump/clicks.tbl' using delimiters '|',' more>', '"' null as '';

I always get this error message: SQLException:importTable:failed to import table ERROR: HEAPextend: failed to extend to 654860960 for 32/3231tail ERROR: TABLETcreate_bats: Failed to create bat of size 163715240

merovingian.log: 2011-03-10 14:12:31 MSG performance[6281]: #GDKmmap(2383544320) fails, try to free up space [memory in use=41411528,virtual memory in use=12310675456] 2011-03-10 14:12:31 MSG performance[6281]: #GDKmmap(2383544320) result [mem=32087208,vm=12310675456]

I am using 64bit openSUSE 11.3, 12GB RAM, 4GB Swap.

Does anybody know how can i overcome this error? Any help would be most welcome :)

Cherers, Tibor.

Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Tibor Barna

11 Mar 11 Mar

12:28 p.m.

New subject: [MonetDB-users] Copy records into table failed

I am still evaluating MonetDB, but if this is what happens when the memory runs out, I am afraid that it is not stable enough for production use.

My database could reach more than 100GB, even if i add more RAM, at some point I will still run out of memory.

Please correct me if I am wrong. What would be the optimal setup for MonetDB for my case?

Cheers, Tibor.

dariuszs wrote:

...

Hi, When you run out of memory and swap space that's what happens.

Tibor Barna wrote:

...
Hi dariuszs,

This sounds like a good idea, but i am still interested what the cause of the problem is. I had run the same script again, not long ago, and it worked (for the 11GB CSV).

Than the next problem: sql>copy 163715212 records into sys.page_urls from '/data/csv_monet_dump/page_urls.tbl' using delimiters '|',' more>', '"' null as '';

MAPI = (monetdb) /home/wired/MonetDB/var/MonetDB5/dbfarm/mapi_socket ACTION= read_line QUERY = copy 163715212 records into sys.page_urls from '/data/csv_monet_dump/page_urls.tbl' using delimiters '|',' ', '"' null as ''; ERROR = !Connection terminated

merovingian.log: 2011-03-10 16:28:51 MSG merovingian[3266]: database 'performance' (17635) was killed by signal SIGSEGV

My database just crashed, I restarted it, run the command again, it worked. Strange :) The second CSV was only 2GB.

Cheers, Tibor.

dariuszs wrote:

...
Hi, Split the data into several tables, load them sequentially.

Tibor Barna wrote:

...
Hello,

I am trying to import data from MySQL into MonetDB usig the bulk loader, the csv has ~11GB, 163Mil Rows. When running:

sql>copy 163715240 records into sys.clicks from '/data/csv_monet_dump/clicks.tbl' using delimiters '|',' more>', '"' null as '';

I always get this error message: SQLException:importTable:failed to import table ERROR: HEAPextend: failed to extend to 654860960 for 32/3231tail ERROR: TABLETcreate_bats: Failed to create bat of size 163715240

merovingian.log: 2011-03-10 14:12:31 MSG performance[6281]: #GDKmmap(2383544320) fails, try to free up space [memory in use=41411528,virtual memory in use=12310675456] 2011-03-10 14:12:31 MSG performance[6281]: #GDKmmap(2383544320) result [mem=32087208,vm=12310675456]

I am using 64bit openSUSE 11.3, 12GB RAM, 4GB Swap.

Does anybody know how can i overcome this error? Any help would be most welcome :)

Cherers, Tibor.

Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- View this message in context: http://old.nabble.com/Copy-records-into-table-failed-tp31115783p31124588.htm... Sent from the monetdb-users mailing list archive at Nabble.com.

Stefan de Konink

12:31 p.m.

New subject: [MonetDB-users] Copy records into table failed

On Fri, 11 Mar 2011, Tibor Barna wrote:

...

I am still evaluating MonetDB, but if this is what happens when the memory runs out, I am afraid that it is not stable enough for production use.

My database could reach more than 100GB, even if i add more RAM, at some point I will still run out of memory.

Did you also implement swap space?

Stefan

Tibor Barna

1:17 p.m.

New subject: [MonetDB-users] Copy records into table failed

Hey Stefan,

I have 4GB Swap space, 12GB RAM, and a 100GB database. I could extend the swap for 1-2x the RAM, but still, if it ever runs out of memory, the database should still not crash.

Is there a rule how the partitions should look like, amount of RAM, Swap?

Tibor.

Stefan de Konink-3 wrote:

...

On Fri, 11 Mar 2011, Tibor Barna wrote:

...
I am still evaluating MonetDB, but if this is what happens when the memory runs out, I am afraid that it is not stable enough for production use.

My database could reach more than 100GB, even if i add more RAM, at some point I will still run out of memory.

Did you also implement swap space?

Stefan

Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- View this message in context: http://old.nabble.com/Copy-records-into-table-failed-tp31115783p31124969.htm... Sent from the monetdb-users mailing list archive at Nabble.com.

Stefan de Konink

1:32 p.m.

New subject: [MonetDB-users] Copy records into table failed

On Fri, 11 Mar 2011, Tibor Barna wrote:

...

I have 4GB Swap space, 12GB RAM, and a 100GB database. I could extend the swap for 1-2x the RAM, but still, if it ever runs out of memory, the database should still not crash.

I guess it should stop the process... but it cannot ever finish it...

...

Is there a rule how the partitions should look like, amount of RAM, Swap?

Rule of thumb is 2x RAM = SWAP.

Stefan

4833

Age (days ago)

4834

Last active (days ago)

List overview

Download

7 comments

3 participants

tags (0)

participants (3)

dariuszs
Stefan de Konink
Tibor Barna