Luis Fernando.
De: Stefan Manegold <Stefan.Manegold@cwi.nl>
Para: Communication channel for MonetDB users <monetdb-users@lists.sourceforge.net>
Enviadas: Sexta-feira, 25 de Junho de 2010 16:42:19
Assunto: Re: [MonetDB-users] MonetDB aborts when importing over 10 million rows
Hi Luis,
Which version of MonetDB are you using?
Is your XP system 32-bit or 64-bit?
If the latter, is your MonetDB 32-bit or
64-bit?
How large is your dataset (.csv file) in total and what is the exact schema
of your table?
How much main memory (RAM) does your system have?
How much free disk space does you system have on the partition (drive) where
your MonetDB database (dbfarm) is located?
Regarding memory usage of MonetDB, you might also find the following thread
interesting and helpful:
http://sourceforge.net/mailarchive/message.php?msg_name=20100319145945.GA9161%40cwi.nlStefan
On Fri, Jun 25, 2010 at 12:18:11PM -0700, Luis Fernando Kauer wrote:
> Hi,
>
> I'm still trying to evaluate MonetDB but I keep getting errors when trying to import over 10 million rows to a table.
> The server simply aborts and closes.
> I'm using MonetDB on Windows XP.
>
Whenener I restart after the server had aborted it shows messages like this:
>
> #GDKmmap(71827456) fails, try to free up space [memory in use=18552560,virtual m
> emory in use=1837957120]
> #GDKmmap(71827456) result [mem=18552560,vm=1837957120]
> #GDKmmap(36175872) fails, try to free up space [memory in use=16499872,virtual m
> emory in use=1844903936]
> #GDKmmap(36175872) result [mem=16499872,vm=1844903936]
> #GDKmmap(143130624) fails, try to free up space [memory in use=15697728,virtual
> memory in use=1744764928]
> #GDKmmap(143130624) result [mem=15697728,vm=1744764928]
> #GDKmmap(143130624) fails, try to free up space [memory in use=13554656,virtual
> memory in use=1654456320]
> #GDKmmap(143130624) result [mem=13554656,vm=1654456320]
> #GDKmmap(143130624) fails, try to free up space [memory in use=8747960,virtual m
> emory in use=1650917376]
>
#GDKmmap(143130624) result [mem=8747960,vm=1650917376]
> #GDKmmap(143130624) fails, try to free up space [memory in use=7946840,virtual m
> emory in use=1531052032]
> #GDKmmap(143130624) result [mem=7946840,vm=1531052032]
> !mvc_init: unable to create system tables
> !SQLException:SQLinit:Catalogue initialization failed
> !ERROR: HEAPextend: failed to extend to 71722596 for 14\1475tail
> !ERROR: HEAPextend: failed to extend to 35861298 for 15\1537tail
> !ERROR: HEAPextend: failed to extend to 142816048 for 15\1543tail
> !ERROR: HEAPextend: failed to extend to 142816048 for 15\1550tail
> !ERROR: HEAPextend: failed to extend to 142816048 for 15\1556tail
> !ERROR: HEAPextend: failed to extend to 142816048 for 15\1557tail
> #function user.main():void;
> # sql.prelude();
> #end main;
>
> First I was trying to import files with about 3 million rows
each but it started giving errors after the 3rd file.
> Then I tried to import through files with about 1 million rows each and I was able to import about 14 million rows, but after about 10 million rows I was only able to import one or two files in a row without MonetDB resetting.
> Is this a bug or a size limit for the available memory.
> Anything I could try?
>
> With regards,
>
> Luis Fernando
>
>
>
> ------------------------------------------------------------------------------
> ThinkGeek and WIRED's GeekDad team up for the Ultimate
> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> lucky parental unit. See the prize list and enter to win:
>
http://p.sf.net/sfu/thinkgeek-promo> _______________________________________________
>
MonetDB-users mailing list
>
MonetDB-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/monetdb-users--
| Dr. Stefan Manegold | mailto:
Stefan.Manegold@cwi.nl |
| CWI, P.O.Box 94079 |
http://www.cwi.nl/~manegold/ |
| 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 |
| The Netherlands | Fax : +31 (20) 592-4199 |
------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate
GeekDad Father's Day Giveaway.
ONE MASSIVE PRIZE to the
lucky parental unit. See the prize list and enter to win:
http://p.sf.net/sfu/thinkgeek-promo_______________________________________________
MonetDB-users mailing list
MonetDB-users@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/monetdb-users