[MonetDB-users] How to set the memory limit for monetDB.
I am using the latest version of monetDB from the web-site(the super source tarball). My machine is a X86-64 machine with a red-hat installed on it. When I insert data into monetDB, it always requires memory from the system. Then it will be crashed with running out of memory. Is there a way to set the limit for memory used by monetDB ? I've read the docs on the web site and can't get what I need from that.
Thank you.
Kang Zhang wrote:
I am using the latest version of monetDB from the web-site(the super source tarball). My machine is a X86-64 machine with a red-hat installed on it. When I insert data into monetDB, it always requires memory from the system. Then it will be crashed with running out of memory. Is there a way to set the limit for memory used by monetDB ? I've read the docs on the web site and can't get what I need from that.
Dear Kang, indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns.
Thank you.
--
Kang Zhang
Computer Science Dept. Shanghai Jiao Tong University
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
The data's size is about 3 Gigabytes. 21,852,641 triples need to be inserted in total. My machine is 64-bit and only 8G memory for that.
Thank you for your immediate reply. Kang.
On Wed, Apr 30, 2008 at 1:21 PM, Martin Kersten Martin.Kersten@cwi.nl wrote:
Kang Zhang wrote:
I am using the latest version of monetDB from the web-site(the super source tarball). My machine is a X86-64 machine with a red-hat installed on it. When I insert data into monetDB, it always requires memory from the system. Then it will be crashed with running out of memory. Is there a way to set the limit for memory used by monetDB ? I've read the docs on the web site and can't get what I need from that.
Dear Kang, indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns.
Thank you.
--
Kang Zhang
Computer Science Dept. Shanghai Jiao Tong University
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Kang Zhang wrote:
The data's size is about 3 Gigabytes. 21,852,641 triples need to be inserted in total. My machine is 64-bit and only 8G memory for that.
Can you say a little more when and how it crashed. Did you use the SQL COPY command? Is it a victim process selected by the OS, due to excessive virtual memory use? Do you have an gdb stack trace where it happen?
Thank you for your immediate reply. Kang.
On Wed, Apr 30, 2008 at 1:21 PM, Martin Kersten <Martin.Kersten@cwi.nl mailto:Martin.Kersten@cwi.nl> wrote:
Kang Zhang wrote: > I am using the latest version of monetDB from the web-site(the super > source tarball). > My machine is a X86-64 machine with a red-hat installed on it. When I > insert data into monetDB, > it always requires memory from the system. Then it will be crashed > with running out of memory. > Is there a way to set the limit for memory used by monetDB ? I've read > the docs on the web site > and can't get what I need from that. Dear Kang, indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns. > > Thank you. > > -- > ------------------------------------------------------------ > Kang Zhang > > Computer Science Dept. > Shanghai Jiao Tong University > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------- > This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save $100. > Use priority code J8TL2D2. > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone > ------------------------------------------------------------------------ > > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net <mailto:MonetDB-users@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/monetdb-users > ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net <mailto:MonetDB-users@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/monetdb-users--
Kang Zhang
Computer Science Dept. Shanghai Jiao Tong University
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
In fact, I've wrapped monetDB using JENA's api. And tried to load DBPedia's RDF data with it. It runs OK on small dataset. But it always requires memory during the insertion. So it failed when there's no memory to be allocated. Since it has been wrapped by JENA, I know few of the background schema of the database. I go with you it may caused by the os's vm strategy. I'll check it. I've seen that there are some debug option for the config file. Can I use some of them to log something that can let you know my problem? Which one should I use ?
On Wed, Apr 30, 2008 at 2:19 PM, Martin Kersten Martin.Kersten@cwi.nl wrote:
Kang Zhang wrote:
The data's size is about 3 Gigabytes. 21,852,641 triples need to be inserted in total. My machine is 64-bit and only 8G memory for that.
Can you say a little more when and how it crashed. Did you use the SQL COPY command? Is it a victim process selected by the OS, due to excessive virtual memory use? Do you have an gdb stack trace where it happen?
Thank you for your immediate reply. Kang.
On Wed, Apr 30, 2008 at 1:21 PM, Martin Kersten <Martin.Kersten@cwi.nl mailto:Martin.Kersten@cwi.nl> wrote:
Kang Zhang wrote: > I am using the latest version of monetDB from the web-site(thesuper
> source tarball). > My machine is a X86-64 machine with a red-hat installed on it. When I > insert data into monetDB, > it always requires memory from the system. Then it will be crashed > with running out of memory. > Is there a way to set the limit for memory used by monetDB ? I've read > the docs on the web site > and can't get what I need from that. Dear Kang, indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns. > > Thank you. > > -- > ------------------------------------------------------------ > Kang Zhang > > Computer Science Dept. > Shanghai Jiao Tong University >
> >
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save $100. > Use priority code J8TL2D2. >http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...
>
> > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net <mailto:MonetDB-users@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/monetdb-users >
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net <mailto:MonetDB-users@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/monetdb-users--
Kang Zhang
Computer Science Dept. Shanghai Jiao Tong University
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
I've found that there are no swap file for this machine. :( I guess this is the reason for that.
On Wed, Apr 30, 2008 at 4:31 PM, Kang Zhang jobo.zh@gmail.com wrote:
In fact, I've wrapped monetDB using JENA's api. And tried to load DBPedia's RDF data with it. It runs OK on small dataset. But it always requires memory during the insertion. So it failed when there's no memory to be allocated. Since it has been wrapped by JENA, I know few of the background schema of the database. I go with you it may caused by the os's vm strategy. I'll check it. I've seen that there are some debug option for the config file. Can I use some of them to log something that can let you know my problem? Which one should I use ?
On Wed, Apr 30, 2008 at 2:19 PM, Martin Kersten Martin.Kersten@cwi.nl wrote:
Kang Zhang wrote:
The data's size is about 3 Gigabytes. 21,852,641 triples need to be inserted in total. My machine is 64-bit and only 8G memory for that.
Can you say a little more when and how it crashed. Did you use the SQL COPY command? Is it a victim process selected by the OS, due to excessive virtual memory use? Do you have an gdb stack trace where it happen?
Thank you for your immediate reply. Kang.
On Wed, Apr 30, 2008 at 1:21 PM, Martin Kersten <Martin.Kersten@cwi.nl mailto:Martin.Kersten@cwi.nl> wrote:
Kang Zhang wrote: > I am using the latest version of monetDB from the web-site(thesuper
> source tarball). > My machine is a X86-64 machine with a red-hat installed on it. When I > insert data into monetDB, > it always requires memory from the system. Then it will becrashed
> with running out of memory. > Is there a way to set the limit for memory used by monetDB ? I've read > the docs on the web site > and can't get what I need from that. Dear Kang, indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns. > > Thank you. > > -- > ------------------------------------------------------------ > Kang Zhang > > Computer Science Dept. > Shanghai Jiao Tong University >
> >
> This SF.net email is sponsored by the 2008 JavaOne(SM)Conference
> Don't miss this year's exciting event. There's still time to save $100. > Use priority code J8TL2D2. >http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...
>
> > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net <mailto:MonetDB-users@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/monetdb-users >
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net <mailto:MonetDB-users@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/monetdb-users--
Kang Zhang
Computer Science Dept. Shanghai Jiao Tong University
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save
$100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao...
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
--
Kang Zhang
Computer Science Dept. Shanghai Jiao Tong University
Is there any strategy that would allow MonetDB to operate in a situation where the size of the columns to be read significantly exceeds the available memory on the machine?
On Wednesday 30 April 2008 01:21:56 Martin Kersten wrote:
indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns.
McKennirey.Matthew wrote:
Is there any strategy that would allow MonetDB to operate in a situation where the size of the columns to be read significantly exceeds the available memory on the machine?
It is part of ongoing research works on partitioned and distributed versions. The primitives to build a distributed version are in the kernel already, e.g. remote execution of MAL blocks. However, this is all experimental.
The current state of affair is that also the SQL compiler can handle partitioned tables, i.e. smaller BATs, which means that it results in better handling of memory on a single processor, but also enables a parallel and distributed execution. We need, however, more confidence on the stability of the often novel approaches taken here.
On Linux systems, the system includes a thread that keeps an eye on the memory consumption and throws pages out when necessary. Often, the performance you see in out-of-memory situations is what you can expect from any system in that case.
See the TPC-H performance figures in http://monetdb.cwi.nl/projects/monetdb//SQL/Benchmark/TPCH/
An update of this table is already available and will be brought online shortly. The message remains the same.
On Wednesday 30 April 2008 01:21:56 Martin Kersten wrote:
indeed, there is no way to limit the memory use. All columns are memory mapped files, which give an upperbound on their size. This is particularly problematic for 32-bits machines and very large columns.
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javao... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
participants (3)
-
Kang Zhang -
Martin Kersten -
McKennirey.Matthew