The problem is HT (hyper-threading), eg. - writing out table output to disk. When I have Logical Processor enabled (BIOS), MonetDB thinks I have 16 cores, but my machine has only *8* (with 2 threads per core). So when I try to write the file out, the second thread will try to release the memory which has been already released by the first thread and will segfault. When I turn off Logical Processor - writing out files works, but MonetDB will use 50% of available memory at most. Dariusz.
Logical Processor Disabled root@monetdb:/var/MonetDB# mserver5 --version MonetDB server v5.18.5 (64-bit), based on kernel v1.36.5 (64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2010 MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/ for further information Found 47.3GiB available memory, 8 available cpu cores Configured for prefix: /usr Libraries: libpcre: 7.8 2008-09-05 (compiled with 7.8) openssl: OpenSSL 0.9.8g 19 Oct 2007 (compiled with OpenSSL 0.9.8g 19 Oct 2007) libxml2: 2.7.5 (compiled with 2.7.5) Compiled by: root@ubuntu-karmic-64 (x86_64-pc-linux-gnu) Compilation: gcc -O2 -Wall -O2 -O6 -fomit-frame-pointer -finline-functions -falign-loops=4 -falign-jumps=4 -falign-functions=4 -fexpensive-optimizations -funroll-loops -frerun-cse-after-loop -frerun-loop-opt -ftree-vectorize Linking : ld -IPA -m elf_x86_64 -Wl,-Bsymbolic-functions
Logical Processor Enabled root@monetdb:/var/MonetDB# mserver5 --version MonetDB server v5.18.5 (64-bit), based on kernel v1.36.5 (64-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2010 MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/ for further information Found 47.3GiB available memory, 16 available cpu cores Configured for prefix: /usr Libraries: libpcre: 7.8 2008-09-05 (compiled with 7.8) openssl: OpenSSL 0.9.8g 19 Oct 2007 (compiled with OpenSSL 0.9.8g 19 Oct 2007) libxml2: 2.7.5 (compiled with 2.7.5) Compiled by: root@ubuntu-karmic-64 (x86_64-pc-linux-gnu) Compilation: gcc -O2 -Wall -O2 -O6 -fomit-frame-pointer -finline-functions -falign-loops=4 -falign-jumps=4 -falign-functions=4 -fexpensive-optimizations -funroll-loops -frerun-cse-after-loop -frerun-loop-opt -ftree-vectorize Linking : ld -IPA -m elf_x86_64 -Wl,-Bsymbolic-functions
Dariusz.
Stefan Manegold wrote:
On Fri, Apr 30, 2010 at 01:33:08PM -0400, dariuszs wrote:
Hi, On processors that support SMT technology if Logical Processors are enabled your software segfaults,
Could you be more specific and verbose?
Does "it" (I suppose you mean MonetDB, in particular MonetDB5/SQL) segfault always (i.e., as soon as you start mserver5 or merovingian), or only under particular circumstances/workloads/situations?
I.e., how (if at all) could we reproduce the segfault(s)? (We cannot do much about problems without detailed description how to reproduce them ...)
In case we cannot reproduce the segfault, can you run MonetDB in a debugger and report where/how/why it segfaults?
if you disable Logical Processors it works but then you use half of the available memory. Dariusz.
How exactly do you come to the conclusion that MonetDB only uses half of the (physically) available memory in case Logical Processors ("Hyperthreading") are disabled? How are Logical Processors related to physical memory usage?
Stefan
Martin Kersten wrote:
Nafees Ur Rehman wrote:
Hi,
Does monetdb has support for parallel processing? If some can guide me on that.
What kind of parallel processing do you have in mind? It runs the queries in parallel on a multi-core engine. regards,Martin
Regards Nafees Ur Rehman