On Thu, Nov 27, 2008 at 11:03:29AM +0000, Jan Flokstra wrote:
> Update of /cvsroot/monetdb/pathfinder/modules/pftijah/tjc
> In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv3513
>
> Modified Files:
> tjc_scan.l
> Log Message:
> - defining an empty ECHO should solve the fwrite() problem I think. I copied
> this solution which first did not make any sense from nexi.l.
Works. Thanks.
Alternative (in case ECHO was used) would have been to disable the strict
checks for (flex-/bison-) generated code, e.g., via
========
Index: buildtools/conf/MonetDB.m4
===================================================================
RCS file: /cvsroot/monetdb/buildtools/conf/MonetDB.m4,v
retrieving revision 1.99
diff -u -r1.99 MonetDB.m4
--- buildtools/conf/MonetDB.m4 27 Nov 2008 08:04:34 -0000 1.99
+++ buildtools/conf/MonetDB.m4 27 Nov 2008 11:46:09 -0000
@@ -758,7 +758,7 @@
dnl locally as possible via "-Wno-undef"
dnl (see also MonetDB4/src/monet/Makefile.ag, sql/src/server/Makefile.ag,
dnl pathfinder/modules/pftijah/Makefile.ag).
- GCC_BISON_CFLAGS="$GCC_BISON_CFLAGS -Wno-undef"
+ GCC_BISON_CFLAGS="$GCC_BISON_CFLAGS -Wno-undef -Wp,-U_FORTIFY_SOURCE"
dnl Likewise, at least with gcc 4.3.0 and flex 2.5.33 and bison 2.3 on Gentoo 1.12.11.1,
dnl gcc complains about some bison/flex-generated code that defines an unused static function:
dnl "warning: 'input' defined but not used".
========
Stefan
>
>
> U tjc_scan.l
> Index: tjc_scan.l
> ===================================================================
> RCS file: /cvsroot/monetdb/pathfinder/modules/pftijah/tjc/tjc_scan.l,v
> retrieving revision 1.2
> retrieving revision 1.3
> diff -u -d -r1.2 -r1.3
> --- tjc_scan.l 27 Nov 2008 10:53:17 -0000 1.2
> +++ tjc_scan.l 27 Nov 2008 11:03:26 -0000 1.3
> @@ -49,6 +49,8 @@
> YY_FATAL_ERROR( "input in tijah flex scanner failed" );
> #endif
>
> +#define ECHO /* empty -- don't echo anything (we shouldn't get there anyway */
> +
> static void
> gotoState (int s)
> {
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Monetdb-pf-checkins mailing list
> Monetdb-pf-checkins(a)lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins
--
| Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl |
| CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ |
| 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 |
| The Netherlands | Fax : +31 (20) 592-4312 |
On Thu, Nov 27, 2008 at 10:43:37AM +0000, Martin Kersten wrote:
> Update of /cvsroot/monetdb/MonetDB5/src/mal
> In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv739
>
> Modified Files:
> mal_instruction.mx
> Log Message:
> If you enter a multiplatform experiment then remove the exception handling too.
>
>
> U mal_instruction.mx
> Index: mal_instruction.mx
> ===================================================================
> RCS file: /cvsroot/monetdb/MonetDB5/src/mal/mal_instruction.mx,v
> retrieving revision 1.374
> retrieving revision 1.375
> diff -u -d -r1.374 -r1.375
> --- mal_instruction.mx 26 Nov 2008 20:57:50 -0000 1.374
> +++ mal_instruction.mx 27 Nov 2008 10:43:34 -0000 1.375
> @@ -1736,7 +1736,7 @@
> #ifdef STRUCT_ALIGNED
> /* experiment against alternative 32 bit platforms */
> #else
> - showException(SYNTAX, "convertConstant", "missing implementation");
> + /* showException(SYNTAX, "convertConstant", "missing implementation");*/
> #endif
... hm, maybe the whole "effectless"
#ifdef STRUCT_ALIGNED
/* ... */
#else
/* ... */
#endif
[cs]hould be removed, now to keep the sources clean of dead code?
Stefan
> if ( vr->vtype != type)
> showException(SYNTAX, "convertConstant", "coercion failed in '%s'", vr->val.sval);
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Monetdb-checkins mailing list
> Monetdb-checkins(a)lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
>
>
--
| Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl |
| CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ |
| 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 |
| The Netherlands | Fax : +31 (20) 592-4312 |
On Wed, Nov 26, 2008 at 08:21:17PM +0000, Martin Kersten wrote:
> Update of /cvsroot/monetdb/MonetDB5/src/mal
> In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv13401
>
> Modified Files:
> mal_instruction.mx
> Log Message:
> An experiment for the nightly tests to address e.g. Darwin color test.
> Should be handled correctly by GDK.
>
>
> U mal_instruction.mx
> Index: mal_instruction.mx
> ===================================================================
> RCS file: /cvsroot/monetdb/MonetDB5/src/mal/mal_instruction.mx,v
> retrieving revision 1.372
> retrieving revision 1.373
> diff -u -d -r1.372 -r1.373
> --- mal_instruction.mx 23 Nov 2008 15:48:14 -0000 1.372
> +++ mal_instruction.mx 26 Nov 2008 20:21:14 -0000 1.373
> @@ -1,4 +1,4 @@
> -@' The contents of this file are subject to the MonetDB Public License
> +' The contents of this file are subject to the MonetDB Public License
> @' Version 1.1 (the "License"); you may not use this file except in
> @' compliance with the License. You may obtain a copy of the License at
> @' http://monetdb.cwi.nl/Legal/MonetDBLicense-1.1.html
> @@ -1715,7 +1715,6 @@
> if( vr->vtype == type)
> return 0;
> if( vr->vtype == TYPE_str){
> -#ifdef STRUCT_ALIGNED
> int ll = 0;
> ptr d = NULL;
>
> @@ -1734,6 +1733,8 @@
> VALset(vr, type, d);
> if (ATOMextern(type) == 0)
> GDKfree(d);
> +#ifdef STRUCT_ALIGNED
> + /* experiment against alternative 32 bit platforms */
The problems does not seem to be 32-bit specific, but possibly big-endian
related; it occurs on both 32- & 64-bit big-endian platforms such as
PowerPC
http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/…
and
SPARC
http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/…http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/…http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/…
similar problems occur with other tests on the same set of platforms:
http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/…http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/…http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/…
Stefan
> #else
> showException(SYNTAX, "convertConstant", "missing implementation");
> #endif
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Monetdb-checkins mailing list
> Monetdb-checkins(a)lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
--
| Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl |
| CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ |
| 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 |
| The Netherlands | Fax : +31 (20) 592-4312 |
On Wed, Nov 26, 2008 at 08:21:17PM +0000, Martin Kersten wrote:
> Update of /cvsroot/monetdb/MonetDB5/src/mal
> In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv13401
>
> Modified Files:
> mal_instruction.mx
> Log Message:
> An experiment for the nightly tests to address e.g. Darwin color test.
> Should be handled correctly by GDK.
>
>
> U mal_instruction.mx
> Index: mal_instruction.mx
> ===================================================================
> RCS file: /cvsroot/monetdb/MonetDB5/src/mal/mal_instruction.mx,v
> retrieving revision 1.372
> retrieving revision 1.373
> diff -u -d -r1.372 -r1.373
> --- mal_instruction.mx 23 Nov 2008 15:48:14 -0000 1.372
> +++ mal_instruction.mx 26 Nov 2008 20:21:14 -0000 1.373
> @@ -1,4 +1,4 @@
> -@' The contents of this file are subject to the MonetDB Public License
> +' The contents of this file are subject to the MonetDB Public License
^^
Was this (also) intended?
Stefan
> @' Version 1.1 (the "License"); you may not use this file except in
> @' compliance with the License. You may obtain a copy of the License at
> @' http://monetdb.cwi.nl/Legal/MonetDBLicense-1.1.html
> @@ -1715,7 +1715,6 @@
> if( vr->vtype == type)
> return 0;
> if( vr->vtype == TYPE_str){
> -#ifdef STRUCT_ALIGNED
> int ll = 0;
> ptr d = NULL;
>
> @@ -1734,6 +1733,8 @@
> VALset(vr, type, d);
> if (ATOMextern(type) == 0)
> GDKfree(d);
> +#ifdef STRUCT_ALIGNED
> + /* experiment against alternative 32 bit platforms */
> #else
> showException(SYNTAX, "convertConstant", "missing implementation");
> #endif
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Monetdb-checkins mailing list
> Monetdb-checkins(a)lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-checkins
>
>
>
--
| Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl |
| CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ |
| 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 |
| The Netherlands | Fax : +31 (20) 592-4312 |
#GDKmalloc(14777222432) fail => BBPtrim(enter)
usage[mem=14840896,vm=27527806976]
#BBPTRIM_ENTER: memsize=14840896,vmsize=27527806976
#BBPTRIM: memtarget=4611686018427387904 vmtarget=0
#BBPTRIM_EXIT: memsize=19120128,vmsize=27527806976
#GDKmalloc(14777222432) fail => BBPtrim(ready)
usage[mem=11538120,vm=27527806976]
#BATpropcheck: BAT tmp_1260(-688): could not allocate hash table for key
test
#GDKmalloc(14777222432) fail => BBPtrim(enter)
usage[mem=11537664,vm=27527806976]
#BBPTRIM_ENTER: memsize=11537664,vmsize=27527806976
#BBPTRIM: memtarget=4611686018427387904 vmtarget=0
#BBPTRIM_EXIT: memsize=19120128,vmsize=27527806976
#GDKmalloc(14777222432) fail => BBPtrim(ready)
usage[mem=10416208,vm=27527806976]
#BATpropcheck: BAT tmp_1135(-605): could not allocate hash table for key
test
MAPI = monetdb@localhost:50000
QUERY = COPY 773410980 RECORDS INTO node_tags from
'/mnt/data2/csv/node_tags.csv' USING DELIMITERS ',', '\n', '''';
ERROR = !MALException: GDKmallocmax: failed for 1892320544 bytes
!ERROR: GDKload: failed name=12/1252, ext=thash
!ERROR: GDKmallocmax: failed for 1892320544 bytes
!ERROR: GDKload: failed name=12/1253, ext=thash
What kind of voodoo is this?
I'm trying to import a 42GB csv file using copy into, on a native
system, 2GB of memory, 4GB of swap. Latest CVS version, this time
compiled for CoreDuo, without debuging, with optimise.
Stefan
Hi Stefan (the non-german .de one)
> Maybe a very strange idea, could be your better solution. If by default
> these operations are mmap'ed, they should be taken care of by a disk based
> memory extension. Hence being independent of RAM/SWAP but having all pro's
> of both.
You are right. In fact, this is what MonetDB is supposed to do. Memory is
organized in contiguous Heap objects. This does not only hold for BATs but
also for Hash Tables. At some moment, I did add the possibility to use a
real memory mapped files for their allocation, as a last resort. The problem
was there that the Heap() API did not have a filename parameter, and a
HEAPextend() thus would not know what file to use. In order not to break the
API, I decided to initialize the filename fields at BAT creation time (this
is why you may sometimes see .hhash and .thash files in a just stopped
MonetDB repository -- even though hash tables are not persistent).
With hindsight, this API decision was bad judgement; because the question
whether real file mapping is used now depends on the question whether the
filename was initialized at BAT or Hash table creation. Now, this should be
the case, but it would be my first question to address in a debugging
session.
I also recall that in case of the Skyserver TB dataset, key checking was
disabled to to similar problems.
Of course, if we would find and fix this bug, life will not be rosy for you,
doing random access (key uniqueness checking) into a 14GB heap that will
never fit your 2GB RAM. That goes back to your question whether it is in
fact reasonable to materialize 14GB. Questions like these led to a redesign
that is much more careful with memory consumption (X100) whose spin-off
development regrettably means that my time for such debugging sessions
currently is limited.
Peter
PS
The likely cause for the problem is a shortage of swap space. Increasing
your swap file size (with a few 10s of GB) should be a workaround for the
problem.