[Monetdb-developers] [Monetdb-pf-checkins] pathfinder/modules/pftijah nexi.c, 1.57, 1.58 nexi_generate_mil.c, 1.27, 1.28 pftijah.mx, 1.116, 1.117

Stefan Manegold Stefan.Manegold at cwi.nl
Mon May 7 11:17:03 CEST 2007


Jan,

On Mon, May 07, 2007 at 09:19:17AM +0200, Jan Flokstra wrote:
> On Monday 07 May 2007 09:09, Stefan Manegold wrote:
> > On Mon, May 07, 2007 at 07:01:43AM +0000, Jan Flokstra wrote:
> > > Update of /cvsroot/monetdb/pathfinder/modules/pftijah
> > > In directory sc8-pr-cvs16:/tmp/cvs-serv20990/modules/pftijah
> > >
> > > Modified Files:
> > > 	nexi.c nexi_generate_mil.c pftijah.mx
> > > Log Message:
> > > * Implement private term- and tag dictionaries. Default is still the
> > > global version so nothing changed yet in the format. I'm having terrible
> > > trouble with the Mtest's again. This version could not be tested by our
> > > Mtest tests because the first query executes OK but all queries after
> > > that seem to 'forget' the instance variables of module pftijah. I will
> > > look if the same happens on other machine and post a bugreport for it.
> >
> > does the same happen if you run the test(s) by hand i.e., without Mtest?
> 
> No, without Mtest the queries run OK. eg, any FIRST query runs OK. Any query 
> after this fails.

do they run OK or NOT ???

If also without Mtest only the first runs OK and any other fails (that what
I read here), it is NOT an Mtest problem, but a general problem, right???

> > How do the queries and the test script(s) look like?
> 
> The normal 28 tests from $PF/modules/pftijah/Tests So a mix of mil scripts 
> and .xq XQueries.
> 
> > How do you (successfully?) run them by hand?
> 
> Just run the scripts in a test directory using MapiClient (-lx)
> > How do you (unsuccesfully) run them via Mtest?
> 
> Mtest.py -d0 --TSTSRCBASE /local/flokstra/software/pathfinder
> 
> in directory /local/flokstra/software/pathfinder/modules/pftijah

Before this mornings changes, everything did work fine, right?

Hence, your latest changes seem to *trigger* (not necessarily cause) the
problem. Looking in detail in your changes might give a hint, which part in
fact triggers the problem, which in turn would help a lot to analyse, locate
and eventually fix the problem.

Speaking of debugging here's a little excercise thet shows that the second
call to tj_init_collection() on the same database does fail, i.e., only the
first call to tj_init_collection() on a virgin database does not fail:

========
.../pathfinder/modules/pftijah/Tests $ export TSTSRCDIR=$PWD
========
.../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); tj_init_collection("testcoll0", new(str,str).insert("stemmer","snowball-english"));' | Mserver --dbname=DB-1
--------
# Monet Database Server V4.17.1
# Copyright (c) 1993-2007, CWI. All rights reserved.
# Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked.
# Visit http://monetdb.cwi.nl/ for further information.
MonetDB># XRPC administrative console at http://localhost:50001/admin
========
.../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); tj_init_collection("testcoll0", new(str,str).insert("stemmer","snowball-english"));' | Mserver --dbname=DB-1
--------
# Monet Database Server V4.17.1
# Copyright (c) 1993-2007, CWI. All rights reserved.
# Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked.
# Visit http://monetdb.cwi.nl/ for further information.
MonetDB># XRPC administrative console at http://localhost:50001/admin
!ERROR: interpret: unknown variable 'verbose'.
========

so, let's "trace" the variables:
========
.../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); vars();' | Mserver --dbname=DB-2 > /tmp/vars.0
========
.../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); vars();' | Mserver --dbname=DB-2 > /tmp/vars.1
========
.../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); tj_init_collection("testcoll0", new(str,str).insert("stemmer","snowball-english"));' | Mserver --dbname=DB-2
--------
# Monet Database Server V4.17.1
# Copyright (c) 1993-2007, CWI. All rights reserved.
# Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked.
# Visit http://monetdb.cwi.nl/ for further information.
MonetDB># XRPC administrative console at http://localhost:50001/admin
========
.../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); vars();' | Mserver --dbname=DB-2 > /tmp/vars.2
========
.../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); tj_init_collection("testcoll0", new(str,str).insert("stemmer","snowball-english"));' | Mserver --dbname=DB-2
--------
# Monet Database Server V4.17.1
# Copyright (c) 1993-2007, CWI. All rights reserved.
# Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked.
# Visit http://monetdb.cwi.nl/ for further information.
MonetDB># XRPC administrative console at http://localhost:50001/admin
!ERROR: interpret: unknown variable 'verbose'.
========
.../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); vars();' | Mserver --dbname=DB-2 > /tmp/vars.3
========
.../pathfinder/modules/pftijah/Tests $ grep --color verbose /tmp/vars.?
--------
/tmp/vars.0:[ "verbose",                          "bit",          "liquid",       "changeable",   "false"                                 ]
/tmp/vars.1:[ "verbose",                          "bit",          "liquid",       "changeable",   "false"                                 ]
========

I have no idea, what tj_init_collection() does but it somehow triggers the
disappeasing of some variables --- see also:
========
.../pathfinder/modules/pftijah/Tests $ diff -dU0 -Bbw /tmp/vars.[12] | cdiff 
--------
--- /tmp/vars.1 2007-05-07 10:40:40.000000000 +0200
+++ /tmp/vars.2 2007-05-07 10:41:03.000000000 +0200
@@ -6 +6 @@
-#-------------------------------------------------------------------------------------------------------------------------#
+#---------------------------------------------------------------------------------------------------------#
@@ -9,2 +9,2 @@
-#-------------------------------------------------------------------------------------------------------------------------#
-[ "pf_logger",                   "logger",       "liquid",       "changeable",   "19792504"                              ]
+#---------------------------------------------------------------------------------------------------------#
+[ "pf_logger",                   "logger",       "liquid",       "changeable",   "19781832"              ]
@@ -41 +41 @@
-[ "pf_chkpt_barrier",            "sema",         "liquid",       "changeable",   "14646584"                              ]
+[ "pf_chkpt_barrier",            "sema",         "liquid",       "changeable",   "15961944"              ]
@@ -43 +43 @@
-[ "pf_chkpt",                    "lock",         "liquid",       "changeable",   "19243832"                              ]
+[ "pf_chkpt",                    "lock",         "liquid",       "changeable",   "19100760"              ]
@@ -47 +47 @@
-[ "pf_collbarrier_lock",         "lock",         "liquid",       "changeable",   "19242456"                              ]
+[ "pf_collbarrier_lock",         "lock",         "liquid",       "changeable",   "19099384"              ]
@@ -63 +63 @@
-[ "ws_log_lock",                 "lock",         "liquid",       "changeable",   "19094712"                              ]
+[ "ws_log_lock",                 "lock",         "liquid",       "changeable",   "19090936"              ]
@@ -219,12 +218,0 @@
-[ "virtRoot",                    "oid",          "frozen",       "constant",     "0 at 0"                                   ]
-[ "tracefile_handle",            "void",         "liquid",       "changeable",   "nil"                                   ]
-[ "equivalences",                "BAT",          "liquid",       "changeable",   "<tmp_162>"                             ]
-[ "tj_coll_mutex",               "lock",         "liquid",       "changeable",   "18815256"                              ]
-[ "GLOBAL_TTBAT",                "bit",          "frozen",       "constant",     "true"                                  ]
-[ "dflt_score_base",             "str",          "frozen",       "constant",     "\"0\""                                 ]
-[ "dflt_bg_index",               "str",          "frozen",       "constant",     "\"DFLT_FT_INDEX\""                     ]
-[ "dflt_ft_index",               "str",          "frozen",       "constant",     "\"DFLT_FT_INDEX\""                     ]
-[ "verbose",                     "bit",          "liquid",       "changeable",   "false"                                 ]
-[ "QENV_SCOREBASE",              "oid",          "frozen",       "constant",     "2 at 0"                                   ]
-[ "QENV_FTIBGNAME",              "oid",          "frozen",       "constant",     "1 at 0"                                   ]
-[ "QENV_FTINAME",                "oid",          "frozen",       "constant",     "0 at 0"                                   ]
@@ -247,3 +235,3 @@
-[ "tj_coll_lockbat",             "BAT",          "liquid",       "changeable",   "<tmp_161>"                             ]
-[ "tj_dep_lock",                 "lock",         "liquid",       "changeable",   "18788840"                              ]
-[ "tj_adm_lock",                 "lock",         "liquid",       "changeable",   "18788504"                              ]
+[ "tj_coll_lockbat",             "BAT",          "liquid",       "changeable",   "<tmp_277>"             ]
+[ "tj_dep_lock",                 "lock",         "liquid",       "changeable",   "18793288"              ]
+[ "tj_adm_lock",                 "lock",         "liquid",       "changeable",   "18792952"              ]
========

Alll these seem to be pftijah variables; hence, it might also be that
loading the pftijah module the second time does fail --- unnoticed because
the pathfinder's `module("pftijah")` is wrapped in a CATCH()
(which is admittedly unhandy for debugging things like these...):

========
.../pathfinder/modules/pftijah/Tests $ grep 'module("pftijah")' ../../../runtime/pathfinder.mx
--------
CATCH(module("pftijah")); # load pftijah only if available
========

Let's try:

========
.../pathfinder/modules/pftijah/Tests $ grep 'module("pftijah")' $PATHFINDER_PREFIX/lib64/MonetDB4/pathfinder.mil
--------
CATCH(module("pftijah")); # load pftijah only if available
========
.../pathfinder/modules/pftijah/Tests $ perl -i -p -e 's|CATCH\((module\("pftijah"\))\)|$1|' $PATHFINDER_PREFIX/lib64/MonetDB4/pathfinder.mil
========
.../pathfinder/modules/pftijah/Tests $ grep 'module("pftijah")' $PATHFINDER_PREFIX/lib64/MonetDB4/pathfinder.mil
--------
module("pftijah"); # load pftijah only if available
========

========
.../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); tj_init_collection("testcoll0", new(str,str).insert("stemmer","snowball-english"));' | Mserver --dbname=DB-3
--------
# Monet Database Server V4.17.1
# Copyright (c) 1993-2007, CWI. All rights reserved.
# Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked.
# Visit http://monetdb.cwi.nl/ for further information.
MonetDB># XRPC administrative console at http://localhost:50001/admin
========
.../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); tj_init_collection("testcoll0", new(str,str).insert("stemmer","snowball-english"));' | Mserver --dbname=DB-3
--------
# Monet Database Server V4.17.1
# Copyright (c) 1993-2007, CWI. All rights reserved.
# Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked.
# Visit http://monetdb.cwi.nl/ for further information.
MonetDB>!ERROR: interpret: no matching MIL operator to 'tj_setCollName(str)'.
========

AHA!!!

========
.../pathfinder/modules/pftijah/Tests $ Mserver --dbname=DB-4
--------
# Monet Database Server V4.17.1
# Copyright (c) 1993-2007, CWI. All rights reserved.
# Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked.
# Visit http://monetdb.cwi.nl/ for further information.
MonetDB>module(pathfinder);
# XRPC administrative console at http://localhost:50001/admin
MonetDB>help("tj_setCollName");
no help available
MonetDB>tj_setCollName();
!ERROR: interpret: no matching MIL operator to 'tj_setCollName()'.
========
.../pathfinder $ find . -type f | xargs grep --color tj_setCollName
--------
./runtime/Tests/procs.stable.out:[ "tj_setCollName(str) : void"                                                                                                                                                                                         ]
./modules/pftijah/Tests/test_nllr.milS:tj_setCollName("testcoll1");
./modules/pftijah/Tests/test_nllr.stable.out:tj_setCollName("testcoll1");
./modules/pftijah/pftijah.mx:           tj_setCollName(name);
./modules/pftijah/nexi.c:    if ( 0 ) { MILPRINTF(MILOUT, "tj_setCollName(\"%s\");\n", parserCtx->collection); /* DELETE */ /* DELETE */ }
========

Apparently, you are still using a PROC ("tj_setCollName") that you removed in
your latest checkin --- running test runtime/Tests/procs.milS (would have)
show(s|ed) you that the respective signature has gone.

Hence, it's neither "Mtest" nor any other "mysterious bug", but quite a
simple problem.


Stefan


ps: ... yes, also this debugging service was free of charge ...


> >
> > Stefan
> 
> The text of the bug report I had in  mind was:
> ====================================================
> I am having Mtest module variable problems again with pftijah. It looks a
> little bit like the never really solved problem of bug:
> 
> [ 1638729 ] Mtest/Mapi module load problem
> 
> The problem here was that the module was loaded but the instance variables
> and the non-ADDHELP'ed functions were unknown. We solved this by loading
> pftijah always from pathfinder but did not find the cause of the problem.
> 
> I'm now having the same problem but in a different form. When I use Mtest
> on the pftijah Testset the first test where module(pftijah) is used works
> fine. But all tests after this fail. The problem is again that the MIL
> functions in the module are executed but the functions do not know the
> instance variables of the module. Nearly all tests fail on the variables
> "verbose" and "dflt_ft_index".
> You can for instance check this by changing $PF/modules/pftijah/Tests/All
> The first test which normally fails is createcoll0 because it is the
> second test. But when you remove loadthesis from All createcoll0 succeeds
> and createcoll1, which is then the second, fails.
> 
> ====================================================
> 
> Jan.
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Monetdb-developers mailing list
> Monetdb-developers at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
> 
> 

-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4312       |




More information about the developers-list mailing list