Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/modules/pftijah nexi.c, 1.57, 1.58 nexi_generate_mil.c, 1.27, 1.28 pftijah.mx, 1.116, 1.117
On Mon, May 07, 2007 at 07:01:43AM +0000, Jan Flokstra wrote:
Update of /cvsroot/monetdb/pathfinder/modules/pftijah In directory sc8-pr-cvs16:/tmp/cvs-serv20990/modules/pftijah
Modified Files: nexi.c nexi_generate_mil.c pftijah.mx Log Message:
- Implement private term- and tag dictionaries. Default is still the global version so nothing changed yet in the format. I'm having terrible trouble with the Mtest's again. This version could not be tested by our Mtest tests because the first query executes OK but all queries after that seem to 'forget' the instance variables of module pftijah. I will look if the same happens on other machine and post a bugreport for it.
does the same happen if you run the test(s) by hand i.e., without Mtest?
How do the queries and the test script(s) look like? How do you (successfully?) run them by hand? How do you (unsuccesfully) run them via Mtest?
Stefan
On Monday 07 May 2007 09:09, Stefan Manegold wrote:
On Mon, May 07, 2007 at 07:01:43AM +0000, Jan Flokstra wrote:
Update of /cvsroot/monetdb/pathfinder/modules/pftijah In directory sc8-pr-cvs16:/tmp/cvs-serv20990/modules/pftijah
Modified Files: nexi.c nexi_generate_mil.c pftijah.mx Log Message:
- Implement private term- and tag dictionaries. Default is still the
global version so nothing changed yet in the format. I'm having terrible trouble with the Mtest's again. This version could not be tested by our Mtest tests because the first query executes OK but all queries after that seem to 'forget' the instance variables of module pftijah. I will look if the same happens on other machine and post a bugreport for it.
does the same happen if you run the test(s) by hand i.e., without Mtest?
No, without Mtest the queries run OK. eg, any FIRST query runs OK. Any query after this fails.
How do the queries and the test script(s) look like?
The normal 28 tests from $PF/modules/pftijah/Tests So a mix of mil scripts and .xq XQueries.
How do you (successfully?) run them by hand?
Just run the scripts in a test directory using MapiClient (-lx)
How do you (unsuccesfully) run them via Mtest?
Mtest.py -d0 --TSTSRCBASE /local/flokstra/software/pathfinder
in directory /local/flokstra/software/pathfinder/modules/pftijah
Stefan
The text of the bug report I had in mind was: ==================================================== I am having Mtest module variable problems again with pftijah. It looks a little bit like the never really solved problem of bug:
[ 1638729 ] Mtest/Mapi module load problem
The problem here was that the module was loaded but the instance variables and the non-ADDHELP'ed functions were unknown. We solved this by loading pftijah always from pathfinder but did not find the cause of the problem.
I'm now having the same problem but in a different form. When I use Mtest on the pftijah Testset the first test where module(pftijah) is used works fine. But all tests after this fail. The problem is again that the MIL functions in the module are executed but the functions do not know the instance variables of the module. Nearly all tests fail on the variables "verbose" and "dflt_ft_index". You can for instance check this by changing $PF/modules/pftijah/Tests/All The first test which normally fails is createcoll0 because it is the second test. But when you remove loadthesis from All createcoll0 succeeds and createcoll1, which is then the second, fails.
====================================================
Jan.
Jan,
On Mon, May 07, 2007 at 09:19:17AM +0200, Jan Flokstra wrote:
On Monday 07 May 2007 09:09, Stefan Manegold wrote:
On Mon, May 07, 2007 at 07:01:43AM +0000, Jan Flokstra wrote:
Update of /cvsroot/monetdb/pathfinder/modules/pftijah In directory sc8-pr-cvs16:/tmp/cvs-serv20990/modules/pftijah
Modified Files: nexi.c nexi_generate_mil.c pftijah.mx Log Message:
- Implement private term- and tag dictionaries. Default is still the
global version so nothing changed yet in the format. I'm having terrible trouble with the Mtest's again. This version could not be tested by our Mtest tests because the first query executes OK but all queries after that seem to 'forget' the instance variables of module pftijah. I will look if the same happens on other machine and post a bugreport for it.
does the same happen if you run the test(s) by hand i.e., without Mtest?
No, without Mtest the queries run OK. eg, any FIRST query runs OK. Any query after this fails.
do they run OK or NOT ???
If also without Mtest only the first runs OK and any other fails (that what I read here), it is NOT an Mtest problem, but a general problem, right???
How do the queries and the test script(s) look like?
The normal 28 tests from $PF/modules/pftijah/Tests So a mix of mil scripts and .xq XQueries.
How do you (successfully?) run them by hand?
Just run the scripts in a test directory using MapiClient (-lx)
How do you (unsuccesfully) run them via Mtest?
Mtest.py -d0 --TSTSRCBASE /local/flokstra/software/pathfinder
in directory /local/flokstra/software/pathfinder/modules/pftijah
Before this mornings changes, everything did work fine, right?
Hence, your latest changes seem to *trigger* (not necessarily cause) the problem. Looking in detail in your changes might give a hint, which part in fact triggers the problem, which in turn would help a lot to analyse, locate and eventually fix the problem.
Speaking of debugging here's a little excercise thet shows that the second call to tj_init_collection() on the same database does fail, i.e., only the first call to tj_init_collection() on a virgin database does not fail:
======== .../pathfinder/modules/pftijah/Tests $ export TSTSRCDIR=$PWD ======== .../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); tj_init_collection("testcoll0", new(str,str).insert("stemmer","snowball-english"));' | Mserver --dbname=DB-1 -------- # Monet Database Server V4.17.1 # Copyright (c) 1993-2007, CWI. All rights reserved. # Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. MonetDB># XRPC administrative console at http://localhost:50001/admin ======== .../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); tj_init_collection("testcoll0", new(str,str).insert("stemmer","snowball-english"));' | Mserver --dbname=DB-1 -------- # Monet Database Server V4.17.1 # Copyright (c) 1993-2007, CWI. All rights reserved. # Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. MonetDB># XRPC administrative console at http://localhost:50001/admin !ERROR: interpret: unknown variable 'verbose'. ========
so, let's "trace" the variables: ======== .../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); vars();' | Mserver --dbname=DB-2 > /tmp/vars.0 ======== .../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); vars();' | Mserver --dbname=DB-2 > /tmp/vars.1 ======== .../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); tj_init_collection("testcoll0", new(str,str).insert("stemmer","snowball-english"));' | Mserver --dbname=DB-2 -------- # Monet Database Server V4.17.1 # Copyright (c) 1993-2007, CWI. All rights reserved. # Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. MonetDB># XRPC administrative console at http://localhost:50001/admin ======== .../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); vars();' | Mserver --dbname=DB-2 > /tmp/vars.2 ======== .../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); tj_init_collection("testcoll0", new(str,str).insert("stemmer","snowball-english"));' | Mserver --dbname=DB-2 -------- # Monet Database Server V4.17.1 # Copyright (c) 1993-2007, CWI. All rights reserved. # Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. MonetDB># XRPC administrative console at http://localhost:50001/admin !ERROR: interpret: unknown variable 'verbose'. ======== .../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); vars();' | Mserver --dbname=DB-2 > /tmp/vars.3 ======== .../pathfinder/modules/pftijah/Tests $ grep --color verbose /tmp/vars.? -------- /tmp/vars.0:[ "verbose", "bit", "liquid", "changeable", "false" ] /tmp/vars.1:[ "verbose", "bit", "liquid", "changeable", "false" ] ========
I have no idea, what tj_init_collection() does but it somehow triggers the disappeasing of some variables --- see also: ======== .../pathfinder/modules/pftijah/Tests $ diff -dU0 -Bbw /tmp/vars.[12] | cdiff -------- --- /tmp/vars.1 2007-05-07 10:40:40.000000000 +0200 +++ /tmp/vars.2 2007-05-07 10:41:03.000000000 +0200 @@ -6 +6 @@ -#-------------------------------------------------------------------------------------------------------------------------# +#---------------------------------------------------------------------------------------------------------# @@ -9,2 +9,2 @@ -#-------------------------------------------------------------------------------------------------------------------------# -[ "pf_logger", "logger", "liquid", "changeable", "19792504" ] +#---------------------------------------------------------------------------------------------------------# +[ "pf_logger", "logger", "liquid", "changeable", "19781832" ] @@ -41 +41 @@ -[ "pf_chkpt_barrier", "sema", "liquid", "changeable", "14646584" ] +[ "pf_chkpt_barrier", "sema", "liquid", "changeable", "15961944" ] @@ -43 +43 @@ -[ "pf_chkpt", "lock", "liquid", "changeable", "19243832" ] +[ "pf_chkpt", "lock", "liquid", "changeable", "19100760" ] @@ -47 +47 @@ -[ "pf_collbarrier_lock", "lock", "liquid", "changeable", "19242456" ] +[ "pf_collbarrier_lock", "lock", "liquid", "changeable", "19099384" ] @@ -63 +63 @@ -[ "ws_log_lock", "lock", "liquid", "changeable", "19094712" ] +[ "ws_log_lock", "lock", "liquid", "changeable", "19090936" ] @@ -219,12 +218,0 @@ -[ "virtRoot", "oid", "frozen", "constant", "0@0" ] -[ "tracefile_handle", "void", "liquid", "changeable", "nil" ] -[ "equivalences", "BAT", "liquid", "changeable", "<tmp_162>" ] -[ "tj_coll_mutex", "lock", "liquid", "changeable", "18815256" ] -[ "GLOBAL_TTBAT", "bit", "frozen", "constant", "true" ] -[ "dflt_score_base", "str", "frozen", "constant", ""0"" ] -[ "dflt_bg_index", "str", "frozen", "constant", ""DFLT_FT_INDEX"" ] -[ "dflt_ft_index", "str", "frozen", "constant", ""DFLT_FT_INDEX"" ] -[ "verbose", "bit", "liquid", "changeable", "false" ] -[ "QENV_SCOREBASE", "oid", "frozen", "constant", "2@0" ] -[ "QENV_FTIBGNAME", "oid", "frozen", "constant", "1@0" ] -[ "QENV_FTINAME", "oid", "frozen", "constant", "0@0" ] @@ -247,3 +235,3 @@ -[ "tj_coll_lockbat", "BAT", "liquid", "changeable", "<tmp_161>" ] -[ "tj_dep_lock", "lock", "liquid", "changeable", "18788840" ] -[ "tj_adm_lock", "lock", "liquid", "changeable", "18788504" ] +[ "tj_coll_lockbat", "BAT", "liquid", "changeable", "<tmp_277>" ] +[ "tj_dep_lock", "lock", "liquid", "changeable", "18793288" ] +[ "tj_adm_lock", "lock", "liquid", "changeable", "18792952" ] ========
Alll these seem to be pftijah variables; hence, it might also be that loading the pftijah module the second time does fail --- unnoticed because the pathfinder's `module("pftijah")` is wrapped in a CATCH() (which is admittedly unhandy for debugging things like these...):
======== .../pathfinder/modules/pftijah/Tests $ grep 'module("pftijah")' ../../../runtime/pathfinder.mx -------- CATCH(module("pftijah")); # load pftijah only if available ========
Let's try:
======== .../pathfinder/modules/pftijah/Tests $ grep 'module("pftijah")' $PATHFINDER_PREFIX/lib64/MonetDB4/pathfinder.mil -------- CATCH(module("pftijah")); # load pftijah only if available ======== .../pathfinder/modules/pftijah/Tests $ perl -i -p -e 's|CATCH((module("pftijah")))|$1|' $PATHFINDER_PREFIX/lib64/MonetDB4/pathfinder.mil ======== .../pathfinder/modules/pftijah/Tests $ grep 'module("pftijah")' $PATHFINDER_PREFIX/lib64/MonetDB4/pathfinder.mil -------- module("pftijah"); # load pftijah only if available ========
======== .../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); tj_init_collection("testcoll0", new(str,str).insert("stemmer","snowball-english"));' | Mserver --dbname=DB-3 -------- # Monet Database Server V4.17.1 # Copyright (c) 1993-2007, CWI. All rights reserved. # Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. MonetDB># XRPC administrative console at http://localhost:50001/admin ======== .../pathfinder/modules/pftijah/Tests $ echo 'module(pathfinder); tj_init_collection("testcoll0", new(str,str).insert("stemmer","snowball-english"));' | Mserver --dbname=DB-3 -------- # Monet Database Server V4.17.1 # Copyright (c) 1993-2007, CWI. All rights reserved. # Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. MonetDB>!ERROR: interpret: no matching MIL operator to 'tj_setCollName(str)'. ========
AHA!!!
======== .../pathfinder/modules/pftijah/Tests $ Mserver --dbname=DB-4 -------- # Monet Database Server V4.17.1 # Copyright (c) 1993-2007, CWI. All rights reserved. # Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. MonetDB>module(pathfinder); # XRPC administrative console at http://localhost:50001/admin MonetDB>help("tj_setCollName"); no help available MonetDB>tj_setCollName(); !ERROR: interpret: no matching MIL operator to 'tj_setCollName()'. ======== .../pathfinder $ find . -type f | xargs grep --color tj_setCollName -------- ./runtime/Tests/procs.stable.out:[ "tj_setCollName(str) : void" ] ./modules/pftijah/Tests/test_nllr.milS:tj_setCollName("testcoll1"); ./modules/pftijah/Tests/test_nllr.stable.out:tj_setCollName("testcoll1"); ./modules/pftijah/pftijah.mx: tj_setCollName(name); ./modules/pftijah/nexi.c: if ( 0 ) { MILPRINTF(MILOUT, "tj_setCollName("%s");\n", parserCtx->collection); /* DELETE */ /* DELETE */ } ========
Apparently, you are still using a PROC ("tj_setCollName") that you removed in your latest checkin --- running test runtime/Tests/procs.milS (would have) show(s|ed) you that the respective signature has gone.
Hence, it's neither "Mtest" nor any other "mysterious bug", but quite a simple problem.
Stefan
ps: ... yes, also this debugging service was free of charge ...
Stefan
The text of the bug report I had in mind was:
I am having Mtest module variable problems again with pftijah. It looks a little bit like the never really solved problem of bug:
[ 1638729 ] Mtest/Mapi module load problem
The problem here was that the module was loaded but the instance variables and the non-ADDHELP'ed functions were unknown. We solved this by loading pftijah always from pathfinder but did not find the cause of the problem.
I'm now having the same problem but in a different form. When I use Mtest on the pftijah Testset the first test where module(pftijah) is used works fine. But all tests after this fail. The problem is again that the MIL functions in the module are executed but the functions do not know the instance variables of the module. Nearly all tests fail on the variables "verbose" and "dflt_ft_index". You can for instance check this by changing $PF/modules/pftijah/Tests/All The first test which normally fails is createcoll0 because it is the second test. But when you remove loadthesis from All createcoll0 succeeds and createcoll1, which is then the second, fails.
====================================================
Jan.
This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
participants (2)
-
Jan Flokstra -
Stefan Manegold