On Mon, Dec 19, 2005 at 01:33:38AM +0100, Stefan Manegold wrote:
Dear all,
here are some more informations for the 3rd MonetDB-Bug-Day today.
Please note, though it's called Bug-*Day*, you don't have to spend the whole day writing test scripts (though no one would mind if you do ;-)). Below you find chunks of 10 bug reports each; simply claim one chunk at a time (by group-reply to this email) whenever you feel like you could spend some minutes on this "community task", check the 10 reports and provide the test script where necessary/applicable. If we don't manage to finish the 21 chunks tomorrow, everybody is welcome to pick more chunk(s) any time later...
Of course, next to (or instead of) adding test scripts, you're also welcome to fix open bugs (and add the test script for each bug you fix)!
Chunks
No Offsets BugIDs # URL
1 550-558 1206432-1206985 9 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
done
2 589-599 1210748-1214335 11 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Did first 4. ... Have to do something else no ;-)
Niels
3 600-609 1214372-1218281 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 4 610-619 1218291-1227493 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 5 620-629 1229018-1238228 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 6 630-639 1238352-1242164 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 7 640-649 1244326-1257463 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 8 650-659 1259668-1263967 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 9 660-669 1265413-1274819 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 10 670-679 1274903-1281996 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 11 680-689 1283842-1293196 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 12 690-698 1294712-1298707 9 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
13 710-719 1313271-1315339 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 14 720-729 1315653-1324575 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 15 730-739 1327854-1334536 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 16 740-749 1337630-1347331 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 17 750-759 1347758-1353883 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 18 760-769 1354321-1358222 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 19 770-779 1359071-1363729 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 20 780-789 1363733-1371524 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 21 790-799 1373391-1380287 10 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... --- 209
General issues
Please pick one chunk at a time and indicate you choice by (group) replying to this mail. I will resolve clashes where necessary. I don't expect more than 21 people participating (concurrently), hence sub-dividing the chunks should not be necessary ;-) In case you cannot finish your last chunk by tonight (for whatever reason), please let me know (via email) which of the reports you finished, and which you didn't. (Please see below for details what to do in case your hit a report that requires a test script, but for which you cannot create a test script.)
Requirements
Obviously, you need to have MonetDB (plus SQL or XQuery for those bugs) installed and running; preferably an up-to-date ssh-based (non anonymous) CVS check out of either the release branches (MonetDB_4-8, SQL_2-8, XQuery_0-8) or the respective CVS heads. Anonymous (pserver-based) CVS check outs or source distributions should also work, but then you cannot check-in your tests directly (see below on how to participate and submit test script in this case).
Stable vs. Current
In principle, it does not matter, whether you're using the Stable release branch(es) or the Current development head(s). Use what ever you like best (or have already running).
Test directories
Your tests must be located (and hence checked in) in the following directories:
MonetDB 4.9.3: MonetDB/tests/BugDay_2005-12-19_4.9.3/Tests/ SQL 2.9.3: sql/src/test/BugDay_2005-12-19_2.9.3/Tests/ XQuery 0.9.3: pathfinder/tests/BugDay_2005-12-19_0.9.3/Tests/
Codes
To indicate the progress (see below for details), please use the following codes when adding comments to the bug report
code explanation ------------------------+--------------------------------------------- RELEASED / REASSIGN | the "claimer" of the bug wasn't able to create a | script from the bug due to insufficient | knowledge. Append reason, e.g. "XQuery". | Also set status of closed bugs to "pending". ALREADY IN TESTWEB | the script was already in CVS and TestWeb | (added location of existing test script). TEST ADDED / SUCCESS | a script was added (append location & name) and | works correctly. TEST ADDED / FAILURE | a script was added, but it fails (bad output). | Re-open closed bug! TEST SUBMITTED | a script was made, but sent to Stefan (for | non-CVS participants) NO TEST / COMPILATION | no test was added because the bug deals with a | compilation problem. NO TEST / DOCUMENTATION | no test was added because the bug deals with a | documentation issue, or misleading help text, | etc. NO TEST / REDUNDANT | no test was added, because either the bug itself | is a duplicate, or the test that follows out of | the bug would clearly be duplicating another | existing bug. When extending another bug to | remove redundancy use TEST ADDED / SUCCESS and | give the name of the already existing updated | bug.
Note: the last action is only meant for experts in their field ;)
An example message on the SF bug could be:
BugDay_2005-12-19, <your name>: TEST ADDED / SUCCESS sql/src/tests/bugday/like_join_union.SF-0123345.SQL
The Task/Procedure
Claim a chunk of bugs as described above, and then process each bug as follows:
Checking for existence
Once claimed, you first check, whether there is already a test script for this bug (i.e., a test script that contains the respective ID in its filename) available in one of
MonetDB/tests/BugsViaSourgeforce/Tests/ sql/src/test/bugs/Tests/ pathfinder/tests/BugsViaSourgeforce/Tests/
If so, you should indicate this be adding a comment like
BugDay_2005-12-19, <your name>: ALREADY IN TESTWEB MonetDB/tests/BugsViaSourgeforce/Tests/ID.602271.*
to the respective bug report on SF, and you're done with this bug, and can proceed to the next one.
If there is no test script, yet, you need to create one:
Naming conventions
Names of test scripts must include a descriptive part (of reasonable length) and the SF bug ID, like
"return_types_of_head_and_tail.SF-701641.*",
i.e., the pattern is
<descriptive-name>.SF-<BugID>[.<extension>]
Creating test scripts
Most bug reports come with some MIL-, SQL-, XQuery-code, or some other description/instructions how to reproduce the bug. The basic task is to put this into a proper "Mtest-compliant" test script. In most cases, this is just a .milS, .sql, or .xq file. See http://monetdb.cwi.nl/monet/src/testing/README for details about Mtest.py . In case of doubt, please feel free to contact any of your colleagues that might be more familiar with Mtest --- or me of course ;-) Of course, you're welcome to extend the test if appropriate.
In case it is obvious that the bug does not require a test script (e.g., compilation, installation, documentation, or web site bugs), please indicate this by adding a comment like
BugDay_2005-12-19, <your name>: NO TEST / <good reason>
to the respective bug report on SF, and proceed with the next bug.
If a test script is required, but you don't know how to make it (for this particular bug), please feel free to ask for help (or "release" the bug report, again).
Running Mtest
Once you made the test script, you need to run the test. In case you're using `source conf/conf.bash` to setup your MonetDB environment, you can simply call one of
Mtest_MonetDB <descriptive-name>.SF-<BugID> Mtest_sql <descriptive-name>.SF-<BugID> Mtest_pathfinder <descriptive-name>.SF-<BugID>
in the respective directory as listed above. See `Mtest.py --help` for command line options.
Note: Do NOT change or check-in the "All" file!
This is not required to run tests that are explicitly given on Mtest's command line. I'll will fill the "All" files at the end of the BugDay.
Creating / fixing stable output
After running Mtest, you need to check the output by pointing your browser to the file given at the end of Mtest's console output.
In case the output is correct and as expected, you can approve it by running
Mapprove_MonetDB <descriptive-name>.SF-<BugID> Mapprove_sql <descriptive-name>.SF-<BugID> Mapprove_pathfinder <descriptive-name>.SF-<BugID>
in the respective directory as listed above.
In case running the test failed, please fix the test and start again with running Mtest.
In case the output is not correct or not as expected, you first approve the output as described above, but then edit the files
<descriptive-name>.SF-<BugID>.stable.out <descriptive-name>.SF-<BugID>.stable.err
by hand to create the correct/expected output.
In any case, you then re-run Mtest, and check the output by pointing your browser to the file given at the end of Mtest's console output.
In case everything is "green" (i.e., OK), you can check-in all files related to the test (including the stable output!), indicate the success by adding a comment like
BugDay_2005-12-19, <your name>: TEST ADDED / SUCCESS tests/BugDay_2005-12-19_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*
to the respective bug report on SF, and proceed with the next bug.
In case there is still a red 'X' (because MonetDB does not produce the correct/expected output), you also check-in all files, but indicate the problem by a comment like
BugDay_2005-12-19, <your name>: TEST ADDED / FAILURE tests/BugDay_2005-12-19_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*
<possibly a description of the failure>
to the respective bug report on SF, and *re-open* this bug report, before proceeding with the next bug.
Anonymous CVS check outs and source distributions
All the above holds, except from the CVS check-in of the new test scripts. Instead, you send me (Stefan.Manegold@cwi.nl) a .tar[.gz] or .zip file including all your new test scripts by the end of your BugDay.
In case you intend to participate without CVS access, please let me know so as soon as you start.
Pooh, I guess that's it for now. I hope, I didn't forget anything --- if so, feel free to ask! (via email, skype, or personally later at CWI).
cu later,
Stefan
On Fri, Dec 16, 2005 at 01:52:30PM +0100, Stefan Manegold wrote:
Dear all,
given the success of the first two MonetDB-Bug-Days on Oct 06 2005 and Nov 09 2005,
--- we managed to check 460 bug reports, added missing test scripts, re-opened bugs that still/again fail and filed new bug reports ---
there will by a 3rd MonetDB-Bug-Day to (hopefully) check the remaining ~225 bug reports, and produce proper test scripts for our automatic Nightly Multi-Platform Regression Test System (aka "TestWeb"; cf. http://monetdb.cwi.nl/Development/TestWeb/):
When: Monday, Dec 19, 2005, "all day" (or as much time as you can spent)
Where: Mainly at CWI, but you're also welcome to join "virtually" working from home or any other place in the world; for those who (again) want to turn the Bug-Day into a Bug-"Party", I reserved our meeting room (C0,01) from 9:00 - 17:00 --- you need to bring your own "equipment" though ...
Who: Primarily everyone who is in touch with MonetDB, and/or one of its companions/front ends (SQL, XQuery, etc.), has rights to check-in to the MonetDB CVS repository on SourceForge; experiences with "Mtest.py" (cf. http://monetdb.cwi.nl/monet/src/testing/README) are helpful, but can also be aquired during the MonetDB-Bug-Day.
What: I will prepare a list of all bugs that have no test script, yet, and chop this in chunks, and assign each chunk to one participant so that [s]he can check them, and prepare the respective test scripts.
More detailed instructions will follow before Monday.Note: Though any help is appreciated, no one is "obliged" to help. Especially, there is no need to spent the whole day (though nine will complain if you do ;-)), anyone can join-in and leave whenever [s]he wants.
For those of you participating at CWI, there will be free snacks and soft-drinks during the day (for the time you're "active"; as long as supplies lasts). Furthermore, there will be "real" drinks at the end of the Bug-Day (for participants, only; as long as supplies lasts; (maximum) amount per person will be proportional to the number of test scripts added).
I hope to see especially (though not only) participants that I haven't seen on the previous MonetDB-Bug-Days ;-)
Once again thank you very much for your cooperation!
CU next Monday,
Stefan
ps: Testing & the TestWeb are not (only?) my "toys"; it's "serious business", and helps all of you to (1) get a stable and reliable MonetDB, (2) monitor when your own bug reports get fixed, and (3) prevent the bugs you once thankfully detected, reported and/or fixed from occurring again!
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |