[Monetdb-developers] TTT goes MonetDB-Bug-Day
[ I deliberately send this email to various related mailing lists; ] [ in case you receive this email more than once, it might be time ] [ to teach your email client to eliminate multiple copies of the ] [ very same email... ]
Dear all,
(admittedly on short notice --- sorry for any related inconvenience)
you're all "invited" to join a community effort to significantly improve the the MonetDB quality control system: Let's (try to) make "Mtest.py-compliant" test scripts from all bug reports that have been filed on SF since July 2002, and thus ensure that these bugs do not re-occur, respectively ensure that we (via the "TestWeb"; cf. http://monetdb.cwi.nl/Development/TestWeb/) immediately notice whenever code changes do trigger an "old" bug.
Summary: ========
When: Thursday, Oct 6, 2005, "all day"
Where: Mainly at CWI, but you're also welcome to join "virtually" working from home or any other place in the world; for those who want to turn the Bug-Day into a Bug-"Party", I reserved our meeting room (C0,01) from 12:30 - 17:00 --- you need to bring your own "equipment" though ...
Who: Primarily everyone who is in touch with MonetDB, and/or one of its companions/front ends (SQL, XQuery, etc.), has rights to check-in to the MonetDB CVS repository on SourceForge, and is reasonably familiar with "Mtest.py" (cf. http://monetdb.cwi.nl/monet/src/testing/README). But I will also (try to) provide "Testing HowTo" by Thursday, so others are welcome to join; further, if you cannot check-in to the MonetDB CVS repository yourself, a colleague of your's (or one of the core developers) might do this for you, once you prepared the complete test.
What: I will prepare a list of all bug that have to test script, yet, and assign them round-robin to all participants (probably, I will try to classify them in say "MIL", "SQL", "XQuery", and "others", and you're welcome to choose your preferred class, given availability).
More detailed instructions will follow by Thursday.
Background: ===========
Next to providing high performance, maintaining correctness and stability of the code base is the major goal of MonetDB and its companions.
First of all, the bug tracking system on SourceForge (http://sourceforge.net/tracker/?group_id=56967&atid=482468) is used intensively to report any problems related to MonetDB --- since MonetDB moved to SourceForge in July 2002, almost 700 bug reports have been filed, most of which have been solved.
Secondly, our automatic Nightly Multi-Platform Regression Test System (aka "TestWeb"; cf. http://monetdb.cwi.nl/Development/TestWeb/) helps us to ensure that bugs that sneak in with modified or newly added code are detected (and fixed) instantly. Obviously, the test system only work for that features/code that are indeed covered by the available test scripts...
Given these two tools, the bug tracker and the test system, we can easily build a system that prevent the (unnoticed) re-occurrence of bug that have been reported and fixed before: We "simply" have to add a proper test script for each bug report.
Unfortunately, so far there are test script only for a fraction of the ~700 bug reports. Given that many bugs are reported by users that are not allowed to add test script to the CVS repository, it is not convenient (not even possible) to require that each bug report includes an "Mtest.py-compliant" (cf. http://monetdb.cwi.nl/monet/src/testing/README) test script. Nevertheless, most bug reports come with a query, a script, or a detailed description how to reproduce the respective bug; hence, it should be rather simple to turn these queries / scripts / description into proper "Mtest.py-compliant" test scripts, and thus significantly improve our automatic quality control system.
Since this task is quite a challenge (in terms of volume, not intellectual complexity ;-)), the idea popped-up to make it a "community effort", as sketched in the "Summary" above. The advantage of this coordinated community effort should be, that all of us dedicate our time to this concurrently, share the same "context" and hence can support each other efficiently when/where necessary.
In advance, thank you very much for your cooperation!
Regards,
Stefan
Dear all,
as promised, here are some more detailed informations for tomorrow.
General issues ==============
To "decentralize" processing and prevent me from becoming the "bottleneck"/"hot spot", I not to assign tests to you; rather, each of you claims the bug that he/she is going to prepare a test script for by adding a comment like
"BugDay_2005-10-06: Claimed [by <name>]."
to the respective bug report; adding a name is optional if you claim a bug for yourself; if you claim a bug for someone who does not have an account with SourceForge (and is not willing to request one) and hence cannot add comments to the bug tracker, adding his/her name is mandatory. See http://sourceforge.net/tracker/?group_id=56967&atid=482468 for a complete list of bug report, and don't for get to set your selection as follows:
Assignee: Status: Category: Group: Any Any Any Any
Sort By: ID Ascending
Of course, you're welcome to first claim multiple bugs, before actually starting to prepare and check-in the tests.
To avoid that everyone starts (trying) to claim the same bugs, here are some bug IDs as entry points. Each one chooses a different entry point, and from there on processes the bugs in in ascending order of bug ID.
602271 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 899586 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 920585 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 929158 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 941693 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 972833 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1010604 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1029099 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1076604 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1119199 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1166456 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1205565 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1214335 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1257463 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1300366 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Please indicate you choice by (group) replying to this mail. I will resolve clashes where necessary. In case there are more than 15 people participating (concurrently), we can easily sub-divide the ranges.
Requirements ============
Obviously, you need to have MonetDB (plus SQL or XQuery for those bugs) installed and running; preferably an up-to-date ssh-based (non anonymous) CVS check out of either the release branches (MonetDB_4-8, SQL_2-8, XQuery_0-8) or the respective CVS heads. Anonymous (pserver-based CVS check outs or source distributions should also work, but then you cannot check-in your tests directly (see below on how to participate and submit test script in this case).
Stable vs. Current ------------------
In principle, it does not matter, whether you're using the Stable release branch(es) or the Current development head(s). Use what ever you like best (or have already running).
Test directories ----------------
Your tests must be located (and hence check in) in different directories depending on the MonetDB version you're using:
Stable branch: MonetDB 4.8.*: MonetDB/tests/BugDay_2005-10-06_4.8/Tests/ SQL 2.8.*: sql/src/test/BugDay_2005-10-06_2.8/Tests/ XQuery 0.8.*: pathfinder/tests/BugDay_2005-10-06_0.8/Tests/
Current head: MonetDB 4.9.3: MonetDB/tests/BugDay_2005-10-06_4.9.3/Tests/ SQL 2.9.3: sql/src/test/BugDay_2005-10-06_2.9.3/Tests/ XQuery 0.9.3: pathfinder/tests/BugDay_2005-10-06_0.9.3/Tests/
The reason for having different direcories is mainly convenience with check-ins and later propagations.
The Task/Procedure ==================
Claiming a bug --------------
As mentioned above, you simply claim a bug by adding a comment like
"BugDay_2005-10-06: Claimed [by <name>]."
to the respective bug report on SF.
Checking for existence ----------------------
Once claimed, you first check, whether there is already a test script for this bug available in one of
MonetDB/tests/BugsViaSourgeforce/Tests/ sql/src/test/bugs/Tests/ pathfinder/tests/BugsViaSourgeforce/Tests/
If so, you should indicate this be adding a comment like
BugDay_2005-10-06: Done. test script available as MonetDB/tests/BugsViaSourgeforce/Tests/ID.602271.*
to the respective bug report on SF, and you're done with this bug, and can proceed to the next one.
If there is no test script, yet, you need to create one:
Naming conventions ------------------
Names of test scripts must include a descriptive part (of reasonable length) and the SF bug ID, like
"return_types_of_head_and_tail.SF-701641.*",
i.e., the pattern is
<descriptive-name>.SF-<BugID>[.<extension>]
Creating test scripts ---------------------
As mentioned in my first email, most bug reports come with some MIL-, SQL-, XQuery-code, or some other description/instructions how to reproduce the bug. The basic task is to put this into a proper "Mtest-compliant" test script. In most cases, this is just a .milS, .sql, or .xq file. See http://monetdb.cwi.nl/monet/src/testing/README for details about Mtest.py . In case of doubt, please feel free to contact any of your colleagues that might be more familiar with Mtest --- or me of course ;-) Of course, you're welcome to extend the test if appropriate.
In case it is obvious that the bug does not require a test script (e.g., compilation, installation, or web site bugs), please indicate this by adding a comment like
BugDay_2005-10-06: Done. No test script added, because <good reason>.
to the respective bug report on SF, and proceed with the next bug.
If a test script is required, but you don't know how to make it (for this particular bug), please feel free to ask for help (or "release" the bug report, again).
Running Mtest -------------
Once you made the test script, you need to run the test. In case you're using `source conf/conf.bash` to setup your MonetDB environment, you can simply call one of
Mtest_MonetDB <descriptive-name>.SF-<BugID> Mtest_sql <descriptive-name>.SF-<BugID> Mtest_pathfinder <descriptive-name>.SF-<BugID>
in the respective directory as listed above. See `Mtest.py --help` for command line options.
Note: Do NOT change or check-in the "All" file!
This is not required to run tests that are explicitly given on Mtest's command line. I'll will fill the "All" files at the end of the BugDay.
Creating / fixing stable output -------------------------------
After running Mtest, you need to check the output by pointing your browser to the file given at the end of Mtest's console output.
In case the output is correct and as expected, you can approve it by running
Mapprove_MonetDB <descriptive-name>.SF-<BugID> Mapprove_sql <descriptive-name>.SF-<BugID> Mapprove_pathfinder <descriptive-name>.SF-<BugID>
in the respective directory as listed above. In case running the test failed, please fix the test and start again with running Mtest.
In case the output is not correct or no as expected, you first approve the output as described above, but then edit the files
<descriptive-name>.SF-<BugID>.stable.out <descriptive-name>.SF-<BugID>.stable.err
by hand to create the correct/expected output.
In any case, you then re-run Mtest, and check the output by pointing your browser to the file given at the end of Mtest's console output.
In case everything is "green" (i.e., OK), you can check-in all files related to the test (including the stable output!), indicate the success by adding a comment like
BugDay_2005-10-06: Done: test script added as tests/BugDay_2005-10-06_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*
to the respective bug report on SF, and proceed with the next bug.
In case there is still a red 'X' (because MonetDB does not produce the correct/expected output), you also check-in all files, but indicate the problem by a comment like
BugDay_2005-10-06: Done: test script added as tests/BugDay_2005-10-06_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*
Note: Test fails! <possibly a description of the failure>
to the respective bug report on SF, and re-open this bug report, before proceeding with the next bug.
Anonymous CVS check outs and source distributions =================================================
All the above holds, except from the CVS check-in of the new test scripts. Instead, you send me (Stefan.Manegold@cwi.nl) a .tar[.gz] or .zip file including all your new test scripts by the end of your BugDay.
In case you intend to participate without CVS access, please let me know so as soon as you start.
Pooh, I guess that's it for now. I hope, I didn't forget anything --- if so, feel free to ask! (via email, or personally tomorrow at CWI).
cu tomorrow,
Stefan
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Stefan Manegold wrote:
To avoid that everyone starts (trying) to claim the same bugs, here are some bug IDs as entry points. Each one chooses a different entry point, and from there on processes the bugs in in ascending order of bug ID.
[snip]
Please indicate you choice by (group) replying to this mail. I will resolve clashes where necessary. In case there are more than 15 people participating (concurrently), we can easily sub-divide the ranges.
Hereby I initially claim: http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Creating test scripts
In case it is obvious that the bug does not require a test script (e.g., compilation, installation, or web site bugs), please indicate this by adding a comment like
BugDay_2005-10-06: Done.
No test script added, because <good reason>.
I'd like to suggest to the following 'codes':
code explanation ------------------------+--------------------------------------------- CLAIMED BY | the bug is being processed by (append name) RELEASED / REASSIGN | the owner of the bug wasn't able to create a | script from the bug due to insufficient | knowledge. Append reason, e.g. "XQuery" ALREADY IN TESTWEB | the script was already in CVS and TestWeb TEST ADDED / SUCCESS | a script was added (append name) and works | correctly TEST ADDED / FAILURE | a script was added, but it fails (bad output) TEST SUBMITTED | a script was made, but sent to Stefan (for | non-CVS participants) NO TEST / COMPILATION | no test was added because the bug deals with a | compilation problem NO TEST / DOCUMENTATION | no test was added because the bug deals with a | documentation issue, or misleading help text, | etc. NO TEST / REDUNDANT | no test was added, because either the bug | itself is a duplicate, or the test that | follows out of the bug would clearly be | duplicating another existing bug. When | extending another bug to remove redundancy | use TEST ADDED / SUCCESS and give the name of | the already existing updated bug.
Note: the last action is only meant for experts in their field ;)
An example message on the SF bug could be:
BugDay_2005-10-06: TEST ADDED / SUCCESS sql/src/tests/bugday/like_join_union.SF-0123345.SQL
On Wed, Oct 05, 2005 at 06:43:17PM +0200, Stefan Manegold wrote:
Dear all,
as promised, here are some more detailed informations for tomorrow.
General issues
To "decentralize" processing and prevent me from becoming the "bottleneck"/"hot spot", I not to assign tests to you; rather, each of you claims the bug that he/she is going to prepare a test script for by adding a comment like
"BugDay_2005-10-06: Claimed [by <name>]."
to the respective bug report; adding a name is optional if you claim a bug for yourself; if you claim a bug for someone who does not have an account with SourceForge (and is not willing to request one) and hence cannot add comments to the bug tracker, adding his/her name is mandatory. See http://sourceforge.net/tracker/?group_id=56967&atid=482468 for a complete list of bug report, and don't for get to set your selection as follows:
Assignee: Status: Category: Group: Any Any Any Any
Sort By: ID Ascending
Of course, you're welcome to first claim multiple bugs, before actually starting to prepare and check-in the tests.
To avoid that everyone starts (trying) to claim the same bugs, here are some bug IDs as entry points. Each one chooses a different entry point, and from there on processes the bugs in in ascending order of bug ID.
602271 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Okay I simply claim the first 50..
Niels
899586 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 920585 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 929158 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 941693 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 972833 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1010604 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1029099 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1076604 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1119199 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1166456 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1205565 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1214335 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1257463 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1300366 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Please indicate you choice by (group) replying to this mail. I will resolve clashes where necessary. In case there are more than 15 people participating (concurrently), we can easily sub-divide the ranges.
Requirements
Obviously, you need to have MonetDB (plus SQL or XQuery for those bugs) installed and running; preferably an up-to-date ssh-based (non anonymous) CVS check out of either the release branches (MonetDB_4-8, SQL_2-8, XQuery_0-8) or the respective CVS heads. Anonymous (pserver-based CVS check outs or source distributions should also work, but then you cannot check-in your tests directly (see below on how to participate and submit test script in this case).
Stable vs. Current
In principle, it does not matter, whether you're using the Stable release branch(es) or the Current development head(s). Use what ever you like best (or have already running).
Test directories
Your tests must be located (and hence check in) in different directories depending on the MonetDB version you're using:
Stable branch: MonetDB 4.8.*: MonetDB/tests/BugDay_2005-10-06_4.8/Tests/ SQL 2.8.*: sql/src/test/BugDay_2005-10-06_2.8/Tests/ XQuery 0.8.*: pathfinder/tests/BugDay_2005-10-06_0.8/Tests/
Current head: MonetDB 4.9.3: MonetDB/tests/BugDay_2005-10-06_4.9.3/Tests/ SQL 2.9.3: sql/src/test/BugDay_2005-10-06_2.9.3/Tests/ XQuery 0.9.3: pathfinder/tests/BugDay_2005-10-06_0.9.3/Tests/
The reason for having different direcories is mainly convenience with check-ins and later propagations.
The Task/Procedure
Claiming a bug
As mentioned above, you simply claim a bug by adding a comment like
"BugDay_2005-10-06: Claimed [by <name>]."
to the respective bug report on SF.
Checking for existence
Once claimed, you first check, whether there is already a test script for this bug available in one of
MonetDB/tests/BugsViaSourgeforce/Tests/ sql/src/test/bugs/Tests/ pathfinder/tests/BugsViaSourgeforce/Tests/
If so, you should indicate this be adding a comment like
BugDay_2005-10-06: Done. test script available as MonetDB/tests/BugsViaSourgeforce/Tests/ID.602271.*
to the respective bug report on SF, and you're done with this bug, and can proceed to the next one.
If there is no test script, yet, you need to create one:
Naming conventions
Names of test scripts must include a descriptive part (of reasonable length) and the SF bug ID, like
"return_types_of_head_and_tail.SF-701641.*",
i.e., the pattern is
<descriptive-name>.SF-<BugID>[.<extension>]
Creating test scripts
As mentioned in my first email, most bug reports come with some MIL-, SQL-, XQuery-code, or some other description/instructions how to reproduce the bug. The basic task is to put this into a proper "Mtest-compliant" test script. In most cases, this is just a .milS, .sql, or .xq file. See http://monetdb.cwi.nl/monet/src/testing/README for details about Mtest.py . In case of doubt, please feel free to contact any of your colleagues that might be more familiar with Mtest --- or me of course ;-) Of course, you're welcome to extend the test if appropriate.
In case it is obvious that the bug does not require a test script (e.g., compilation, installation, or web site bugs), please indicate this by adding a comment like
BugDay_2005-10-06: Done.
No test script added, because <good reason>.
to the respective bug report on SF, and proceed with the next bug.
If a test script is required, but you don't know how to make it (for this particular bug), please feel free to ask for help (or "release" the bug report, again).
Running Mtest
Once you made the test script, you need to run the test. In case you're using `source conf/conf.bash` to setup your MonetDB environment, you can simply call one of
Mtest_MonetDB <descriptive-name>.SF-<BugID> Mtest_sql <descriptive-name>.SF-<BugID> Mtest_pathfinder <descriptive-name>.SF-<BugID>
in the respective directory as listed above. See `Mtest.py --help` for command line options.
Note: Do NOT change or check-in the "All" file!
This is not required to run tests that are explicitly given on Mtest's command line. I'll will fill the "All" files at the end of the BugDay.
Creating / fixing stable output
After running Mtest, you need to check the output by pointing your browser to the file given at the end of Mtest's console output.
In case the output is correct and as expected, you can approve it by running
Mapprove_MonetDB <descriptive-name>.SF-<BugID> Mapprove_sql <descriptive-name>.SF-<BugID> Mapprove_pathfinder <descriptive-name>.SF-<BugID>
in the respective directory as listed above.
In case running the test failed, please fix the test and start again with running Mtest.
In case the output is not correct or no as expected, you first approve the output as described above, but then edit the files
<descriptive-name>.SF-<BugID>.stable.out <descriptive-name>.SF-<BugID>.stable.err
by hand to create the correct/expected output.
In any case, you then re-run Mtest, and check the output by pointing your browser to the file given at the end of Mtest's console output.
In case everything is "green" (i.e., OK), you can check-in all files related to the test (including the stable output!), indicate the success by adding a comment like
BugDay_2005-10-06: Done: test script added as tests/BugDay_2005-10-06_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*
to the respective bug report on SF, and proceed with the next bug.
In case there is still a red 'X' (because MonetDB does not produce the correct/expected output), you also check-in all files, but indicate the problem by a comment like
BugDay_2005-10-06: Done: test script added as tests/BugDay_2005-10-06_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*
Note: Test fails!
<possibly a description of the failure>
to the respective bug report on SF, and re-open this bug report, before proceeding with the next bug.
Anonymous CVS check outs and source distributions
All the above holds, except from the CVS check-in of the new test scripts. Instead, you send me (Stefan.Manegold@cwi.nl) a .tar[.gz] or .zip file including all your new test scripts by the end of your BugDay.
In case you intend to participate without CVS access, please let me know so as soon as you start.
Pooh, I guess that's it for now. I hope, I didn't forget anything --- if so, feel free to ask! (via email, or personally tomorrow at CWI).
cu tomorrow,
Stefan
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Niels Nes wrote:
On Wed, Oct 05, 2005 at 06:43:17PM +0200, Stefan Manegold wrote:
Dear all,
as promised, here are some more detailed informations for tomorrow.
General issues
To "decentralize" processing and prevent me from becoming the "bottleneck"/"hot spot", I not to assign tests to you; rather, each of you claims the bug that he/she is going to prepare a test script for by adding a comment like
"BugDay_2005-10-06: Claimed [by <name>]."
to the respective bug report; adding a name is optional if you claim a bug for yourself; if you claim a bug for someone who does not have an account with SourceForge (and is not willing to request one) and hence cannot add comments to the bug tracker, adding his/her name is mandatory. See http://sourceforge.net/tracker/?group_id=56967&atid=482468 for a complete list of bug report, and don't for get to set your selection as follows:
Assignee: Status: Category: Group: Any Any Any Any
Sort By: ID Ascending
Of course, you're welcome to first claim multiple bugs, before actually starting to prepare and check-in the tests.
To avoid that everyone starts (trying) to claim the same bugs, here are some bug IDs as entry points. Each one chooses a different entry point, and from there on processes the bugs in in ascending order of bug ID.
602271 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Okay I simply claim the first 50..
Niels
I'll claim the following 50:
899586 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
920585 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 929158 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 941693 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 972833 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1010604 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1029099 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1076604 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1119199 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1166456 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1205565 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1214335 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1257463 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1300366 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Please indicate you choice by (group) replying to this mail. I will resolve clashes where necessary. In case there are more than 15 people participating (concurrently), we can easily sub-divide the ranges.
Requirements
Obviously, you need to have MonetDB (plus SQL or XQuery for those bugs) installed and running; preferably an up-to-date ssh-based (non anonymous) CVS check out of either the release branches (MonetDB_4-8, SQL_2-8, XQuery_0-8) or the respective CVS heads. Anonymous (pserver-based CVS check outs or source distributions should also work, but then you cannot check-in your tests directly (see below on how to participate and submit test script in this case).
Stable vs. Current
In principle, it does not matter, whether you're using the Stable release branch(es) or the Current development head(s). Use what ever you like best (or have already running).
Test directories
Your tests must be located (and hence check in) in different directories depending on the MonetDB version you're using:
Stable branch: MonetDB 4.8.*: MonetDB/tests/BugDay_2005-10-06_4.8/Tests/ SQL 2.8.*: sql/src/test/BugDay_2005-10-06_2.8/Tests/ XQuery 0.8.*: pathfinder/tests/BugDay_2005-10-06_0.8/Tests/
Current head: MonetDB 4.9.3: MonetDB/tests/BugDay_2005-10-06_4.9.3/Tests/ SQL 2.9.3: sql/src/test/BugDay_2005-10-06_2.9.3/Tests/ XQuery 0.9.3: pathfinder/tests/BugDay_2005-10-06_0.9.3/Tests/
The reason for having different direcories is mainly convenience with check-ins and later propagations.
The Task/Procedure
Claiming a bug
As mentioned above, you simply claim a bug by adding a comment like
"BugDay_2005-10-06: Claimed [by <name>]."
to the respective bug report on SF.
Checking for existence
Once claimed, you first check, whether there is already a test script for this bug available in one of
MonetDB/tests/BugsViaSourgeforce/Tests/ sql/src/test/bugs/Tests/ pathfinder/tests/BugsViaSourgeforce/Tests/
If so, you should indicate this be adding a comment like
BugDay_2005-10-06: Done. test script available as MonetDB/tests/BugsViaSourgeforce/Tests/ID.602271.*
to the respective bug report on SF, and you're done with this bug, and can proceed to the next one.
If there is no test script, yet, you need to create one:
Naming conventions
Names of test scripts must include a descriptive part (of reasonable length) and the SF bug ID, like
"return_types_of_head_and_tail.SF-701641.*",
i.e., the pattern is
<descriptive-name>.SF-<BugID>[.<extension>]
Creating test scripts
As mentioned in my first email, most bug reports come with some MIL-, SQL-, XQuery-code, or some other description/instructions how to reproduce the bug. The basic task is to put this into a proper "Mtest-compliant" test script. In most cases, this is just a .milS, .sql, or .xq file. See http://monetdb.cwi.nl/monet/src/testing/README for details about Mtest.py . In case of doubt, please feel free to contact any of your colleagues that might be more familiar with Mtest --- or me of course ;-) Of course, you're welcome to extend the test if appropriate.
In case it is obvious that the bug does not require a test script (e.g., compilation, installation, or web site bugs), please indicate this by adding a comment like
BugDay_2005-10-06: Done.
No test script added, because <good reason>.
to the respective bug report on SF, and proceed with the next bug.
If a test script is required, but you don't know how to make it (for this particular bug), please feel free to ask for help (or "release" the bug report, again).
Running Mtest
Once you made the test script, you need to run the test. In case you're using `source conf/conf.bash` to setup your MonetDB environment, you can simply call one of
Mtest_MonetDB <descriptive-name>.SF-<BugID> Mtest_sql <descriptive-name>.SF-<BugID> Mtest_pathfinder <descriptive-name>.SF-<BugID>
in the respective directory as listed above. See `Mtest.py --help` for command line options.
Note: Do NOT change or check-in the "All" file!
This is not required to run tests that are explicitly given on Mtest's command line. I'll will fill the "All" files at the end of the BugDay.
Creating / fixing stable output
After running Mtest, you need to check the output by pointing your browser to the file given at the end of Mtest's console output.
In case the output is correct and as expected, you can approve it by running
Mapprove_MonetDB <descriptive-name>.SF-<BugID> Mapprove_sql <descriptive-name>.SF-<BugID> Mapprove_pathfinder <descriptive-name>.SF-<BugID>
in the respective directory as listed above.
In case running the test failed, please fix the test and start again with running Mtest.
In case the output is not correct or no as expected, you first approve the output as described above, but then edit the files
<descriptive-name>.SF-<BugID>.stable.out <descriptive-name>.SF-<BugID>.stable.err
by hand to create the correct/expected output.
In any case, you then re-run Mtest, and check the output by pointing your browser to the file given at the end of Mtest's console output.
In case everything is "green" (i.e., OK), you can check-in all files related to the test (including the stable output!), indicate the success by adding a comment like
BugDay_2005-10-06: Done: test script added as tests/BugDay_2005-10-06_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*
to the respective bug report on SF, and proceed with the next bug.
In case there is still a red 'X' (because MonetDB does not produce the correct/expected output), you also check-in all files, but indicate the problem by a comment like
BugDay_2005-10-06: Done: test script added as tests/BugDay_2005-10-06_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*
Note: Test fails!
<possibly a description of the failure>
to the respective bug report on SF, and re-open this bug report, before proceeding with the next bug.
Anonymous CVS check outs and source distributions
All the above holds, except from the CVS check-in of the new test scripts. Instead, you send me (Stefan.Manegold@cwi.nl) a .tar[.gz] or .zip file including all your new test scripts by the end of your BugDay.
In case you intend to participate without CVS access, please let me know so as soon as you start.
Pooh, I guess that's it for now. I hope, I didn't forget anything --- if so, feel free to ask! (via email, or personally tomorrow at CWI).
cu tomorrow,
Stefan
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
On Thu, Oct 06, 2005 at 10:05:58AM +0200, Sjoerd Mullender wrote:
Niels Nes wrote:
On Wed, Oct 05, 2005 at 06:43:17PM +0200, Stefan Manegold wrote:
Dear all,
as promised, here are some more detailed informations for tomorrow.
General issues
To "decentralize" processing and prevent me from becoming the "bottleneck"/"hot spot", I not to assign tests to you; rather, each of you claims the bug that he/she is going to prepare a test script for by adding a comment like
"BugDay_2005-10-06: Claimed [by <name>]."
to the respective bug report; adding a name is optional if you claim a bug for yourself; if you claim a bug for someone who does not have an account with SourceForge (and is not willing to request one) and hence cannot add comments to the bug tracker, adding his/her name is mandatory. See http://sourceforge.net/tracker/?group_id=56967&atid=482468 for a complete list of bug report, and don't for get to set your selection as follows:
Assignee: Status: Category: Group: Any Any Any Any
Sort By: ID Ascending
Of course, you're welcome to first claim multiple bugs, before actually starting to prepare and check-in the tests.
To avoid that everyone starts (trying) to claim the same bugs, here are some bug IDs as entry points. Each one chooses a different entry point, and from there on processes the bugs in in ascending order of bug ID.
602271 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Okay I simply claim the first 50..
Niels
I'll claim the following 50:
899586 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Then, I'll start with then next package:
920585 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Stefan
929158 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 941693 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 972833 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1010604 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1029099 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1076604 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1119199 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1166456 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1205565 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1214335 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1257463 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1300366 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Please indicate you choice by (group) replying to this mail. I will resolve clashes where necessary. In case there are more than 15 people participating (concurrently), we can easily sub-divide the ranges.
Requirements
Obviously, you need to have MonetDB (plus SQL or XQuery for those bugs) installed and running; preferably an up-to-date ssh-based (non anonymous) CVS check out of either the release branches (MonetDB_4-8, SQL_2-8, XQuery_0-8) or the respective CVS heads. Anonymous (pserver-based CVS check outs or source distributions should also work, but then you cannot check-in your tests directly (see below on how to participate and submit test script in this case).
Stable vs. Current
In principle, it does not matter, whether you're using the Stable release branch(es) or the Current development head(s). Use what ever you like best (or have already running).
Test directories
Your tests must be located (and hence check in) in different directories depending on the MonetDB version you're using:
Stable branch: MonetDB 4.8.*: MonetDB/tests/BugDay_2005-10-06_4.8/Tests/ SQL 2.8.*: sql/src/test/BugDay_2005-10-06_2.8/Tests/ XQuery 0.8.*: pathfinder/tests/BugDay_2005-10-06_0.8/Tests/
Current head: MonetDB 4.9.3: MonetDB/tests/BugDay_2005-10-06_4.9.3/Tests/ SQL 2.9.3: sql/src/test/BugDay_2005-10-06_2.9.3/Tests/ XQuery 0.9.3: pathfinder/tests/BugDay_2005-10-06_0.9.3/Tests/
The reason for having different direcories is mainly convenience with check-ins and later propagations.
The Task/Procedure
Claiming a bug
As mentioned above, you simply claim a bug by adding a comment like
"BugDay_2005-10-06: Claimed [by <name>]."
to the respective bug report on SF.
Checking for existence
Once claimed, you first check, whether there is already a test script for this bug available in one of
MonetDB/tests/BugsViaSourgeforce/Tests/ sql/src/test/bugs/Tests/ pathfinder/tests/BugsViaSourgeforce/Tests/
If so, you should indicate this be adding a comment like
BugDay_2005-10-06: Done. test script available as MonetDB/tests/BugsViaSourgeforce/Tests/ID.602271.*
to the respective bug report on SF, and you're done with this bug, and can proceed to the next one.
If there is no test script, yet, you need to create one:
Naming conventions
Names of test scripts must include a descriptive part (of reasonable length) and the SF bug ID, like
"return_types_of_head_and_tail.SF-701641.*",
i.e., the pattern is
<descriptive-name>.SF-<BugID>[.<extension>]
Creating test scripts
As mentioned in my first email, most bug reports come with some MIL-, SQL-, XQuery-code, or some other description/instructions how to reproduce the bug. The basic task is to put this into a proper "Mtest-compliant" test script. In most cases, this is just a .milS, .sql, or .xq file. See http://monetdb.cwi.nl/monet/src/testing/README for details about Mtest.py . In case of doubt, please feel free to contact any of your colleagues that might be more familiar with Mtest --- or me of course ;-) Of course, you're welcome to extend the test if appropriate.
In case it is obvious that the bug does not require a test script (e.g., compilation, installation, or web site bugs), please indicate this by adding a comment like
BugDay_2005-10-06: Done.
No test script added, because <good reason>.
to the respective bug report on SF, and proceed with the next bug.
If a test script is required, but you don't know how to make it (for this particular bug), please feel free to ask for help (or "release" the bug report, again).
Running Mtest
Once you made the test script, you need to run the test. In case you're using `source conf/conf.bash` to setup your MonetDB environment, you can simply call one of
Mtest_MonetDB <descriptive-name>.SF-<BugID> Mtest_sql <descriptive-name>.SF-<BugID> Mtest_pathfinder <descriptive-name>.SF-<BugID>
in the respective directory as listed above. See `Mtest.py --help` for command line options.
Note: Do NOT change or check-in the "All" file!
This is not required to run tests that are explicitly given on Mtest's command line. I'll will fill the "All" files at the end of the BugDay.
Creating / fixing stable output
After running Mtest, you need to check the output by pointing your browser to the file given at the end of Mtest's console output.
In case the output is correct and as expected, you can approve it by running
Mapprove_MonetDB <descriptive-name>.SF-<BugID> Mapprove_sql <descriptive-name>.SF-<BugID> Mapprove_pathfinder <descriptive-name>.SF-<BugID>
in the respective directory as listed above.
In case running the test failed, please fix the test and start again with running Mtest.
In case the output is not correct or no as expected, you first approve the output as described above, but then edit the files
<descriptive-name>.SF-<BugID>.stable.out <descriptive-name>.SF-<BugID>.stable.err
by hand to create the correct/expected output.
In any case, you then re-run Mtest, and check the output by pointing your browser to the file given at the end of Mtest's console output.
In case everything is "green" (i.e., OK), you can check-in all files related to the test (including the stable output!), indicate the success by adding a comment like
BugDay_2005-10-06: Done: test script added as tests/BugDay_2005-10-06_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*
to the respective bug report on SF, and proceed with the next bug.
In case there is still a red 'X' (because MonetDB does not produce the correct/expected output), you also check-in all files, but indicate the problem by a comment like
BugDay_2005-10-06: Done: test script added as tests/BugDay_2005-10-06_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*
Note: Test fails!
<possibly a description of the failure>
to the respective bug report on SF, and re-open this bug report, before proceeding with the next bug.
Anonymous CVS check outs and source distributions
All the above holds, except from the CVS check-in of the new test scripts. Instead, you send me (Stefan.Manegold@cwi.nl) a .tar[.gz] or .zip file including all your new test scripts by the end of your BugDay.
In case you intend to participate without CVS access, please let me know so as soon as you start.
Pooh, I guess that's it for now. I hope, I didn't forget anything --- if so, feel free to ask! (via email, or personally tomorrow at CWI).
cu tomorrow,
Stefan
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
-- Sjoerd Mullender
602271 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Okay I simply claim the first 50..
Niels
I'll claim the following 50:
899586 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Then, I'll start with then next package:
920585 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Stefan
929158 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 941693 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 972833 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Then, I will start with this one (1010604):
1010604 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1029099 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1076604 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1119199 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1166456 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1205565 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1214335 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1257463 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&... 1300366 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
I'll start from offset 150:
929158 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Then I guess I'll just continue from 200
941693 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Sandor
250, c'est moi
Sandor Heman wrote:
Then I guess I'll just continue from 200
941693 http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&...
Sandor
participants (8)
-
Fabian -
Niels Nes -
Roberto Cornacchia -
Sandor Heman -
Sjoerd Mullender -
Stefan Manegold -
Wouter Alink (CWI) -
Ying (Jennie) Zhang