[Monetdb-developers] Re: TTT goes MonetDB-Bug-Day

Stefan Manegold Stefan.Manegold at cwi.nl
Wed Oct 5 18:43:17 CEST 2005


Dear all,

as promised, here are some more detailed informations for tomorrow.

General issues
==============

To "decentralize" processing and prevent me from becoming the
"bottleneck"/"hot spot", I not to assign tests to you; rather, each of you
claims the bug that he/she is going to prepare a test script for
by adding a comment like

	"BugDay_2005-10-06: Claimed [by <name>]."

to the respective bug report; adding a name is optional if you claim a bug
for yourself; if you claim a bug for someone who does not have an account
with SourceForge (and is not willing to request one) and hence cannot add
comments to the bug tracker, adding his/her name is mandatory.
See http://sourceforge.net/tracker/?group_id=56967&atid=482468 for a
complete list of bug report, and don't for get to set your selection as
follows:

	Assignee:	Status:		Category:	Group:
	Any		Any		Any		Any

	Sort By:	ID	Ascending

Of course, you're welcome to first claim multiple bugs, before actually
starting to prepare and check-in the tests.

To avoid that everyone starts (trying) to claim the same bugs, here are some
bug IDs as entry points. Each one chooses a different entry point, and from
there on processes the bugs in in ascending order of bug ID.

 602271	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=0
 899586	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=50
 920585	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=100
 929158	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=150
 941693	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=200
 972833	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=250
1010604	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=300
1029099	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=350
1076604	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=400
1119199	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=450
1166456	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=500
1205565	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=550
1214335	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=600
1257463	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=650
1300366	http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=700

Please indicate you choice by (group) replying to this mail. I will resolve
clashes where necessary.  In case there are more than 15 people
participating (concurrently), we can easily sub-divide the ranges.


Requirements
============

Obviously, you need to have MonetDB (plus SQL or XQuery for those bugs)
installed and running; preferably an up-to-date ssh-based (non anonymous)
CVS check out of either the release branches (MonetDB_4-8, SQL_2-8,
XQuery_0-8) or the respective CVS heads. Anonymous (pserver-based CVS check
outs or source distributions should also work, but then you cannot check-in
your tests directly (see below on how to participate and submit test script
in this case).

Stable vs. Current
------------------

In principle, it does not matter, whether you're using the Stable release
branch(es) or the Current development head(s). Use what ever you like best
(or have already running).

Test directories
----------------

Your tests must be located (and hence check in) in different directories
depending on the MonetDB version you're using:

Stable branch:
	MonetDB 4.8.*:    MonetDB/tests/BugDay_2005-10-06_4.8/Tests/
	SQL     2.8.*:        sql/src/test/BugDay_2005-10-06_2.8/Tests/
	XQuery  0.8.*: pathfinder/tests/BugDay_2005-10-06_0.8/Tests/

Current head:
	MonetDB 4.9.3:    MonetDB/tests/BugDay_2005-10-06_4.9.3/Tests/
	SQL     2.9.3:        sql/src/test/BugDay_2005-10-06_2.9.3/Tests/
	XQuery  0.9.3: pathfinder/tests/BugDay_2005-10-06_0.9.3/Tests/

The reason for having different direcories is mainly convenience with
check-ins and later propagations.


The Task/Procedure
==================

Claiming a bug
--------------

As mentioned above, you simply claim a bug by adding a comment like

	"BugDay_2005-10-06: Claimed [by <name>]."

to the respective bug report on SF.

Checking for existence
----------------------

Once claimed, you first check, whether there is already a test script for
this bug available in one of

	MonetDB/tests/BugsViaSourgeforce/Tests/
	sql/src/test/bugs/Tests/
	pathfinder/tests/BugsViaSourgeforce/Tests/

If so, you should indicate this be adding a comment like

	BugDay_2005-10-06: Done.
	test script available as
	MonetDB/tests/BugsViaSourgeforce/Tests/ID.602271.*

to the respective bug report on SF, and you're done with this bug, and can
proceed to the next one.

If there is no test script, yet, you need to create one:

Naming conventions
------------------

Names of test scripts must include a descriptive part (of reasonable length)
and the SF bug ID, like

	"return_types_of_head_and_tail.SF-701641.*",

i.e., the pattern is 

	<descriptive-name>.SF-<BugID>[.<extension>]

Creating test scripts
---------------------

As mentioned in my first email, most bug reports come with some MIL-, SQL-,
XQuery-code, or some other description/instructions how to reproduce the
bug. The basic task is to put this into a proper "Mtest-compliant" test script.
In most cases, this is just a .milS, .sql, or .xq file.
See http://monetdb.cwi.nl/monet/src/testing/README for details about Mtest.py .
In case of doubt, please feel free to contact any of your colleagues that might 
be more familiar with Mtest --- or me of course ;-)
Of course, you're welcome to extend the test if appropriate.

In case it is obvious that the bug does not require a test script (e.g.,
compilation, installation, or web site bugs), please indicate this by adding
a comment like

	BugDay_2005-10-06: Done.
	
	No test script added, because <good reason>.

to the respective bug report on SF, and proceed with the next bug.

If a test script is required, but you don't know how to make it (for this
particular bug), please feel free to ask for help (or "release" the bug
report, again).

Running Mtest
-------------

Once you made the test script, you need to run the test. In case you're
using `source conf/conf.bash` to setup your MonetDB environment, you can
simply call one of

	Mtest_MonetDB <descriptive-name>.SF-<BugID>
	Mtest_sql <descriptive-name>.SF-<BugID>
	Mtest_pathfinder <descriptive-name>.SF-<BugID>

in the respective directory as listed above.
See `Mtest.py --help` for command line options.

Note: Do NOT change or check-in the "All" file!

This is not required to run tests that are explicitly given on Mtest's
command line. I'll will fill the "All" files at the end of the BugDay.

Creating / fixing stable output
-------------------------------

After running Mtest, you need to check the output by pointing your browser
to the file given at the end of Mtest's console output.

In case the output is correct and as expected, you can approve it by running

	Mapprove_MonetDB <descriptive-name>.SF-<BugID>
	Mapprove_sql <descriptive-name>.SF-<BugID>
	Mapprove_pathfinder <descriptive-name>.SF-<BugID>

in the respective directory as listed above.
	
In case running the test failed, please fix the test and start again with
running Mtest.

In case the output is not correct or no as expected, you first approve the
output as described above, but then edit the files

	<descriptive-name>.SF-<BugID>.stable.out
	<descriptive-name>.SF-<BugID>.stable.err

by hand to create the correct/expected output.

In any case, you then re-run Mtest, and check the output by pointing your
browser to the file given at the end of Mtest's console output.

In case everything is "green" (i.e., OK), you can check-in all files related
to the test (including the stable output!), indicate the success by adding a
comment like

	BugDay_2005-10-06: Done:
	test script added as
	tests/BugDay_2005-10-06_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*

to the respective bug report on SF, and proceed with the next bug.

In case there is still a red 'X' (because MonetDB does not produce the
correct/expected output), you also check-in all files, but indicate the
problem by a comment like

	BugDay_2005-10-06: Done:
	test script added as
	tests/BugDay_2005-10-06_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*

	Note: Test fails!
	<possibly a description of the failure>

to the respective bug report on SF, and re-open this bug report,
before proceeding with the next bug.


Anonymous CVS check outs and source distributions
=================================================

All the above holds, except from the CVS check-in of the new test scripts.
Instead, you send me (Stefan.Manegold at cwi.nl) a .tar[.gz] or .zip file
including all your new test scripts by the end of your BugDay.

In case you intend to participate without CVS access, please let me know so
as soon as you start.


Pooh, I guess that's it for now.
I hope, I didn't forget anything --- if so, feel free to ask!
(via email, or personally tomorrow at CWI).

cu tomorrow,

Stefan

--
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4312       |




More information about the developers-list mailing list