[Monetdb-developers] Re: 2nd MonetDB-Bug-Day: Wed, Nov 9 2005 (tomorrow)

Stefan Manegold Stefan.Manegold at cwi.nl
Tue Nov 8 23:18:18 CET 2005


Dear all,

here are some more informations for the 2nd MonetDB-Bug-Day tomorrow.

Please note, though it's call Bug-*Day*, you don't have to spend the whole
day writing test scripts (though no one would mind if you did ;-)). 
Below you find chunks of 10 bug reports each; so simple claim one chunk at a
time (by group-reply to this email) whenever you feel like you could spend
some minutes on this "community task", check the 10 reports and provide the
test script where necessary/applicable.
I we don't manage to finish 40 chunks tomorrow, everybody is welcome to
pick one any of the next days.

Of course, next to (or instead of) adding test scripts, you're also welcome
to fix open bugs (and add the test script for each bug you fix)!


General issues
==============

As with the first MonetDB-Bug-Day, I split the (remaining 391) bug reports
in chunks. However, this time, they're smaller --- only 10 instead 50
reports --- to allow more "fine-grained" processing:

No  Offsets      BugIDs        #  URL
--------------------------------------
 1  213-219   947600- 949071   7  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=213
 2  220-229   949135- 952758  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=220
 3  230-239   953545- 963620  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=230
 4  240-249   963759- 971158  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=240
 5  330-339  1023645-1025005  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=330
 6  340-349  1025062-1029058  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=340
 7  350-359  1029099-1034613  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=350
 8  360-369  1034922-1037591  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=360
 9  370-379  1037993-1043526  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=370
10  380-389  1044534-1063089  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=380
11  390-399  1065373-1076344  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=390
12  400-409  1076604-1081218  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=400
13  410-419  1081274-1085120  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=410
14  420-429  1085355-1099345  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=420
15  430-439  1100291-1113994  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=430
16  440-449  1114000-1119186  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=440
17  524-529  1190980-1194362   6  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=524
18  530-539  1195753-1201291  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=530
19  540-549  1201604-1205010  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=540
20  550-559  1205565-1206985  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=550
21  560-569  1207029-1207516  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=560
22  570-579  1207625-1208129  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=570
23  580-589  1208599-1210743  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=580
24  590-599  1210748-1212707  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=590
25  600-609  1214335-1218268  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=600
26  610-619  1218281-1227464  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=610
27  620-629  1227493-1237444  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=620
28  630-639  1238228-1240701  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=630
29  640-649  1242164-1256794  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=640
30  650-659  1257463-1263928  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=650
31  660-669  1263967-1274649  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=660
32  670-679  1274819-1281176  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=670
33  680-689  1281996-1292727  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=680
34  690-699  1293196-1298707  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=690
35  700-709  1300366-1312972  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=700
36  710-719  1313008-1314995  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=710
37  720-729  1315339-1324219  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=720
38  730-739  1324575-1334486  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=730
39  740-749  1334536-1347183  10  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=740
40  750-757  1347331-1351516   8  http://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=482468&set=custom&_assigned_to=0&_status=100&_category=100&_group=100&order=artifact_id&sort=ASC&offset=750
                             ---
                             391 

Please pick one chunk at a time and indicate you choice by (group) replying
to this mail. I will resolve clashes where necessary.  I don't expect more
than 40 people participating (concurrently), hence sub-dividing the chunks
should not be necessary ;-)
In case you cannot finish your last chunk by tomorrow night (for whatever
reason), please let me know (via email) which of the reports you finished,
and which you didn't.
(Please see below for details what to do in case your hit a report that
requires a test script, but for which you cannot create a test script.)


Requirements
============

Obviously, you need to have MonetDB (plus SQL or XQuery for those bugs)
installed and running; preferably an up-to-date ssh-based (non anonymous)
CVS check out of either the release branches (MonetDB_4-8, SQL_2-8,
XQuery_0-8) or the respective CVS heads. Anonymous (pserver-based) CVS check
outs or source distributions should also work, but then you cannot check-in
your tests directly (see below on how to participate and submit test script
in this case).

Stable vs. Current
------------------

In principle, it does not matter, whether you're using the Stable release
branch(es) or the Current development head(s). Use what ever you like best
(or have already running).

Test directories
----------------

Your tests must be located (and hence checked in) in different directories
depending on the MonetDB version you're using:

Stable branch:
	MonetDB 4.8.*:    MonetDB/tests/BugDay_2005-11-09_4.8/Tests/
	SQL     2.8.*:     sql/src/test/BugDay_2005-11-09_2.8/Tests/
	XQuery  0.8.*: pathfinder/tests/BugDay_2005-11-09_0.8/Tests/

Current head:
	MonetDB 4.9.3:    MonetDB/tests/BugDay_2005-11-09_4.9.3/Tests/
	SQL     2.9.3:     sql/src/test/BugDay_2005-11-09_2.9.3/Tests/
	XQuery  0.9.3: pathfinder/tests/BugDay_2005-11-09_0.9.3/Tests/

The reason for having different directories is mainly convenience with
check-ins and later propagations.


Codes
=====
 
To indicate the progress (see below for details), please use the following
codes when adding comments to the bug report

code                      explanation
------------------------+---------------------------------------------
RELEASED / REASSIGN     | the "claimer" of the bug wasn't able to create a
                        | script from the bug due to insufficient
                        | knowledge.  Append reason, e.g. "XQuery".
                        | Also set status of closed bugs to "pending".
ALREADY IN TESTWEB      | the script was already in CVS and TestWeb 
                        | (added location of existing test script).
TEST ADDED / SUCCESS    | a script was added (append location & name) and
                        | works correctly.
TEST ADDED / FAILURE    | a script was added, but it fails (bad output).
                        | Re-open closed bug!
TEST SUBMITTED          | a script was made, but sent to Stefan (for
                        | non-CVS participants)
NO TEST / COMPILATION   | no test was added because the bug deals with a
                        | compilation problem. 
NO TEST / DOCUMENTATION | no test was added because the bug deals with a
                        | documentation issue, or misleading help text,
                        | etc.
NO TEST / REDUNDANT     | no test was added, because either the bug itself
                        | is a duplicate, or the test that follows out of
                        | the bug would clearly be duplicating another
                        | existing bug.  When extending another bug to
                        | remove redundancy use TEST ADDED / SUCCESS and
                        | give the name of the already existing updated
                        | bug.

Note: the last action is only meant for experts in their field ;)

An example message on the SF bug could be:

  	BugDay_2005-11-09, <your name>: TEST ADDED / SUCCESS
  	sql/src/tests/bugday/like_join_union.SF-0123345.SQL


The Task/Procedure
==================

Claim a chunk of bugs as described above, and then process each bug as
follows:

Checking for existence
----------------------

Once claimed, you first check, whether there is already a test script for
this bug (i.e., a test script that contains the respective ID in its
filename) available in one of

	MonetDB/tests/BugsViaSourgeforce/Tests/
	sql/src/test/bugs/Tests/
	pathfinder/tests/BugsViaSourgeforce/Tests/

If so, you should indicate this be adding a comment like

	BugDay_2005-11-09, <your name>: ALREADY IN TESTWEB
	MonetDB/tests/BugsViaSourgeforce/Tests/ID.602271.*

to the respective bug report on SF, and you're done with this bug, and can
proceed to the next one.

If there is no test script, yet, you need to create one:

Naming conventions
------------------

Names of test scripts must include a descriptive part (of reasonable length)
and the SF bug ID, like

	"return_types_of_head_and_tail.SF-701641.*",

i.e., the pattern is 

	<descriptive-name>.SF-<BugID>[.<extension>]

Creating test scripts
---------------------

Most bug reports come with some MIL-, SQL-, XQuery-code, or some other
description/instructions how to reproduce the bug. The basic task is to put
this into a proper "Mtest-compliant" test script. In most cases, this is
just a .milS, .sql, or .xq file.
See http://monetdb.cwi.nl/monet/src/testing/README for details about Mtest.py .
In case of doubt, please feel free to contact any of your colleagues that might 
be more familiar with Mtest --- or me of course ;-)
Of course, you're welcome to extend the test if appropriate.

In case it is obvious that the bug does not require a test script (e.g.,
compilation, installation, documentation, or web site bugs), please indicate
this by adding a comment like

	BugDay_2005-11-09, <your name>: NO TEST / <good reason>

to the respective bug report on SF, and proceed with the next bug.

If a test script is required, but you don't know how to make it (for this
particular bug), please feel free to ask for help (or "release" the bug
report, again).

Running Mtest
-------------

Once you made the test script, you need to run the test. In case you're
using `source conf/conf.bash` to setup your MonetDB environment, you can
simply call one of

	Mtest_MonetDB <descriptive-name>.SF-<BugID>
	Mtest_sql <descriptive-name>.SF-<BugID>
	Mtest_pathfinder <descriptive-name>.SF-<BugID>

in the respective directory as listed above.
See `Mtest.py --help` for command line options.

Note: Do NOT change or check-in the "All" file!

This is not required to run tests that are explicitly given on Mtest's
command line. I'll will fill the "All" files at the end of the BugDay.

Creating / fixing stable output
-------------------------------

After running Mtest, you need to check the output by pointing your browser
to the file given at the end of Mtest's console output.

In case the output is correct and as expected, you can approve it by running

	Mapprove_MonetDB <descriptive-name>.SF-<BugID>
	Mapprove_sql <descriptive-name>.SF-<BugID>
	Mapprove_pathfinder <descriptive-name>.SF-<BugID>

in the respective directory as listed above.
	
In case running the test failed, please fix the test and start again with
running Mtest.

In case the output is not correct or no as expected, you first approve the
output as described above, but then edit the files

	<descriptive-name>.SF-<BugID>.stable.out
	<descriptive-name>.SF-<BugID>.stable.err

by hand to create the correct/expected output.

In any case, you then re-run Mtest, and check the output by pointing your
browser to the file given at the end of Mtest's console output.

In case everything is "green" (i.e., OK), you can check-in all files related
to the test (including the stable output!), indicate the success by adding a
comment like

	BugDay_2005-11-09, <your name>: TEST ADDED / SUCCESS
	tests/BugDay_2005-11-09_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*

to the respective bug report on SF, and proceed with the next bug.

In case there is still a red 'X' (because MonetDB does not produce the
correct/expected output), you also check-in all files, but indicate the
problem by a comment like

	BugDay_2005-11-09, <your name>: TEST ADDED / FAILURE
	tests/BugDay_2005-11-09_4.9.3/Tests/return_types_of_head_and_tail.SF-701641.*

	<possibly a description of the failure>

to the respective bug report on SF, and *re-open* this bug report,
before proceeding with the next bug.


Anonymous CVS check outs and source distributions
=================================================

All the above holds, except from the CVS check-in of the new test scripts.
Instead, you send me (Stefan.Manegold at cwi.nl) a .tar[.gz] or .zip file
including all your new test scripts by the end of your BugDay.

In case you intend to participate without CVS access, please let me know so
as soon as you start.


Pooh, I guess that's it for now.
I hope, I didn't forget anything --- if so, feel free to ask!
(via email, skype, or personally tomorrow at CWI).

cu tomorrow,

Stefan


On Fri, Nov 04, 2005 at 03:34:19PM +0100, Stefan Manegold wrote:
> 
> (some of you might already have see this one coming; 
>  cf. http://www.cwi.nl/htbin/cwwwi/agenda?datum=2005_11+dag=09 ;-))
> 
> 
> Dear all,
> 
> given the success of the 1st MonetDB-Bug-Day on Oct 06 2005,
> 
> 8 participants managed to check 347 of the then 721 bug reports, added 97
> new test scripts, re-opened some bugs that still/again fail and filed 9 new
> bug reports; see
> https://sourceforge.net/mailarchive/forum.php?thread_id=8403629&forum_id=39803
> for details,
> 
> there will by a 2nd MonetDB-Bug-Day to (hopefully) check the remaining 347
> bug reports (plus those filed since Oct 06 2005), and produce proper test
> scripts for our automatic Nightly Multi-Platform Regression Test System (aka
> "TestWeb"; cf. http://monetdb.cwi.nl/Development/TestWeb/):
> 
> 
> When:   Wednesday, Nov 9, 2005, "all day" (or as much time as you can spent)
> 
> Where:  Mainly at CWI, but you're also welcome to join "virtually" working
>         from home or any other place in the world;
>         for those who (again) want to turn the Bug-Day into a Bug-"Party", I
>         reserved our meeting room (C0,01) from 8:00 - 23:00 --- you need to
>         bring your own "equipment" though ...
> 
> Who:    Primarily everyone who is in touch with MonetDB, and/or one of its
>         companions/front ends (SQL, XQuery, etc.), has rights to check-in to
>         the MonetDB CVS repository on SourceForge, and is reasonably
>         familiar with "Mtest.py" (cf.
>         http://monetdb.cwi.nl/monet/src/testing/README).
> 
> What:   I will prepare a list of all bugs that have no test script, yet, and
> 	chop this in chunks, and assign each chunk to one participant so
> 	that [s]he can check them, and prepare the respective test scripts.
> 
>         More detailed instructions can be found at 
> 	https://sourceforge.net/mailarchive/forum.php?thread_id=8403629&forum_id=39803
> 	and/or will follow before next Wednesday.
> 
> Note:	Though any help is appreciated, no one is "obliged" to help.
> 	Especially, there is no need to spent the whole day (though nine
> 	will complain if you do ;-)), anyone can join-in and leave whenever
> 	[s]he wants.
> 
> 	For those of you participating at CWI, there will be free snacks and
> 	soft-drinks during the day (for the time you're "active"; as long as
> 	supplies lasts).
> 	Furthermore, there will be "real" drinks at the end of the Bug-Day
> 	(for participants, only; as long as supplies lasts; (maximum) amount
> 	 per person will be proportional to the number of test scripts added).
> 
> 
> I hope to see especially (though not only) participants that I haven't seen
> on the 1st MonetDB-Bug-Day ;-)
> 
> Once again thank you very much for your cooperation!
> 
> 
> CU next Wednesday,
> 
> Stefan
> 
> 
> ps: Testing & the TestWeb are not (only?) my "toys"; it's "serious
>     business", and helps all of you to (1) get a stable and reliable
>     MonetDB, (2) monitor when your own bug reports get fixed, and (3)
>     prevent the bugs you once thankfully detected, reported and/or fixed
>     from occurring again!
> 
> -- 
> | Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
> | CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
> | 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
> | The Netherlands     | Fax : +31 (20) 592-4312       |

-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4312       |




More information about the developers-list mailing list