[Monetdb-developers] MonetDB Bug-Etiquette
Dear MonetDB-users and -developers,
bug reports and especially test script play a crucial role with maintaining and improving the stability and correct functioning of MonetDB.
However, adding test scripts "a posteriori" as we did recently during the two "MonetDB-Bug-Days" (a third one is still to come before the end of this year!) is a very tedious and time-consuming task, especially since who ever is adding a test script first needs to get familiar with the details of the very bug report.
Since both the user who submits a bug report and the developer(s) who take(s) care of fixing a bug are by definition very familiar with the details of a bug report, it seems more than obvious that they should share the task of adding a test script to the CVS repository while dealing with the respective bug, i.e., *before* closing the bug report.
Ideally, the user/submitter provides the query or script that triggers the bug as well as the expected/correct output with his/her bug report. In case the user/submitter has CVS check-in privileges, he/she could add the test script to the CVS repository him-/herself. Otherwise, the developer/assignee should add the user's test script to the CVS repository. The test script helps both the developer/assignee and the user/submitter to monitor the status of the bug by running "Mtest[.py]" (cf., http://monetdb.cwi.nl/monet/src/testing/README) or by checking the TestWeb (http://monetdb.cwi.nl/Development/TestWeb/).
To coordinate this procedure, I made a first draft for a "MonetDB Bug Etiquette", which you find attached to this mail.
I'd be very pleased, if you could comment on this.
Thank you all very much for your contribution to make MonetDB even better than it already is!
Kind regards,
Stefan
On 21-11-2005 02:38:01 +0100, Stefan Manegold wrote:
Since both the user who submits a bug report and the developer(s) who take(s) care of fixing a bug are by definition very familiar with the details of a bug report, it seems more than obvious that they should share the task of adding a test script to the CVS repository while dealing with the respective bug, i.e., *before* closing the bug report.
It is not obvious that the submitter knows the problem very well to me. Hence, I'd like to reduce the 'options' here, and let the developer add the script when (s)he closes the bug.
User/Submitter:
Please fill in Category and Group.
Feel free to set the Priority if you think the default is no appropriate.
not appropriate
- Please provide a detailed description how to repeat the bug you report, or even a concise test script to do so. Preferably, the test script should also be added to the standard test-suite (in the CVS repository; see "Adding test scripts" below for more details!).
I strongly suggest to drop the CVS thing. Many users don't have it, and it's confusing and more work for devs, see below.
- Please feel free to close your bug report, in case you notice it does not occur any more (your test script in the standard test-suite is your free-of-charge monitor; simply check our "TestWeb" at http://monetdb.cwi.nl/Development/TestWeb/ ;-)). See "Closing bug reports" below for more details!
This is weird. So a bug can suddenly vanish? Yes it can, but then the exact cause of the bug was not found, so it can be gone just by coincidence, which is *not* good. Submitters should only close their bug if they think they made a mistake and that their bug is invalid.
- Once a developer closes your bug report, please double-check whether the solution indeed works for you (simply check the status of your test script in our "TestWeb" at http://monetdb.cwi.nl/Development/TestWeb/ ;-)). If is does not work for you, feel free to re-open the bug report, or file a new one. In either case, please give a detailed description of the remaining or new problem!
Why not let the developer *always* add the test script before closing the bug. Checking that a bug is already there (hoping it complies to the naming conventions as pointed out below) is just some extra hassle. Besides that, IMHO it's just more well organised if the dev adds the script once (s)he fixes the problem and as such the correct output can be determined as well as the test possibly made somewhat more extensive to cover more possible problems. The dev that fixes the problem should ideally have the best knowledge on why something went wrong.
Just my e0.02
On Mon, Nov 21, 2005 at 09:15:59AM +0100, Fabian Groffen wrote:
On 21-11-2005 02:38:01 +0100, Stefan Manegold wrote:
Since both the user who submits a bug report and the developer(s) who take(s) care of fixing a bug are by definition very familiar with the details of a bug report, it seems more than obvious that they should share the task of adding a test script to the CVS repository while dealing with the respective bug, i.e., *before* closing the bug report.
It is not obvious that the submitter knows the problem very well to me. Hence, I'd like to reduce the 'options' here, and let the developer add the script when (s)he closes the bug.
User/Submitter:
Please fill in Category and Group.
Feel free to set the Priority if you think the default is no appropriate.
not appropriate
- Please provide a detailed description how to repeat the bug you report, or even a concise test script to do so. Preferably, the test script should also be added to the standard test-suite (in the CVS repository; see "Adding test scripts" below for more details!).
I strongly suggest to drop the CVS thing. Many users don't have it, and it's confusing and more work for devs, see below.
- Please feel free to close your bug report, in case you notice it does not occur any more (your test script in the standard test-suite is your free-of-charge monitor; simply check our "TestWeb" at http://monetdb.cwi.nl/Development/TestWeb/ ;-)). See "Closing bug reports" below for more details!
This is weird. So a bug can suddenly vanish? Yes it can, but then the exact cause of the bug was not found, so it can be gone just by coincidence, which is *not* good. Submitters should only close their bug if they think they made a mistake and that their bug is invalid.
- Once a developer closes your bug report, please double-check whether the solution indeed works for you (simply check the status of your test script in our "TestWeb" at http://monetdb.cwi.nl/Development/TestWeb/ ;-)). If is does not work for you, feel free to re-open the bug report, or file a new one. In either case, please give a detailed description of the remaining or new problem!
Why not let the developer *always* add the test script before closing the bug. Checking that a bug is already there (hoping it complies to the naming conventions as pointed out below) is just some extra hassle. Besides that, IMHO it's just more well organised if the dev adds the script once (s)he fixes the problem and as such the correct output can be determined as well as the test possibly made somewhat more extensive to cover more possible problems. The dev that fixes the problem should ideally have the best knowledge on why something went wrong.
All okay but your are forgetting one very important problem here. Developers do not have the right attitude towards testing. They construct while for testing a destructive attitude is needed. So leaving it all to the developer as your suggesting will also leave some bugs.
Niels
Just my e0.02
This SF.Net email is sponsored by the JBoss Inc. Get Certified Today Register for a JBoss Training Course. Free Certification Exam for All Training Attendees Through End of 2005. For more info visit: http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On 21-11-2005 10:58:34 +0100, Niels Nes wrote:
All okay but your are forgetting one very important problem here. Developers do not have the right attitude towards testing. They construct while for testing a destructive attitude is needed. So leaving it all to the developer as your suggesting will also leave some bugs.
I don't understand this. The chance that a user has CVS write access is very small. I did not intend to suggest that we shouldn't encourage users to make nice reproduction scripts of their bug(s). On the contrary. I just don't think it's a good idea to have the user check the script in, unless approved by the dev.
Fabian Groffen wrote:
On 21-11-2005 02:38:01 +0100, Stefan Manegold wrote:
Since both the user who submits a bug report and the developer(s) who take(s) care of fixing a bug are by definition very familiar with the details of a bug report, it seems more than obvious that they should share the task of adding a test script to the CVS repository while dealing with the respective bug, i.e., *before* closing the bug report.
It is not obvious that the submitter knows the problem very well to me. Hence, I'd like to reduce the 'options' here, and let the developer add the script when (s)he closes the bug.
I guess it depends on the submitter. For some it is (almost) trivial to create a test script, and in any case, it helps the developper enormously if a small test can be made that shows the problem. In fact I would go so far as to say that for all real bugs (i.e. not compilation issues and typos and such) a small test is essential. I do think it is the developper's responsibility that the test gets added to the test web (that doesn't mean the developper has to do it, it means the developper must make sure it is done).
User/Submitter:
Please fill in Category and Group.
Feel free to set the Priority if you think the default is no appropriate.
not appropriate
- Please provide a detailed description how to repeat the bug you report, or
even a concise test script to do so. Preferably, the test script should also be added to the standard test-suite (in the CVS repository; see "Adding test scripts" below for more details!).
I strongly suggest to drop the CVS thing. Many users don't have it, and it's confusing and more work for devs, see below.
I'm sort of with Fabian here. Maybe the "preferably" should be changed in "if possible".
- Please feel free to close your bug report, in case you notice it does not
occur any more (your test script in the standard test-suite is your free-of-charge monitor; simply check our "TestWeb" at http://monetdb.cwi.nl/Development/TestWeb/ ;-)). See "Closing bug reports" below for more details!
This is weird. So a bug can suddenly vanish? Yes it can, but then the exact cause of the bug was not found, so it can be gone just by coincidence, which is *not* good. Submitters should only close their bug if they think they made a mistake and that their bug is invalid.
Bugs do sometimes disappear because of fixes that were made for some other reason. It is appropriate that the report gets closed in that case. The resolution should then be "Out of Date".
- Once a developer closes your bug report, please double-check whether the
solution indeed works for you (simply check the status of your test script in our "TestWeb" at http://monetdb.cwi.nl/Development/TestWeb/ ;-)). If is does not work for you, feel free to re-open the bug report, or file a new one. In either case, please give a detailed description of the remaining or new problem!
Why not let the developer *always* add the test script before closing the bug. Checking that a bug is already there (hoping it complies to the naming conventions as pointed out below) is just some extra hassle. Besides that, IMHO it's just more well organised if the dev adds the script once (s)he fixes the problem and as such the correct output can be determined as well as the test possibly made somewhat more extensive to cover more possible problems. The dev that fixes the problem should ideally have the best knowledge on why something went wrong.
As I mentioned above, it is the developer's responsibility. That doesn't mean the developer must do it, and if the submitter can create and check in a test script, I would encourage that.
Just my e0.02
This SF.Net email is sponsored by the JBoss Inc. Get Certified Today Register for a JBoss Training Course. Free Certification Exam for All Training Attendees Through End of 2005. For more info visit: http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
On Mon, Nov 21, 2005 at 11:45:35AM +0100, Sjoerd Mullender wrote:
I guess it depends on the submitter. For some it is (almost) trivial to create a test script, and in any case, it helps the developper enormously if a small test can be made that shows the problem. In fact I would go so far as to say that for all real bugs (i.e. not compilation issues and typos and such) a small test is essential. I do think it is the developper's responsibility that the test gets added to the test web (that doesn't mean the developper has to do it, it means the developper must make sure it is done).
That exactly what I intended to say --- apparently, I need to work on the "efficiency" of my words ... ;-)
Users and developers should not be "eminies" blaming each other, rather they should work together, and support each other as good as possible to solve bug with minimal overhead. The user know how to trigger the bug, hence (s)he should provide the developer with as much information as possible to resproduce the bug and thus check whether her/his fixed indeed solve the problem --- and a test query/script is the most efficient way to describe and reproduce a bug. Likewise, the developer is responsible for ensuring that his/her fix solves the problem *permanently*; again, running the users test script/query interatively via Mtest AND automatically every night in our test system is the most efficient/convenient/effective way to do so. Since we cannot assume that every user is familiar with our testing facilities, it is the developer's responsibility that test scripts indeed get added to CVS (and hence the testing system and test web) before a bug is closed.
Both user and developer share the reposibility of verifying that the provided solution indeed solved the problem permanently.
- Please provide a detailed description how to repeat the bug you report, or
even a concise test script to do so. Preferably, the test script should also be added to the standard test-suite (in the CVS repository; see "Adding test scripts" below for more details!).
I strongly suggest to drop the CVS thing. Many users don't have it, and it's confusing and more work for devs, see below.
I'm sort of with Fabian here. Maybe the "preferably" should be changed in "if possible".
fine with me.
- Please feel free to close your bug report, in case you notice it does not
occur any more (your test script in the standard test-suite is your free-of-charge monitor; simply check our "TestWeb" at http://monetdb.cwi.nl/Development/TestWeb/ ;-)). See "Closing bug reports" below for more details!
This is weird. So a bug can suddenly vanish? Yes it can, but then the exact cause of the bug was not found, so it can be gone just by coincidence, which is *not* good. Submitters should only close their bug if they think they made a mistake and that their bug is invalid.
Bugs do sometimes disappear because of fixes that were made for some other reason. It is appropriate that the report gets closed in that case. The resolution should then be "Out of Date".
Exactly. ;-)
- Once a developer closes your bug report, please double-check whether the
solution indeed works for you (simply check the status of your test script in our "TestWeb" at http://monetdb.cwi.nl/Development/TestWeb/ ;-)). If is does not work for you, feel free to re-open the bug report, or file a new one. In either case, please give a detailed description of the remaining or new problem!
Why not let the developer *always* add the test script before closing the bug. Checking that a bug is already there (hoping it complies to the naming conventions as pointed out below) is just some extra hassle. Besides that, IMHO it's just more well organised if the dev adds the script once (s)he fixes the problem and as such the correct output can be determined as well as the test possibly made somewhat more extensive to cover more possible problems. The dev that fixes the problem should ideally have the best knowledge on why something went wrong.
As I mentioned above, it is the developer's responsibility. That doesn't mean the developer must do it, and if the submitter can create and check in a test script, I would encourage that.
This (c|sh)ould have been my words ;-)
Anyway, thank you all very much for your comments so far. I'll try to include them (as well as the comments that are still to come) in the MonetDB-Bug-Etiquette.
Stefan
On 21-11-2005 12:45:48 +0100, Stefan Manegold wrote:
Users and developers should not be "eminies" blaming each other, rather they should work together, and support each other as good as possible to solve bug with minimal overhead. The user know how to trigger the bug, hence (s)he should provide the developer with as much information as possible to resproduce the bug and thus check whether her/his fixed indeed solve the
Sure. We misunderstood each other I think.
Both user and developer share the reposibility of verifying that the provided solution indeed solved the problem permanently.
A user should be 'stimulated' to produce a test script. He/She shouldn't be responsible for in any way if you want to have some decent QA IMHO. Devs are responsible for it to get into CVS and that it is correct. Users are most welcome to help working out such script.
This is weird. So a bug can suddenly vanish? Yes it can, but then the exact cause of the bug was not found, so it can be gone just by coincidence, which is *not* good. Submitters should only close their bug if they think they made a mistake and that their bug is invalid.
Bugs do sometimes disappear because of fixes that were made for some other reason. It is appropriate that the report gets closed in that case. The resolution should then be "Out of Date".
Exactly. ;-)
Then it is a dev that decides that a bug is out of date. Which was my point. I don't think a (regular) user (not a dev reporting a bug to another dev!) should be the one responsible for closing a bug. QA again.
As I mentioned above, it is the developer's responsibility. That doesn't mean the developer must do it, and if the submitter can create and check in a test script, I would encourage that.
This (c|sh)ould have been my words ;-)
I think my second mail should make clear that I wanted to say exactly the same thing.
Stefan Manegold wrote:
To coordinate this procedure, I made a first draft for a "MonetDB Bug Etiquette", which you find attached to this mail.
Two highly relevant links:
http://catb.org/~esr/faqs/smart-questions.html http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
Stefan Manegold wrote:
Test scripts must be named according to the following conventions:
SF-<bug-report-id>.<descriptive-name>.*
While working with the bugs again, I found out why I did it different like this before:
<descriptive-name><separator>SF-<bug-report-id><separator>*
The reason for this is twofold: 1) tab completion goes faster for words. I'm not that good with SF numbers, sorry. 2) in testweb, the SF bug number will be aligned to the right this way, and hence is easier to read (as well as the first word of the description is easier to find this way).
Hence, I'd strongly suggest (I actually propose) to use the format as used for BugDay and as was in use before.
On Thu, Nov 24, 2005 at 06:04:16PM +0100, Fabian Groffen wrote:
Stefan Manegold wrote:
Test scripts must be named according to the following conventions:
SF-<bug-report-id>.<descriptive-name>.*
While working with the bugs again, I found out why I did it different like this before:
<descriptive-name><separator>SF-<bug-report-id><separator>*
The reason for this is twofold:
- tab completion goes faster for words. I'm not that good with SF
numbers, sorry. 2) in testweb, the SF bug number will be aligned to the right this way, and hence is easier to read (as well as the first word of the description is easier to find this way).
Hence, I'd strongly suggest (I actually propose) to use the format as used for BugDay and as was in use before.
as simple as that: I (almost, see below) totally agree, "aplogise" for not thinking about this when proposing the above, and will fix the comment's in the all files as well as rename the (few) existing tests asap.
I propose to set <separator> fixed to '.' .
Stefan
Stefan Manegold wrote:
Hence, I'd strongly suggest (I actually propose) to use the format as used for BugDay and as was in use before.
as simple as that: I (almost, see below) totally agree, "aplogise" for not thinking about this when proposing the above, and will fix the comment's in the all files as well as rename the (few) existing tests asap.
I didn't object at first because I didn't see any problem with it at first ;) Just when working with it, I experienced 'difficulties'.
I propose to set <separator> fixed to '.' .
Ist mir egal. I really don't care whatever it is, hence the choice for <separator> ;) As long as it isn't some hard to generate char, I'm fine with it.
participants (4)
-
Fabian Groffen -
Niels Nes -
Sjoerd Mullender -
Stefan Manegold