MonetDB: default - rewrote result parser to make it eat multiple...

Gijs Molenaar g.j.molenaar at uva.nl
Tue Mar 12 11:08:47 CET 2013


Hi Stefan,

yep sorry, guilty, I keep forgetting that adding a test changes the test
result output, I should know that by now.

After a long day of hacking code my brain doesn't function as good as I
would like to.

 - Gijs


On 12/03/13 10:39, Stefan Manegold wrote:
> Dear Gijs,
>    
> thank you very much for continuously taking care of MonetDB's python
> interface / API!
> We very muchappreciate your work and contribution!
>    
> 
> However, we'd appreciate even more, it you could also keep the stable output
> up-to-date whenever you change / extend you test or your changes intendedly
> change the output of your tests.
> 
> See
> http://monetdb.cwi.nl/testweb/web/status.php
> http://monetdb.cwi.nl/testweb/web/testgrid.php?serial=47169:7d1c376e995f&order=platform,arch,compiler
> http://monetdb.cwi.nl/testweb/web/testgrid.php?serial=47169:7d1c376e995f&order=platform,arch,compiler&targets=GNU-Darwin-i386-propcheck,GNU-Darwin-powerpc,GNU-Darwin-x86_64-propcheck,GNU-Fedora-i386-propcheck-dbfarm,GNU-Fedora-x86_64-dist,GNU-Fedora-x86_64-propcheck-assert,GNU-Fedora-x86_64-propcheck-dbfarm,GNU-Fedora-x86_64-propcheck-oid32-dbfarm,GNU-Fedora-x86_64-propcheck-thrs=1,GNU-Fedora-x86_64-propcheck-thrs=32,GNU-Fedora-x86_64-propcheck-thrs=53,GNU-Fedora-x86_64-thrs=1,GNU-FreeBSD-x86_64-propcheck,GNU-Gentoo-powerpc-propcheck-dbfarm,GNU-OpenIndiana-x86_64-propcheck,GNU-Solaris-sparc-dbfarm,GNU-Solaris-sparcv9-dbfarm,GNU-Solaris-sparcv9-oid32-dbfarm,GNU-Ubuntu-i386-propcheck,Int-Fedora-x86_64-propcheck-oid32,Int-Fedora-x86_64-propcheck,Int-Windows2008-x86_64-propcheck-oid32,Int-Windows7-x86_64-propcheck,Int-WindowsXP-i386-propcheck&module=sql&tstlimit=sql/test/mapi
> http://monetdb.cwi.nl/testweb/web/showtestoutput.php?serial=47169:7d1c376e995f&target=GNU-Fedora-x86_64-propcheck-assert&module=sql&test=sql%2Ftest%2Fmapi%2Fpython2_test_monetdb_sql&which=err
> http://monetdb.cwi.nl/testweb/web/testchange.php?test=47169:7d1c376e995f/GNU-Fedora-x86_64-propcheck-assert/sql/mTests/sql%2Ftest%2Fmapi%2Fpython2_test_monetdb_sql
> 
> 
> If nothing else, simply go to .../MonetDB/sql/test/mapi/Tests/ and run
> `Mtest.py` to run (among others) your pyhton test (preferably on a system
> with both Python2 & python3 installed, and follow the instructions.
> 
> See also `Mtest.py --help`, `Mapprove.py --help` and
> http://dev.monetdb.org/hg/MonetDB/file/tip/testing/README
> 
> 
> Thanks!
> 
> Stefan
> 
> 
> On Mon, Mar 11, 2013 at 06:44:43PM +0100, Gijs Molenaar wrote:
>> Changeset: 39b2905626b8 for MonetDB
>> URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=39b2905626b8
>> Modified Files:
>> 	clients/python2/monetdb/sql/cursors.py
>> 	clients/python2/test/capabilities.py
>> 	clients/python3/monetdb/sql/cursors.py
>> 	clients/python3/test/capabilities.py
>> Branch: default
>> Log Message:
>>
>> rewrote result parser to make it eat multiple resultsets
>>
>>
>> diffs (truncated from 463 to 300 lines):
>>
>> diff --git a/clients/python2/monetdb/sql/cursors.py b/clients/python2/monetdb/sql/cursors.py
>> --- a/clients/python2/monetdb/sql/cursors.py
>> +++ b/clients/python2/monetdb/sql/cursors.py
>> @@ -380,133 +380,99 @@ class Cursor(object):
>>          if not block:
>>              block = ""
>>  
>> -        lines = block.split("\n")
>> -        firstline = lines[0]
>> +        for line in block.split("\n"):
>> +            if line.startswith(mapi.MSG_INFO):
>> +                logger.info(line[1:])
>> +                self.messages.append((Warning, line[1:]))
>>  
>> -        while firstline.startswith(mapi.MSG_INFO):
>> -            logger.info(firstline[1:])
>> -            self.messages.append((Warning, firstline[1:]))
>> -            lines = lines[1:]
>> -            firstline = lines[0]
>> +            elif line.startswith(mapi.MSG_QTABLE):
>> +                (self.__query_id, rowcount, columns, tuples) = line[2:].split()
>>  
>> -        if firstline.startswith(mapi.MSG_QTABLE):
>> -            (id, rowcount, columns, tuples) = firstline[2:].split()
>> -            columns = int(columns)   # number of columns in result
>> -            rowcount = int(rowcount) # total number of rows
>> -            tuples = int(tuples)     # number of rows in this set
>> -            rows = []
>> +                columns = int(columns)   # number of columns in result
>> +                self.rowcount = int(rowcount)  # total number of rows
>> +                tuples = int(tuples)     # number of rows in this set
>> +                self.__rows = []
>>  
>> -            # set up fields for description
>> -            table_name = [None]*columns
>> -            column_name = [None]*columns
>> -            type_ = [None]*columns
>> -            display_size = [None]*columns
>> -            internal_size = [None]*columns
>> -            precision = [None]*columns
>> -            scale = [None]*columns
>> -            null_ok = [None]*columns
>> +                # set up fields for description
>> +                table_name = [None] * columns
>> +                column_name = [None] * columns
>> +                type_ = [None] * columns
>> +                display_size = [None] * columns
>> +                internal_size = [None] * columns
>> +                precision = [None] * columns
>> +                scale = [None] * columns
>> +                null_ok = [None] * columns
>> +                typesizes = [(0, 0)] * columns
>> +                self.__offset = 0
>> +                self.lastrowid = None
>>  
>> -            typesizes = [(0,0)]*columns
>> +            elif line.startswith(mapi.MSG_HEADER):
>> +                (data, identity) = line[1:].split("#")
>> +                values = [x.strip() for x in data.split(",")]
>> +                identity = identity.strip()
>>  
>> -            for line in lines[1:]:
>> -                if line.startswith(mapi.MSG_HEADER):
>> -                    (data, identity) = line[1:].split("#")
>> -                    values = [x.strip() for x in data.split(",")]
>> -                    identity = identity.strip()
>> +                if identity == "table_name":
>> +                    table_name = values
>> +                elif identity == "name":
>> +                    column_name = values
>> +                elif identity == "type":
>> +                    type_ = values
>> +                elif identity == "length":
>> +                    length = values
>> +                elif identity == "typesizes":
>> +                    typesizes = [[int(j) for j in i.split()] for i in values]
>> +                    internal_size = [x[0] for x in typesizes]
>> +                    for num, typeelem in enumerate(type_):
>> +                        if typeelem in ['decimal']:
>> +                            precision[num] = typesizes[num][0]
>> +                            scale[num] = typesizes[num][1]
>> +                else:
>> +                    msg = "unknown header field"
>> +                    self.messages.append((InterfaceError, msg))
>> +                    self.__exception_handler(InterfaceError, msg)
>>  
>> -                    if identity == "table_name":
>> -                        table_name = values   # not used
>> -                    elif identity == "name":
>> -                        column_name = values
>> -                    elif identity == "type":
>> -                        type_ = values
>> -                    elif identity == "length":
>> -                        pass # not used
>> -                    elif identity == "typesizes":
>> -                        typesizes = [[int(j) for j in i.split()] for i in values]
>> -                        internal_size = [x[0] for x in typesizes]
>> -                        for num, typeelem in enumerate(type_):
>> -                            if typeelem in ['decimal']:
>> -                                precision[num] = typesizes[num][0]
>> -                                scale[num] = typesizes[num][1]
>> -                    else:
>> -                        self.messages.append((InterfaceError,
>> -                            "unknown header field"))
>> -                        self.__exception_handler(InterfaceError,
>> -                                "unknown header field")
>> +                self.description = list(zip(column_name, type_, display_size,
>> +                                            internal_size, precision, scale,
>> +                                            null_ok))
>> +                self.__offset = 0
>> +                self.lastrowid = None
>>  
>> -                    self.description = list(zip(column_name, type_,
>> -                        display_size, internal_size, precision, scale, null_ok))
>>  
>> -                if line.startswith(mapi.MSG_TUPLE):
>> -                    values = self.__parse_tuple(line)
>> -                    rows.append(values)
>> +            elif line.startswith(mapi.MSG_TUPLE):
>> +                values = self.__parse_tuple(line)
>> +                self.__rows .append(values)
>>  
>> -                elif line == mapi.MSG_PROMPT:
>> -                    self.__query_id = id
>> -                    self.__rows = rows
>> -                    self.__offset = 0
>> +            elif line.startswith(mapi.MSG_QBLOCK):
>> +                self.__rows = []
>>  
>> -                    self.rowcount = rowcount
>> -                    self.lastrowid = None
>> -                    logger.debug("II store result finished")
>> -                    return
>> -
>> -        elif firstline.startswith(mapi.MSG_QBLOCK):
>> -            rows = []
>> -            for line in lines[1:]:
>> -                if line.startswith(mapi.MSG_TUPLE):
>> -                    values = self.__parse_tuple(line)
>> -                    rows.append(values)
>> -                elif line == mapi.MSG_PROMPT:
>> -                    logger.debug("II store result finished")
>> -                    self.__rows = rows
>> -                    return
>> -
>> -        elif firstline.startswith(mapi.MSG_QSCHEMA):
>> -           if lines[1] == mapi.MSG_PROMPT:
>> +            elif line.startswith(mapi.MSG_QSCHEMA):
>> +                self.__offset = 0
>> +                self.lastrowid = None
>>                  self.__rows = []
>> -                self.__offset = 0
>>                  self.description = None
>>                  self.rowcount = -1
>> -                self.lastrowid = None
>> -                logger.debug("II schema finished")
>> -                return
>>  
>> -        elif firstline.startswith(mapi.MSG_QUPDATE):
>> -           if lines[1] == mapi.MSG_PROMPT:
>> -                (affected, identity) = firstline[2:].split()
>> +            elif line.startswith(mapi.MSG_QUPDATE):
>> +                (affected, identity) = line[2:].split()
>> +                self.__offset = 0
>>                  self.__rows = []
>> -                self.__offset = 0
>>                  self.description = None
>>                  self.rowcount = int(affected)
>>                  self.lastrowid = int(identity)
>>                  self.__query_id = -1
>> -                logger.debug("II update finished")
>> +
>> +            elif line.startswith(mapi.MSG_QTRANS):
>> +                self.__offset = 0
>> +                self.lastrowid = None
>> +                self.__rows = []
>> +                self.description = None
>> +                self.rowcount = -1
>> +
>> +            elif line.startswith(mapi.MSG_PROMPT):
>>                  return
>>  
>> -        elif firstline.startswith(mapi.MSG_ERROR):
>> -            self.__exception_handler(ProgrammingError, firstline[1:])
>> -
>> -        elif firstline.startswith(mapi.MSG_QTRANS):
>> -           if lines[1] == mapi.MSG_PROMPT:
>> -                self.__rows = []
>> -                self.__offset = 0
>> -                self.description = None
>> -                self.rowcount = -1
>> -                self.lastrowid = None
>> -                logger.debug("II transaction finished")
>> -                return
>> -
>> -        elif firstline.startswith(mapi.MSG_PROMPT):
>> -            self.__query_id = -1
>> -            self.__rows = []
>> -            self.__offset = 0
>> -
>> -            self.rowcount = 0
>> -            self.lastrowid = None
>> -            logger.debug("II empty response, assuming everything is ok")
>> -            return
>> +            elif line.startswith(mapi.MSG_ERROR):
>> +                self.__exception_handler(ProgrammingError, line[1:])
>>  
>>          # you are not supposed to be here
>>          self.__exception_handler(InterfaceError, "Unknown state, %s" % block)
>> diff --git a/clients/python2/test/capabilities.py b/clients/python2/test/capabilities.py
>> --- a/clients/python2/test/capabilities.py
>> +++ b/clients/python2/test/capabilities.py
>> @@ -352,4 +352,16 @@ class DatabaseTest(unittest.TestCase):
>>          t = ["list", "test"]
>>          self.assertRaises(ProgrammingError, self.db_module.monetize.convert, t)
>>          self.db_module.monetize.mapping[list] = str
>> -        self.assertEqual(self.db_module.monetize.convert(t), "['list', 'test']")
>> \ No newline at end of file
>> +        self.assertEqual(self.db_module.monetize.convert(t), "['list', 'test']")
>> +
>> +    def test_multiple_queries(self):
>> +        table1 = self.new_table_name()
>> +        table2 = table1[:-1] + 'bla"'
>> +        self.cursor.execute("create table %s (a int)" % table1)
>> +        self.cursor.execute("create table %s (a int, b int)" % table2)
>> +        self.cursor.execute("insert into %s VALUES (100)" % table1)
>> +        self.cursor.execute("insert into %s VALUES (50, 50)" % table2)
>> +        self.cursor.execute('select * from %s; select * from %s;' %
>> +                            (table1, table2))
>> +        result = self.cursor.fetchall()
>> +        self.assertEqual(result, [(50, 50)])
>> \ No newline at end of file
>> diff --git a/clients/python3/monetdb/sql/cursors.py b/clients/python3/monetdb/sql/cursors.py
>> --- a/clients/python3/monetdb/sql/cursors.py
>> +++ b/clients/python3/monetdb/sql/cursors.py
>> @@ -381,133 +381,99 @@ class Cursor(object):
>>          if not block:
>>              block = ""
>>  
>> -        lines = block.split("\n")
>> -        firstline = lines[0]
>> +        for line in block.split("\n"):
>> +            if line.startswith(mapi.MSG_INFO):
>> +                logger.info(line[1:])
>> +                self.messages.append((Warning, line[1:]))
>>  
>> -        while firstline.startswith(mapi.MSG_INFO):
>> -            logger.info(firstline[1:])
>> -            self.messages.append((Warning, firstline[1:]))
>> -            lines = lines[1:]
>> -            firstline = lines[0]
>> +            elif line.startswith(mapi.MSG_QTABLE):
>> +                (self.__query_id, rowcount, columns, tuples) = line[2:].split()
>>  
>> -        if firstline.startswith(mapi.MSG_QTABLE):
>> -            (id, rowcount, columns, tuples) = firstline[2:].split()
>> -            columns = int(columns)   # number of columns in result
>> -            rowcount = int(rowcount) # total number of rows
>> -            tuples = int(tuples)     # number of rows in this set
>> -            rows = []
>> +                columns = int(columns)   # number of columns in result
>> +                self.rowcount = int(rowcount)  # total number of rows
>> +                tuples = int(tuples)     # number of rows in this set
>> +                self.__rows = []
>>  
>> -            # set up fields for description
>> -            table_name = [None]*columns
>> -            column_name = [None]*columns
>> -            type_ = [None]*columns
>> -            display_size = [None]*columns
>> -            internal_size = [None]*columns
>> -            precision = [None]*columns
>> -            scale = [None]*columns
>> -            null_ok = [None]*columns
>> +                # set up fields for description
>> +                table_name = [None] * columns
>> +                column_name = [None] * columns
>> +                type_ = [None] * columns
>> +                display_size = [None] * columns
>> +                internal_size = [None] * columns
>> +                precision = [None] * columns
>> +                scale = [None] * columns
>> +                null_ok = [None] * columns
>> +                typesizes = [(0, 0)] * columns
>> +                self.__offset = 0
>> +                self.lastrowid = None
>>  
>> -            typesizes = [(0,0)]*columns
>> +            elif line.startswith(mapi.MSG_HEADER):
>> +                (data, identity) = line[1:].split("#")
>> +                values = [x.strip() for x in data.split(",")]
>> +                identity = identity.strip()
>>  
>> -            for line in lines[1:]:
>> -                if line.startswith(mapi.MSG_HEADER):
>> -                    (data, identity) = line[1:].split("#")
>> -                    values = [x.strip() for x in data.split(",")]
>> -                    identity = identity.strip()
>> +                if identity == "table_name":
>> +                    table_name = values
>> _______________________________________________
>> checkin-list mailing list
>> checkin-list at monetdb.org
>> http://mail.monetdb.org/mailman/listinfo/checkin-list
> 


-- 
Gijs Molenaar
http://www.astro.uva.nl/people/gijs-molenaar/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 263 bytes
Desc: OpenPGP digital signature
URL: <http://www.monetdb.org/pipermail/developers-list/attachments/20130312/a1a7d4a4/attachment.sig>


More information about the developers-list mailing list