MonetDB: default - rewrote result parser to make it eat multiple...

Stefan Manegold Stefan.Manegold at cwi.nl
Tue Mar 12 11:18:35 CET 2013


Hi Gijs,

no problem, I/we are there to remind you ... ;-)

Stefan

----- Original Message -----
> 
> Hi Stefan,
> 
> yep sorry, guilty, I keep forgetting that adding a test changes the
> test
> result output, I should know that by now.
> 
> After a long day of hacking code my brain doesn't function as good as
> I
> would like to.
> 
>  - Gijs
> 
> 
> On 12/03/13 10:39, Stefan Manegold wrote:
> > Dear Gijs,
> >    
> > thank you very much for continuously taking care of MonetDB's
> > python
> > interface / API!
> > We very muchappreciate your work and contribution!
> >    
> > 
> > However, we'd appreciate even more, it you could also keep the
> > stable output
> > up-to-date whenever you change / extend you test or your changes
> > intendedly
> > change the output of your tests.
> > 
> > See
> > http://monetdb.cwi.nl/testweb/web/status.php
> > http://monetdb.cwi.nl/testweb/web/testgrid.php?serial=47169:7d1c376e995f&order=platform,arch,compiler
> > http://monetdb.cwi.nl/testweb/web/testgrid.php?serial=47169:7d1c376e995f&order=platform,arch,compiler&targets=GNU-Darwin-i386-propcheck,GNU-Darwin-powerpc,GNU-Darwin-x86_64-propcheck,GNU-Fedora-i386-propcheck-dbfarm,GNU-Fedora-x86_64-dist,GNU-Fedora-x86_64-propcheck-assert,GNU-Fedora-x86_64-propcheck-dbfarm,GNU-Fedora-x86_64-propcheck-oid32-dbfarm,GNU-Fedora-x86_64-propcheck-thrs=1,GNU-Fedora-x86_64-propcheck-thrs=32,GNU-Fedora-x86_64-propcheck-thrs=53,GNU-Fedora-x86_64-thrs=1,GNU-FreeBSD-x86_64-propcheck,GNU-Gentoo-powerpc-propcheck-dbfarm,GNU-OpenIndiana-x86_64-propcheck,GNU-Solaris-sparc-dbfarm,GNU-Solaris-sparcv9-dbfarm,GNU-Solaris-sparcv9-oid32-dbfarm,GNU-Ubuntu-i386-propcheck,Int-Fedora-x86_64-propcheck-oid32,Int-Fedora-x86_64-propcheck,Int-Windows2008-x86_64-propcheck-oid32,Int-Windows7-x86_64-propcheck,Int-WindowsXP-i386-propcheck&module=sql&tstlimit=sql/test/mapi
> > http://monetdb.cwi.nl/testweb/web/showtestoutput.php?serial=47169:7d1c376e995f&target=GNU-Fedora-x86_64-propcheck-assert&module=sql&test=sql%2Ftest%2Fmapi%2Fpython2_test_monetdb_sql&which=err
> > http://monetdb.cwi.nl/testweb/web/testchange.php?test=47169:7d1c376e995f/GNU-Fedora-x86_64-propcheck-assert/sql/mTests/sql%2Ftest%2Fmapi%2Fpython2_test_monetdb_sql
> > 
> > 
> > If nothing else, simply go to .../MonetDB/sql/test/mapi/Tests/ and
> > run
> > `Mtest.py` to run (among others) your pyhton test (preferably on a
> > system
> > with both Python2 & python3 installed, and follow the instructions.
> > 
> > See also `Mtest.py --help`, `Mapprove.py --help` and
> > http://dev.monetdb.org/hg/MonetDB/file/tip/testing/README
> > 
> > 
> > Thanks!
> > 
> > Stefan
> > 
> > 
> > On Mon, Mar 11, 2013 at 06:44:43PM +0100, Gijs Molenaar wrote:
> >> Changeset: 39b2905626b8 for MonetDB
> >> URL:
> >> http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=39b2905626b8
> >> Modified Files:
> >> 	clients/python2/monetdb/sql/cursors.py
> >> 	clients/python2/test/capabilities.py
> >> 	clients/python3/monetdb/sql/cursors.py
> >> 	clients/python3/test/capabilities.py
> >> Branch: default
> >> Log Message:
> >>
> >> rewrote result parser to make it eat multiple resultsets
> >>
> >>
> >> diffs (truncated from 463 to 300 lines):
> >>
> >> diff --git a/clients/python2/monetdb/sql/cursors.py
> >> b/clients/python2/monetdb/sql/cursors.py
> >> --- a/clients/python2/monetdb/sql/cursors.py
> >> +++ b/clients/python2/monetdb/sql/cursors.py
> >> @@ -380,133 +380,99 @@ class Cursor(object):
> >>          if not block:
> >>              block = ""
> >>  
> >> -        lines = block.split("\n")
> >> -        firstline = lines[0]
> >> +        for line in block.split("\n"):
> >> +            if line.startswith(mapi.MSG_INFO):
> >> +                logger.info(line[1:])
> >> +                self.messages.append((Warning, line[1:]))
> >>  
> >> -        while firstline.startswith(mapi.MSG_INFO):
> >> -            logger.info(firstline[1:])
> >> -            self.messages.append((Warning, firstline[1:]))
> >> -            lines = lines[1:]
> >> -            firstline = lines[0]
> >> +            elif line.startswith(mapi.MSG_QTABLE):
> >> +                (self.__query_id, rowcount, columns, tuples) =
> >> line[2:].split()
> >>  
> >> -        if firstline.startswith(mapi.MSG_QTABLE):
> >> -            (id, rowcount, columns, tuples) =
> >> firstline[2:].split()
> >> -            columns = int(columns)   # number of columns in
> >> result
> >> -            rowcount = int(rowcount) # total number of rows
> >> -            tuples = int(tuples)     # number of rows in this set
> >> -            rows = []
> >> +                columns = int(columns)   # number of columns in
> >> result
> >> +                self.rowcount = int(rowcount)  # total number of
> >> rows
> >> +                tuples = int(tuples)     # number of rows in this
> >> set
> >> +                self.__rows = []
> >>  
> >> -            # set up fields for description
> >> -            table_name = [None]*columns
> >> -            column_name = [None]*columns
> >> -            type_ = [None]*columns
> >> -            display_size = [None]*columns
> >> -            internal_size = [None]*columns
> >> -            precision = [None]*columns
> >> -            scale = [None]*columns
> >> -            null_ok = [None]*columns
> >> +                # set up fields for description
> >> +                table_name = [None] * columns
> >> +                column_name = [None] * columns
> >> +                type_ = [None] * columns
> >> +                display_size = [None] * columns
> >> +                internal_size = [None] * columns
> >> +                precision = [None] * columns
> >> +                scale = [None] * columns
> >> +                null_ok = [None] * columns
> >> +                typesizes = [(0, 0)] * columns
> >> +                self.__offset = 0
> >> +                self.lastrowid = None
> >>  
> >> -            typesizes = [(0,0)]*columns
> >> +            elif line.startswith(mapi.MSG_HEADER):
> >> +                (data, identity) = line[1:].split("#")
> >> +                values = [x.strip() for x in data.split(",")]
> >> +                identity = identity.strip()
> >>  
> >> -            for line in lines[1:]:
> >> -                if line.startswith(mapi.MSG_HEADER):
> >> -                    (data, identity) = line[1:].split("#")
> >> -                    values = [x.strip() for x in data.split(",")]
> >> -                    identity = identity.strip()
> >> +                if identity == "table_name":
> >> +                    table_name = values
> >> +                elif identity == "name":
> >> +                    column_name = values
> >> +                elif identity == "type":
> >> +                    type_ = values
> >> +                elif identity == "length":
> >> +                    length = values
> >> +                elif identity == "typesizes":
> >> +                    typesizes = [[int(j) for j in i.split()] for
> >> i in values]
> >> +                    internal_size = [x[0] for x in typesizes]
> >> +                    for num, typeelem in enumerate(type_):
> >> +                        if typeelem in ['decimal']:
> >> +                            precision[num] = typesizes[num][0]
> >> +                            scale[num] = typesizes[num][1]
> >> +                else:
> >> +                    msg = "unknown header field"
> >> +                    self.messages.append((InterfaceError, msg))
> >> +                    self.__exception_handler(InterfaceError, msg)
> >>  
> >> -                    if identity == "table_name":
> >> -                        table_name = values   # not used
> >> -                    elif identity == "name":
> >> -                        column_name = values
> >> -                    elif identity == "type":
> >> -                        type_ = values
> >> -                    elif identity == "length":
> >> -                        pass # not used
> >> -                    elif identity == "typesizes":
> >> -                        typesizes = [[int(j) for j in i.split()]
> >> for i in values]
> >> -                        internal_size = [x[0] for x in typesizes]
> >> -                        for num, typeelem in enumerate(type_):
> >> -                            if typeelem in ['decimal']:
> >> -                                precision[num] =
> >> typesizes[num][0]
> >> -                                scale[num] = typesizes[num][1]
> >> -                    else:
> >> -                        self.messages.append((InterfaceError,
> >> -                            "unknown header field"))
> >> -                        self.__exception_handler(InterfaceError,
> >> -                                "unknown header field")
> >> +                self.description = list(zip(column_name, type_,
> >> display_size,
> >> +                                            internal_size,
> >> precision, scale,
> >> +                                            null_ok))
> >> +                self.__offset = 0
> >> +                self.lastrowid = None
> >>  
> >> -                    self.description = list(zip(column_name,
> >> type_,
> >> -                        display_size, internal_size, precision,
> >> scale, null_ok))
> >>  
> >> -                if line.startswith(mapi.MSG_TUPLE):
> >> -                    values = self.__parse_tuple(line)
> >> -                    rows.append(values)
> >> +            elif line.startswith(mapi.MSG_TUPLE):
> >> +                values = self.__parse_tuple(line)
> >> +                self.__rows .append(values)
> >>  
> >> -                elif line == mapi.MSG_PROMPT:
> >> -                    self.__query_id = id
> >> -                    self.__rows = rows
> >> -                    self.__offset = 0
> >> +            elif line.startswith(mapi.MSG_QBLOCK):
> >> +                self.__rows = []
> >>  
> >> -                    self.rowcount = rowcount
> >> -                    self.lastrowid = None
> >> -                    logger.debug("II store result finished")
> >> -                    return
> >> -
> >> -        elif firstline.startswith(mapi.MSG_QBLOCK):
> >> -            rows = []
> >> -            for line in lines[1:]:
> >> -                if line.startswith(mapi.MSG_TUPLE):
> >> -                    values = self.__parse_tuple(line)
> >> -                    rows.append(values)
> >> -                elif line == mapi.MSG_PROMPT:
> >> -                    logger.debug("II store result finished")
> >> -                    self.__rows = rows
> >> -                    return
> >> -
> >> -        elif firstline.startswith(mapi.MSG_QSCHEMA):
> >> -           if lines[1] == mapi.MSG_PROMPT:
> >> +            elif line.startswith(mapi.MSG_QSCHEMA):
> >> +                self.__offset = 0
> >> +                self.lastrowid = None
> >>                  self.__rows = []
> >> -                self.__offset = 0
> >>                  self.description = None
> >>                  self.rowcount = -1
> >> -                self.lastrowid = None
> >> -                logger.debug("II schema finished")
> >> -                return
> >>  
> >> -        elif firstline.startswith(mapi.MSG_QUPDATE):
> >> -           if lines[1] == mapi.MSG_PROMPT:
> >> -                (affected, identity) = firstline[2:].split()
> >> +            elif line.startswith(mapi.MSG_QUPDATE):
> >> +                (affected, identity) = line[2:].split()
> >> +                self.__offset = 0
> >>                  self.__rows = []
> >> -                self.__offset = 0
> >>                  self.description = None
> >>                  self.rowcount = int(affected)
> >>                  self.lastrowid = int(identity)
> >>                  self.__query_id = -1
> >> -                logger.debug("II update finished")
> >> +
> >> +            elif line.startswith(mapi.MSG_QTRANS):
> >> +                self.__offset = 0
> >> +                self.lastrowid = None
> >> +                self.__rows = []
> >> +                self.description = None
> >> +                self.rowcount = -1
> >> +
> >> +            elif line.startswith(mapi.MSG_PROMPT):
> >>                  return
> >>  
> >> -        elif firstline.startswith(mapi.MSG_ERROR):
> >> -            self.__exception_handler(ProgrammingError,
> >> firstline[1:])
> >> -
> >> -        elif firstline.startswith(mapi.MSG_QTRANS):
> >> -           if lines[1] == mapi.MSG_PROMPT:
> >> -                self.__rows = []
> >> -                self.__offset = 0
> >> -                self.description = None
> >> -                self.rowcount = -1
> >> -                self.lastrowid = None
> >> -                logger.debug("II transaction finished")
> >> -                return
> >> -
> >> -        elif firstline.startswith(mapi.MSG_PROMPT):
> >> -            self.__query_id = -1
> >> -            self.__rows = []
> >> -            self.__offset = 0
> >> -
> >> -            self.rowcount = 0
> >> -            self.lastrowid = None
> >> -            logger.debug("II empty response, assuming everything
> >> is ok")
> >> -            return
> >> +            elif line.startswith(mapi.MSG_ERROR):
> >> +                self.__exception_handler(ProgrammingError,
> >> line[1:])
> >>  
> >>          # you are not supposed to be here
> >>          self.__exception_handler(InterfaceError, "Unknown state,
> >>          %s" % block)
> >> diff --git a/clients/python2/test/capabilities.py
> >> b/clients/python2/test/capabilities.py
> >> --- a/clients/python2/test/capabilities.py
> >> +++ b/clients/python2/test/capabilities.py
> >> @@ -352,4 +352,16 @@ class DatabaseTest(unittest.TestCase):
> >>          t = ["list", "test"]
> >>          self.assertRaises(ProgrammingError,
> >>          self.db_module.monetize.convert, t)
> >>          self.db_module.monetize.mapping[list] = str
> >> -        self.assertEqual(self.db_module.monetize.convert(t),
> >> "['list', 'test']")
> >> \ No newline at end of file
> >> +        self.assertEqual(self.db_module.monetize.convert(t),
> >> "['list', 'test']")
> >> +
> >> +    def test_multiple_queries(self):
> >> +        table1 = self.new_table_name()
> >> +        table2 = table1[:-1] + 'bla"'
> >> +        self.cursor.execute("create table %s (a int)" % table1)
> >> +        self.cursor.execute("create table %s (a int, b int)" %
> >> table2)
> >> +        self.cursor.execute("insert into %s VALUES (100)" %
> >> table1)
> >> +        self.cursor.execute("insert into %s VALUES (50, 50)" %
> >> table2)
> >> +        self.cursor.execute('select * from %s; select * from %s;'
> >> %
> >> +                            (table1, table2))
> >> +        result = self.cursor.fetchall()
> >> +        self.assertEqual(result, [(50, 50)])
> >> \ No newline at end of file
> >> diff --git a/clients/python3/monetdb/sql/cursors.py
> >> b/clients/python3/monetdb/sql/cursors.py
> >> --- a/clients/python3/monetdb/sql/cursors.py
> >> +++ b/clients/python3/monetdb/sql/cursors.py
> >> @@ -381,133 +381,99 @@ class Cursor(object):
> >>          if not block:
> >>              block = ""
> >>  
> >> -        lines = block.split("\n")
> >> -        firstline = lines[0]
> >> +        for line in block.split("\n"):
> >> +            if line.startswith(mapi.MSG_INFO):
> >> +                logger.info(line[1:])
> >> +                self.messages.append((Warning, line[1:]))
> >>  
> >> -        while firstline.startswith(mapi.MSG_INFO):
> >> -            logger.info(firstline[1:])
> >> -            self.messages.append((Warning, firstline[1:]))
> >> -            lines = lines[1:]
> >> -            firstline = lines[0]
> >> +            elif line.startswith(mapi.MSG_QTABLE):
> >> +                (self.__query_id, rowcount, columns, tuples) =
> >> line[2:].split()
> >>  
> >> -        if firstline.startswith(mapi.MSG_QTABLE):
> >> -            (id, rowcount, columns, tuples) =
> >> firstline[2:].split()
> >> -            columns = int(columns)   # number of columns in
> >> result
> >> -            rowcount = int(rowcount) # total number of rows
> >> -            tuples = int(tuples)     # number of rows in this set
> >> -            rows = []
> >> +                columns = int(columns)   # number of columns in
> >> result
> >> +                self.rowcount = int(rowcount)  # total number of
> >> rows
> >> +                tuples = int(tuples)     # number of rows in this
> >> set
> >> +                self.__rows = []
> >>  
> >> -            # set up fields for description
> >> -            table_name = [None]*columns
> >> -            column_name = [None]*columns
> >> -            type_ = [None]*columns
> >> -            display_size = [None]*columns
> >> -            internal_size = [None]*columns
> >> -            precision = [None]*columns
> >> -            scale = [None]*columns
> >> -            null_ok = [None]*columns
> >> +                # set up fields for description
> >> +                table_name = [None] * columns
> >> +                column_name = [None] * columns
> >> +                type_ = [None] * columns
> >> +                display_size = [None] * columns
> >> +                internal_size = [None] * columns
> >> +                precision = [None] * columns
> >> +                scale = [None] * columns
> >> +                null_ok = [None] * columns
> >> +                typesizes = [(0, 0)] * columns
> >> +                self.__offset = 0
> >> +                self.lastrowid = None
> >>  
> >> -            typesizes = [(0,0)]*columns
> >> +            elif line.startswith(mapi.MSG_HEADER):
> >> +                (data, identity) = line[1:].split("#")
> >> +                values = [x.strip() for x in data.split(",")]
> >> +                identity = identity.strip()
> >>  
> >> -            for line in lines[1:]:
> >> -                if line.startswith(mapi.MSG_HEADER):
> >> -                    (data, identity) = line[1:].split("#")
> >> -                    values = [x.strip() for x in data.split(",")]
> >> -                    identity = identity.strip()
> >> +                if identity == "table_name":
> >> +                    table_name = values
> >> _______________________________________________
> >> checkin-list mailing list
> >> checkin-list at monetdb.org
> >> http://mail.monetdb.org/mailman/listinfo/checkin-list
> > 
> 
> 
> --
> Gijs Molenaar
> http://www.astro.uva.nl/people/gijs-molenaar/
> 
> 
> _______________________________________________
> developers-list mailing list
> developers-list at monetdb.org
> http://mail.monetdb.org/mailman/listinfo/developers-list
> 

-- 
| Stefan.Manegold at CWI.nl | DB Architectures   (DA) |
| www.CWI.nl/~manegold/  | Science Park 123 (L321) |
| +31 (0)20 592-4212     | 1098 XG Amsterdam  (NL) |




More information about the developers-list mailing list