MonetDB: default - rewrote result parser to make it eat multiple...

Stefan Manegold Stefan.Manegold at cwi.nl
Tue Mar 12 10:39:43 CET 2013


Dear Gijs,
   
thank you very much for continuously taking care of MonetDB's python
interface / API!
We very muchappreciate your work and contribution!
   

However, we'd appreciate even more, it you could also keep the stable output
up-to-date whenever you change / extend you test or your changes intendedly
change the output of your tests.

See
http://monetdb.cwi.nl/testweb/web/status.php
http://monetdb.cwi.nl/testweb/web/testgrid.php?serial=47169:7d1c376e995f&order=platform,arch,compiler
http://monetdb.cwi.nl/testweb/web/testgrid.php?serial=47169:7d1c376e995f&order=platform,arch,compiler&targets=GNU-Darwin-i386-propcheck,GNU-Darwin-powerpc,GNU-Darwin-x86_64-propcheck,GNU-Fedora-i386-propcheck-dbfarm,GNU-Fedora-x86_64-dist,GNU-Fedora-x86_64-propcheck-assert,GNU-Fedora-x86_64-propcheck-dbfarm,GNU-Fedora-x86_64-propcheck-oid32-dbfarm,GNU-Fedora-x86_64-propcheck-thrs=1,GNU-Fedora-x86_64-propcheck-thrs=32,GNU-Fedora-x86_64-propcheck-thrs=53,GNU-Fedora-x86_64-thrs=1,GNU-FreeBSD-x86_64-propcheck,GNU-Gentoo-powerpc-propcheck-dbfarm,GNU-OpenIndiana-x86_64-propcheck,GNU-Solaris-sparc-dbfarm,GNU-Solaris-sparcv9-dbfarm,GNU-Solaris-sparcv9-oid32-dbfarm,GNU-Ubuntu-i386-propcheck,Int-Fedora-x86_64-propcheck-oid32,Int-Fedora-x86_64-propcheck,Int-Windows2008-x86_64-propcheck-oid32,Int-Windows7-x86_64-propcheck,Int-WindowsXP-i386-propcheck&module=sql&tstlimit=sql/test/mapi
http://monetdb.cwi.nl/testweb/web/showtestoutput.php?serial=47169:7d1c376e995f&target=GNU-Fedora-x86_64-propcheck-assert&module=sql&test=sql%2Ftest%2Fmapi%2Fpython2_test_monetdb_sql&which=err
http://monetdb.cwi.nl/testweb/web/testchange.php?test=47169:7d1c376e995f/GNU-Fedora-x86_64-propcheck-assert/sql/mTests/sql%2Ftest%2Fmapi%2Fpython2_test_monetdb_sql


If nothing else, simply go to .../MonetDB/sql/test/mapi/Tests/ and run
`Mtest.py` to run (among others) your pyhton test (preferably on a system
with both Python2 & python3 installed, and follow the instructions.

See also `Mtest.py --help`, `Mapprove.py --help` and
http://dev.monetdb.org/hg/MonetDB/file/tip/testing/README


Thanks!

Stefan


On Mon, Mar 11, 2013 at 06:44:43PM +0100, Gijs Molenaar wrote:
> Changeset: 39b2905626b8 for MonetDB
> URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=39b2905626b8
> Modified Files:
> 	clients/python2/monetdb/sql/cursors.py
> 	clients/python2/test/capabilities.py
> 	clients/python3/monetdb/sql/cursors.py
> 	clients/python3/test/capabilities.py
> Branch: default
> Log Message:
> 
> rewrote result parser to make it eat multiple resultsets
> 
> 
> diffs (truncated from 463 to 300 lines):
> 
> diff --git a/clients/python2/monetdb/sql/cursors.py b/clients/python2/monetdb/sql/cursors.py
> --- a/clients/python2/monetdb/sql/cursors.py
> +++ b/clients/python2/monetdb/sql/cursors.py
> @@ -380,133 +380,99 @@ class Cursor(object):
>          if not block:
>              block = ""
>  
> -        lines = block.split("\n")
> -        firstline = lines[0]
> +        for line in block.split("\n"):
> +            if line.startswith(mapi.MSG_INFO):
> +                logger.info(line[1:])
> +                self.messages.append((Warning, line[1:]))
>  
> -        while firstline.startswith(mapi.MSG_INFO):
> -            logger.info(firstline[1:])
> -            self.messages.append((Warning, firstline[1:]))
> -            lines = lines[1:]
> -            firstline = lines[0]
> +            elif line.startswith(mapi.MSG_QTABLE):
> +                (self.__query_id, rowcount, columns, tuples) = line[2:].split()
>  
> -        if firstline.startswith(mapi.MSG_QTABLE):
> -            (id, rowcount, columns, tuples) = firstline[2:].split()
> -            columns = int(columns)   # number of columns in result
> -            rowcount = int(rowcount) # total number of rows
> -            tuples = int(tuples)     # number of rows in this set
> -            rows = []
> +                columns = int(columns)   # number of columns in result
> +                self.rowcount = int(rowcount)  # total number of rows
> +                tuples = int(tuples)     # number of rows in this set
> +                self.__rows = []
>  
> -            # set up fields for description
> -            table_name = [None]*columns
> -            column_name = [None]*columns
> -            type_ = [None]*columns
> -            display_size = [None]*columns
> -            internal_size = [None]*columns
> -            precision = [None]*columns
> -            scale = [None]*columns
> -            null_ok = [None]*columns
> +                # set up fields for description
> +                table_name = [None] * columns
> +                column_name = [None] * columns
> +                type_ = [None] * columns
> +                display_size = [None] * columns
> +                internal_size = [None] * columns
> +                precision = [None] * columns
> +                scale = [None] * columns
> +                null_ok = [None] * columns
> +                typesizes = [(0, 0)] * columns
> +                self.__offset = 0
> +                self.lastrowid = None
>  
> -            typesizes = [(0,0)]*columns
> +            elif line.startswith(mapi.MSG_HEADER):
> +                (data, identity) = line[1:].split("#")
> +                values = [x.strip() for x in data.split(",")]
> +                identity = identity.strip()
>  
> -            for line in lines[1:]:
> -                if line.startswith(mapi.MSG_HEADER):
> -                    (data, identity) = line[1:].split("#")
> -                    values = [x.strip() for x in data.split(",")]
> -                    identity = identity.strip()
> +                if identity == "table_name":
> +                    table_name = values
> +                elif identity == "name":
> +                    column_name = values
> +                elif identity == "type":
> +                    type_ = values
> +                elif identity == "length":
> +                    length = values
> +                elif identity == "typesizes":
> +                    typesizes = [[int(j) for j in i.split()] for i in values]
> +                    internal_size = [x[0] for x in typesizes]
> +                    for num, typeelem in enumerate(type_):
> +                        if typeelem in ['decimal']:
> +                            precision[num] = typesizes[num][0]
> +                            scale[num] = typesizes[num][1]
> +                else:
> +                    msg = "unknown header field"
> +                    self.messages.append((InterfaceError, msg))
> +                    self.__exception_handler(InterfaceError, msg)
>  
> -                    if identity == "table_name":
> -                        table_name = values   # not used
> -                    elif identity == "name":
> -                        column_name = values
> -                    elif identity == "type":
> -                        type_ = values
> -                    elif identity == "length":
> -                        pass # not used
> -                    elif identity == "typesizes":
> -                        typesizes = [[int(j) for j in i.split()] for i in values]
> -                        internal_size = [x[0] for x in typesizes]
> -                        for num, typeelem in enumerate(type_):
> -                            if typeelem in ['decimal']:
> -                                precision[num] = typesizes[num][0]
> -                                scale[num] = typesizes[num][1]
> -                    else:
> -                        self.messages.append((InterfaceError,
> -                            "unknown header field"))
> -                        self.__exception_handler(InterfaceError,
> -                                "unknown header field")
> +                self.description = list(zip(column_name, type_, display_size,
> +                                            internal_size, precision, scale,
> +                                            null_ok))
> +                self.__offset = 0
> +                self.lastrowid = None
>  
> -                    self.description = list(zip(column_name, type_,
> -                        display_size, internal_size, precision, scale, null_ok))
>  
> -                if line.startswith(mapi.MSG_TUPLE):
> -                    values = self.__parse_tuple(line)
> -                    rows.append(values)
> +            elif line.startswith(mapi.MSG_TUPLE):
> +                values = self.__parse_tuple(line)
> +                self.__rows .append(values)
>  
> -                elif line == mapi.MSG_PROMPT:
> -                    self.__query_id = id
> -                    self.__rows = rows
> -                    self.__offset = 0
> +            elif line.startswith(mapi.MSG_QBLOCK):
> +                self.__rows = []
>  
> -                    self.rowcount = rowcount
> -                    self.lastrowid = None
> -                    logger.debug("II store result finished")
> -                    return
> -
> -        elif firstline.startswith(mapi.MSG_QBLOCK):
> -            rows = []
> -            for line in lines[1:]:
> -                if line.startswith(mapi.MSG_TUPLE):
> -                    values = self.__parse_tuple(line)
> -                    rows.append(values)
> -                elif line == mapi.MSG_PROMPT:
> -                    logger.debug("II store result finished")
> -                    self.__rows = rows
> -                    return
> -
> -        elif firstline.startswith(mapi.MSG_QSCHEMA):
> -           if lines[1] == mapi.MSG_PROMPT:
> +            elif line.startswith(mapi.MSG_QSCHEMA):
> +                self.__offset = 0
> +                self.lastrowid = None
>                  self.__rows = []
> -                self.__offset = 0
>                  self.description = None
>                  self.rowcount = -1
> -                self.lastrowid = None
> -                logger.debug("II schema finished")
> -                return
>  
> -        elif firstline.startswith(mapi.MSG_QUPDATE):
> -           if lines[1] == mapi.MSG_PROMPT:
> -                (affected, identity) = firstline[2:].split()
> +            elif line.startswith(mapi.MSG_QUPDATE):
> +                (affected, identity) = line[2:].split()
> +                self.__offset = 0
>                  self.__rows = []
> -                self.__offset = 0
>                  self.description = None
>                  self.rowcount = int(affected)
>                  self.lastrowid = int(identity)
>                  self.__query_id = -1
> -                logger.debug("II update finished")
> +
> +            elif line.startswith(mapi.MSG_QTRANS):
> +                self.__offset = 0
> +                self.lastrowid = None
> +                self.__rows = []
> +                self.description = None
> +                self.rowcount = -1
> +
> +            elif line.startswith(mapi.MSG_PROMPT):
>                  return
>  
> -        elif firstline.startswith(mapi.MSG_ERROR):
> -            self.__exception_handler(ProgrammingError, firstline[1:])
> -
> -        elif firstline.startswith(mapi.MSG_QTRANS):
> -           if lines[1] == mapi.MSG_PROMPT:
> -                self.__rows = []
> -                self.__offset = 0
> -                self.description = None
> -                self.rowcount = -1
> -                self.lastrowid = None
> -                logger.debug("II transaction finished")
> -                return
> -
> -        elif firstline.startswith(mapi.MSG_PROMPT):
> -            self.__query_id = -1
> -            self.__rows = []
> -            self.__offset = 0
> -
> -            self.rowcount = 0
> -            self.lastrowid = None
> -            logger.debug("II empty response, assuming everything is ok")
> -            return
> +            elif line.startswith(mapi.MSG_ERROR):
> +                self.__exception_handler(ProgrammingError, line[1:])
>  
>          # you are not supposed to be here
>          self.__exception_handler(InterfaceError, "Unknown state, %s" % block)
> diff --git a/clients/python2/test/capabilities.py b/clients/python2/test/capabilities.py
> --- a/clients/python2/test/capabilities.py
> +++ b/clients/python2/test/capabilities.py
> @@ -352,4 +352,16 @@ class DatabaseTest(unittest.TestCase):
>          t = ["list", "test"]
>          self.assertRaises(ProgrammingError, self.db_module.monetize.convert, t)
>          self.db_module.monetize.mapping[list] = str
> -        self.assertEqual(self.db_module.monetize.convert(t), "['list', 'test']")
> \ No newline at end of file
> +        self.assertEqual(self.db_module.monetize.convert(t), "['list', 'test']")
> +
> +    def test_multiple_queries(self):
> +        table1 = self.new_table_name()
> +        table2 = table1[:-1] + 'bla"'
> +        self.cursor.execute("create table %s (a int)" % table1)
> +        self.cursor.execute("create table %s (a int, b int)" % table2)
> +        self.cursor.execute("insert into %s VALUES (100)" % table1)
> +        self.cursor.execute("insert into %s VALUES (50, 50)" % table2)
> +        self.cursor.execute('select * from %s; select * from %s;' %
> +                            (table1, table2))
> +        result = self.cursor.fetchall()
> +        self.assertEqual(result, [(50, 50)])
> \ No newline at end of file
> diff --git a/clients/python3/monetdb/sql/cursors.py b/clients/python3/monetdb/sql/cursors.py
> --- a/clients/python3/monetdb/sql/cursors.py
> +++ b/clients/python3/monetdb/sql/cursors.py
> @@ -381,133 +381,99 @@ class Cursor(object):
>          if not block:
>              block = ""
>  
> -        lines = block.split("\n")
> -        firstline = lines[0]
> +        for line in block.split("\n"):
> +            if line.startswith(mapi.MSG_INFO):
> +                logger.info(line[1:])
> +                self.messages.append((Warning, line[1:]))
>  
> -        while firstline.startswith(mapi.MSG_INFO):
> -            logger.info(firstline[1:])
> -            self.messages.append((Warning, firstline[1:]))
> -            lines = lines[1:]
> -            firstline = lines[0]
> +            elif line.startswith(mapi.MSG_QTABLE):
> +                (self.__query_id, rowcount, columns, tuples) = line[2:].split()
>  
> -        if firstline.startswith(mapi.MSG_QTABLE):
> -            (id, rowcount, columns, tuples) = firstline[2:].split()
> -            columns = int(columns)   # number of columns in result
> -            rowcount = int(rowcount) # total number of rows
> -            tuples = int(tuples)     # number of rows in this set
> -            rows = []
> +                columns = int(columns)   # number of columns in result
> +                self.rowcount = int(rowcount)  # total number of rows
> +                tuples = int(tuples)     # number of rows in this set
> +                self.__rows = []
>  
> -            # set up fields for description
> -            table_name = [None]*columns
> -            column_name = [None]*columns
> -            type_ = [None]*columns
> -            display_size = [None]*columns
> -            internal_size = [None]*columns
> -            precision = [None]*columns
> -            scale = [None]*columns
> -            null_ok = [None]*columns
> +                # set up fields for description
> +                table_name = [None] * columns
> +                column_name = [None] * columns
> +                type_ = [None] * columns
> +                display_size = [None] * columns
> +                internal_size = [None] * columns
> +                precision = [None] * columns
> +                scale = [None] * columns
> +                null_ok = [None] * columns
> +                typesizes = [(0, 0)] * columns
> +                self.__offset = 0
> +                self.lastrowid = None
>  
> -            typesizes = [(0,0)]*columns
> +            elif line.startswith(mapi.MSG_HEADER):
> +                (data, identity) = line[1:].split("#")
> +                values = [x.strip() for x in data.split(",")]
> +                identity = identity.strip()
>  
> -            for line in lines[1:]:
> -                if line.startswith(mapi.MSG_HEADER):
> -                    (data, identity) = line[1:].split("#")
> -                    values = [x.strip() for x in data.split(",")]
> -                    identity = identity.strip()
> +                if identity == "table_name":
> +                    table_name = values
> _______________________________________________
> checkin-list mailing list
> checkin-list at monetdb.org
> http://mail.monetdb.org/mailman/listinfo/checkin-list

-- 
| Stefan.Manegold at CWI.nl | DB Architectures   (DA) |
|  www.CWI.nl/~manegold  | Science Park 123 (L321) |
|   +31 (0)20 592-4212   | 1098 XG Amsterdam  (NL) |



More information about the developers-list mailing list