June 2010 - developers-list

[Monetdb-developers] Basics about MonetDB
by 叶亮 01 Jun '11

01 Jun '11

Hi, I'm a new comer with some questions about basic conceptions of MonetDB. 1. The executor I've ever found the executor of postgreSQL at src\backend\executor, but cannot make sure where it is like that in MonetDB src. MonetDB\MonetDB\src\gdk or MonetDB5-server\MonetDB5\src\mal ? 3. MAL MAL (MonetDB Assembly Language) is a script tool only for MonetDB or a common script language that can be applied to other softwares. Why it is a must in MonetDB ? 4. .mx files In MonetDB, .mx files are compiled to make .h and .c files generated.(in this way ?) Why is this routines ? (not just coding in .h and .c files directly) Yel ---- Best Regards richardroky(a)gmail.com

8 10

[Monetdb-developers] python adapter: transaction semantics and connection.close()
by Mark Bucciarelli 30 Jun '10

30 Jun '10

Hey Gijs, I'm seeing a difference in how MonetDB and SQLite handle transactions inside Django. Here's my current unit test in Django (it passes w/ SQLite and fails with MonetDB): 1 : s1 = Simple(name="mark") 2 : s1.save() 3 : 4 : s2 = Simple(name="mark") 5 : try: 6 : s2.save() 7 : except IntegrityError: 8 : pass 9 : 10 : s = Simple.objects.get(name="mark") 11 : print s What happens under SQLite is the save on line 6 fails, but the select on line 10 works. In MonetDB, at line 10, I get the error current transaction is aborted (please ROLLBACK) Here's the trace of mapi2 commands, starting from line 2 mapi2.py, cmd, operation = sINSERT INTO "testapp_simple" ("name") VALUES ('mark'); mapi2.py, cmd, operation = Xreply_size 1 mapi2.py, cmd, operation = sCOMMIT; mapi2.py, cmd, operation = Xreply_size 1000 mapi2.py, cmd, operation = sINSERT INTO "testapp_simple" ("name") VALUES ('mark'); mapi2.py, cmd, operation = Xreply_size 1000 mapi2.py, cmd, operation = sSELECT "testapp_simple"."id", "testapp_simple"."name" FROM "testapp_simple" WHERE "testapp_simple"."name" = 'mark' ; mapi2.py, cmd, operation = Xreply_size 1 mapi2.py, cmd, operation = sROLLBACK; It looks like the ROLLBACK isn't executed until the connection closes. With auto commit off, shouldn't the ROLLBACK fire as soon as the insert fails? If so, how do I fix this? Thanks m

1 1

[Monetdb-developers] python adapter: registering new Monetizer types?
by Mark Bucciarelli 30 Jun '10

30 Jun '10

Hi, I'm working on the Django adapter again, and have run into a snag. I think the Python MonetDB wrapper may need a new method, but before building a patch I wanted to share the idea and get your feedback. The issue is this: Django uses the field type SafeUnicode. Monetizer doesn't have a mapping for this, and thus raises a ProgrammerError exception. (This is unicode that is safe to disply on an HTML page, and is used for all slug fields, which are common in Django.) The SQLite adapter handles this via a register_adapter method on the Database object. PostgreSQL uses psycopg2.extensions.register_adapter(). MySQL Python adapter exposes a encoders() method on the connection object. It seems the cleanest solution would be to mimic SQLite (if you want this for one connection, you most likely want it for all connections), and be able to do something like this: Database.register_adapter( type(SafeUnicode), lambda s:unicode(s) ) But it wasn't clear to me how to implement this. Somehow that call needs to make it into Montizer mapping dictionary. Thanks, Mark

2 4

[Monetdb-developers] encoding of strings returned by Python client
by Mark Bucciarelli 28 Jun '10

28 Jun '10

Any chance the Python client could be modified to return strings in Unicode instead of UTF-8? Seems like a better fit for Python; for example, that's how Python's sqlite3 module behaves. An alternative approach is to expose a set_encoding method on the connection that let's you specify the encoding of all text fields returned. If backward compatibility is an issue, the latter idea seems better even though it adds a knob. Any comments? (Is the maintainer of the Python adapter active even on this list? I didn't see many Python-specific posts in the archives.) Thanks, m

3 12

Re: [Monetdb-developers] MonetDB: Jun2010 - make it possible to use '?' in offset and limit
by Sjoerd Mullender 23 Jun '10

23 Jun '10

Niels, can you create a Changlog entry, please? On 2010-06-22 15:25, Niels Nes wrote: > Changeset: 600a35cbf6c0 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=600a35cbf6c0 > Modified Files: > sql/src/server/bin_optimizer.mx > sql/src/server/rel_bin.mx > sql/src/server/rel_schema.mx > sql/src/server/rel_select.mx > sql/src/server/sql_parser.mx > sql/src/server/sql_statement.mx > sql/src/server/sql_symbol.mx > sql/src/test/BugTracker-2009/Tests/POWER_vs_prod.SF-2596114.stable.out > sql/src/test/Dependencies/Tests/Dependencies.stable.out > Branch: Jun2010 > Log Message: > > make it possible to use '?' in offset and limit > (transplanted from 61c37c0d76f8ee3359bda98cc25e750b5201ae56) > > > diffs (truncated from 482 to 300 lines): > > diff -r d83c56c4bba7 -r 600a35cbf6c0 sql/src/server/bin_optimizer.mx > --- a/sql/src/server/bin_optimizer.mx Tue Jun 22 14:39:09 2010 +0200 > +++ b/sql/src/server/bin_optimizer.mx Mon Jun 21 23:35:09 2010 +0200 > @@ -288,9 +288,9 @@ > stmt *r = stmt_dup(j->op2.stval); > > l = stmt_limit(l, > - s->op2.stval->op1.aval->data.val.wval, > - s->op3.stval->op1.aval->data.val.wval, > - s->flag); > + stmt_dup(s->op2.stval), > + stmt_dup(s->op3.stval), > + s->flag); > s = stmt_join(l, r, cmp_equal); > ns = bin_optimizer(c, s); > stmt_destroy(s); > @@ -311,9 +311,9 @@ > stmt *r = stmt_dup(j->op2.stval); > > r = stmt_limit(r, > - s->op2.stval->op1.aval->data.val.wval, > - s->op3.stval->op1.aval->data.val.wval, > - s->flag); > + stmt_dup(s->op2.stval), > + stmt_dup(s->op3.stval), > + s->flag); > s = stmt_join(l, r, cmp_equal); > ns = bin_optimizer(c, s); > stmt_destroy(s); > @@ -325,8 +325,8 @@ > /* try to push the limit through the order */ > if (j->type == st_order) { > s = stmt_order(stmt_limit(stmt_dup(j->op1.stval), > - s->op2.stval->op1.aval->data.val.wval, > - s->op3.stval->op1.aval->data.val.wval, > + stmt_dup(s->op2.stval), > + stmt_dup(s->op3.stval), > LIMIT_DIRECTION(j->flag,1,0)), j->flag); > ns = bin_optimizer(c, s); > stmt_destroy(s); > @@ -338,8 +338,8 @@ > /* try to push the limit through the reverse */ > if (!s->flag && j->type == st_reverse) { > s = stmt_reverse(stmt_limit(stmt_dup(j->op1.stval), > - s->op2.stval->op1.aval->data.val.wval, > - s->op3.stval->op1.aval->data.val.wval, > + stmt_dup(s->op2.stval), > + stmt_dup(s->op3.stval), > 0)); > ns = bin_optimizer(c, s); > stmt_destroy(s); > @@ -351,8 +351,9 @@ > /* try to push the limit through the mark (only if there is no offset) */ > if (!s->op2.stval->op1.aval->data.val.wval && j->type == st_mark) { > s = stmt_mark_tail(stmt_limit(stmt_dup(j->op1.stval), > - s->op2.stval->op1.aval->data.val.wval, > - s->op3.stval->op1.aval->data.val.wval, s->flag), > + stmt_dup(s->op2.stval), > + stmt_dup(s->op3.stval), > + s->flag), > j->op2.stval->op1.aval->data.val.ival); > ns = bin_optimizer(c, s); > stmt_destroy(s); > diff -r d83c56c4bba7 -r 600a35cbf6c0 sql/src/server/rel_bin.mx > --- a/sql/src/server/rel_bin.mx Tue Jun 22 14:39:09 2010 +0200 > +++ b/sql/src/server/rel_bin.mx Mon Jun 21 23:35:09 2010 +0200 > @@ -1478,32 +1478,26 @@ > return stmt_list(l); > } > > -static wrd > +static sql_exp* > topn_limit( sql_rel *rel ) > { > if (rel->exps) { > sql_exp *limit = rel->exps->h->data; > - atom *a = limit->l; > - > - return a->data.val.wval; > + > + return limit; > } > - return -1; > + return NULL; > } > > -static wrd > +static sql_exp* > topn_offset( sql_rel *rel ) > { > - wrd o = 0; > - > if (rel->exps && list_length(rel->exps) > 1) { > sql_exp *offset = rel->exps->h->next->data; > - atom *a = offset->l; > - > - o = a->data.val.wval; > - if (o <= 0) > - o = 0; > + > + return offset; > } > - return o; > + return NULL; > } > > static sql_table * > @@ -1524,15 +1518,24 @@ > list *pl; > node *en, *n; > stmt *sub = NULL, *psub = NULL; > - wrd l = -1; > + stmt *l = NULL; > > if (topn) { > - wrd o = topn_offset(topn); > - l = topn_limit(topn); > - if (l < 0) /* for now only handle topn > + sql_exp *le = topn_limit(topn); > + sql_exp *oe = topn_offset(topn); > + > + if (!le) { /* for now only handle topn > including limit, ie not just offset */ > topn = NULL; > - l += o; > + } else { > + l = exp_bin(sql, le, NULL, NULL, NULL, NULL); > + if (oe) { > + sql_subtype *wrd = sql_bind_localtype("wrd"); > + sql_subfunc *add = sql_bind_func_result(sql->session->schema, "sql_add", wrd, wrd, wrd); > + stmt *o = exp_bin(sql, oe, NULL, NULL, NULL, NULL); > + l = stmt_binop(l, o, add); > + } > + } > } > > if (!rel->exps) > @@ -1591,7 +1594,7 @@ > stmt_destroy(sub); > return NULL; > } > - limit = stmt_limit(orderbycols, 0, l, LIMIT_DIRECTION(is_ascending(orderbycole), 1, before_project)); > + limit = stmt_limit(orderbycols, stmt_atom_wrd(0), l, LIMIT_DIRECTION(is_ascending(orderbycole), 1, before_project)); > for ( n=pl->h ; n; n = n->next) { > list_append(npl, > stmt_semijoin(column(stmt_dup(n->data)), > @@ -1902,8 +1905,8 @@ > rel2bin_topn( mvc *sql, sql_rel *rel, list *refs) > { > list *newl; > - wrd l = -1, o = 0; > - stmt *sub = NULL, *order = NULL; > + sql_exp *oe = NULL, *le = NULL; > + stmt *sub = NULL, *order = NULL, *l = NULL, *o = NULL; > node *n; > > if (rel->l) { /* first construct the sub relation */ > @@ -1918,8 +1921,8 @@ > if (!sub) > return NULL; > > - l = topn_limit(rel); > - o = topn_offset(rel); > + le = topn_limit(rel); > + oe = topn_offset(rel); > > if (sub->type == st_ordered) { > stmt *s = stmt_dup(sub->op2.stval); > @@ -1932,18 +1935,25 @@ > > if (n) { > stmt *limit = NULL, *p, *j; > - wrd lmt = l; > - > - if (l < 0) { > + sql_exp *lmt = le; > + > + if (le) > + l = exp_bin(sql, le, NULL, NULL, NULL, NULL); > + if (oe) > + o = exp_bin(sql, oe, NULL, NULL, NULL, NULL); > + > + if (!le) { > l = o; > - o = 0; > + o = stmt_atom_wrd(0); > } > + if (!o) > + o = stmt_atom_wrd(0); > > if (order) { > limit = stmt_limit(stmt_dup(order), o, l, 0); > } else { > limit = stmt_limit(column(stmt_dup(n->data)), o, l, 0); > - if (lmt >= 0) { > + if (lmt) { > n = n->next; > list_append(newl, limit); > } > @@ -1953,7 +1963,7 @@ > j = find_projection_join(limit); > if (j && 0) { > p = find_pivot(j); > - if (lmt < 0) > + if (!lmt) > p = stmt_diff(stmt_dup(p), stmt_dup(limit)); > else > p = stmt_semijoin(stmt_dup(p), stmt_dup(limit)); > @@ -1971,7 +1981,7 @@ > for ( ; n; n = n->next) { > stmt *s; > > - if (lmt < 0) > + if (!lmt) > s = stmt_diff(column(stmt_dup(n->data)), stmt_dup(limit)); > else > s = stmt_semijoin(column(stmt_dup(n->data)), stmt_dup(limit)); > @@ -1979,7 +1989,7 @@ > } > } > if (order) { > - if (lmt < 0) { > + if (!lmt) { > order = stmt_diff(order, limit); > } else { > stmt_destroy(order); > diff -r d83c56c4bba7 -r 600a35cbf6c0 sql/src/server/rel_schema.mx > --- a/sql/src/server/rel_schema.mx Tue Jun 22 14:39:09 2010 +0200 > +++ b/sql/src/server/rel_schema.mx Mon Jun 21 23:35:09 2010 +0200 > @@ -859,7 +859,7 @@ > if (query->token == SQL_SELECT) { > SelectNode *sn = (SelectNode *) query; > > - if (sn->limit >= 0) > + if (sn->limit) > return sql_error(sql, 01, "CREATE VIEW: LIMIT not supported"); > if (sn->orderby) > return sql_error(sql, 01, "CREATE VIEW: ORDER BY not supported"); > diff -r d83c56c4bba7 -r 600a35cbf6c0 sql/src/server/rel_select.mx > --- a/sql/src/server/rel_select.mx Tue Jun 22 14:39:09 2010 +0200 > +++ b/sql/src/server/rel_select.mx Mon Jun 21 23:35:09 2010 +0200 > @@ -4644,12 +4644,31 @@ > if (!rel) > return NULL; > > - if (sn->limit > 0 || sn->offset > 0) { > + if (sn->limit || sn->offset) { > + sql_subtype *wrd = sql_bind_localtype("wrd"); > list *exps = new_exp_list(); > > - append(exps, exp_atom_wrd(sn->limit)); > - if (sn->offset > 0) > - append(exps, exp_atom_wrd(sn->offset)); > + if (sn->limit) { > + sql_exp *l = rel_value_exp( sql, NULL, sn->limit, 0, ek); > + > + if (!l || !(l=rel_check_type(sql, wrd, l, type_equal))) > + return NULL; > + if ((ek.card != card_relation && sn->limit) && > + (ek.card == card_value && sn->limit)) { > + sql_subaggr *zero_or_one = sql_bind_aggr(sql->session->schema, "zero_or_one", exp_subtype(l)); > + > + l = exp_aggr1(l, zero_or_one, 0, 0, CARD_ATOM, 0); > + } > + // return sql_error(sql, 01, "SELECT: LIMIT only allowed on outermost SELECT"); > + append(exps, l); > + } else > + append(exps, NULL); > + if (sn->offset) { > + sql_exp *o = rel_value_exp( sql, NULL, sn->offset, 0, ek); > + if (!o || !(o=rel_check_type(sql, wrd, o, type_equal))) > + return NULL; > + append(exps, o); > + } > rel = rel_topn(rel, exps); > } > return rel; > @@ -4670,10 +4689,6 @@ > return table_ref(sql, rel, sq); > assert(sn->s.token == SQL_SELECT); > > - if ((ek.card != card_relation && sn->limit >= 0) && > - (ek.card == card_value && sn->limit != 1)) > - return sql_error(sql, 01, "SELECT: LIMIT only allowed on outermost SELECT"); > - > if (ek.card != card_relation && sn->orderby) > return sql_error(sql, 01, "SELECT: ORDER BY only allowed on outermost SELECT"); > > diff -r d83c56c4bba7 -r 600a35cbf6c0 sql/src/server/sql_parser.mx > --- a/sql/src/server/sql_parser.mx Tue Jun 22 14:39:09 2010 +0200 > +++ b/sql/src/server/sql_parser.mx Mon Jun 21 23:35:09 2010 +0200 > @@ -388,6 +388,9 @@ > ordering_spec > simple_table > table_ref > + opt_limit > + opt_offset > + param > case_exp > case_scalar_exp > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- Sjoerd Mullender

1 0

[Monetdb-developers] specifying NULL when using ODBC param_query
by kss ram 20 Jun '10

20 Jun '10

Hello Everyone, Is there any way to specify a null value when using ODBC param_query, in case of SQL_Query we do not really end up facing this problem as we build our own query string and then execute sql_query, sql_query is potentially unsafe of sql injection attacks, and thus using param_query makes alot of sense, but param query makes it hard to specify the data type nuetral NULL value, if we try specifying blank values like 0 in case of number and "" in case of string, these blank values will end up taking space in the database and thus would be expensive in terms of memory consumption, a database specific NULL value would leave the field null with no memory consumption. Thank you everyone in advance. kss ram

2 1

Re: [Monetdb-developers] MonetDB: default - MAP_ANONYMOUS instead of MAP_ANON
by Stefan Manegold 17 Jun '10

17 Jun '10

Niels, is this just precaution --- `man mmap` says " MAP_ANON Synonym for MAP_ANONYMOUS. Deprecated. " --- or bug fix that should be back-ported to the Jun2010 branch (I could do that)? Thanks. Stefan On Thu, Jun 17, 2010 at 09:47:40PM +0200, Niels Nes wrote: > Changeset: 57878a477b7c for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=57878a477b7c > Modified Files: > MonetDB/src/gdk/gdk_posix.mx > Branch: default > Log Message: > > MAP_ANONYMOUS instead of MAP_ANON > > > diffs (14 lines): > > diff -r 99d16aff6ce6 -r 57878a477b7c MonetDB/src/gdk/gdk_posix.mx > --- a/MonetDB/src/gdk/gdk_posix.mx Thu Jun 17 14:39:08 2010 +0200 > +++ b/MonetDB/src/gdk/gdk_posix.mx Thu Jun 17 21:47:00 2010 +0200 > @@ -2105,8 +2105,8 @@ > > #define MT_PAGESIZE(s) ((((s)-1)/MT_pagesize()+1)*MT_pagesize()) > > -#if defined(MAP_ANON) > -#define MMAP_FLAGS(f) f|MAP_ANON > +#if defined(MAP_ANONYMOUS) > +#define MMAP_FLAGS(f) f|MAP_ANONYMOUS > #define MMAP_FD -1 > #define MMAP_OPEN_DEV_ZERO int fd = 1 > #define MMAP_CLOSE_DEV_ZERO (void)fd > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

1 0

[Monetdb-developers] Parallel insertions in monetdb table - some records do not insert
by kss ram 16 Jun '10

16 Jun '10

Hello Everyone! Brief Overview of the issue 1. Sample Data that erlang program tries to insert in monetdb {data, [{"a","string"}, {"b","string"}], %% a,b are column names [ [{"a","a1"}, {"b","b1"}], %% key, value pair of the data - {columnname, data} [{"a","a2"}, {"b","b2"}], [{"a","a3"}, {"b","b3"}], [{"a","a4"}, {"b","b4"}], [{"a","a5"}, {"b","b5"}], [{"a","a6"}, {"b","b6"}], [{"a","a7"}, {"b","b7"}], [{"a","a8"}, {"b","b8"}], [{"a","a9"}, {"b","b9"}], [{"a","a10"}, {"b","b10"}], [{"a","a11"}, {"b","b11"}] ], {"no_of_records","12"} } 2. no of threads/processes that would do parallel inserts would be determined by the thread size i.e how many of records the thread would insert, if say 5 records then 3 threads, if 3 records then 4 threads. 3. in order to connect to monetdb from erlang, the monetdb ODBC driver is specified in ODBCConfig utility of ubuntu, driver file is /usr/lib/libMonetODBC.so 4. threads are successfully created and the ODBC module quits successfully after inserting in the erlang program, but when we look in the monetdb table, some of the rows get missed, for example row from 1 to 11 get inserted without row 3 or 4 getting inserted. could this be a problem with the ODBC driver?, am I missing any configuration?, any help would be deeply appreciated. Thank you, Ram.

3 2

Re: [Monetdb-developers] MonetDB: default - fix compilation failure opt_mapreduce.mx:141:...
by Sjoerd Mullender 10 Jun '10

10 Jun '10

Why not fix this "properly" by changing the type of cnt (and the function return value and several other variables)? At the very least you could have an assert to check that the value fits in an int. On 2010-06-10 10:54, Fabian Groffen wrote: > Changeset: 820a7637cf5e for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=820a7637cf5e > Modified Files: > MonetDB5/src/optimizer/opt_mapreduce.mx > Branch: default > Log Message: > > fix compilation failure opt_mapreduce.mx:141: '=' : conversion from 'size_t' to 'int', possible loss of data > > > diffs (12 lines): > > diff -r c7d1cd0d0bbb -r 820a7637cf5e MonetDB5/src/optimizer/opt_mapreduce.mx > --- a/MonetDB5/src/optimizer/opt_mapreduce.mx Wed Jun 09 15:30:45 2010 +0200 > +++ b/MonetDB5/src/optimizer/opt_mapreduce.mx Thu Jun 10 10:54:23 2010 +0200 > @@ -138,7 +138,7 @@ > return 0; > } > cloud = BATdescriptor(bid); > - cnt = BATcount(cloud); > + cnt = (int)BATcount(cloud); > BBPreleaseref(bid); /* we're done with it */ > return(cnt); > } > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list -- Sjoerd Mullender

2 1

Re: [Monetdb-developers] MonetDB: default - Add test to prevent warning
by Stefan Manegold 01 Jun '10

01 Jun '10

Arjen, is this a fix that should also be include in the upcoming Jun2010 release? If so, it would need to be back-ported to the Jun2010 branch ... Stefan On Tue, Jun 01, 2010 at 11:40:20AM +0200, Arjen de Rijke wrote: > Changeset: 74ee41649425 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=74ee41649425 > Modified Files: > clients/src/php/native/lib/php_mapi.inc > Branch: default > Log Message: > > Add test to prevent warning > > > diffs (15 lines): > > diff -r 80afe14ca780 -r 74ee41649425 clients/src/php/native/lib/php_mapi.inc > --- a/clients/src/php/native/lib/php_mapi.inc Tue Jun 01 10:02:37 2010 +0200 > +++ b/clients/src/php/native/lib/php_mapi.inc Tue Jun 01 11:37:25 2010 +0200 > @@ -768,8 +768,10 @@ > global $connection_pool; > > $connections = array(); > - foreach($connection_pool as $conn) { > + if ($connection_pool !== null) { > + foreach($connection_pool as $conn) { > $connections[] = $conn["id"]; > + } > } > > $id = hash("sha1", time()); > _______________________________________________ > Checkin-list mailing list > Checkin-list(a)monetdb.org > http://mail.monetdb.org/mailman/listinfo/checkin-list > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

2 1