September 2014 - developers-list

Re: MonetDB: default - Disable -pthread for Clang 6.*
by Sjoerd Mullender 19 Sep '14

19 Sep '14

Shouldn't this go to the Oct2014 branch? On 19/09/14 11:45, Dimitar Nedev wrote: > Changeset: 10ae6a3308bf for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=10ae6a3308bf > Modified Files: > configure.ag > Branch: default > Log Message: > > Disable -pthread for Clang 6.* > This is in line OS X Yosemite and more specifically Xcode 6.0.1 shipping with Clang verion 6 > > > diffs (19 lines): > > diff --git a/configure.ag b/configure.ag > --- a/configure.ag > +++ b/configure.ag > @@ -1885,11 +1885,11 @@ if test "x$have_pthread" != xno; then > save_LIBS="$LIBS" > save_CPPFLAGS="$CPPFLAGS" > case $GCC-$have_pthread-$CC_ver in > - yes-auto-clang-5.*|yes-yes-clang-5.*) > - # clang 5.* (at least on Mac OSX Mavericks) does not > + yes-auto-clang-5.*|yes-yes-clang-5.*|yes-auto-clang-6.*|yes-yes-clang-6.*) > + # clang 5.*/6.* (Xcode 6.0) does not > # seem to have / require -pthread as compiler > - # option; on Mac OSX Mavericks, "Apple LLVM version > - # 5.0 (clang-500.2.79) (based on LLVM 3.3svn)" > + # option; on Mac OS X Yosamite, "Apple LLVM > + # version 6.0 (clang-600.0.51) (based on LLVM 3.5svn)" > # complains about "error: argument unused during > # compilation: '-pthread'". > # However, disabling -pthread completely might be > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

2 2

How varchar/text column works ?
by Mike 16 Sep '14

16 Sep '14

Hi, i have been working on monetdb for quite some time and am trying to understand how string column works so as i can (try to ) make it import string columns as fast as possible.The tool is external and generates monetdb binary compatible columns( and may be used to optimize tables in offline mode). As far as reading code and understanding it. here is my part String columns is kept in 2 parts, x.tail and x.theap ( where x is number ) x.tail contains pointers into x.theap x.theap has 2 regions/parts first part contains a 8KB hash table, to detect collisions and second part contains 8 zeroes,8 Byte hash, variable length String which is null terminated and aligned on 8 byte boundary (i am talking about x64 bit arch) and next string follows and so on However, the problem i am facing is , the tools works and generates columns till x.tail contains pointers addressing x.theap positions less than 255. The moment pointers are 2 byte tail entry, monetdb is confused. The reason is tail pointers are varsized for efficiency, and if there are few small string, pointers will fit in small memory and not waste 8 bytes per entry. However, i am unable to understand how monetdb figures out, how big entry is currently being used ? in case of small strings, it's 1 byte (tested) in case of some bigger string, it's 2 bytes (tested) and even in large string tables, it 4 bytes (tested) and large (>4 GB) sting tables, it can be 8 bytes ??? ( this case has not been tested) i have understood, it has something to with bat->T->shift , bat->T->width but still it doesn't work.So, i am definitely missing something. So, my questions are 1) is it an efficient method ( not talking about space efficiency) to have var-sized tail pointers. because it might involve lots of casting and so on during calculations.. 2) how to disable var-sized tail pointers (if it's possible) so as all pointers are 4byte or 8 byte long ( any compile time option or runtime option ) 3) if 2) not possible , then how does monetdb calculate tail pointer width. for example, i am using following code, to initialize new bat with my tail and theap pointers cap=num_of_tail_entries; bs = BATcreatedesc(TYPE_void, TYPE_str, 1); bn = &bs->B; BATsetdims(bn); BATkey(bn, TRUE); BATsetcapacity(bn, cap); BATsetcount(bn, cap); bn->tsorted = 0; bn->trevsorted = 0; bn->tdense = 0; bn->tkey = 0; bn->tvarsized=1; // try to set shift to 1 and thus width to 2 bytes //bn->T->shift=1; // this property seems to be used to calculate tail ptr size //bn->T->width=2; // this property seems to be used to calculate tail ptr size bn->T->varsized=1; then i load both heap and theap. The above code works and i am able to query the table as long as tail pointers are 1 byte long. However, as soon as pointers size is 2 bytes,it fails, so i must set something in BAT init to tell tail ptr size. Any clues are welcome... ( if more information is required, please ask.) Thanks for any help . Rgds mike

1 1

Re: MonetDB: Oct2014 - split update bat within the catalog interface...
by Sjoerd Mullender 09 Sep '14

09 Sep '14

This causes an assertion failed during upgrade from a Jan2014 database. On 2014-09-09 08:24, Niels Nes wrote: > Changeset: 1c3d356ffdc2 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=1c3d356ffdc2 > Modified Files: > sql/backends/monet5/rel_bin.c > sql/backends/monet5/sql.c > sql/backends/monet5/sql_gencode.c > sql/backends/monet5/sql_optimizer.c > sql/backends/monet5/sql_statistics.c > sql/include/sql_catalog.h > sql/storage/bat/bat_storage.c > Branch: Oct2014 > Log Message: > > split update bat within the catalog interface to remove dependency on > BATselect. > > > diffs (truncated from 395 to 300 lines): > > diff --git a/sql/backends/monet5/rel_bin.c b/sql/backends/monet5/rel_bin.c > --- a/sql/backends/monet5/rel_bin.c > +++ b/sql/backends/monet5/rel_bin.c > @@ -733,7 +733,7 @@ stmt_col( mvc *sql, sql_column *c, stmt > (c->base.flag != TR_NEW || c->t->base.flag != TR_NEW /* alter */) && > (c->t->persistence == SQL_PERSIST || c->t->persistence == SQL_DECLARED_TABLE) && !c->t->commit_action) { > stmt *i = stmt_bat(sql->sa, c, RD_INS); > - stmt *u = stmt_bat(sql->sa, c, RD_UPD); > + stmt *u = stmt_bat(sql->sa, c, RD_UPD_ID); > sc = stmt_project_delta(sql->sa, sc, u, i); > sc = stmt_project(sql->sa, del, sc); > } else if (del) { /* always handle the deletes */ > @@ -751,7 +751,7 @@ stmt_idx( mvc *sql, sql_idx *i, stmt *de > (i->base.flag != TR_NEW || i->t->base.flag != TR_NEW /* alter */) && > (i->t->persistence == SQL_PERSIST || i->t->persistence == SQL_DECLARED_TABLE) && !i->t->commit_action) { > stmt *ic = stmt_idxbat(sql->sa, i, RD_INS); > - stmt *u = stmt_idxbat(sql->sa, i, RD_UPD); > + stmt *u = stmt_idxbat(sql->sa, i, RD_UPD_ID); > sc = stmt_project_delta(sql->sa, sc, u, ic); > sc = stmt_project(sql->sa, del, sc); > } else if (del) { /* always handle the deletes */ > diff --git a/sql/backends/monet5/sql.c b/sql/backends/monet5/sql.c > --- a/sql/backends/monet5/sql.c > +++ b/sql/backends/monet5/sql.c > @@ -489,7 +489,7 @@ table_has_updates(sql_trans *tr, sql_tab > > for ( n = t->columns.set->h; !cnt && n; n = n->next) { > sql_column *c = n->data; > - BAT *b = store_funcs.bind_col(tr, c, RD_UPD); > + BAT *b = store_funcs.bind_col(tr, c, RD_UPD_ID); > cnt |= BATcount(b) > 0; > BBPunfix(b->batCacheid); > } > @@ -1644,6 +1644,7 @@ mvc_bind_wrap(Client cntxt, MalBlkPtr mb > bn = BATslice(b, part_nr * psz, (part_nr + 1 == nr_parts) ? cnt : ((part_nr + 1) * psz)); > BATseqbase(bn, part_nr * psz); > } else { > + /* BAT b holds the UPD_ID bat */ > oid l, h; > BAT *c = mvc_bind(m, *sname, *tname, *cname, 0); > cnt = BATcount(c); > @@ -1651,23 +1652,39 @@ mvc_bind_wrap(Client cntxt, MalBlkPtr mb > l = part_nr * psz; > h = (part_nr + 1 == nr_parts) ? cnt : ((part_nr + 1) * psz); > h--; > - bn = BATmirror(BATselect(BATmirror(b), &l, &h)); > + bn = BATsubselect(b, NULL, &l, &h, 1, 0, 0); > BBPreleaseref(c->batCacheid); > } > BBPreleaseref(b->batCacheid); > b = bn; > + } else if (upd) { > + BAT *uv = mvc_bind(m, *sname, *tname, *cname, RD_UPD_VAL); > + int *uvl = (int *) getArgReference(stk, pci, 1); > + > + BBPkeepref(*bid = b->batCacheid); > + BBPkeepref(*uvl = uv->batCacheid); > + return MAL_SUCCEED; > } > if (upd) { > int *uvl = (int *) getArgReference(stk, pci, 1); > > if (BATcount(b)) { > - BAT *id = BATmirror(BATmark(b, 0)); > - BAT *vl = BATmirror(BATmark(BATmirror(b), 0)); > + BAT *uv = mvc_bind(m, *sname, *tname, *cname, RD_UPD_VAL); > + BAT *ui = mvc_bind(m, *sname, *tname, *cname, RD_UPD_ID); > + BAT *id = BATproject(b, ui); > + BAT *vl = BATproject(b, uv); > + bat_destroy(b); > + bat_destroy(ui); > + bat_destroy(uv); > BBPkeepref(*bid = id->batCacheid); > BBPkeepref(*uvl = vl->batCacheid); > } else { > + sql_schema *s = mvc_bind_schema(m, *sname); > + sql_table *t = mvc_bind_table(m, s, *tname); > + sql_column *c = mvc_bind_column(m, t, *cname); > + > *bid = e_bat(TYPE_oid); > - *uvl = e_bat(b->T->type); > + *uvl = e_bat(c->type.type->localtype); > } > BBPreleaseref(b->batCacheid); > } else { > @@ -1711,6 +1728,7 @@ mvc_bind_idxbat_wrap(Client cntxt, MalBl > bn = BATslice(b, part_nr * psz, (part_nr + 1 == nr_parts) ? cnt : ((part_nr + 1) * psz)); > BATseqbase(bn, part_nr * psz); > } else { > + /* BAT b holds the UPD_ID bat */ > oid l, h; > BAT *c = mvc_bind_idxbat(m, *sname, *tname, *iname, 0); > cnt = BATcount(c); > @@ -1718,23 +1736,37 @@ mvc_bind_idxbat_wrap(Client cntxt, MalBl > l = part_nr * psz; > h = (part_nr + 1 == nr_parts) ? cnt : ((part_nr + 1) * psz); > h--; > - bn = BATmirror(BATselect(BATmirror(b), &l, &h)); > + bn = BATsubselect(b, NULL, &l, &h, 1, 0, 0); > BBPreleaseref(c->batCacheid); > } > BBPreleaseref(b->batCacheid); > b = bn; > + } else if (upd) { > + BAT *uv = mvc_bind_idxbat(m, *sname, *tname, *iname, RD_UPD_VAL); > + int *uvl = (int *) getArgReference(stk, pci, 1); > + BBPkeepref(*bid = b->batCacheid); > + BBPkeepref(*uvl = uv->batCacheid); > + return MAL_SUCCEED; > } > if (upd) { > int *uvl = (int *) getArgReference(stk, pci, 1); > > if (BATcount(b)) { > - BAT *id = BATmirror(BATmark(b, 0)); > - BAT *vl = BATmirror(BATmark(BATmirror(b), 0)); > + BAT *uv = mvc_bind_idxbat(m, *sname, *tname, *iname, RD_UPD_VAL); > + BAT *ui = mvc_bind_idxbat(m, *sname, *tname, *iname, RD_UPD_ID); > + BAT *id = BATproject(b, ui); > + BAT *vl = BATproject(b, uv); > + bat_destroy(b); > + bat_destroy(ui); > + bat_destroy(uv); > BBPkeepref(*bid = id->batCacheid); > BBPkeepref(*uvl = vl->batCacheid); > } else { > + sql_schema *s = mvc_bind_schema(m, *sname); > + sql_idx *i = mvc_bind_idx(m, s, *iname); > + > *bid = e_bat(TYPE_oid); > - *uvl = e_bat(b->T->type); > + *uvl = e_bat((i->type==join_idx)?TYPE_oid:TYPE_wrd); > } > BBPreleaseref(b->batCacheid); > } else { > @@ -3959,7 +3991,7 @@ SQLcluster1(Client cntxt, MalBlkPtr mb, > c = o->data; > if (first) { > first = 0; > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > msg = CLUSTER_key(&hid, &b->batCacheid); > BBPreleaseref(b->batCacheid); > if (msg) > @@ -3973,7 +4005,7 @@ SQLcluster1(Client cntxt, MalBlkPtr mb, > throw(SQL, "sql.cluster", "Can not access descriptor"); > } > > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > if (b == NULL) > throw(SQL, "sql.cluster", "Can not access descriptor"); > msg = CLUSTER_apply(&bid, b, map); > @@ -4039,7 +4071,7 @@ SQLcluster2(Client cntxt, MalBlkPtr mb, > bat psum; > unsigned int bits = 10, off = 0; > first = 0; > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > msg = MKEYbathash(&hid, &b->batCacheid); > BBPreleaseref(b->batCacheid); > if (msg) > @@ -4051,7 +4083,7 @@ SQLcluster2(Client cntxt, MalBlkPtr mb, > return msg; > } > > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > if (b == NULL) > throw(SQL, "sql.cluster", "Can not access descriptor"); > msg = CLS_map(&bid, &mid, &b->batCacheid); > @@ -4132,7 +4164,7 @@ vacuum(Client cntxt, MalBlkPtr mb, MalSt > bids[i] = 0; > for (o = t->columns.set->h; o; o = o->next, i++) { > c = o->data; > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > if (b == NULL || (msg = (*func) (&bid, &(b->batCacheid), &(del->batCacheid))) != NULL) { > for (i--; i >= 0; i--) > BBPdecref(bids[i], TRUE); > @@ -4226,7 +4258,7 @@ SQLvacuum(Client cntxt, MalBlkPtr mb, Ma > > for (o = t->columns.set->h; o && ordered == 0; o = o->next) { > c = o->data; > - b = store_funcs.bind_col(tr, c, 0); > + b = store_funcs.bind_col(tr, c, RDONLY); > if (b == NULL) > throw(SQL, "sql.vacuum", "Can not access descriptor"); > ordered |= BATtordered(b); > @@ -4409,7 +4441,7 @@ sql_storage(Client cntxt, MalBlkPtr mb, > for (ncol = (t)->columns.set->h; ncol; ncol = ncol->next) { > sql_base *bc = ncol->data; > sql_column *c = (sql_column *) ncol->data; > - BAT *bn = store_funcs.bind_col(tr, c, 0); > + BAT *bn = store_funcs.bind_col(tr, c, RDONLY); > lng sz; > > /*printf("schema %s.%s.%s" , b->name, bt->name, bc->name); */ > @@ -4474,7 +4506,7 @@ sql_storage(Client cntxt, MalBlkPtr mb, > sql_base *bc = ncol->data; > sql_idx *c = (sql_idx *) ncol->data; > if (c->type != no_idx) { > - BAT *bn = store_funcs.bind_idx(tr, c, 0); > + BAT *bn = store_funcs.bind_idx(tr, c, RDONLY); > lng sz; > > /*printf("schema %s.%s.%s" , b->name, bt->name, bc->name); */ > diff --git a/sql/backends/monet5/sql_gencode.c b/sql/backends/monet5/sql_gencode.c > --- a/sql/backends/monet5/sql_gencode.c > +++ b/sql/backends/monet5/sql_gencode.c > @@ -897,7 +897,7 @@ static int > q = newStmt2(mb, sqlRef, bindRef); > if (q == NULL) > return -1; > - if (s->flag == RD_UPD) { > + if (s->flag == RD_UPD_ID) { > q = pushReturn(mb, q, newTmpVariable(mb, newBatType(ht, tt))); > } else > setVarType(mb, getArg(q, 0), newBatType(ht, tt)); > @@ -910,7 +910,7 @@ static int > return -1; > s->nr = getDestVar(q); > > - if (s->flag == RD_UPD) { > + if (s->flag == RD_UPD_ID) { > /* rename second result */ > renameVariable(mb, getArg(q, 1), "r1_%d", s->nr); > } > @@ -924,7 +924,7 @@ static int > q = newStmt2(mb, sqlRef, bindidxRef); > if (q == NULL) > return -1; > - if (s->flag == RD_UPD) { > + if (s->flag == RD_UPD_ID) { > q = pushReturn(mb, q, newTmpVariable(mb, newBatType(ht, tt))); > } else > setVarType(mb, getArg(q, 0), newBatType(ht, tt)); > @@ -937,7 +937,7 @@ static int > return -1; > s->nr = getDestVar(q); > > - if (s->flag == RD_UPD) { > + if (s->flag == RD_UPD_ID) { > /* rename second result */ > renameVariable(mb, getArg(q, 1), "r1_%d", s->nr); > } > diff --git a/sql/backends/monet5/sql_optimizer.c b/sql/backends/monet5/sql_optimizer.c > --- a/sql/backends/monet5/sql_optimizer.c > +++ b/sql/backends/monet5/sql_optimizer.c > @@ -138,7 +138,7 @@ SQLgetStatistics(Client cntxt, mvc *m, M > if (i && (!isRemote(i->t) && !isMergeTable(i->t))) { > cnt = store_funcs.count_idx(tr, i, 1); > assert(cnt <= (size_t) GDK_oid_max); > - b = store_funcs.bind_idx(m->session->tr, i, 0); > + b = store_funcs.bind_idx(m->session->tr, i, RDONLY); > if (b) { > str loc; > if (b->batPersistence == PERSISTENT && BATlocation(&loc, &b->batCacheid) && loc) > @@ -158,7 +158,7 @@ SQLgetStatistics(Client cntxt, mvc *m, M > > cnt = store_funcs.count_col(tr, c, 1); > assert(cnt <= (size_t) GDK_oid_max); > - b = store_funcs.bind_col(m->session->tr, c, 0); > + b = store_funcs.bind_col(m->session->tr, c, RDONLY); > if (b) { > str loc; > if (b->batPersistence == PERSISTENT && BATlocation(&loc, &b->batCacheid) && loc) > diff --git a/sql/backends/monet5/sql_statistics.c b/sql/backends/monet5/sql_statistics.c > --- a/sql/backends/monet5/sql_statistics.c > +++ b/sql/backends/monet5/sql_statistics.c > @@ -98,7 +98,7 @@ sql_analyze(Client cntxt, MalBlkPtr mb, > for (ncol = (t)->columns.set->h; ncol; ncol = ncol->next) { > sql_base *bc = ncol->data; > sql_column *c = (sql_column *) ncol->data; > - BAT *bn = store_funcs.bind_col(tr, c, 0), *br; > + BAT *bn = store_funcs.bind_col(tr, c, RDONLY), *br; > BAT *bsample; > lng sz = BATcount(bn); > > diff --git a/sql/include/sql_catalog.h b/sql/include/sql_catalog.h > --- a/sql/include/sql_catalog.h > +++ b/sql/include/sql_catalog.h > @@ -92,8 +92,9 @@ > > #define RDONLY 0 > #define RD_INS 1 > -#define RD_UPD 2 > -#define QUICK 3 > +#define RD_UPD_ID 2 > +#define RD_UPD_VAL 3 > +#define QUICK 4 > > #define FRAME_ROWS 0 > #define FRAME_RANGE 1 > diff --git a/sql/storage/bat/bat_storage.c b/sql/storage/bat/bat_storage.c > --- a/sql/storage/bat/bat_storage.c > +++ b/sql/storage/bat/bat_storage.c > @@ -52,7 +52,7 @@ delta_bind_del(sql_dbat *bat, int access > (void) access; /* satisfy compiler */ > #endif > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

1 0

AND / OR evaluation inconsistency
by Scott Mathieson 03 Sep '14

03 Sep '14

Hi, I've come across something that is either a bug or is just my misunderstanding of how MonetDB deals with NULL results from boolean expressions. The issue I'm having is the OR and AND expressions behave differently in when evaluating a bollean expression where one of the variable is NULL (This is using version 11.15.19) If I run a query: SELECT ([some expression that evaluates to NULL] OR [some expression that evaluates to True] ) as "Result" - the returned value is <null> If I try SELECT ([some expression that evaluates to NULL] AND [some expression that evaluates to True] ) as "Result" - the returned value is True This would seem to suggest the AND evaluation is ignoring NULL or Treating NULL as meaning True whereas OR is propagating it. Which is correct and is this a bug? Thanks, Scott Mathieson ________________________________

3 2