developers-list November 2016

developers-list@monetdb.org

8 participants
9 discussions

MAL: How to interate through group values?

by Gatis Ozoliņš

I am trying to create my own MAL aggregate function. I have implemented main function and it works OK: function group_concat(b:bat[:oid, :str],s:bat[:oid, :str]):str; sep := ""; barrier (h, t) := iterator.new(s); sep := t; exit (h, t); value := nil:str; barrier (h, t) := iterator.new(b); barrier bnormal := value; value := value + sep; value := value + t; exit bnormal; barrier bnil := calc.isnil(value); value := t; exit bnil; redo (h, t) := iterator.next(b); exit (h, t); return value; end group_concat; Now I need to implemented subfunction with following definition: function subgroup_concat(b:bat[:oid,:str],s:bat[:oid,:str],g:bat[:oid,:oid],e:bat[:oid,:any_2],skip_nils:bit) :bat[:oid,:str]; I can iterate over all groups g, but I can't find correct MAL function to get all values (b,s) for this specific group. Which MAL function would return list of oid for specific group g entry ? Gatis

5 years, 8 months

.gdk_lock and the GDK locking scheme

by Eyal Rozenberg

I used to think .gdk_lock was used by mserver the same way Linux daemons use lock files - the filesystem enforces serialization, so if you do: 1. if (lock file present) then locking failed 2. create the lock file 3. if (creation failed) then locking failed 4. locking successful that's a valid non-blocking locking procedure. However, it appears that: * .gdk_lock is not always empty, and can have text such as USR=12096163 PID=5844 TIME=Thu Nov 24 15:35:08 2016 @ LOGON USR=12096163 PID=5844 TIME=Thu Nov 24 15:42:26 2016 @ LOGOFF * If .gdk_lock is not always created on DB startup, nor deleted on shutdown. * The locking using .gdk_lock is actually an flock() on its 5th byte (offset 4) and never any other one. * The GDKunlockhome() file will cause a second file descriptor opening in addition to GDKlockfile So I'm a bit confused about the locking scheme. Help? Eyal

5 years, 8 months

Questions about BATs and their names in BBP's

by Eyal Rozenberg

So, I'm listing the BATs in a BBP (of a TPCH database), and I'm noticing several strange things: 1. There's a h-u-g-e number of BATs. Actually I knew this to be the case already, since BBPs have so many files, but, it's still striking. 863 BATs for a database with 61 bona fide columns (in lineitem, customer etc.) 2. There are a lot of BATs with null storage. And I do mean a lot - 514 out of 863. What are these? why do they need to exist? And note this is a database in which nothing has occurred except one-time loading. 3. Most BATs have dummy generated names, e.g. tmp_547 - not, say, a concatenation of their table name and column name. Yet - some BATs _do_ have meaningful names: environment_key, sql_dsnapshots, stat_opt_calls and so on. Why not have meaningful names for all BATs then? and lastly, and perhaps most importantly: 5. What's the best/most appropriate/most quick-and-dirty way of obtaining table and column names for those BATs which have them? 6. If it's through the storage table - is there a way to determine which BATs correspond to its columns (table, column, location at the very least)?

5 years, 8 months

Can I expect to be able to use BBP code without GDKinit'ing?

by Eyal Rozenberg

Bottom line question: BBPinit() is not exposed in gdk_bbp.h, but rather only called from GDKinit() (which sees it through gdk_private.h). I just want to load data from persisted BAT files - not into MonetDB and not within the mserver5 process - so, speaking conceptually, I do not want to initialize the GDK, but do want to initialize the BBP. Questions: * Should BBPinit() work outside the scope of a GDKinit()? * If not, is it adaptable so as to allow this? * More generally, how much of GDK do I need to have running, just so as load persisted data into memory, using the code in gdk_bbp (or a slight variation thereof)? Now for the introduction and the motivation: As I mentioned a few MADADMs ago, I'll need to load persisted MonetDB columns into my GPU kernel testbench. This is relatively simple for numeric columns, given some scripting work to extract catalog data in a parsable format or to build named symlinks to columns (eg /path/to/dbfarm/tpch-sf-1/named_bats/lineitem/l_shipdate -> /path/to/dbfarm/tpch-sf-1/bat/12/34.tail). But it won't do for other kinds of data, most importantly strings - which I do want to work on. Plus, writing a wider-scope loader will let me avoid depending on MonetDB running for access to persisted columns. So, I'm writing a (selective) loader of persisted MonetDB columns, or a BBP loader if you will. My strategy is the following: 1. Copying the files in the GDK codebase which are necessary for building a binary which can call all code in gdk_bbp.h and not have unmet dependencies (this is mostly done). 2. Get that leaner slice of code, with a small main(), to actually work, i.e. not fail due to weird errors or result in junk data. 3. Peel away functions from the code I've copied from the repository which are not actually used. 4. Peel away the parts of the code which are not necessary for the actual loading - from within functions. In this initial stage this may involve work that becomes unnecessary when you have auxiliary data obtained from querying the DB (such as table-column name to BAT filename mapping). 5. Expand functionality and/or optimize performance after the peeling and/or C++ify for integration with my code 6. If other people / MDBS are interested, collaborate on making this support different BBP versions - so that the result is an inter-version export/forensics library.

5 years, 8 months

Re: developers-list Digest, Vol 51, Issue 4

by Joweria Nakibuuka

the command i used: ./configure --prefix=/home/joweria/MonetDB-Nov2016-SP4 --enable-geom=yes --enable-pyintegration=yes --with-geos=yes and then after compilation under the LDFLAGS-enabled componets geom is one of the enabled componet 2016-11-10 13:00 GMT+02:00 <developers-list-request(a)monetdb.org>: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka) > 2. Re: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Ying Zhang) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 9 Nov 2016 17:39:17 +0100 > From: Joweria Nakibuuka <joweria.nakibuuka(a)unitn.it> > To: developers-list(a)monetdb.org > Subject: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > Message-ID: > <CANbNUqZfu_pfzsZ=xUesfyLQPw3Ji3DW_fYeNOSZwfkTw= > 1Hsg(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Dear Admin, > I am running Ubuntu linux 14.04 LTS, x86_64-Ubuntu-linux-gnu/64bit with > 128bit integers. i recently installed MonetDB5 default branch from the > mercurial repository. > > I tried to rebuild and enable geom module but to no avail, it shows no > error, after compilation it shows that the module is enabled,However when i > run the mserver the monetDB/GIS module is not loaded as expected. I am > stuck and i don't seem to know how to proceed forward in order to run my > spatial queries > > Are there any setting i need to do explicitly to load this module? > > Thank you > > Regards, > Joweria nakibuuka >

5 years, 8 months

Need Help on enabling GIS (MonetDB/GIS) module in MonetDB

by Joweria Nakibuuka

Dear Admin, I am running Ubuntu linux 14.04 LTS, x86_64-Ubuntu-linux-gnu/64bit with 128bit integers. i recently installed MonetDB5 default branch from the mercurial repository. I tried to rebuild and enable geom module but to no avail, it shows no error, after compilation it shows that the module is enabled,However when i run the mserver the monetDB/GIS module is not loaded as expected. I am stuck and i don't seem to know how to proceed forward in order to run my spatial queries Are there any setting i need to do explicitly to load this module? Thank you Regards, Joweria nakibuuka

5 years, 8 months

Two new juicy assertions in the SQL front-end

by Dean De Leo

Dear MonetDB developers, I'm reporting two other assertions I hit in the front-end. The first is related to scalar subqueries, see query3.sql for a repro and backtrace3.txt. I guess it could be rather considered a feature request... BTW I appreciate this style of treating unsupported features, I'm starting to use in my changes as well :) The second occurs somewhere in the query rewriter when issuing the whole query, see query4.sql and backtrace4.txt. On Linux, tip of the default branch, debug build, empty tables, schema attached. Thanks, Dean

5 years, 8 months

Two sweet crashes in the sql front-end

by Dean De Leo

Dear MonetDB developers, I would like to report a crash due to a null pointer using the attached schema.sql and query1a.sql, see backtrace1.txt. Eventually in the process of simplyfying the query to make this report, I stumbled upon a second assertion, see query2_crash_repro.sql and backtrace2.txt On Linux, tip of the default branch, debug build, empty tables. Dean

5 years, 8 months

Re: MonetDB: malupgrade - Clear variables before resetting mb->vtop.

by Martin Kersten

WOW .... that took me quite some time nailing the example and looking into the wrong direction thx On 07/11/16 12:38, Sjoerd Mullender wrote: > Changeset: 8ac922886df2 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=8ac922886df2 > Modified Files: > monetdb5/mal/mal_instruction.c > Branch: malupgrade > Log Message: > > Clear variables before resetting mb->vtop. > > > diffs (20 lines): > > diff --git a/monetdb5/mal/mal_instruction.c b/monetdb5/mal/mal_instruction.c > --- a/monetdb5/mal/mal_instruction.c > +++ b/monetdb5/mal/mal_instruction.c > @@ -197,13 +197,13 @@ freeMalBlk(MalBlkPtr mb) > mb->stmt[i] = NULL; > } > mb->stop = 0; > + for(i=0; i< mb->vtop; i++) > + if (isVarConstant(mb, i)) > + VALclear(&getVarConstant(mb,i)); > mb->vtop = 0; > mb->vid = 0; > GDKfree(mb->stmt); > mb->stmt = 0; > - for(i=0; i< mb->vtop; i++) > - if (isVarConstant(mb, i)) > - VALclear(&getVarConstant(mb,i)); > GDKfree(mb->var); > mb->var = 0; > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

5 years, 8 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

developers-list November 2016