- developers-list - monetdb.org

New Oct2020 stable branch created
by Sjoerd Mullender 21 Aug '20

21 Aug '20

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. >From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Oct2020. So in each of your stable checkout directories you can do hg pull hg update -rOct2020 to update to this new branch. -- Sjoerd Mullender

1 0

Possible leaks (MonetDB-11.37.7\common\stream\stream.c)
by Ranier Vilela 04 Aug '20

04 Aug '20

Possible leaks: MonetDB-11.37.7\common\stream\stream. 1. At line 1306 if (stream_bzread(s, buf, 1, UTF8BOMLENGTH) == UTF8BOMLENGTH && strncmp(buf, UTF8BOM, UTF8BOMLENGTH) == 0) { s->isutf8 = true; } else if (s->stream_data.p) { bzp = s->stream_data.p; BZ2_bzReadClose(&err, bzp->b); rewind(bzp->f); bzp->b = BZ2_bzReadOpen(&err, bzp->f, 0, 0, NULL, 0); } If stream_bzread at line 1302 is successful, and (s->stream_data.p) is not null at line 1305, bzp (original pointer malloc) is not lost at line 1306? bzp = s->stream_data.p; 2. At line 1322: if (err != BZ_OK) { stream_bzclose(s); destroy(s); return NULL; } miss free(bzp);? destroy(s) do not free bzp, according, with line 1283. 3. At line 2350 miss free(c); regards, Ranier Vilela

2 4

Possible type with bitwise operator (&)
by Ranier Vilela 03 Aug '20

03 Aug '20

Hi, At gdk\gdk_bat.c (line 1384) variable b->type is bool Would be a typo? if (((b->ttype != TYPE_void) & b->tkey) && b->batCount > 1) { Should be: if (((b->ttype != TYPE_void) && b->tkey) && b->batCount > 1) { regards, Ranier Vilela

4 16

Possible typo with bitwise operator (&)
by Ranier Vilela 01 Aug '20

01 Aug '20

Hi, At gdk\gdk_bat.c (line 1384) variable b->type is bool Would be a typo? if (((b->ttype != TYPE_void) & b->tkey) && b->batCount > 1) { Should be: if (((b->ttype != TYPE_void) && b->tkey) && b->batCount > 1) { regards, Ranier Vilela

1 0

Announcement: New Jun2020-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 27 Jul '20

27 Jul '20

The MonetDB team at MonetDB BV is pleased to announce the Jun2020-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Jun 2020-SP1 bugfix release (11.37.11) MonetDB Common * Hash buckets come in variable widths. But if a BAT grows long enough so that the BAT indexes that are stored in the buckets don't fit anymore, the buckets need to be widened. This is now fixed. Bug Fixes * 6844: sys.getUser('https://me:pw@www.monetdb.org/Doc') does not return the user: me * 6845: the url sys.get...(url) functions do not allow null as a parameter * 6858: json.keyarray(json '{ "":0 }') fails with error: Could not allocate space * 6859: only first character of the separator string in json.text(js json, sep string) is used * 6873: sys.hot_snapshot() creates incomplete snapshots if the write-ahead log is very large * 6876: tar files created by sys.hot_snapshot() produce warnings on some implementations of tar * 6877: MonetDB produces malformed LZ4 files * 6880: Left fuzzy queries are much slower than other fuzzy queries. * 6882: cgroups limits no longer respected? * 6883: SQLancer crash on delete query * 6884: SQLancer generates query with unclear error message * 6885: SQLancer causes assertion error on UTF8_strlen * 6886: SQLancer alter table add unique gives strange error message * 6887: SQLancer crash on complex query * 6888: SQLancer crash on cross join on view * 6889: SQLancer crash on long query * 6892: SQLancer crash on query with HAVING * 6893: SQLancer inner join reporting GDK error * 6894: SQLancer crash on rtrim function * 6895: SQLancer causing 'algebra.select' undefined error * 6896: SQLancer algebra.select' undefined 2 * 6897: SQLancer distinct aggregate with error on group by constant * 6898: SQLancer crash on join query * 6899: SQLancer TLP query with wrong results * 6900: SQLancer generated SIGFPE * 6901: SQLancer TLP query with wrong results 2 * 6902: SQLancer query: batcalc.between undefined * 6903: SQLancer calc.abs undefined * 6904: SQLancer aggr.subavg undefined * 6905: SQLancer TLP query with wrong results 3 * 6906: SQLancer crash on complex join * 6907: SQLancer algebra.select undefined * 6908: SQLancer inputs not the same size * 6909: SQLancer query with wrong results * 6911: SQLancer query: 'calc.bit' undefined * 6917: Decimal parsing fails * 6918: SQLancer query compilation error * 6919: SQLancer insert function doesn't handle utf-8 strings * 6920: SQLancer project_bte: does not match always * 6922: Timestamp columns not migrated to new format * 6923: Imprints data files for timestamp BAT not migrated to the new format * 6924: SQLancer query copy on unique pair of columns fails and complex query with GDK error * 6925: Count string rows in union of string tables leaks (RSS) memory * 6926: SQLancer query with wrong results * 6927: SQLancer inputs not the same size * 6928: SQLancer crash on coalesce * 6929: SQLancer calc.date undefined * 6932: Syntax error while parsing JSON numbers with exponent * 6934: sys.isauuid() returns wrong answer for some invalid uuid strings

1 0

Announcement: New Jun2020 Feature release of MonetDB suite
by Sjoerd Mullender 03 Jun '20

03 Jun '20

The MonetDB team at MonetDB BV is pleased to announce the Jun2020 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Jun 2020 feature release (11.37.7) MonetDB5 Server * The mserver5 option --verbose (-v) was removed. A similar effect can be had by issuing the query CALL logging.setcomplevel('SQL_TRANS', 'INFO'); as the monetdb user. * The functions algebra.intersect, algebra.semijoin, and algebra.crossproduct have an extra argument, bool max_one, which indicates that there must be no more than one match in the join. * The example module opt_sql_append is not installed in the binary packages anymore. * Added mserver5 option (--dbtrace=<path>) in order to be able to specify the output file any produced traces. * Removed function bat.setKey(). * algebra.rangejoin now has two extra arguments: anti:bit and symmetric:bit. * Added session identifier, number of workers and memory claim to the sysmon queue. * The worker (number of threads), memory (in MB) and optimizer pipeline limits can now be set per user session basis. The query and session timeouts are now set in seconds. * With required privileges an user can set resource limits for a session. * There are now versions of group.(sub)group(done) that produce a single output containing just the groups. * algebra.join and algebra.leftjoin now have forms which return a single column. The column that is returned is the left column of the two column version. * The MAL profiler now assigns the SQL TRACE output to the client record thereby avoiding the interaction with other queries, but loosing insight of competing queries. The stethoscope should be used for that. SELinux Support * There was a problem with the MonetDB SELinux support on Fedora 32. That is fixed in this release. In order to do a proper upgrade of the package if you have already installed MonetDB-selinux on Fedora 32, you may need to uninstall (dnf remove) the old package and then install the new. Whole System * A new system to deal with debug output has been implemented. There is now an option --dbtrace to mserver5 that takes a file argument to which debug output is written. The default value is the file mdbtrace.log inside the database directory. This option can also be set through the monetdb program. * The home directory of the automatically created monetdb user was changed from /var/MonetDB to /var/lib/monetdb (RPM based systems only). This home directory is (currently) not used for anything, though. * Python 2 support has been removed. There is now only support for using Python 3. * Removed support for bam and sam files. * Added mserver5 option (--set raw_strings=true|false) and monetdb database property (raw_strings=yes|no) to control interpretation of strings. Merovingian * On Fedora and RHEL systems (not RHEL 6), if monetdbd runs under systemd, when the package is updated, monetdbd (and hence any mserver5 process it runs) is restarted. * Added dbtrace mserver5 option to the daemon in order to set mserver5's output directory for the produced traces. Client Package * The monetdb-client-tools (Debian/Ubuntu) and MonetDB-client-tools (Fedora/RH) containing the stethoscope, tachograph, and tomograph has been removed. A completely new version of stethoscope will be released to replace the old version. * Removed the possibility of using the MD5 checksum for authentication purposes. It was never actively used but was there as an option. Now the option has been removed. * Add port and host as fields in the .monetdb file. * Added 'sessionid' column to system function sys.queue(), so each query gets tagged with the current session identifier * Allow monetdb user to control session and query time out and selectively stopping a client sessions with a soft termination request. ODBC Driver * The NUMERIC and FLOAT types are now handled fully. Before only DECIMAL, FLOAT, and DOUBLE were handled fully. * Some bugs were fixed in the passing back and forth between application and server of values of type GUID (UUID). MonetDB Common * The functions BATintersect, BATsemijoin, and BATsubcross have an extra argument, bool max_one, which indicates that there must be no more than one match in the join. * The "unique" property on BATs was removed. The property indicated that all values in a BAT *had* to be distinct, but this was not actually used. * A new type of candidate list has been introduced. Candidate lists are used internally to specify which rows of a column participate in an operation. Before, candidate lists always contained a list of candidate row IDs. The new candidate list type specifies a list of row IDs that should NOT be considered (negative candidates). * The maximum number of BATs in the system has been increased for 64 bit architectures. * The hash tables used internally by the system now uses a technique based on Linear Hashing which allows them to grow gracefully. This means that hash tables aren't removed and recreated nearly as often anymore. This also meant that the hash table had to be split into two files, which means that after an upgrade the hash tables have to be recreated. * Implemented a version of BATproject, called BATproject2, with two "right" arguments which conceptually follow each other. * Removed MT_mmap and MT_munmap from the list of exported functions. Use GDKmmap and GDKmunmap with the same parameters instead. * Changed the interface of the atom "fix" and "unfix" functions. They now return a value of type gdk_return to indicate success/failure. * Removed the tunique property. The tunique property indicated that all values in the column had to be distinct. It was removed because it wasn't used. * BATrangeselect now has two extra arguments: anti and symmetric (both bool). SQL Frontend * Updating the value of a sequence now requires privilege on its own schema. * The sys.querylog_enable(threshold integer) now actually enables the querylog and uses a threshold in milliseconds. * Removed UNION JOIN statements. They were dropped by the SQL:2003 standard, plus MonetDB implementation was not fully compliant. * The OFFSET value in the COPY INTO query now counts uninterpreted newlines. Before it counted "unquoted record separators" which meant that if you had a single quote on a line that you want to skip, you could not use the feature. * Added support for FROM RANGE MINVALUE TO RANGE MAXVALUE and FROM RANGE MINVALUE TO RANGE MAXVALUE WITH NULL VALUES cases in partitioned tables by range (before they weren't). * Removed support for Python 2. Python 2 itself is no longer supported. Use Python 3 instead. Functions that were declared as LANGUAGE PYTHON2 or LANGUAGE PYTHON2_MAP are changed to LANGUAGE PYTHON and LANGUAGE PYTHON_MAP respectively (without changing the Python code). * Implemented 'covar_pop' and 'covar_samp' aggregate functions, as well as their window function counterparts. Implemented 'stddev_samp', 'stddev_pop', 'var_samp', 'var_pop', 'corr' and 'group_concat' window function correspondents. * Extended SQL catalog with CREATE WINDOW syntax for user-defined SQL:2003 window functions. At the moment, window functions must be defined on the backend engine, i.e. on this case MAL. In the current implementation, the backend code generation creates two additional columns of type lng with the start and end offsets for each row. * Added prepared_statements_args view, which details the arguments for the prepared statements created in the current session. * Added sys.prepared_statements view, which lists the available prepared statements in the current session. * Added deallocate statements with the syntax 'DEALLOCATE [PREPARE] { number | ALL }', to close an existing prepared statement or all, through the SQL layer. Previously this feature was available via MAPI exclusively with the "release" command. * Added support for raw strings using the syntax r'' or R''. This means that C-like escapes will remain uninterpreted within those strings. For instance SELECT r'\"' returns a string of length two. The user needs to escape single quotes by doubling them: SELECT r''''. * Implemented ROLLUP, CUBE and GROUPING SETS from SQL:1999. They define grouping subsets used with the GROUP BY clause in order to compute partial groupings. Also, the GROUPING aggregate was added. This aggregate is a bitmask identifying the grouping columns not present in the generated grouping row when used with the operators described above. * Updated user session procedures by adding the possibility to set properties based on a session identifier. Optimizer pipeline: sys.setoptimizer(int, string) Number of worker threads: sys.setworkerlimit(int, int) Memory limits (in MB): sys.setmemorylimit(int, int) Query timeout (in ms): sys.setquerytimeout(int, int) Session timeout (in ms): sys.setsessiontimeout(int, int) The first argument corresponds to the id of the session to modify, and these procedures are bound to the monetdb user exclusively. The versions of the mentioned procedures with just the second argument were added as well, where the changes are reflected in the current user session, and therefore every user can call them. * The procedures sys.settimeout(bigint), sys.settimeout(bigint,bigint) and sys.session(bigint) are now deprecated. Instead use sys.setquerytimeout and sys.setsessiontimeout mentioned above. * Added SQL procedure sys.hot_snapshot() which can be used to write a snapshot of the database to a tar file. For example, sys.hot_snapshot('/tmp/snapshot.tar'). If compression support is compiled in, snapshots can also be compressed ('/tmp/snapshot.tar.gz'). The tar file expands to a single directory with the same name as the database that was snapshotted. This directory can be passed directly as the --dbpath argument of mserver5 or it can be copied into an existing dbfarm and started from monetdbd. Bug Fixes * 6298: unexpectedly slow execution of SELECT length(fieldname) FROM tablename LIMIT 1 queries * 6401: Suspected memory leak in mserver5 when creating/dropping tables * 6687: Count distinct very slow and use too much the hard drive * 6723: columns aliases duplicates should not be allowed. automatic aliasing required. * 6731: Add system view to allow querying of available prepared statements and their parameters * 6732: Add SQL command to close a specific prepared statement * 6750: Executing a query on a non-existing column on a remote table crashes the remote server * 6785: function sys.isaURL(url) should have been declared as sys.isaURL(string) * 6808: reveal the alarm.sleep procedure in SQL * 6813: function not_uniques(bigint) returns error when called * 6818: usage of multiple column expressions in where-clause (f(a), f(b)) in (select a, b) causes assertion failure on mserver5 * 6821: Failed to start monetdb with embedded python * 6828: Server crashes when executing a window query with ordering by EXTRACT date * 6846: Global temporary table not accessible in other connections / sessions * 6847: A simple way of speeding up impscheck for dense canditers * 6850: Idle timestamp not set * 6851: json parser doesn't parse integers correctly * 6863: thash files not released upon drop table * 6864: (I)LIKE with multiple % doen't find matches

1 0

Re: MonetDB: Jun2020 - added support for single join, ie join and ch...
by Martin Kersten 02 May '20

02 May '20

Hoi Niels, I saw the check-in and ran the SF-100 test. The new results are: [mk@zoom 2020-05-02T21:20:46:SF-100]$ grep all timings.csv SF-100,all, 1, 263.423964 seconds SF-100,all, 2, 267.062088 seconds SF-100,all, 3, 267.019248 seconds Compared to yesterday's email: [mk@zoom 2020-04-29T16:09:48:SF-100]$ grep all timings.csv SF-100,all, 1, 263.779505 seconds SF-100,all, 2, 268.900062 seconds SF-100,all, 3, 269.732425 seconds On 04/04/2020 14:13, Martin Kersten wrote: > I ran the Jun2020 SF100 version on zoom again. > > [mk@zoom 2020-04-04T11:57:10:SF-100]$ grep all timings.csv > SF-100,all, 1, 238.894476 seconds > SF-100,all, 2, 246.591707 seconds > SF-100,all, 3, 244.333189 seconds > > average 243

1 0

Re: MonetDB: msk-type - Initial support for the msk type.
by Martin Kersten 24 Apr '20

24 Apr '20

wil je dat ik het doortrek naar MAL niveau? > On 24 Apr 2020, at 17:30, Sjoerd Mullender <commits+sjoerd=acm.org(a)monetdb.org> wrote: > > Changeset: abf00b42a524 for MonetDB > URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=abf00b42a524 > Modified Files: > clients/Tests/exports.stable.out > gdk/gdk.h > gdk/gdk_atoms.c > gdk/gdk_bat.c > gdk/gdk_batop.c > gdk/gdk_calc.c > gdk/gdk_hash.c > gdk/gdk_value.c > monetdb5/mal/mal_type.c > monetdb5/modules/atoms/inet.c > monetdb5/modules/kernel/bat5.c > sql/backends/monet5/UDF/capi/capi.c > sql/backends/monet5/sql_result.c > Branch: msk-type > Log Message: > > Initial support for the msk type. > > > diffs (truncated from 1402 to 300 lines): > > diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out > --- a/clients/Tests/exports.stable.out > +++ b/clients/Tests/exports.stable.out > @@ -1273,7 +1273,7 @@ str INETnetmask(inet *retval, const inet > str INETnetwork(inet *retval, const inet *val); > str INETnew(inet *retval, str *in); > const inet *INETnull(void); > -str INETsetmasklen(inet *retval, const inet *val, const int *msk); > +str INETsetmasklen(inet *retval, const inet *val, const int *mask); > str INETtext(str *retval, const inet *val); > ssize_t INETtoString(str *retval, size_t *len, const inet *handle, bool external); > str INSPECTatom_names(bat *ret); > diff --git a/gdk/gdk.h b/gdk/gdk.h > --- a/gdk/gdk.h > +++ b/gdk/gdk.h > @@ -425,7 +425,8 @@ > > enum { > TYPE_void = 0, > - TYPE_bit, > + TYPE_msk, /* bit mask */ > + TYPE_bit, /* TRUE, FALSE, or nil */ > TYPE_bte, > TYPE_sht, > TYPE_bat, /* BAT id: index in BBPcache */ > @@ -445,6 +446,7 @@ enum { > TYPE_any = 255, /* limit types to <255! */ > }; > > +typedef bool msk; > typedef int8_t bit; > typedef int8_t bte; > typedef int16_t sht; > @@ -602,6 +604,7 @@ typedef struct { > oid oval; > sht shval; > bte btval; > + msk mval; > flt fval; > ptr pval; > bat bval; > @@ -626,7 +629,7 @@ gdk_export void VALclear(ValPtr v); > gdk_export ValPtr VALset(ValPtr v, int t, void *p); > gdk_export void *VALget(ValPtr v); > gdk_export int VALcmp(const ValRecord *p, const ValRecord *q); > -gdk_export int VALisnil(const ValRecord *v); > +gdk_export bool VALisnil(const ValRecord *v); > > /* > * @- The BAT record > @@ -746,7 +749,10 @@ typedef struct BAT { > > typedef struct BATiter { > BAT *b; > - oid tvid; > + union { > + oid tvid; > + bool tmsk; > + }; > } BATiter; > > /* macros to hide complexity of the BAT structure */ > @@ -772,6 +778,33 @@ typedef struct BATiter { > #define tprops T.props > > > +/* some access functions for the bitmask type */ > +static inline void > +mskSet(BAT *b, BUN p) > +{ > + ((uint32_t *) b->theap.base)[p / 32] |= 1U << (p % 32); > +} > + > +static inline void > +mskClr(BAT *b, BUN p) > +{ > + ((uint32_t *) b->theap.base)[p / 32] &= ~(1U << (p % 32)); > +} > + > +static inline void > +mskSetVal(BAT *b, BUN p, msk v) > +{ > + if (v) > + mskSet(b, p); > + else > + mskClr(b, p); > +} > + > +static inline msk > +mskGet(BAT *b, BUN p) > +{ > + return ((uint32_t *) b->theap.base)[p / 32] & (1U << (p % 32)); > +} > > /* > * @- Heap Management > @@ -925,7 +958,11 @@ gdk_export BUN BUNfnd(BAT *b, const void > > #define Tsize(b) ((b)->twidth) > > -#define tailsize(b,p) ((b)->ttype?((size_t)(p))<<(b)->tshift:0) > +#define tailsize(b,p) ((b)->ttype ? \ > + (ATOMstorage((b)->ttype) == TYPE_msk ? \ > + (((size_t) (p) + 31) / 32) * 4 : \ > + ((size_t) (p)) << (b)->tshift) : \ > + 0) > > #define Tloc(b,p) ((void *)((b)->theap.base+(((size_t)(p))<<(b)->tshift))) > > @@ -935,7 +972,7 @@ typedef var_t stridx_t; > > #define BUNtvaroff(bi,p) VarHeapVal((bi).b->theap.base, (p), (bi).b->twidth) > > -#define BUNtloc(bi,p) Tloc((bi).b,p) > +#define BUNtloc(bi,p) (ATOMstorage((bi).b->ttype) == TYPE_msk ? Tmsk(&(bi), p) : Tloc((bi).b,p)) > #define BUNtpos(bi,p) Tpos(&(bi),p) > #define BUNtvar(bi,p) (assert((bi).b->ttype && (bi).b->tvarsized), (void *) (Tbase((bi).b)+BUNtvaroff(bi,p))) > #define BUNtail(bi,p) ((bi).b->ttype?(bi).b->tvarsized?BUNtvar(bi,p):BUNtloc(bi,p):BUNtpos(bi,p)) > @@ -1519,6 +1556,8 @@ Tputvalue(BAT *b, BUN p, const void *v, > break; > #endif > } > + } else if (b->ttype == TYPE_msk) { > + mskSetVal(b, p, * (msk *) v); > } else { > return ATOMputFIX(b->ttype, Tloc(b, p), v); > } > @@ -1528,7 +1567,11 @@ Tputvalue(BAT *b, BUN p, const void *v, > static inline gdk_return __attribute__((__warn_unused_result__)) > tfastins_nocheck(BAT *b, BUN p, const void *v, int s) > { > - b->theap.free += s; > + if (ATOMstorage(b->ttype) == TYPE_msk) { > + if (p % 32 == 0) > + b->theap.free += 4; > + } else > + b->theap.free += s; > return Tputvalue(b, p, v, false); > } > > @@ -1676,6 +1719,7 @@ VALptr(const ValRecord *v) > { > switch (ATOMstorage(v->vtype)) { > case TYPE_void: return (const void *) &v->val.oval; > + case TYPE_msk: return (const void *) &v->val.mval; > case TYPE_bte: return (const void *) &v->val.btval; > case TYPE_sht: return (const void *) &v->val.shval; > case TYPE_int: return (const void *) &v->val.ival; > @@ -1734,8 +1778,6 @@ gdk_export void *THRdata[THREADDATA]; > #define THRget_errbuf(t) ((char*)t->data[2]) > #define THRset_errbuf(t,b) (t->data[2] = b) > > -#ifndef GDK_NOLINK > - > static inline bat > BBPcheck(bat x, const char *y) > { > @@ -1773,7 +1815,12 @@ Tpos(BATiter *bi, BUN p) > return (void*)&bi->tvid; > } > > -#endif > +static inline void * > +Tmsk(BATiter *bi, BUN p) > +{ > + bi->tmsk = mskGet(bi->b, p); > + return &bi->tmsk; > +} > > /* > * @+ Transaction Management > diff --git a/gdk/gdk_atoms.c b/gdk/gdk_atoms.c > --- a/gdk/gdk_atoms.c > +++ b/gdk/gdk_atoms.c > @@ -36,6 +36,12 @@ > * NaN). */ > > static int > +mskCmp(const msk *l, const msk *r) > +{ > + return (*l > *r) - (*l < *r); > +} > + > +static int > bteCmp(const bte *l, const bte *r) > { > return (*l > *r) - (*l < *r); > @@ -450,6 +456,42 @@ voidWrite(const void *a, stream *s, size > * experiments showed that library function is even slightly faster and we > * now also support True/False (and trUe/FAlSE should this become a thing). > */ > +static ssize_t > +mskFromStr(const char *src, size_t *len, msk **dst, bool external) > +{ > + const char *p = src; > + > + (void) external; > + atommem(sizeof(msk)); > + > + if (strNil(src)) > + return -1; > + > + while (GDKisspace(*p)) > + p++; > + if (*p == '0') { > + **dst = 0; > + p++; > + } else if (*p == '1') { > + **dst = 1; > + p++; > + } else { > + return -1; > + } > + while (GDKisspace(*p)) > + p++; > + return (ssize_t) (p - src); > +} > + > +static ssize_t > +mskToStr(char **dst, size_t *len, const msk *src, bool external) > +{ > + (void) external; > + atommem(2); > + strcpy(*dst, *src ? "1" : "0"); > + return 1; > +} > + > ssize_t > bitFromStr(const char *src, size_t *len, bit **dst, bool external) > { > @@ -1202,6 +1244,17 @@ atomDesc BATatoms[MAXATOMS] = { > .atomCmp = (int (*)(const void *, const void *)) bteCmp, > .atomHash = (BUN (*)(const void *)) bteHash, > }, > + [TYPE_msk] = { > + .name = "msk", > + .storage = TYPE_msk, > + .linear = false, > + .size = 1, /* really 1/8 */ > + .atomFromStr = (ssize_t (*)(const char *, size_t *, void **, bool)) mskFromStr, > + .atomToStr = (ssize_t (*)(char **, size_t *, const void *, bool)) mskToStr, > +// .atomRead = (void *(*)(void *, stream *, size_t)) mskRead, > +// .atomWrite = (gdk_return (*)(const void *, stream *, size_t)) mskWrite, > + .atomCmp = (int (*)(const void *, const void *)) mskCmp, > + }, > [TYPE_bte] = { > .name = "bte", > .storage = TYPE_bte, > diff --git a/gdk/gdk_bat.c b/gdk/gdk_bat.c > --- a/gdk/gdk_bat.c > +++ b/gdk/gdk_bat.c > @@ -184,7 +184,9 @@ COLnew(oid hseq, int tt, BUN cap, role_t > /* round up to multiple of BATTINY */ > if (cap < BUN_MAX - BATTINY) > cap = (cap + BATTINY - 1) & ~(BATTINY - 1); > - if (cap < BATTINY) > + if (ATOMstorage(tt) == TYPE_msk && cap < 8*BATTINY) > + cap = 8*BATTINY; > + else if (cap < BATTINY) > cap = BATTINY; > /* limit the size */ > if (cap > BUN_MAX) > @@ -197,6 +199,9 @@ COLnew(oid hseq, int tt, BUN cap, role_t > BATsetdims(bn); > bn->batCapacity = cap; > > + if (ATOMstorage(tt) == TYPE_msk) > + cap /= 8; /* 8 values per byte */ > + > /* alloc the main heaps */ > if (tt && HEAPalloc(&bn->theap, cap, bn->twidth) != GDK_SUCCEED) { > goto bailout; > @@ -339,12 +344,16 @@ BATattach(int tt, const char *heapfile, > GDKerror("heapfile size not integral number of atoms\n"); > return NULL; > } > - if ((size_t) (st.st_size / atomsize) > (size_t) BUN_MAX) { > + if (ATOMstorage(tt) == TYPE_msk ? > + (st.st_size > (off_t) (BUN_MAX / 8)) : > + ((size_t) (st.st_size / atomsize) > (size_t) BUN_MAX)) { > fclose(f); > GDKerror("heapfile too large\n"); > return NULL; > } > - cap = (BUN) (st.st_size / atomsize); > + cap = (BUN) (ATOMstorage(tt) == TYPE_msk ? > + st.st_size * 8 : > + st.st_size / atomsize); > bn = COLnew(0, tt, cap, role); > if (bn == NULL) { > fclose(f); > @@ -371,8 +380,8 @@ BATattach(int tt, const char *heapfile, > bn->trevsorted = false; > bn->tkey = false; > } else { > - bn->tsorted = true; > - bn->trevsorted = true; > + bn->tsorted = ATOMlinear(tt); > + bn->trevsorted = ATOMlinear(tt); > bn->tkey = true; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

2 1

Re: MonetDB: msk-type - Initial support for the msk type.
by Martin Kersten 24 Apr '20

24 Apr '20

wil je dat ik het doortrek naar MAL niveau? > On 24 Apr 2020, at 17:30, Sjoerd Mullender <commits+sjoerd=acm.org(a)monetdb.org> wrote: > > Changeset: abf00b42a524 for MonetDB > URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=abf00b42a524 > Modified Files: > clients/Tests/exports.stable.out > gdk/gdk.h > gdk/gdk_atoms.c > gdk/gdk_bat.c > gdk/gdk_batop.c > gdk/gdk_calc.c > gdk/gdk_hash.c > gdk/gdk_value.c > monetdb5/mal/mal_type.c > monetdb5/modules/atoms/inet.c > monetdb5/modules/kernel/bat5.c > sql/backends/monet5/UDF/capi/capi.c > sql/backends/monet5/sql_result.c > Branch: msk-type > Log Message: > > Initial support for the msk type. > > > diffs (truncated from 1402 to 300 lines): > > diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out > --- a/clients/Tests/exports.stable.out > +++ b/clients/Tests/exports.stable.out > @@ -1273,7 +1273,7 @@ str INETnetmask(inet *retval, const inet > str INETnetwork(inet *retval, const inet *val); > str INETnew(inet *retval, str *in); > const inet *INETnull(void); > -str INETsetmasklen(inet *retval, const inet *val, const int *msk); > +str INETsetmasklen(inet *retval, const inet *val, const int *mask); > str INETtext(str *retval, const inet *val); > ssize_t INETtoString(str *retval, size_t *len, const inet *handle, bool external); > str INSPECTatom_names(bat *ret); > diff --git a/gdk/gdk.h b/gdk/gdk.h > --- a/gdk/gdk.h > +++ b/gdk/gdk.h > @@ -425,7 +425,8 @@ > > enum { > TYPE_void = 0, > - TYPE_bit, > + TYPE_msk, /* bit mask */ > + TYPE_bit, /* TRUE, FALSE, or nil */ > TYPE_bte, > TYPE_sht, > TYPE_bat, /* BAT id: index in BBPcache */ > @@ -445,6 +446,7 @@ enum { > TYPE_any = 255, /* limit types to <255! */ > }; > > +typedef bool msk; > typedef int8_t bit; > typedef int8_t bte; > typedef int16_t sht; > @@ -602,6 +604,7 @@ typedef struct { > oid oval; > sht shval; > bte btval; > + msk mval; > flt fval; > ptr pval; > bat bval; > @@ -626,7 +629,7 @@ gdk_export void VALclear(ValPtr v); > gdk_export ValPtr VALset(ValPtr v, int t, void *p); > gdk_export void *VALget(ValPtr v); > gdk_export int VALcmp(const ValRecord *p, const ValRecord *q); > -gdk_export int VALisnil(const ValRecord *v); > +gdk_export bool VALisnil(const ValRecord *v); > > /* > * @- The BAT record > @@ -746,7 +749,10 @@ typedef struct BAT { > > typedef struct BATiter { > BAT *b; > - oid tvid; > + union { > + oid tvid; > + bool tmsk; > + }; > } BATiter; > > /* macros to hide complexity of the BAT structure */ > @@ -772,6 +778,33 @@ typedef struct BATiter { > #define tprops T.props > > > +/* some access functions for the bitmask type */ > +static inline void > +mskSet(BAT *b, BUN p) > +{ > + ((uint32_t *) b->theap.base)[p / 32] |= 1U << (p % 32); > +} > + > +static inline void > +mskClr(BAT *b, BUN p) > +{ > + ((uint32_t *) b->theap.base)[p / 32] &= ~(1U << (p % 32)); > +} > + > +static inline void > +mskSetVal(BAT *b, BUN p, msk v) > +{ > + if (v) > + mskSet(b, p); > + else > + mskClr(b, p); > +} > + > +static inline msk > +mskGet(BAT *b, BUN p) > +{ > + return ((uint32_t *) b->theap.base)[p / 32] & (1U << (p % 32)); > +} > > /* > * @- Heap Management > @@ -925,7 +958,11 @@ gdk_export BUN BUNfnd(BAT *b, const void > > #define Tsize(b) ((b)->twidth) > > -#define tailsize(b,p) ((b)->ttype?((size_t)(p))<<(b)->tshift:0) > +#define tailsize(b,p) ((b)->ttype ? \ > + (ATOMstorage((b)->ttype) == TYPE_msk ? \ > + (((size_t) (p) + 31) / 32) * 4 : \ > + ((size_t) (p)) << (b)->tshift) : \ > + 0) > > #define Tloc(b,p) ((void *)((b)->theap.base+(((size_t)(p))<<(b)->tshift))) > > @@ -935,7 +972,7 @@ typedef var_t stridx_t; > > #define BUNtvaroff(bi,p) VarHeapVal((bi).b->theap.base, (p), (bi).b->twidth) > > -#define BUNtloc(bi,p) Tloc((bi).b,p) > +#define BUNtloc(bi,p) (ATOMstorage((bi).b->ttype) == TYPE_msk ? Tmsk(&(bi), p) : Tloc((bi).b,p)) > #define BUNtpos(bi,p) Tpos(&(bi),p) > #define BUNtvar(bi,p) (assert((bi).b->ttype && (bi).b->tvarsized), (void *) (Tbase((bi).b)+BUNtvaroff(bi,p))) > #define BUNtail(bi,p) ((bi).b->ttype?(bi).b->tvarsized?BUNtvar(bi,p):BUNtloc(bi,p):BUNtpos(bi,p)) > @@ -1519,6 +1556,8 @@ Tputvalue(BAT *b, BUN p, const void *v, > break; > #endif > } > + } else if (b->ttype == TYPE_msk) { > + mskSetVal(b, p, * (msk *) v); > } else { > return ATOMputFIX(b->ttype, Tloc(b, p), v); > } > @@ -1528,7 +1567,11 @@ Tputvalue(BAT *b, BUN p, const void *v, > static inline gdk_return __attribute__((__warn_unused_result__)) > tfastins_nocheck(BAT *b, BUN p, const void *v, int s) > { > - b->theap.free += s; > + if (ATOMstorage(b->ttype) == TYPE_msk) { > + if (p % 32 == 0) > + b->theap.free += 4; > + } else > + b->theap.free += s; > return Tputvalue(b, p, v, false); > } > > @@ -1676,6 +1719,7 @@ VALptr(const ValRecord *v) > { > switch (ATOMstorage(v->vtype)) { > case TYPE_void: return (const void *) &v->val.oval; > + case TYPE_msk: return (const void *) &v->val.mval; > case TYPE_bte: return (const void *) &v->val.btval; > case TYPE_sht: return (const void *) &v->val.shval; > case TYPE_int: return (const void *) &v->val.ival; > @@ -1734,8 +1778,6 @@ gdk_export void *THRdata[THREADDATA]; > #define THRget_errbuf(t) ((char*)t->data[2]) > #define THRset_errbuf(t,b) (t->data[2] = b) > > -#ifndef GDK_NOLINK > - > static inline bat > BBPcheck(bat x, const char *y) > { > @@ -1773,7 +1815,12 @@ Tpos(BATiter *bi, BUN p) > return (void*)&bi->tvid; > } > > -#endif > +static inline void * > +Tmsk(BATiter *bi, BUN p) > +{ > + bi->tmsk = mskGet(bi->b, p); > + return &bi->tmsk; > +} > > /* > * @+ Transaction Management > diff --git a/gdk/gdk_atoms.c b/gdk/gdk_atoms.c > --- a/gdk/gdk_atoms.c > +++ b/gdk/gdk_atoms.c > @@ -36,6 +36,12 @@ > * NaN). */ > > static int > +mskCmp(const msk *l, const msk *r) > +{ > + return (*l > *r) - (*l < *r); > +} > + > +static int > bteCmp(const bte *l, const bte *r) > { > return (*l > *r) - (*l < *r); > @@ -450,6 +456,42 @@ voidWrite(const void *a, stream *s, size > * experiments showed that library function is even slightly faster and we > * now also support True/False (and trUe/FAlSE should this become a thing). > */ > +static ssize_t > +mskFromStr(const char *src, size_t *len, msk **dst, bool external) > +{ > + const char *p = src; > + > + (void) external; > + atommem(sizeof(msk)); > + > + if (strNil(src)) > + return -1; > + > + while (GDKisspace(*p)) > + p++; > + if (*p == '0') { > + **dst = 0; > + p++; > + } else if (*p == '1') { > + **dst = 1; > + p++; > + } else { > + return -1; > + } > + while (GDKisspace(*p)) > + p++; > + return (ssize_t) (p - src); > +} > + > +static ssize_t > +mskToStr(char **dst, size_t *len, const msk *src, bool external) > +{ > + (void) external; > + atommem(2); > + strcpy(*dst, *src ? "1" : "0"); > + return 1; > +} > + > ssize_t > bitFromStr(const char *src, size_t *len, bit **dst, bool external) > { > @@ -1202,6 +1244,17 @@ atomDesc BATatoms[MAXATOMS] = { > .atomCmp = (int (*)(const void *, const void *)) bteCmp, > .atomHash = (BUN (*)(const void *)) bteHash, > }, > + [TYPE_msk] = { > + .name = "msk", > + .storage = TYPE_msk, > + .linear = false, > + .size = 1, /* really 1/8 */ > + .atomFromStr = (ssize_t (*)(const char *, size_t *, void **, bool)) mskFromStr, > + .atomToStr = (ssize_t (*)(char **, size_t *, const void *, bool)) mskToStr, > +// .atomRead = (void *(*)(void *, stream *, size_t)) mskRead, > +// .atomWrite = (gdk_return (*)(const void *, stream *, size_t)) mskWrite, > + .atomCmp = (int (*)(const void *, const void *)) mskCmp, > + }, > [TYPE_bte] = { > .name = "bte", > .storage = TYPE_bte, > diff --git a/gdk/gdk_bat.c b/gdk/gdk_bat.c > --- a/gdk/gdk_bat.c > +++ b/gdk/gdk_bat.c > @@ -184,7 +184,9 @@ COLnew(oid hseq, int tt, BUN cap, role_t > /* round up to multiple of BATTINY */ > if (cap < BUN_MAX - BATTINY) > cap = (cap + BATTINY - 1) & ~(BATTINY - 1); > - if (cap < BATTINY) > + if (ATOMstorage(tt) == TYPE_msk && cap < 8*BATTINY) > + cap = 8*BATTINY; > + else if (cap < BATTINY) > cap = BATTINY; > /* limit the size */ > if (cap > BUN_MAX) > @@ -197,6 +199,9 @@ COLnew(oid hseq, int tt, BUN cap, role_t > BATsetdims(bn); > bn->batCapacity = cap; > > + if (ATOMstorage(tt) == TYPE_msk) > + cap /= 8; /* 8 values per byte */ > + > /* alloc the main heaps */ > if (tt && HEAPalloc(&bn->theap, cap, bn->twidth) != GDK_SUCCEED) { > goto bailout; > @@ -339,12 +344,16 @@ BATattach(int tt, const char *heapfile, > GDKerror("heapfile size not integral number of atoms\n"); > return NULL; > } > - if ((size_t) (st.st_size / atomsize) > (size_t) BUN_MAX) { > + if (ATOMstorage(tt) == TYPE_msk ? > + (st.st_size > (off_t) (BUN_MAX / 8)) : > + ((size_t) (st.st_size / atomsize) > (size_t) BUN_MAX)) { > fclose(f); > GDKerror("heapfile too large\n"); > return NULL; > } > - cap = (BUN) (st.st_size / atomsize); > + cap = (BUN) (ATOMstorage(tt) == TYPE_msk ? > + st.st_size * 8 : > + st.st_size / atomsize); > bn = COLnew(0, tt, cap, role); > if (bn == NULL) { > fclose(f); > @@ -371,8 +380,8 @@ BATattach(int tt, const char *heapfile, > bn->trevsorted = false; > bn->tkey = false; > } else { > - bn->tsorted = true; > - bn->trevsorted = true; > + bn->tsorted = ATOMlinear(tt); > + bn->trevsorted = ATOMlinear(tt); > bn->tkey = true; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

1 0

New Jun2020 stable branch created
by Sjoerd Mullender 18 Mar '20

18 Mar '20

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. >From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jun2020. So in each of your stable checkout directories you can do hg pull hg update -rJun2020 to update to this new branch. -- Sjoerd Mullender

1 0