users-list June 2019

users-list@monetdb.org

12 participants
10 discussions

Announcement: New Apr2019 Feature release of MonetDB suite

by Sjoerd Mullender

The MonetDB team at CWI/MonetDB BV is pleased to announce the Apr2019 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Apr 2019 feature release (11.33.3) MonetDB5 Server * Changed the way blobs are ordered. Before, shorter blobs came before longer, now the contents is compared first. * Removed function blob.tostring() since it cannot guarantee that the resulting string is properly encoded in UTF-8. * Removed the type "sqlblob", changed the external format of the type "blob" to be like what "sqlblob" used to be. In other words, the "blob" type is now SQL-compatible. * The MAL functions algebra.sort and algebra.firstn now have a new, extra second-to-last argument, nilslast:bit, which indicates where NIL values are to be sorted: at the beginning, or at the end. The old behavior is when for algebra.sort, the value of nilslast is equal to the value or the reverse:bit parameter, for algebra.firstn when the value of nilslast is opposite to the value of the asc:bit parameter. Build Environment * We now use the C11 atomic operations if available, although libatomic_ops is still used if present and not disabled on the configure command line. * We now also build the Python 3 integration packages for Fedora, Debian, Ubuntu, and Windows (not for Debian 8, Ubuntu 14.04, or EPEL 6 and 7 which are too old). SELinux Support * There is one tunable parameter, mserver5_can_read_home, which can be set using "setsebool -P mserver5_can_read_home=true" to allow an mserver5 process started by monetdbd under the control of systemd to read files in users' home directories. Merovingian * Added a new database property named `profilerbeatfreq`. Its value, <freq>, should be an integer and, if it is set the command `profilerstart` will call stethoscope with this value as the argument to -b. This will instruct stethoscope to collect heartbeat events from the MonetDB server every <freq> milliseconds. Please note that no type checking is done, and if <freq> is not an integer, stethoscope will silently ignore it. Client Package * The mclient program can now be quit by typing quit or exit on the command-line. This makes it easier for novice users to quit the program if they do not know that the standard command to quit is \q. Mapi Library * Removed function mapi_cache_shuffle. * Removed function mapi_stream_query. Stream Library * The functions mnstr_write{Sht,Int,Lng,Hge} and their Array variants now swap bytes of the written values if the stream is set to swap bytes (i.e. big-endian on a little-endian machine and v.v.). * Removed defines ST_ASCII, ST_BIN, ST_READ, ST_WRITE from stream.h. * Changed function mnstr_set_byteorder(stream, char) to mnstr_set_bigendian(stream, bool) where the second argument should be `true' for specifying that the stream is bigendian. This sets an internal flag whether or not to swap bytes which can be retrieved with mnstr_get_swapbytes(stream). * Changed function `int mnstr_byteorder(stream)' to `bool mnstr_get_swapbytes(stream)'; it now returns whether bytes are swapped. * Changed function `int mnstr_type(stream)' to `bool mnstr_isbinary(stream)'. MonetDB Common * During processing, we now try to maintain the smallest and largest values that occur in a BAT. This information is not saved across server restarts. * Added an extra argument of type `bool' to the atom to string and string to atom methods to indicate whether the string is for internal use (casting the value to or from a string) of external use (to be printed). * Implemented a nilslast option for BATfirstn. If set, NILs come last in the ordering that BATfirstn simulates, so non-NIL values are preferentially returned. The old behavior can be obtained by setting nilslast to !asc(ending). * Implemented a nilslast option for BATsort. This option should be equal to the reverse option for stable sort (it is not implemented for stable sort), but can be different from reverse for non-stable sort. The functions BATsort and GDKqsort have extra parameters, the function GDKqsort_rev has been removed (superseded by GDKqsort with the new `reverse' parameter). * The BUNtail, BUNtvar, BUNtloc, and BUNtpos macros (and Tloc and Tpos) now return a `void *' instead of a `char *'. * Function PROPdestroy now takes a BAT* parameter. * A whole bunch of functions now take a bool argument instead of an int argument. Some functions now return a bool instead of an int. In all these cases, the int was used as a Boolean value. SQL Frontend * Implemented two-argument least/greatest functions. If one argument is NULL, the functions return the other value, otherwise they return the least/greatest of the two arguments. * Implemented SQL2003 natural logarithm function: ln(num_expr). * Function octet_length now also accepts a BLOB argument and returns the length of the BLOB (i.e. the same as length when called with a BLOB argument). * Implemented length(blob) and comparison between blobs. * Implemented a two argument function log which calculates the logarithm of the first argument using the second argument as base. * Next to the functions log (natural logarithm) and log10 (base 10 logarithm), there is now also log2 (base 2 logarithm). * Removed unused global SQL variable "history". * Allow usage of custom expressions in GROUP BY and PARTITION BY clauses. The same expressions can be used in the projection phase as long as it is exactly the same used in the grouping clause (textual lookup is performed). As an example the query: SELECT col1*2 FROM t1 GROUP BY col1*2; is correct, while SELECT sin(col1+5) FROM t1 GROUP BY col1*2; is wrong. * Allow usage of WITH clauses in insert, update, delete and merge statements. * Implemented merge statements from SQL:2003 standard. Using a source relation R, a target table T is updating depending on the result of the merge. In case of a match, the table T's row is either updated or deleted with R's row. In a non-match case, R's row is inserted into T. The grammar is the follows: > MERGE INTO target [ [AS] ident ] USING source ON search_condition { WHEN MATCHED [ AND search_condition ] THEN { UPDATE SET assignment_list | DELETE } } | { WHEN NOT MATCHED [ AND search_condition ] THEN INSERT [ column_list ] [ { DEFAULT VALUES | VALUES row_values } ] } * Added possibility to change the schema of a table with the syntax: > ALTER TABLE [ IF EXISTS ] qname SET SCHEMA ident * Added optional alias option for target table in update and delete statements: (e.g. UPDATE target AS alias SET col1=1 WHERE alias.col2 < 5) * Improved and extended storagemodel functionality. Changes include: * Added views sys."tablestorage" and sys."schemastorage" for easy and quick assesment of aggregated storage usage per table or per schema. * Excluded listing system tables in views sys."storage", sys."tablestorage" and sys."schemastorage". You can still use sys."storage"() to query storage of system tables and columns. * In procedure sys.storagemodelinit() when populating table sys.storagemodelinput it now excludes system tables as system tables are not useful to be modeled for storagesize by application users. It now also computes and populates the atomwidth column more correctly for variable size data types (strings, blobs) when the table has representative data in those columns. * System function sys.columnsize() has been corrected for types: tinyint, real, date, time, timetz, sec_interval, month_interval, decimal, uuid, mbr, char, clob, json, url, blob, geometry and geometrya. For variable size data types (varchar, char, clob, json, url, blob, geometry, geometrya) it now returns the columnsize excluding the variable heapsize. The heapsize is retrievable via sys.heapsize(). * System function sys.heapsize() has been corrected for all variable size data types: varchar, char, clob, json, url, blob, geometry and geometrya. * System function sys.imprintsize() has been corrected. The imprintsize depends on the width (1 or 2 or 4 or 8 or 16 bytes) of the data type. Also instead of 12% it now uses 20%, which is a better estimate. * System function sys.storagemodel() has been removed as it outputs the same data as view sys.storagemodel. Use view sys.storagemodel instead. * Corrected views sys.storagemodel and sys.tablestoragemodel by returning a computed orderidxsize when the count has been changed in the sys.storagemodelinput table. Also the views now return data ordered by schema, table and column names. * Extended view sys.tablestoragemodel with column: "storages". Besides columns also keys (primary, foreign and unique keys) and indexes (ordered, imprints) use storage, so the "storages" count can be higher than the number of columns per table. * Corrected the data type of columns "schema", "table", "column", "type", "mode" and location in table sys.storagemodelinput and functions sys."storage"(), sys."storage"(sname), sys."storage"(sname, tname) and sys."storage"(sname, tname, cname) from string into varchar(1024). Consequently also the views based on the table or functions will be reporting varchar(1024) as column meta data instead of clob for those columns. This allows faster querying and reporting by generic SQL programs which treat clob querying different from varchar columns. * Extended windowing functions catalog with SQL standard semantics. Standard aggregation functions can now be used in windowing functions: (avg,max,min,sum,prod,count). Other windowing specific functions were also implemented: (percent_rank,cume_dist,ntile, lag,lead,first_value,last_value,nth_value). * The standard frame specification was implemented for aggregation functions as well as first_value, last_value and nth_value functions. The available frames are rows, range and groups. Bounds can be unbounded (partition limit), the current row, a fixed number of rows (constant), or variable (column as input). (e.g SELECT COUNT(col1) OVER (PARTITION BY col2 ORDER BY col3 RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) FROM t1;). * Added WINDOW keyword which is optionally provided after the FROM clause, with window specifications used in the projection (e.g SELECT SUM(col1) OVER w1, LAST_VALUE(col2) OVER w2 FROM t1 WINDOW w1 AS (ROWS BETWEEN 5 PRECEDING AND 0 FOLLOWING), w2 AS (w1);). * Our previous partitioning implementation didn’t impose order in the input. With this reexamination, partitioning now imposes ascending order by default, thus pairing with the industry standard implementation. * Implemented X'...' style binary string literals. * Implemented U&'...' Unicode character string literals and U&"..." Unicode delimited identifiers, including UESCAPE. For the string literals, you can have U&'...' '...' '...' UESCAPE '...' where the escape must be as single character and the other '...' strings are also Unicode character string literals. For now, these latter strings also undergo C-style backslash interpretation. * Implemented PostgreSQL-like E'...' strings. The strings can contain C-style backslash escapes. The old format strings '...' currently still also accept C-style escapes, but that feature will be removed in a future release. * Implemented the NULLS FIRST and NULLS LAST option to ORDER BY. The default is NULLS FIRST for ASC(ending) and NULLS LAST for DESC(ending). * Added possibility to rename SQL schemas, tables and columns with ALTER statements: > ALTER SCHEMA [ IF EXISTS ] ident RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME [ COLUMN ] ident TO ident Also added optional IF EXISTS clause to other existing ALTER TABLE statements. * The sql SAMPLE syntax is extended to include an optional integral SEED parameter that sets the seed for the internal random number generator in the sample algorithm. When the seed is set, the user can obtain a reproducible sample set from a static dataset with the same sample query. The new syntax is SELECT ... FROM ... WHERE ... SAMPLE <expr> [SEED <integer>] * Implemented a version of COPY INTO that reads/writes files from/to the client instead of doing it in the server. This has the advantage that COPY INTO is then no longer restricted to only the "super user" monetdb, nor only to absolute file names. The syntax to have the server communicate with the client for file content is COPY INTO table FROM file ON CLIENT ...; and COPY query INTO file ON CLIENT ...;. This also works for COPY BINARY INTO. There is also the possibility to specify that files are to be read/written by the server by using ON SERVER. This is also the default when ON CLIENT or ON SERVER is not specified. * The sys.functions table now has an extra column called "system" of type BOOLEAN whose value is TRUE for system functions (i.e. functions that should not be dumped). The table sys.systemfunctions has been changed to a view and is now officially deprecated. * Extended merge tables with partitioning using a predicate: > CREATE MERGE TABLE [ IF NOT EXISTS ] table_name (... columns ...) [ PARTITION BY { RANGE | VALUES } { ON '(' column_name ')' | USING '(' expression ')' } ] The partitioning can occur by range or list of values using one of the table's columns or an expression. The domain of the partitioning scheme will be covered by each child table added with alter statements depending if the table is partitioned by range or list of values: > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION IN '(' expression [ ',' ... ] ')' [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION BETWEEN { RANGE MINVALUE | expression } AND { RANGE MAXVALUE | expression } [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION WITH NULL When updating a single partition range/list of values, the SET keyword should be used instead of ADD. INSERT, UPDATE and DELETE statements are possible on partitioned tables with corresponding validation on each partition domain. The system tables sys.table_partitions, sys.range_partitions and sys.value_partitions were added to store meta-information about each existing partition. Bug Fixes * 2403: stream: http read support in streams * 2416: GDK: file/dir creation mask * 2478: SQL: rename columns * 2496: SQL: implement greatest / least * 3384: Auxiliary window functions * 3416: log() SQL function only supports natural logarithm * 3448: Make lbatomic_ops optional if installed * 3520: Add support for scalar function LN(num) * 3530: sigabort on window aggr function * 3567: Add support for: ALTER TABLE [schema1.]oldtablename RENAME [TO] [schema2.]newtablename * 3743: DELETE FROM merge table not supported. * 3832: Cannot use expressions in GROUP BY clause * 3946: Expose C log2 function to SQL * 4056: mapi_timeout is not implemented * 6181: comments cause error with return in function * 6326: Eliminate unused UNION tables * 6347: Mserver returns with "memory exhausted in" error if query size exceeds 2.5MB * 6475: Remove unused SQL global variable "history" * 6588: Table aliasing are not supported in SQL update queries * 6591: Name mangling in the SQL catalog results in ambiguity * 6608: Sqlitelogictest error message in group by with coalesce * 6609: Rename Schema in MonetDB * 6636: sys.queue() extension: allow DBA to see all running queries * 6647: Add suport to Python 3 on Windows * 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records * 6671: Error when running user function on merge table with remote part * 6674: R UDF with Date type fails to convert RAPI * 6676: Max data length of 2048 for column sys._tables.query is too small for the actual data (2811 for view sys.ids) * 6678: Binding NULL parameter to parametrized query results in syntax error on execution * 6680: Copy cannot open CSV file if systemctl enabled * 6684: Inserting into a replica table crashes the server * 6685: adding a view to a merge table gives unexpected behaviour * 6690: Unable to fetch empty string with ODBC driver * 6696: Re-use of MAL explain plans with complex type coercions

2 years, 10 months

GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory

by RanchoYuan

Hi Stefan, Thanks for your reply! We have run the query a few times with different size of data. There we used 16G RAM(actually 13.5G was used), and find the size of 10G's data is the critical point that can run the query. All of the data files' size are listed below, each file name is a table name(there are only a few tables are refered -- store_sales, date_dim, item, customer, catalog_sales, web_sales): 7.4K call_center.dat 1.6M catalog_page.dat 212M catalog_returns.dat 2.9G catalog_sales.dat 27M customer_address.dat 64M customer.dat 77M customer_demographics.dat 9.9M date_dim.dat 77B dbgen_version.dat 149K household_demographics.dat 328B income_band.dat 2.6G inventory.dat 28M item.dat 61K promotion.dat 1.7K reason.dat 1.1K ship_mode.dat 27K store.dat 323M store_returns.dat 3.8G store_sales.dat 4.9M time_dim.dat 1.2K warehouse.dat 19K web_page.dat 98M web_returns.dat 1.5G web_sales.dat 12K web_site.dat So we guess that the monetdb has no memory management? For the output of `mserver5 --version` is: MonetDB 5 server v11.27.13 "Jul2017-SP4" (64-bit, 128-bit integers) Copyright (c) 1993 - July 2008 CWI Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved Visit https://www.monetdb.org/ for further information Found 17.0GiB available memory, 40 available cpu cores Libraries: libpcre: 8.38 2015-11-23 (compiled with 8.38) openssl: OpenSSL 1.0.2g 1 Mar 2016 (compiled with OpenSSL 1.0.2g 1 Mar 2016) libxml2: 2.9.3 (compiled with 2.9.3) Compiled by: monetdb(a)MonetDB-0.0 (x86_64-pc-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 And the size of processes is not limited. To let you reproduce the problem conveniently, I'll provide more details here: you can get tpc-ds from its website(we use version 2.6.0). Install the tpc-ds, access the directory v2.6.0/tools and run `./dsdgen -scale 10 -dir /home/monetdb/tpc-ds_test_data10G` to generate the data. When data has been generated, using the script /expe.sh/ to create tables and load the data. The query script is 123.tpcds.23.sql.(The syntaxs of other queries that tpc-ds generates is not suitable for monetdb all, we don't modify them all when the problem occurred). One more question, I can't get your reply email, so I don't know how to reply you, for this case, I could only send a new mail echo time. Thanks! Regards, Rancho

3 years

Using the Monet API to connect to MonetDB if MonetDB comes up AFTER our application

by Gotwisner, Dave

I am currently trying to update my code that uses the Mapi API to retry connections if we start an application when MonetDB is down. Our programs do not run interactively, so detecting the failure, terminating the program, and restarting the application is problematic. There are a couple of issues here that I can use help with. What we are doing is calling mapi_connect(). Mapi_connect returns what appears to be a pointer (it's non-zero). So we call mapi_error with this return value. If we are not connected, we get an MERROR, not an MTIMEOUT, which is what I would expect. With an MERROR failure, we call mapi_error_str() to see what kind of error. We then must do a string compare (actually, a strstr() call looking for a "Connection refused" pattern inside the string. Is there a way to get an enum or #define value instead, with a different function call? Sort of like looking at errno rather than calling strerror(errno) and doing a string compare on the result. This would be way more efficient if we could do this. Yes, errors should be rare, but determining the type of error (or protecting against a string text change on an RPM update) shouldn't be expensive. We then call mapi_explain() to stderr. Is there a routine that returns the value of the output of explain() to a std::string or a "const char *"? We have a logger that does stuff with the strings before logging (timestamping, __line__ __file__, putting error codes, etc. Having the multiline output of the explain() information would be very useful for us. The above process is executed multiple times until we connect. Since the failed connects return what appears to be a pointer, do we need to do anything to free the pointers up? It doesn't appear that we have a memory leak (I ran valgrind). Note that if the connect fails, calling mapi_destroy() crashes. From the documentation, "mapi_destroy()" does "Free handle resources", so it seems reasonable to call it after a failed (and non-null) mapi_connect(). It should probably do nothing, it should not crash. Also, it appears that if I connect multiple times without disconnecting (I had a bug in my code), I can't connect more than 65 times (simultaneous connections). Is this an API issue or a server issue? The reason I ask, is that if it a client issue, we will be having a single client that potentially does queries against hundreds of servers, and aggregates the result. If it is a server issue, is there a parameter that we can tweak to increase this? We do multiple queries in parallel, each on their own connections. Thanks, Dave

3 years, 1 month

Guidelines for MonetDB in production environments

by Roberto Cornacchia

Hi all, I'm struggling with optimizing resource sharing of MonetDB in production environments (see also: https://www.monetdb.org/pipermail/users-list/2018-June/010276.html). We run MonetDB instances for several projects / customers on each of our servers. Each MonetDB instance is a docker container (used mainly because of ease of deployment and environment isolation). It is not unusual to have 5-10 MonetDB containers on the same server. In principle, Docker does not even have much to do with this, but it puts everything in a realistic context. ** Memory mserver5 checks the system memory and calibrates on that. When 10 instances are running, they all assume they have the whole memory for themselves. Docker allows to set limits on the container memory. It does that by using cgroups (so Docker just makes things easier, but it's really about cgroups). However, memory limits set by cgroups are not namespaced ( https://ops.tips/blog/why-top-inside-container-wrong-memory/#memory-limits-… ). This means that each container will still see the whole memory and will simply get killed when the container limit has been reached (definitely not a solution). So far, the only mechanism I know to obtain the correect behavior is to run actual VMs for MonetDB. But this is very cumbersome and I want to avoid that as much as possible. Should we let 10 instances believe they each have the whole memory, and let them fight for it? (well, that's what's happening now, and I know for sure it's bad). Perhaps the solution can be as easy as allowing an explicit max memory setting, together with some documentation on the consequences of using low / high values. ** CPU Again, Docker allows to set quotas per container. I think cgroups CPU limits are namespaced, so perhaps this would just work well, I haven't really tried yet. ** I/O Same issue. It would be ideal to be able to set priorities, so that mserver5 instances that do background work get a lower I/O priority than instances serving online queries. Also, recommendations on swap settings would be interesting. How much swap? How to tune swappiness kernel settings? I am very aware that there is no simple answer to most of these questions. Many variables are in the picture. Still, some general thoughts from the developers would be appreciated. I think I have read pretty much everything has ever been written about MonetDB, but when it comes to resource utilization I have always bumped into the very unrealistic assumption that each MonetDB instance has a whole server for itself. As I mentioned above, things could get already much better with simple improvements, like allowing to set the maximum memory usable by each instance. But more in general, I feel there is much need for some guidelines for production environments. Or at least, to start the discussion. Best regards, Roberto

3 years, 1 month

Error Upgrading to 11.33.3 (Apr2019)

by Ariel Abadi

Hi! I have a database that It started on version Dec2016-SP5. I have been upgrading from version to version without any problem. Now, Im trying to do the last upgrade (this means from Aug2018-SP2 to Apr2019. But Im facing an error. This is what I see on the log: 2019-06-06 15:17:55 ERR cli_gm[2188]: #main thread:!ERROR: BBPcheckbats: file /opt/monetdb/cli_gm/bat/50/65/506571.tail too small (expected 127992, actual 125580) 2019-06-06 15:17:55 MSG cli_gm[2188]: !ERROR: BBPcheckbats: file /opt/monetdb/cli_gm/bat/50/65/506571.tail too small (expected 127992, actual 125580) 2019-06-06 15:17:55 ERR cli_gm[2188]: !FATAL: BBPinit: BBPcheckbats failed 2019-06-06 15:17:58 MSG merovingian[2180]: database 'cli_gm' (2188) has exited with exit status 1 2019-06-06 15:17:58 ERR control[2180]: (local): failed to fork mserver: database 'cli_gm' appears to shut itself down after starting, check monetdbd's logfile (merovingian.log) for possible hints Is there anything I can do to solve this problem? Thks in advance Ariel PD.If I go back to version Aug2018-SP2 the database keeps working with no problem.

3 years, 1 month

Questions about using COPY INTO when using GRANTS on tables

by Gotwisner, Dave

Our application was working quite well when using COPY INTO when inserting data into our tables as the default MonetDB user "monetdb" into the system schema (sys). We have subsequently added users and granted table access to the various users. At this point, INSERT works successfully, but COPY INTO does not. The error message for the COPY INTO statement indicated that we needed to use "ON CLIENT" in our load. That makes no difference. Here are the relevant commands to the system (note the user names, passwords, schema names, and tables names were modified for this example): Program that sets up our basic database does this: START TRANSACTION; CREATE USER admin WITH UNENCRYPTED PASSWORD '***' NAME 'Admin User' SCHEMA sys; CREATE SCHEMA test AUTHORIZATION admin; ALTER USER admin SET SCHEMA test; CREATE USER update WITH UNENCRYPTED PASSWORD '***' NAME 'Update User' SCHEMA test; COMMIT; Then, the program that creates the tables does the following: START TRANSACTION; CREATE MERGE TABLE x (...); GRANT ALL ON x TO "update"; CREATE TABLE x0 (...); GRANT ALL ON x0 TO "update"; CREATE TABLE x1 (...); GRANT ALL ON x1 TO "update"; CREATE TABLE x2 (...); GRANT ALL ON x2 TO "update"; CREATE TABLE x3 (...); GRANT ALL ON x3 TO "update"; COMMIT; And we insert (or try to) (4 threads ingest into their own table, the merge table is used to combine the result set on queries): START TRANSACTION; COPY 100000 OFFSET 2 RECORDS INTO x0 FROM '/dev/shm/MonetDB/foo.csv' (columnlist) ON CLIENT DELIMETERS ',','\n'; This returns an error: !/dev/shm/MonetDB/foo.csv': cannot retrieve files. Because of company policy, I can't provide the schema, but the columns in the csv file and the schema differ, which is why I am using the (...) and (columnlist) in the example. Again, when I omit the CREATE USER, CREATE SCHEMA, and GRANT clauses, everything works correctly. Any ideas on what is needed to make it work? Right now, the ingest application and MonetDB are on the same server. In the future, that might not be true. We are running 11.33.3 on CentOS 7. Thanks, Dave

3 years, 1 month

FW: Cancel query

by Zingman, Jonathan

Yes, sys.queue(), sorry. A couple of clarifications. The cookbook shows mclient, but we're using the API and we assume you can do the same thing from the API as well. Also, we use autogenerated queries that can be hundreds if not thousands of characters long. Matching in a case independent way would be very resource intensive. It's fairly clear from the documentation that MonetDB doesn't stop immediately in the middle of whatever it is doing, so that's understood. But cancelling queries has to be part of a robust database system. My model of how this might work is that the handle that’s returned when we issue a query could be used to associate with some data returned by sys.queue(), preferably tag or start time since those should be unique and fairly short. Then we stash then in some structure, and if a cancel request comes in, we do a much simpler search of the sys.queue() table that's guaranteed to work. JONATHAN ZINGMAN | Manager – TrueCall Data | DIRECT: 510.809.0423 | CELL: 510.823.5638 | www.netscout.com -----Original Message----- From: users-list <users-list-bounces+jonathan.zingman=netscout.com(a)monetdb.org> On Behalf Of Martin Kersten Sent: Tuesday, June 11, 2019 1:56 PM To: users-list(a)monetdb.org Subject: Re: Cancel query This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe. On 11/06/2019 22:38, Ying (Jennie) Zhang wrote: > > >> On 11 Jun 2019, at 00:58, Zingman, Jonathan <Jonathan.Zingman(a)netscout.com> wrote: >> >> In the cookbook, you give a good way to find out the status of queries by using the sys.query() call, > > Hai, Jonathan, > > MonetDB doesn’t have a function called sys.query(). Did you mean sys.queue() instead? > >> which returns the list of queries tagged in various ways. Our question is whether there is any way to access, e.g., the tag value. In our application, we need to occasionally cancel queries from our UI. The UI has no direct knowledge of the query. While we might be able to find the query we want to cancel by comparing the query text, our automatically generated queries can be very long, and if there’s some internal reformatting by Monetdb, we might not get an exact match. So is there anything in the handle or elsewhere that we can use to identify the query we want to cancel? > > Correct me if I’m wrong, but I’m not aware of a feature with which you can query the qtag of a query. Some code study will be needed to see if it’s feasible to add such feature, and how. > > You can probably get away with string matching, since MonetDB doesn’t do much/any reformatting next to to-lowercase. Not sure how much text the “query” field of sys.queue() holds. > > In addition, please be aware that > i) sys.queue() is originally develop for debugging purpose… > ii) the minimal cancellation entity is a function, i.e., when MonetDB is busy executing a very long running function, a cancellation will only take effect *after* that function has finished => then MonetDB won’t continue executing the next function. Small correction here, with function we mean a MAL function call. So, if the system is executing a very time consuming join, it will *not* be interrupted, but rather at the end of it the query in which is rans is terminated. > > Regards, > Jennie > >> >> JONATHAN ZINGMAN >> Manager – TrueCall Data >> >> <image001.png> >> 2855 Telegraph Ave, Suite 200 >> Berkeley, CA, 94705-1169 >> >> DIRECT: 510.809.0423 >> CELL: 510.823.5638 >> http://www.netscout.com >> >> >> _______________________________________________ >> users-list mailing list >> users-list(a)monetdb.org >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_mailma… > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_mailma… > _______________________________________________ users-list mailing list users-list(a)monetdb.org https://urldefense.proofpoint.com/v2/url?u=https-3A__www.monetdb.org_mailma…

3 years, 1 month

Cancel query

by Zingman, Jonathan

In the cookbook, you give a good way to find out the status of queries by using the sys.query() call, which returns the list of queries tagged in various ways. Our question is whether there is any way to access, e.g., the tag value. In our application, we need to occasionally cancel queries from our UI. The UI has no direct knowledge of the query. While we might be able to find the query we want to cancel by comparing the query text, our automatically generated queries can be very long, and if there's some internal reformatting by Monetdb, we might not get an exact match. So is there anything in the handle or elsewhere that we can use to identify the query we want to cancel? JONATHAN ZINGMAN Manager - TrueCall Data [logos] 2855 Telegraph Ave, Suite 200 Berkeley, CA, 94705-1169 DIRECT: 510.809.0423 CELL: 510.823.5638 www.netscout.com<http://www.netscout.com/>

3 years, 1 month

MonetDB - Case Sensitive

by Suresh Devarakonda

Hi, I am new to the MonetDB and interested to know about its case sensitivity. We noticed that objects are being created in lower case if they are not specified in quotes so it appears to me that default case for objects created in Monet is LOWERCASE. But, we got a requirement that objects should be created in upper case even though they are not specified in quotes in DDL statements. In other words, can we make the default to UPPERCASE for all the objects when they created. Please let us know if that is possible. Your early response is very much appreciated. ~Suresh D

3 years, 1 month

Installation error on AWS Linux

by Gerald Sangudi

Hello, I'm trying to install MonetDB on AWS Linux. The first command succeeds: sudo yum install https://dev.monetdb.org/downloads/epel/MonetDB-release-epel.noarch.rpm The second command fails: sudo yum install MonetDB-SQL-server5 MonetDB-client Error: *[ec2-user@ip-172-31-23-149 ~]$ sudo yum install MonetDB-SQL-server5 MonetDB-clientLoaded plugins: extras_suggestions, langpacks, priorities, update-motdhttps://dev.monetdb.org/downloads/ <https://dev.monetdb.org/downloads/>epel/2/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not FoundTrying other mirror. One of the configured repositories failed (MonetDB 2 - x86_64), and yum doesn't have enough cached data to continue. At this point the only safe thing yum can do is fail. There are a few ways to work "fix" this:* It seems to me there's a reference to epel/2 which is out of date. Should that be epel/7? Is it possible to get an updated rpm? Thanks, Gerald

3 years, 1 month

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

users-list June 2019