- users-list - monetdb.org

Re: Re: RSS memory
by Vojtech Kurka 21 Jun '21

21 Jun '21

Hello Roberto, Sjoerd, I think I am hitting the same problem described in this thread. I am running MonetDB in Docker container on a server with 30GB of memory, allocating 21GB of it to MonetDB container. Sometimes, under heavier load I get OOM - process mserver5 uses more memory than is the memory.limit_in_bytes for a cgroup of this container. It looks like this: [Mon Jun 21 12:56:36 2021] memory: usage 20971484kB, limit 20971520kB, failcnt 48706810 [Mon Jun 21 12:56:36 2021] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0 [Mon Jun 21 12:56:36 2021] kmem: usage 143696kB, limit 9007199254740988kB, failcnt 0 .. [Mon Jun 21 12:56:36 2021] Tasks state (memory values in pages): [Mon Jun 21 12:56:36 2021] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [Mon Jun 21 12:56:36 2021] [ 26927] 0 26927 1371 151 53248 0 0 run.sh [Mon Jun 21 12:56:36 2021] [ 27054] 0 27054 134763 326 139264 0 0 monetdbd [Mon Jun 21 12:56:36 2021] [ 27068] 0 27068 40646510 5209845 64159744 0 0 mserver5 [Mon Jun 21 12:56:36 2021] [ 27289] 0 27289 1019 69 45056 0 0 tail [Mon Jun 21 12:56:36 2021] [ 29503] 0 29503 6425 547 86016 0 0 mclient [Mon Jun 21 12:56:36 2021] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=d37591a31d4d010a7b4e34ebb19ba11220a84a264733f70a00ee92899c7918fc,mems_allowed=0,oom_memcg=/docker/d37591a31d4d010a7b4e34ebb19ba11220a84a264733f70a00ee92899c7918fc,task_memcg=/docker/d37591a31d4d010a7b4e34ebb19ba11220a84a264733f70a00ee92899c7918fc,task=mserver5,pid=27068,uid=0 [Mon Jun 21 12:56:36 2021] Memory cgroup out of memory: Killed process 27068 (mserver5) total-vm:162586040kB, anon-rss:20823812kB, file-rss:15568kB, shmem-rss:0kB, UID:0 pgtables:62656kB oom_score_adj:0 Have you made any progress in resolving the OOMs? Things I am trying to test are (I am using docker-compose v2 for running the container, so settings are from there): mem_limit: 21G # this sets memory.limit_in_bytes for the cgroup of the container mem_reservation: 14G # this sets memory.soft_limit_in_bytes oom_score_adj: -1000 # to lower the chance OOM will kill this process Exploring "select * from env();" in MonetDB tells me that the gdk_mem_maxsize is 12251394211, which corresponds to the 81.5% of the mem_reservation, or memory.soft_limit_in_bytes. I am monitoring memory usage of the container and it still is spiking at 19G, which is uncomfortably high, very close to the hard limit of the cgroup. Is this normal? FYI, originally meant only for PostgreSQL, but it became our standard - we are using vm.overcommit = 2 and vm.overcommit_ration = 100, if it changes anything. I would be thankful for any clue on how to run MonetDB in the container reliably as right now it is causing me quite a headache. Running it on a host solely for the MonetDB would be a last resort for me. Thank you and have a great day!

2 1

New MonetDB JDBC driver released
by Sjoerd Mullender 16 Jun '21

16 Jun '21

The MonetDB team at MonetDB BV is pleased to announce a bugfix release of the MonetDB Java jars and JDBC driver. We recommend all users of MonetDB JDBC driver to upgrade to version: monetdb-jdbc-3.1.jre8.jar The download location is as usual: <https://www.monetdb.org/downloads/Java/>. For a list of changes please read: <https://www.monetdb.org/downloads/Java/ChangeLog>. For information and tips on using the MonetDB JDBC driver, see: <https://www.monetdb.org/downloads/Java/release.txt>. Note: as of release monetdb-jdbc-3.0.jre8.jar the MonetDB JDBC Driver class name has been changed to: org.monetdb.jdbc.MonetDriver It used to be nl.cwi.monetdb.jdbc.MonetDriver which is now marked as deprecated. Please use the new driver class name asap in your configuration files/scripts and/or Java code. The old driver class name is still included in the jar file to ease the transition for existing deployments. However it will be removed in a future release of the MonetDB JDBC driver. -- Sjoerd Mullender

1 0

DBSeeder Version 2.9.1
by Walter Weinmann 12 Jun '21

12 Jun '21

The latest version of DBSeeder is now available on GitHub: https://github.com/KonnexionsGmbH/db_seeder. This also includes from MonetDB the latest available versions on Docker Hub (Oct2020-SP5) and https://www.monetdb.org/downloads/Java (JDBC monet-jdbc-3.0.jre8). MonetDB now shows a huge performance improvement after a manual commit (instead of autocommit) is executed in DBSeeder after each batch of insert operations. [image: Company_9.9.9_win10.png]

1 0

Linux kernel vm settings
by Roberto Cornacchia 06 Jun '21

06 Jun '21

Hi there, I looked around but couldn't find any recommendation about kernel vm settings in Linux for MonetDB. In particular: - vm.overcommit_memory: 0 (default) : a heuristics decides whether overcommitting is allowed 1: no check, overcommit is always allowed 2: overcommitting is regulated by vm.overcommit_ratio (default = 50%) Do I understand correctly that using vm.overcommit_memory=1 will only make the OOM kill mserver5 when the total VM available is exhausted? If that is true, should it be reasonably safe to use on a server that is mainly intended for MonetDB, as long as sufficient disk space is available? - vm.swappiness Generic recommendations are usually 60 for a desktop and 30 for a server. Oracle recommends 10. Redis recommends 1. Are there studies / recommendations for MonetDB?

5 8

DBSeeder Version 2.9.0
by Walter Weinmann 05 Jun '21

05 Jun '21

We are happy to announce that version 2.9.0 of DBSeeder ( https://github.com/KonnexionsGmbH/db_seeder) is now available for general use. DBSeeder also supports the latest version of MonetDB available on DockerHub (Oct2020-SP5).

1 0

for the Cyrillic alphabet, lower does not work
by Илья Хайбуллин 17 May '21

17 May '21

2 3

gcc optimization level?
by Roberto Cornacchia 14 May '21

14 May '21

I was wondering, -DCMAKE_BUILD_TYPE=Release (or RelWithDebInfo) doesn't seem to enable -O2 on gcc. Is there a good reason for it or was it just forgotten?

2 2

Announcement: New Oct2020-SP5 Bugfix release of MonetDB suite
by Sjoerd Mullender 10 May '21

10 May '21

The MonetDB team at MonetDB BV is pleased to announce the Oct2020-SP5 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct2020-SP5 Bugfix Release (11.39.17) Windows Packaging * Added the monetdbe library to the Windows installer. MonetDB Common * A bug that would very occasionally produce an error "strPut: incorrectly encoded UTF-8", even when no incorrectly coded characters are used at all, has been fixed. It was the result of a rare combination of strings having been added to the column that caused essentially an off-by-one type of error to occur. SQL Frontend * A bug where a sequence of TRUNCATE TABLE and COPY INTO the just truncated table would result in success being reported to both queries, but the table still being empty afterwards, has been fixed. Merovingian * When stopping monetdbd using the `monetdbd stop' command, this command now waits for 5 seconds longer than the exittimeout value before it kills the monetdbd daemon instead of only 30 seconds total (or until that daemon stops earlier). This gives the daemon enough time to terminate the mserver5 processes that it is managing. If exittimeout is negative, the daemon and the monetdbd process initiating the stop wait indefinitely until the mserver5 processes have stopped. Bug Fixes * 3336: DB files not removed if all rows are deleted, even after restart * 7104: Monetdbe NTILE function does not produce correct ordering * 7108: Monetdb crashes on query execution * 7109: MERGE Statement incorrectly reports that input relation matches multiple rows * 7110: Monetdb Query parsing consistency issues in the latest release (Remote Table)

1 0

Announcement: New Oct2020-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 12 Apr '21

12 Apr '21

The MonetDB team at MonetDB BV is pleased to announce the Oct2020-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct2020-SP4 Bugfix Release (11.39.15) ODBC Driver * When connecting using a DSN (Data Source Name), information about the data source is retrieved from the ODBC.INI file. Now we also get the location of the LOGFILE from this file. The logfile can be used to log all calls to the MonetDB ODBC driver to a file which can be used for debugging. * The ODBC driver now only passes on information about HUGEINT columns as HUGEINT when the application has indicated interest by querying about the SQL_HUGEINT extension type using the SQLGetTypeInfo function or by specifying the type in a call to SQLSetDescField. Otherwise the driver silently translates the HUGEINT type to BIGINT. This means that most application will see BIGINT columns when the server produced a HUGEINT column and only give an error if the value in the HUGEINT column didn't fit into a BIGINT. Bug Fixes * 6786: function json.isvalid(js json) is not useful, could be removed * 7016: Database crashes when use similarity function on a table with more than 200k records * 7037: Clearer err msg for ALTER USER with insufficient privileges * 7042: AddressSanitizer:DEADLYSIGNAL in Oct2020/gdk/gdk_tracer.c:494 * 7050: file descriptor leak when forward=redirect * 7057: ODBC driver installer on Windows is missing some DLLs * 7058: MonetDBe: COPY INTO csv file does not produce any output * 7059: MonetDBe: 'reverse' C UDF crashes * 7061: Have bulk load support combined gzip files * 7064: Temporary hashes created in hash and unique logic should try to use transient data farm first * 7066: percent_rank function with wrong results * 7070: double free error when running MonetDBe Example * 7076: mserver5 ignores memory.low from cgroups v2 * 7077: Oct2020: new default privileges not effectively communicated * 7083: MonetDBe C++ Compiling Error * 7085: Mitosis and filter functions * 7087: SIGSEGV caused by error in subquery's function being ignored by top-level query * 7089: Data consistency problem of query results in the latest release of Monetdb (Remote Table)

1 0

Need info about --in-memory option
by Lakshmipathy Jois 09 Apr '21

09 Apr '21

Hi All, I am new to MonetDB and this is my first email to this group. I came across the option --in-memory for mserver5. I understood that if this option is set, then MonetDB operates out of memory and data is not persisted. I wanted to know more details about the working of MonetDB if this option is set: 1. If this option is set, what is the maximum data size that MonetDB can operate with? 2. If this option is set, will MonetDB use mmap only for reading purposes or not use mmap at all? Any other information is really welcome. Thanks in advance. Lakshmi Pathy S N

2 1