users-list October 2018

users-list@monetdb.org

10 participants
6 discussions

GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory

by RanchoYuan

Hi Stefan, Thanks for your reply! We have run the query a few times with different size of data. There we used 16G RAM(actually 13.5G was used), and find the size of 10G's data is the critical point that can run the query. All of the data files' size are listed below, each file name is a table name(there are only a few tables are refered -- store_sales, date_dim, item, customer, catalog_sales, web_sales): 7.4K call_center.dat 1.6M catalog_page.dat 212M catalog_returns.dat 2.9G catalog_sales.dat 27M customer_address.dat 64M customer.dat 77M customer_demographics.dat 9.9M date_dim.dat 77B dbgen_version.dat 149K household_demographics.dat 328B income_band.dat 2.6G inventory.dat 28M item.dat 61K promotion.dat 1.7K reason.dat 1.1K ship_mode.dat 27K store.dat 323M store_returns.dat 3.8G store_sales.dat 4.9M time_dim.dat 1.2K warehouse.dat 19K web_page.dat 98M web_returns.dat 1.5G web_sales.dat 12K web_site.dat So we guess that the monetdb has no memory management? For the output of `mserver5 --version` is: MonetDB 5 server v11.27.13 "Jul2017-SP4" (64-bit, 128-bit integers) Copyright (c) 1993 - July 2008 CWI Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved Visit https://www.monetdb.org/ for further information Found 17.0GiB available memory, 40 available cpu cores Libraries: libpcre: 8.38 2015-11-23 (compiled with 8.38) openssl: OpenSSL 1.0.2g 1 Mar 2016 (compiled with OpenSSL 1.0.2g 1 Mar 2016) libxml2: 2.9.3 (compiled with 2.9.3) Compiled by: monetdb(a)MonetDB-0.0 (x86_64-pc-linux-gnu) Compilation: gcc -g -O2 Linking : /usr/bin/ld -m elf_x86_64 And the size of processes is not limited. To let you reproduce the problem conveniently, I'll provide more details here: you can get tpc-ds from its website(we use version 2.6.0). Install the tpc-ds, access the directory v2.6.0/tools and run `./dsdgen -scale 10 -dir /home/monetdb/tpc-ds_test_data10G` to generate the data. When data has been generated, using the script /expe.sh/ to create tables and load the data. The query script is 123.tpcds.23.sql.(The syntaxs of other queries that tpc-ds generates is not suitable for monetdb all, we don't modify them all when the problem occurred). One more question, I can't get your reply email, so I don't know how to reply you, for this case, I could only send a new mail echo time. Thanks! Regards, Rancho

3 years

Circular Memory Only Database

by Doherty, Adam

Hello all: I am wondering if anyone has a remote table strategy such that "hot data" is stored in a table that is backed by RAM. I have a table with approximately 1 billion events that grows by approximately 1,000 events a minute. Table is a merge table of merge tables, the parent merge table has a child merge table for every year of data under those child tables are source tables that make up data for areas. So Area1_2017, Area2_2017 -> Events_2017 Area1_2018, Area2_2018 -> Events_2018 Events_2017, Events_2018 -> Events I am finding that as I add more data the system seems to slow down. I am wanting to keep a cyclical table of say the last 6 months from the Events table in a table backed solely by RAM So a query to Recent_Events would remain very fast or at the very least constant. This should be achievable with tmpfs. I am currently running 11.29.3 on CentOS 7 on VMware 6.7 Server has 128GB of RAM and 16 cores. Adam

3 years, 9 months

SQLSTATE Error not showing up in mclient Aug2018 version (Aug2018 (11.31.7))

by Sharma, Sreejith

All, Do we have a way to compile mclient along with server source code? We are not able to see SQLSTATE error code on concurrency with Aug2018 (11.31.7) server version along with mclient Aug2018. Would like to see if we need to specifically compile client too? ERROR = !COMMIT: transaction is aborted because of concurrency conflicts, will ROLLBACK instead CODE: 40000 Regards, Sreejith

3 years, 9 months

Re: users-list Digest, Vol 74, Issue 3

by Oxana Dolmatova

Hi Stefan, Thank you for the information, now I see the point. >Did you by any chance try any other DBMSs with your "non-standard" workload? >If so, how do those experiences and limitations compare to MonetDB? I did not run tests over conventional DBMSs because they have stricter limits compared to MonetDB (e.g. Postgres is limited to 1600 columns), but only over array DBMSs or math packages. Obviously, their column scalability is better, but they can not handle RA that efficiently. Best regards, Oksana On Tue, 16 Oct 2018 at 12:16, <users-list-request(a)monetdb.org> wrote: > Send users-list mailing list submissions to > users-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/users-list > or, via email, send a message with subject or body 'help' to > users-list-request(a)monetdb.org > > You can reach the person managing the list at > users-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of users-list digest..." > > > Today's Topics: > > 1. Re: Column limit (Oxana Dolmatova) > 2. Re: Column limit (Stefan Manegold) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 15 Oct 2018 17:14:14 +0200 > From: Oxana Dolmatova <oxana.dolmatova(a)gmail.com> > To: E.Rozenberg(a)cwi.nl > Cc: users-list(a)monetdb.org > Subject: Re: Column limit > Message-ID: > <CAJJ2wAQi+4S+= > e9BfbStuJxD7izNh8z-mcBF8FYKxagjj5oRkA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Eyal, > > Sorry for the late reply. Yes, there is no problem with large number of > columns in sparse representation of arrays (i.e., dimension, dimension, > value). > However, in order to perform relational operations, this structure should > be pivoted first. When matrices are represented by tables, both algebras > can be applied to the same object, > so I'm specifically looking for the main bottleneck of processing hundreds > of thousands of columns in MonetDB. > > Best regards, > Oksana > > > On Wed, 10 Oct 2018 at 13:40, Eyal Rozenberg <E.Rozenberg(a)cwi.nl> wrote: > > > If you want to model an NxM matrix of elements of type MYTYPE in a DBMS > > (not just MonetDB), you would typically have a table with 3 columns: > > > > CREATE TABLE my_matrix ( > > "row" INTEGER NOT NULL, > > "col" INTEGER NOT NULL, > > "cell_value" MYTYPE NOT NULL, > > PRIMARY KEY ("row", "col") > > ); > > > > of length N x M. > > > > Anyways, I'm not the expert on Arrays in our group so I'll let others > > chime in. > > > > Eyal > > > > > > On 10/10/2018 13:01, Oxana Dolmatova wrote: > > > Hello, > > > > > > Thank you for all the replies and information! > > > The version is 11.23, plus I'm using the cloud platform > > with 4cpu-16ram-hpc. > > > The number of rows was just 1K for 15K attributes in a table. > > > I understand that MonetDB was made with no intention to create and > > > process tables with that many columns. But I'm interested in matrix > > > operations integration, so I would like to know what is the main > > > bottleneck to process such data (matrix-like sized) in the system. > > > > > > Best regards, > > > Oksana > > > > > > > > > On Tue, 9 Oct 2018 at 21:02, Eyal Rozenberg <eyalroz(a)technion.ac.il > > > <mailto:eyalroz@technion.ac.il>> wrote: > > > > > > I'll add two observations to what Zhang Ying said. > > > > > > 1. Internally in MonetDB, the index of a column (or a BAT) in > > MonetDB's > > > BBP (BAT Buffer Pool) is a signed 32-bit integer (on most > > platforms), so > > > there isn't a theoretical problem having more than 15,000 columns; > > but > > > MonetDB was not designed with that many tables in mind. The "long > > > dimension" is expected to be the length of columns. > > > > > > 2. When you have as many as 15,000 columns, you're not simply > finding > > > more attributes of the same items. So, for example, a person has a > > first > > > name, last name, year of birth, etc. etc. - but after several > dozens > > of > > > these you're probably adding columns such as "data relevant to this > > > person at point X", "data relevant to this person at point Y" and > so > > on. > > > > > > In your case, I'm guessing you have a lot of points and hence a lot > > of > > > columns with data for your different items. What you should > consider > > > doing instead is having an extra "coordinate" column. Thus, in the > > > example I gave, I would drop most of the columns, in favor of a > > second > > > table named "person_data". This table's columns would be > "person_id" > > (or > > > some other key columns), "sample_point" and "value". Thus instead > of > > M > > > columns and N records you would have N x M records with 3 columns. > > > > > > Eyal > > > > > > > > > > > > On 09/10/2018 16:50, Ying Zhang wrote: > > > > Hai Oksana, > > > > > > > > A.f.a.I.k., there is no explicit limit on the #columns you can > put > > > in a relation (did you mean ?table??). The reason that mserver5 > > > stopped responding is probably because it is busy with I/O. > > > > > > > > A long time ago, I once tried to create 1M tables, each with 3 > > > columns. After about a day or something, MonetDB has created ~400K > > > tables, and I could hear my HDD being constantly busy. MonetDB was > > > still working, but that was already a sufficient evidence that that > > > idea wasn?t going to work. > > > > > > > > So the question is actually what do you want to achieve? > > > > > > > > Regards, > > > > Jennie > > > > > > > > > > > >> On 9 Oct 2018, at 13:43, Oxana Dolmatova > > > <oxana.dolmatova(a)gmail.com <mailto:oxana.dolmatova@gmail.com>> > > wrote: > > > >> > > > >> Hello, > > > >> > > > >> Could you tell the limitation for number of columns in a > relation > > > and the main reason for that limitation. So far I could create > > > relations with up to 15000 columns. When the number of columns is > > > bigger mserver5 stops responding. > > > >> > > > >> Thank you and best regards, > > > >> Oksana Dolmatova > > > >> _______________________________________________ > > > >> users-list mailing list > > > >> users-list(a)monetdb.org <mailto:users-list@monetdb.org> > > > >> https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > _______________________________________________ > > > > users-list mailing list > > > > users-list(a)monetdb.org <mailto:users-list@monetdb.org> > > > > https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Oksana Dolmatova > > > > > > _______________________________________________ > > > users-list mailing list > > > users-list(a)monetdb.org > > > https://www.monetdb.org/mailman/listinfo/users-list > > > > > > > > -- > Best regards, > Oksana Dolmatova >

3 years, 9 months

Announcement: New Aug2018-SP1 Bugfix release of MonetDB suite

by Sjoerd Mullender

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2018-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Aug 2018-SP1 bugfix release (11.31.11) Build Environment * On Ubuntu 18.10 (Cosmic Cuttlefish), the libmonetdb5-server-bam package cannot be built because of an incompatibility in the libbam library (it cannot be used in a shared object. Whole System * Some subtle dependencies between RPMs have been fixed. Bug Fixes * 6640: timestamp_to_str returning incorrectly adjusted results * 6641: race condition in SQL UDF with update * 6642: Hanging query * 6648: key property potentially wrong after type conversion * 6649: Projection inside within transaction gives wrong results

3 years, 9 months

Column limit

by Oxana Dolmatova

Hello, Could you tell the limitation for number of columns in a relation and the main reason for that limitation. So far I could create relations with up to 15000 columns. When the number of columns is bigger mserver5 stops responding. Thank you and best regards, Oksana Dolmatova

3 years, 9 months

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

users-list October 2018