Hi Stefan, Thanks for explaining it.
I am wondering if columnar structure will be helpful in compressing data given very low cardinality of more than half of these columns? Actually, I want to estimate what kind of physical memory will be sufficient to have this database in memory to avoid swapping and give good performance of ad-hoc queries. 64 GB looks insufficient, is there anyway, I can have some estimate to avoid multiple iterations?
Regards, Manish
On Thu, Mar 23, 2017 at 7:14 PM, Stefan Manegold Stefan.Manegold@cwi.nl wrote:
Dear Manish,
since MonetDB uses memory mapped files (also on Windows), the (only) "utility" that knows for sure which data is in physical memory at what time is the OS (operating system) that does the virtual memory management.
a back-of-the-envelope calculation yields:
- assuming 4 byte wide columns (e.g., integer, real) (on average): 200M * 70 * 4 byte ~= 56 GB
- assuming 8 byte wide columns (e.g., bigint, double, float, timestamp)
(on average): 200M * 70 * 8 byte ~= 112 GB
additionally, MonetDB might have decided to build (and persist) some indexes.
Note that this calculation only considers persistent base data, i.e., no transient intermediate results that might also consume (temporary) disk space.
Best, Stefan
----- On Mar 23, 2017, at 2:31 PM, Manish gupta gahoimnshg@gmail.com wrote:
Dear all, I am looking at some memory profiling of monet database. Basically, the database size at disk is ~160 GB ( Although I am not very convinced with this big data size, there are 200M records with ~70
columns
distributed among two big tables and several smaller ( relatively )
tables) .
Right now, I have 64GB physical RAM dedicated machine for this database,
but
soon after firing queries on these tables ( with all sort of
permutations on
columns, but no joins between tables ), the memory is almost fully
occupied,
and resource crunch kills the performance. Is there any utility, which shows which all columns are in-memory and
what is
the size? And better some setting through which I can guide which columns should remain in memory and which should immediately trimmed after query returns?
I am using windows 16 core machine on 64 bit architecture.
Regards, Manish
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list