MonetDB causing disk space to fill up

Anu Raj Srivastava Anu.Srivastava at
Mon Jul 20 23:28:23 CEST 2015

I am sorry didn¹t realize mailing list is not their in my reply.

On 7/20/15, 14:24, "Ying Zhang" <Y.Zhang at> wrote:

>Hello Anu,
>Would you please post this to the MonetDB users mailing list?  Thanks!
>> On Jul 20, 2015, at 19:44, Anu Raj Srivastava <Anu.Srivastava at>
>> Hi Hannes,
>> I am using apache dbcp2 connection Pool. I am running queries in a
>>constant loop with 8 concurrent invocations. And each 8 threads are
>>running 9 queries each again and again.
>> Java code is quite straight forward:-
>> private ConnectionPoolPG() {
>>        try {
>>            final Properties properties = new Properties();
>>            dataSource =
>>            schemaName = properties.getProperty("schemaname");
>>        } catch (Exception e) {
>>            throw new ExposureStoreException("Unable to instantiate
>>connection pool: " + e.getMessage(),
>>                    EAPAException.ERR_STORE_CONNECTION_CONFIG_ERROR);
>>        }
>>    }
>> public void AllQueryPerformance() throws Exception {
>>        while (true) {
>>            if (queries != null) {
>>                for (String query : queries) {
>>                    try (Connection connection =
>>ConnectionPoolPG.getInstance().getConnection()) {
>>                        try (Statement statement =
>>connection.createStatement()) {
>>                            statement.execute(query);
>>                        }
>>                    }
>>                }
>>            }
>>        }
>>    }
>> -----Original Message-----
>> From: Hannes Mühleisen [mailto:Hannes.Muehleisen at]
>> Sent: Sunday, July 19, 2015 4:47 AM
>> To: Communication channel for developers of the MonetDB suite.
>> Cc: Ying Zhang; Anu Raj Srivastava
>> Subject: Re: MonetDB causing disk space to fill up
>>> On 19 Jul 2015, at 13:23, Hannes Mühleisen <Hannes.Muehleisen at>
>>> Hi,
>>>> On 16 Jul 2015, at 23:11, Ying Zhang <Y.Zhang at> wrote:
>>>> On Jul 13, 2015, at 22:32, Anu Raj Srivastava
>>>><Anu.Srivastava at> wrote:
>>>>> Hey Jennie,
>>>>> I did upgrade to latest version but I am facing the same issue.
>>>>>Further debugging into the issue it seems like it is ConnectionPool
>>>>>issue. When I use new connection for each query MonetDB seems to be
>>>>>able to clean up the resources
>>>> Ok, MonetDB is doing what it¹s supposed to do.
>>>>> but if I use ConnectionPool for some reason DiskSpace keep filling
>>>> If a connection is kept open, I¹d expect MonetDB to be less eager to
>>>>clean up resources.
>>>> However, we generally recommend using a ConnectionPool, to avoid too
>>>>much overhead caused by making a new connection for every query.  So
>>>>my suggestion for now would be to keep using the ConnectionPool, but
>>>>try to find the reason why your disk is filled up.
>>>>> Do you have any idea why that could happen?
>>>> Hmmm, no.  We¹ve fixed quite some leaks in Jul2015...
>>>> Some more thinking:
>>>> - After how many iterations of repeatedly executing the queries is
>>>>your disk filled up?
>>>> - Can you please try to run your queries with mclient to observe the
>>>>behaviour of MonetDB (and disk usage)?
>>>> @Hannes: sorry to disturb you during holiday.  But if possible, would
>>>>you please have a look at this thread.  Does anything ring a bell?
>>> I tested this with ac3p0 connection pool (limited to 4 concurrent
>>>connections) and repeating queries from 10 threads (SELECT
>>>MIN(something) FROM Š). I was not able to observe any (!) increase in
>>>storage footprint. So without a minimal reproducible example of Java
>>>code + DB schema + sample data, it is hard to tell what is going on. Is
>>>this running in auto commit? Using temporary tables?
>> Small follow-up: When forcing the allocation of a large intermediate
>>(SELECT MIN(something+1) FROM Š), additional storage space is indeed
>>claimed. The attached plot shows dbfarm size over time while 10 threads
>>are running 1000 queries each using a 5-connection pool. You can see how
>>the amount of additional storage drops back to exactly where it was once
>>the pool shuts down at around 350s, and that there is no accumulation of
>>additional storage over the course of the pool being active.
>> Hannes

More information about the developers-list mailing list