MonetDB causing disk space to fill up

Anu Raj Srivastava Anu.Srivastava at rms.com
Mon Jul 20 23:28:23 CEST 2015


I am sorry didn¹t realize mailing list is not their in my reply.

On 7/20/15, 14:24, "Ying Zhang" <Y.Zhang at cwi.nl> wrote:

>Hello Anu,
>
>Would you please post this to the MonetDB users mailing list?  Thanks!
>
>Regards,
>
>Jennie
>
>> On Jul 20, 2015, at 19:44, Anu Raj Srivastava <Anu.Srivastava at rms.com>
>>wrote:
>> 
>> Hi Hannes,
>> 
>> I am using apache dbcp2 connection Pool. I am running queries in a
>>constant loop with 8 concurrent invocations. And each 8 threads are
>>running 9 queries each again and again.
>> 
>> Java code is quite straight forward:-
>> 
>> private ConnectionPoolPG() {
>>        try {
>>            final Properties properties = new Properties();
>>            
>>properties.load(ConnectionPoolPG.class.getClassLoader().getResourceAsStre
>>am("monetDB.properties"));
>>            dataSource =
>>BasicDataSourceFactory.createDataSource(properties);
>>            schemaName = properties.getProperty("schemaname");
>>        } catch (Exception e) {
>>            throw new ExposureStoreException("Unable to instantiate
>>connection pool: " + e.getMessage(),
>>                    EAPAException.ERR_STORE_CONNECTION_CONFIG_ERROR);
>>        }
>>    }
>> 
>> public void AllQueryPerformance() throws Exception {
>>        while (true) {
>>            if (queries != null) {
>>                for (String query : queries) {
>>                    try (Connection connection =
>>ConnectionPoolPG.getInstance().getConnection()) {
>>                        try (Statement statement =
>>connection.createStatement()) {
>>                            statement.execute(query);
>>                        }
>>                    }
>>                }
>>            }
>>        }
>>    }
>> 
>> -----Original Message-----
>> From: Hannes Mühleisen [mailto:Hannes.Muehleisen at cwi.nl]
>> Sent: Sunday, July 19, 2015 4:47 AM
>> To: Communication channel for developers of the MonetDB suite.
>> Cc: Ying Zhang; Anu Raj Srivastava
>> Subject: Re: MonetDB causing disk space to fill up
>> 
>> 
>>> On 19 Jul 2015, at 13:23, Hannes Mühleisen <Hannes.Muehleisen at cwi.nl>
>>>wrote:
>>> 
>>> Hi,
>>> 
>>>> On 16 Jul 2015, at 23:11, Ying Zhang <Y.Zhang at cwi.nl> wrote:
>>>> 
>>>> On Jul 13, 2015, at 22:32, Anu Raj Srivastava
>>>><Anu.Srivastava at rms.com> wrote:
>>>>> 
>>>>> Hey Jennie,
>>>>> 
>>>>> I did upgrade to latest version but I am facing the same issue.
>>>>>Further debugging into the issue it seems like it is ConnectionPool
>>>>>issue. When I use new connection for each query MonetDB seems to be
>>>>>able to clean up the resources
>>>> 
>>>> Ok, MonetDB is doing what it¹s supposed to do.
>>>> 
>>>>> but if I use ConnectionPool for some reason DiskSpace keep filling
>>>>>up.
>>>> 
>>>> If a connection is kept open, I¹d expect MonetDB to be less eager to
>>>>clean up resources.
>>>> However, we generally recommend using a ConnectionPool, to avoid too
>>>>much overhead caused by making a new connection for every query.  So
>>>>my suggestion for now would be to keep using the ConnectionPool, but
>>>>try to find the reason why your disk is filled up.
>>>> 
>>>>> Do you have any idea why that could happen?
>>>> 
>>>> Hmmm, no.  We¹ve fixed quite some leaks in Jul2015...
>>>> Some more thinking:
>>>> - After how many iterations of repeatedly executing the queries is
>>>>your disk filled up?
>>>> - Can you please try to run your queries with mclient to observe the
>>>>behaviour of MonetDB (and disk usage)?
>>>> 
>>>> @Hannes: sorry to disturb you during holiday.  But if possible, would
>>>>you please have a look at this thread.  Does anything ring a bell?
>>>>Thanks!
>>> 
>>> I tested this with ac3p0 connection pool (limited to 4 concurrent
>>>connections) and repeating queries from 10 threads (SELECT
>>>MIN(something) FROM Š). I was not able to observe any (!) increase in
>>>storage footprint. So without a minimal reproducible example of Java
>>>code + DB schema + sample data, it is hard to tell what is going on. Is
>>>this running in auto commit? Using temporary tables?
>> 
>> Small follow-up: When forcing the allocation of a large intermediate
>>(SELECT MIN(something+1) FROM Š), additional storage space is indeed
>>claimed. The attached plot shows dbfarm size over time while 10 threads
>>are running 1000 queries each using a 5-connection pool. You can see how
>>the amount of additional storage drops back to exactly where it was once
>>the pool shuts down at around 350s, and that there is no accumulation of
>>additional storage over the course of the pool being active.
>> 
>> Hannes
>> 
>



More information about the developers-list mailing list