HI All,
While calculating monetdb size I am observing below :
When I am inserting records to Monet DB using JDBC total memory and VM usage is going to 10 G + but when I see file system size of the database directory it is showing something like 1 GB .
Is some compression coming to picture here while storing data in file system?
Any pointers will be helpful.
Regards,
Ashish
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Also it will be interesting to know if large memory usage is due to intermediate results then when can be expect them to be cleaned? ON statement and connection close??
Regards,
Ashish
From: Ashish Kumar Singh Sent: Saturday, November 10, 2012 3:09 PM To: users-list@monetdb.org Cc: Ashish Kumar Singh Subject: Database Memory Usage
HI All,
While calculating monetdb size I am observing below :
When I am inserting records to Monet DB using JDBC total memory and VM usage is going to 10 G + but when I see file system size of the database directory it is showing something like 1 GB .
Is some compression coming to picture here while storing data in file system?
Any pointers will be helpful.
Regards,
Ashish
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Hi,
I ran some test on a simple flat schema and below are results hinting at some bug! Vmem and RES are collected from top command. It seems when you close connection at that time only monetdb releases server side resources which is quite undesirable. All queries are insert queries sent using JDBC.
Insertion Model
Number of record
Vmem (MB)
RES
System Monet DB Health
Single Connection Multiple Statement
45603
3596m
2.8g
Crash
Single Connection Single Statement
69699
2508m
1.9g
Crash
Single Connection Single Statement Disposed and recreated after every 100 record
396980
713 M
66 M
Good Health
Single Connection Single Statement Disposed and recreated after every 100 record
800000
713 M
66 M
Good Health
Single Connection Single Statement Disposed and recreated after every 100 record
396980
773 M
67 M
Good Health
Single Connection Single Statement Disposed and recreated after every 100 record
8000000
1603 M
97 M
Good Health
Schema:
CREATE TABLE rc_event(
roe_type bigint,
dc_servername varchar(50),
roe_start_datetime timestamp ,
resources_used_vmem double,
resource_list_vmem double,
roe_source_id varchar(120),
walltime_splitt bigint,
ncpu_walltime bigint,
custom_resource_1 bigint,
custom_resource_2 bigint,
custom_resource_3 bigint,
custom_resource_4 bigint,
custom_resource_5 bigint);
I ran these test on CentOS 5.8 2 core 4 GB ram .
Regards,
Ashish
From: Ashish Kumar Singh [mailto:ashishkumar.singh@altair.com] Sent: Saturday, November 10, 2012 5:28 PM To: users-list@monetdb.org Cc: ashishk@altair.com Subject: RE: Database Memory Usage
Also it will be interesting to know if large memory usage is due to intermediate results then when can be expect them to be cleaned? ON statement and connection close??
Regards,
Ashish
From: Ashish Kumar Singh Sent: Saturday, November 10, 2012 3:09 PM To: users-list@monetdb.org Cc: Ashish Kumar Singh Subject: Database Memory Usage
HI All,
While calculating monetdb size I am observing below :
When I am inserting records to Monet DB using JDBC total memory and VM usage is going to 10 G + but when I see file system size of the database directory it is showing something like 1 GB .
Is some compression coming to picture here while storing data in file system?
Any pointers will be helpful.
Regards,
Ashish
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Hi,
Any pointers on this?
Regards,
Ashish
From: Ashish Kumar Singh [mailto:ashishkumar.singh@altair.com] Sent: Sunday, November 11, 2012 2:51 PM To: users-list@monetdb.org Cc: ashishk@altair.com Subject: RE: Database Memory Usage
Hi,
I ran some test on a simple flat schema and below are results hinting at some bug! Vmem and RES are collected from top command. It seems when you close connection at that time only monetdb releases server side resources which is quite undesirable. All queries are insert queries sent using JDBC.
Insertion Model
Number of record
Vmem (MB)
RES
System Monet DB Health
Single Connection Multiple Statement
45603
3596m
2.8g
Crash
Single Connection Single Statement
69699
2508m
1.9g
Crash
Single Connection Single Statement Disposed and recreated after every 100 record
396980
713 M
66 M
Good Health
Single Connection Single Statement Disposed and recreated after every 100 record
800000
713 M
66 M
Good Health
Single Connection Single Statement Disposed and recreated after every 100 record
396980
773 M
67 M
Good Health
Single Connection Single Statement Disposed and recreated after every 100 record
8000000
1603 M
97 M
Good Health
Schema:
CREATE TABLE rc_event(
roe_type bigint,
dc_servername varchar(50),
roe_start_datetime timestamp ,
resources_used_vmem double,
resource_list_vmem double,
roe_source_id varchar(120),
walltime_splitt bigint,
ncpu_walltime bigint,
custom_resource_1 bigint,
custom_resource_2 bigint,
custom_resource_3 bigint,
custom_resource_4 bigint,
custom_resource_5 bigint);
I ran these test on CentOS 5.8 2 core 4 GB ram .
Regards,
Ashish
From: Ashish Kumar Singh [mailto:ashishkumar.singh@altair.com] Sent: Saturday, November 10, 2012 5:28 PM To: users-list@monetdb.org Cc: ashishk@altair.com Subject: RE: Database Memory Usage
Also it will be interesting to know if large memory usage is due to intermediate results then when can be expect them to be cleaned? ON statement and connection close??
Regards,
Ashish
From: Ashish Kumar Singh Sent: Saturday, November 10, 2012 3:09 PM To: users-list@monetdb.org Cc: Ashish Kumar Singh Subject: Database Memory Usage
HI All,
While calculating monetdb size I am observing below :
When I am inserting records to Monet DB using JDBC total memory and VM usage is going to 10 G + but when I see file system size of the database directory it is showing something like 1 GB .
Is some compression coming to picture here while storing data in file system?
Any pointers will be helpful.
Regards,
Ashish
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
participants (1)
-
Ashish Kumar Singh