GDK reported error. MT_mremap: mremap failed. !OS: Cannot allocate memory
The free disk space is 800G. And the query also fail when I run query right after a restart of the MonetDB server. And I'm not very sure what the mserver5 size means. When I run the query, the log file provide some information below, is that your mentioned? #GDKmremap(2465464320) fails, try to free up space [memory in use=149750256,virtual memory in use=4497801712]
Also, I'm sorry that the information(query statement and result) I put in is incomplete by my negligence. I copy the query statement and result and error message again as followed:
result and error message:
GDK reported error. MT_mremap: mremap(0x7ffc80000000,2054553600,2465464320) failed !OS: Cannot allocate memory HEAPextend: failed to extend to 2465464320 for 55/5577.tail: GDKmremap() failed +-------------+--------------+-------+ | c_last_name | c_first_name | sales | +=============+==============+=======+ +-------------+--------------+-------+ 0 tuples
query statement:
with frequent_ss_items as ( select substr(i_item_desc,1,30) itemdesc ,i_item_sk item_sk ,d_date solddate ,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (1999,1999+1,1999+2,1999+3) group by itemdesc ,i_item_sk ,d_date having count(*) >4 ), max_store_sales as ( select max(csales) tpcds_cmax from ( select c_customer_sk ,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (1999,1999+1,1999+2,1999+3) group by c_customer_sk ) x ), best_ss_customer as ( select c_customer_sk ,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (95/100.0) * ( select * from max_store_sales ) ) select sum(sales) from ( select cs_quantity*cs_list_price sales from catalog_sales ,date_dim where d_year = 1999 and d_moy = 1 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) union all select ws_quantity*ws_list_price sales from web_sales ,date_dim where d_year = 1999 and d_moy = 1 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer)) y limit 100; with frequent_ss_items as (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date solddate,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (1999,1999 + 1,1999 + 2,1999 + 3) group by itemdesc,i_item_sk,d_date having count(*) >4), max_store_sales as (select max(csales) tpcds_cmax from (select c_customer_sk,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (1999,1999+1,1999+2,1999+3) group by c_customer_sk) x), best_ss_customer as (select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (95/100.0) * (select * from max_store_sales)) select c_last_name,c_first_name,sales from (select c_last_name,c_first_name,sum(cs_quantity*cs_list_price) sales from catalog_sales ,customer ,date_dim where d_year = 1999 and d_moy = 1 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) and cs_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name union all select c_last_name,c_first_name,sum(ws_quantity*ws_list_price) sales from web_sales ,customer ,date_dim where d_year = 1999 and d_moy = 1 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer) and ws_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name) x order by c_last_name,c_first_name,sales limit 100;
Thanks! Regards,
Rancho
Hi Rancho,
thanks for the info!
With the size of mserver5, I was aiming at the virtual and memory-resident sizes as reported by, e.g., top, but the numbers for the log are also fine.
Given the 8GB RAM of your machine and the 800 GB free disk space, I honestly have no idea why the mremap of less than 2.5 GB files with a mserver5 that is less than 4.5 GB in total size (of which less than 1.5 GB in memory) --- all of this assuming that you indeed run on a 64-bit Ubuntu with a 64-bit MonetDB.
Just in case, maybe you can also share the output of `mserver5 --version` ?
Do you happen to have any OS limits activated to restrict the size of processes, e.g., via ulimit ?
While we haven't experimented with TPC-DS, yet, we regularly run TPC-H up to scale factor 100 (100 GB data) on a 8 GB RAM machine without any problems.
Could you possibly share your scripts to run TPC-DS with MonetDB, i.e., in particular your scripts to generate and load the data, and possible as well as the (MonetDB-)SQL code of the (remaining) queries?
Then we could ("easily") try to reproduce the problem, in order to analyze and fix it.
Thanks!
Best, Stefan
----- On Mar 15, 2018, at 10:22 AM, RanchoYuan yuanshijia@ww-it.cn wrote:
The free disk space is 800G. And the query also fail when I run query right after a restart of the MonetDB server. And I'm not very sure what the mserver5 size means. When I run the query, the log file provide some information below, is that your mentioned? #GDKmremap(2465464320) fails, try to free up space [memory in use=149750256,virtual memory in use=4497801712]
Also, I'm sorry that the information(query statement and result) I put in is incomplete by my negligence. I copy the query statement and result and error message again as followed:
result and error message:
GDK reported error. MT_mremap: mremap(0x7ffc80000000,2054553600,2465464320) failed !OS: Cannot allocate memory HEAPextend: failed to extend to 2465464320 for 55/5577.tail: GDKmremap() failed +-------------+--------------+-------+ | c_last_name | c_first_name | sales | +=============+==============+=======+ +-------------+--------------+-------+ 0 tuples
query statement:
with frequent_ss_items as ( select substr(i_item_desc,1,30) itemdesc ,i_item_sk item_sk ,d_date solddate ,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (1999,1999+1,1999+2,1999+3) group by itemdesc ,i_item_sk ,d_date having count(*) >4 ), max_store_sales as ( select max(csales) tpcds_cmax from ( select c_customer_sk ,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (1999,1999+1,1999+2,1999+3) group by c_customer_sk ) x ), best_ss_customer as ( select c_customer_sk ,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (95/100.0) * ( select * from max_store_sales ) ) select sum(sales) from ( select cs_quantity*cs_list_price sales from catalog_sales ,date_dim where d_year = 1999 and d_moy = 1 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) union all select ws_quantity*ws_list_price sales from web_sales ,date_dim where d_year = 1999 and d_moy = 1 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer)) y limit 100; with frequent_ss_items as (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date solddate,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (1999,1999 + 1,1999 + 2,1999 + 3) group by itemdesc,i_item_sk,d_date having count(*) >4), max_store_sales as (select max(csales) tpcds_cmax from (select c_customer_sk,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (1999,1999+1,1999+2,1999+3) group by c_customer_sk) x), best_ss_customer as (select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (95/100.0) * (select * from max_store_sales)) select c_last_name,c_first_name,sales from (select c_last_name,c_first_name,sum(cs_quantity*cs_list_price) sales from catalog_sales ,customer ,date_dim where d_year = 1999 and d_moy = 1 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) and cs_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name union all select c_last_name,c_first_name,sum(ws_quantity*ws_list_price) sales from web_sales ,customer ,date_dim where d_year = 1999 and d_moy = 1 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer) and ws_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name) x order by c_last_name,c_first_name,sales limit 100;
Thanks! Regards,
Rancho
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
participants (2)
-
RanchoYuan
-
Stefan Manegold