I am running MonetDB5 Oct 2012 SP3 on my work Windows 7 desktop computer. My office has what I imagine is a pretty typical mix of networked machines, with a few more powerful servers and larger shared drives that everyone can connect to. (1) When I attempt to import a large CSV file into a MonetDB server on the _local_ disk (C:\My Directory\MonetDB\dbfarm), the importation command runs as expected. (2) When I attempt to import the same CSV file to a MonetDB server stored on a networked drive, the import command fails. (3) Finally, when I *remote desktop* into the server that the networked drive is actually attached to, the importation succeeds again! Here's the error seen in scenario (2) --
GDKerror:!ERROR: GDKunlink(bat\13\1327.tail) !OS: The process cannot access the file because it is being used by another process. GDKunlink(bat\13\1327.theap) !OS: The process cannot access the file because it is being used by another process.
So it seems pretty clear that something about windows networked drives is the culprit. This error does not occur on every file (larger files seem more prone to the error), but if an import command does not work on a file, it _consistently_ does not work on that file. As an example, importing this large file works with methods (1) and (3) but not (2) --
http://downloads.cms.gov/BSAPUF/2008_BSA_Carrier_Line_Items_PUF_1.zip
I tried disabling the windows firewall and using different server ports, but I still hit the above error.
I spoke with Hannes about it briefly, he said: "The issue is probably related to the Windows mmap() implementation when talking to a SMB filesystem. It is far from certain that there is something we can do about this, but in either case, we'd like to know about it."
Should I submit an official bug report to http://bugs.monetdb.org/ ? Perhaps we are overlooking something obvious about our windows network's configuration..
Thanks!!!
Here is my MonetDB server info, followed by the COPY INTO commands run in -X mode for each of the three scenarios:
# MonetDB 5 server v11.13.9 "Oct2012-SP3" # Serving database 'bsa', using 2 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 15.873 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50001/ # MonetDB/JAQL module loaded # MonetDB/SQL module loaded
************************** (1) - local disk - works -----
fetch next block: start at:196 got next block: length:0 text: got complete block: text:☺
read_line:☺ Database: MonetDB v11.13.9 (Oct2012-SP3), 'bsa' closing result set Type \q to quit, ? for a list of available commands auto commit mode: on mapi_query:46:SET TIME ZONE INTERVAL '-05:00' HOUR TO MINUTE fetch next block: start at:198 got next block: length:3 text:&3
got complete block: text:&3
read_line:&3 allocating new result set fetch next block: start at:201 got next block: length:0 text: got complete block: text:☺
read_line:☺ closing result set sql>create table mytable ( mapi_query_part:23:create table mytable (
fetch next block: start at:203 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:206 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_id VARCHAR(255), mapi_query_part:27: car_line_id VARCHAR(255),
fetch next block: start at:208 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:211 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> bene_sex_ident_cd INTEGER, mapi_query_part:28: bene_sex_ident_cd INTEGER,
fetch next block: start at:213 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:216 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> bene_age_cat_cd INTEGER, mapi_query_part:26: bene_age_cat_cd INTEGER,
fetch next block: start at:218 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:221 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_icd9_dgns_cd VARCHAR(255), mapi_query_part:37: car_line_icd9_dgns_cd VARCHAR(255),
fetch next block: start at:223 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:226 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_hcpcs_cd VARCHAR(255), mapi_query_part:33: car_line_hcpcs_cd VARCHAR(255),
fetch next block: start at:228 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:231 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_betos_cd VARCHAR(255), mapi_query_part:33: car_line_betos_cd VARCHAR(255),
fetch next block: start at:233 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:236 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_srvc_cnt INTEGER, mapi_query_part:28: car_line_srvc_cnt INTEGER,
fetch next block: start at:238 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:241 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_prvdr_type_cd INTEGER, mapi_query_part:33: car_line_prvdr_type_cd INTEGER,
fetch next block: start at:243 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:246 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_cms_type_srvc_cd VARCHAR(255), mapi_query_part:41: car_line_cms_type_srvc_cd VARCHAR(255),
fetch next block: start at:248 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:251 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_place_of_srvc_cd INTEGER, mapi_query_part:36: car_line_place_of_srvc_cd INTEGER,
fetch next block: start at:253 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:256 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_hcpcs_pmt_amt INTEGER mapi_query_part:27: car_hcpcs_pmt_amt INTEGER
fetch next block: start at:258 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:261 got next block: length:0 text: got complete block: text:☺
read_line:☺ more>) ; mapi_query_part:4:) ;
fetch next block: start at:263 got next block: length:3 text:&3
got complete block: text:&3
read_line:&3 allocating new result set fetch next block: start at:266 got next block: length:0 text: got complete block: text:☺
read_line:☺ operation successful (118.294ms) closing result set sql> fetch next block: start at:268 got next block: length:0 text: got complete block: text:☺
read_line:☺ sql>COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Directory\BSAPUF\2008 \2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n','"' NULL AS ' ' ; mapi_query_part:160:COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Direc tory\BSAPUF\2008\2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n ','"' NULL AS '' ;
fetch next block: start at:270 got next block: length:14 text:&2 9676439 -1
got complete block: text:&2 9676439 -1
read_line:&2 9676439 -1 allocating new result set fetch next block: start at:284 got next block: length:0 text: got complete block: text:☺
read_line:☺ 9676439 affected rows (3m 26s) closing result set sql>
************************** (2) - networked disk - fails -----
read_line:☺ more> car_line_id VARCHAR(255), mapi_query_part:27: car_line_id VARCHAR(255),
fetch next block: start at:208 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:211 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> bene_sex_ident_cd INTEGER, mapi_query_part:28: bene_sex_ident_cd INTEGER,
fetch next block: start at:213 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:216 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> bene_age_cat_cd INTEGER, mapi_query_part:26: bene_age_cat_cd INTEGER,
fetch next block: start at:218 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:221 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_icd9_dgns_cd VARCHAR(255), mapi_query_part:37: car_line_icd9_dgns_cd VARCHAR(255),
fetch next block: start at:223 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:226 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_hcpcs_cd VARCHAR(255), mapi_query_part:33: car_line_hcpcs_cd VARCHAR(255),
fetch next block: start at:228 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:231 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_betos_cd VARCHAR(255), mapi_query_part:33: car_line_betos_cd VARCHAR(255),
fetch next block: start at:233 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:236 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_srvc_cnt INTEGER, mapi_query_part:28: car_line_srvc_cnt INTEGER,
fetch next block: start at:238 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:241 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_prvdr_type_cd INTEGER, mapi_query_part:33: car_line_prvdr_type_cd INTEGER,
fetch next block: start at:243 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:246 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_cms_type_srvc_cd VARCHAR(255), mapi_query_part:41: car_line_cms_type_srvc_cd VARCHAR(255),
fetch next block: start at:248 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:251 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_place_of_srvc_cd INTEGER, mapi_query_part:36: car_line_place_of_srvc_cd INTEGER,
fetch next block: start at:253 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:256 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_hcpcs_pmt_amt INTEGER mapi_query_part:27: car_hcpcs_pmt_amt INTEGER
fetch next block: start at:258 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:261 got next block: length:0 text: got complete block: text:☺
read_line:☺ more>) ; mapi_query_part:4:) ;
fetch next block: start at:263 got next block: length:3 text:&3
got complete block: text:&3
read_line:&3 allocating new result set fetch next block: start at:266 got next block: length:0 text: got complete block: text:☺
read_line:☺ operation successful (289.145ms) closing result set sql> fetch next block: start at:268 got next block: length:0 text: got complete block: text:☺
read_line:☺ sql>COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Directory\BSAPUF\2008 \2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n','"' NULL AS ' ' ; mapi_query_part:160:COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Direc tory\BSAPUF\2008\2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n ','"' NULL AS '' ;
fetch next block: start at:270 got next block: length:252 text:!GDKerror:!ERROR: GDKunlink(bat\13\1327.tail) !!OS: The process cannot access the file because it is being used by another pro cess. !GDKunlink(bat\13\1327.theap) !!OS: The process cannot access the file because it is being used by another pro cess. !
got complete block: text:!GDKerror:!ERROR: GDKunlink(bat\13\1327.tail) !!OS: The process cannot access the file because it is being used by another pro cess. !GDKunlink(bat\13\1327.theap) !!OS: The process cannot access the file because it is being used by another pro cess. !
read_line:!GDKerror:!ERROR: GDKunlink(bat\13\1327.tail) allocating new result set got complete block: text:!!OS: The process cannot access the file because it is being used by anothe r process. !GDKunlink(bat\13\1327.theap) !!OS: The process cannot access the file because it is being used by another pro cess. !
read_line:!!OS: The process cannot access the file because it is being used by a nother process. got complete block: text:!GDKunlink(bat\13\1327.theap) !!OS: The process cannot access the file because it is being used by another pro cess. !
read_line:!GDKunlink(bat\13\1327.theap) got complete block: text:!!OS: The process cannot access the file because it is being used by anothe r process. !
read_line:!!OS: The process cannot access the file because it is being used by a nother process. got complete block: text:!
read_line:! fetch next block: start at:522 got next block: length:0 text: got complete block: text:☺
read_line:☺ GDKerror:!ERROR: GDKunlink(bat\13\1327.tail) !OS: The process cannot access the file because it is being used by another proc ess. GDKunlink(bat\13\1327.theap) !OS: The process cannot access the file because it is being used by another proc ess.
closing result set sql>
************************** (3) - remote desktop into server so networked disk becomes local disk - works -----
read_line:[ "monet_release", "Oct2012-SP3" ] fetch next block: start at:196 got next block: length:0 text: got complete block: text:☺
read_line:☺ Database: MonetDB v11.13.9 (Oct2012-SP3), 'bsa' closing result set Type \q to quit, ? for a list of available commands auto commit mode: on mapi_query:46:SET TIME ZONE INTERVAL '-05:00' HOUR TO MINUTE fetch next block: start at:198 got next block: length:3 text:&3
got complete block: text:&3
read_line:&3 allocating new result set fetch next block: start at:201 got next block: length:0 text: got complete block: text:☺
read_line:☺ closing result set sql>create table mytable3 ( mapi_query_part:24:create table mytable3 (
fetch next block: start at:203 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:206 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_id VARCHAR(255), mapi_query_part:27: car_line_id VARCHAR(255),
fetch next block: start at:208 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:211 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> bene_sex_ident_cd INTEGER, mapi_query_part:28: bene_sex_ident_cd INTEGER,
fetch next block: start at:213 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:216 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> bene_age_cat_cd INTEGER, mapi_query_part:26: bene_age_cat_cd INTEGER,
fetch next block: start at:218 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:221 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_icd9_dgns_cd VARCHAR(255), mapi_query_part:37: car_line_icd9_dgns_cd VARCHAR(255),
fetch next block: start at:223 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:226 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_hcpcs_cd VARCHAR(255), mapi_query_part:33: car_line_hcpcs_cd VARCHAR(255),
fetch next block: start at:228 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:231 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_betos_cd VARCHAR(255), mapi_query_part:33: car_line_betos_cd VARCHAR(255),
fetch next block: start at:233 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:236 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_srvc_cnt INTEGER, mapi_query_part:28: car_line_srvc_cnt INTEGER,
fetch next block: start at:238 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:241 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_prvdr_type_cd INTEGER, mapi_query_part:33: car_line_prvdr_type_cd INTEGER,
fetch next block: start at:243 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:246 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_cms_type_srvc_cd VARCHAR(255), mapi_query_part:41: car_line_cms_type_srvc_cd VARCHAR(255),
fetch next block: start at:248 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:251 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_line_place_of_srvc_cd INTEGER, mapi_query_part:36: car_line_place_of_srvc_cd INTEGER,
fetch next block: start at:253 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:256 got next block: length:0 text: got complete block: text:☺
read_line:☺ more> car_hcpcs_pmt_amt INTEGER mapi_query_part:27: car_hcpcs_pmt_amt INTEGER
fetch next block: start at:258 got next block: length:3 text:☺☻
got complete block: text:☺☻
read_line:☺☻ fetch next block: start at:261 got next block: length:0 text: got complete block: text:☺
read_line:☺ more>) ; mapi_query_part:4:) ;
fetch next block: start at:263 got next block: length:3 text:&3
got complete block: text:&3
read_line:&3 allocating new result set fetch next block: start at:266 got next block: length:0 text: got complete block: text:☺
read_line:☺ operation successful (276.500ms) closing result set sql> fetch next block: start at:268 got next block: length:0 text: got complete block: text:☺
read_line:☺ sql>COPY 9676440 offset 2 records INTO mytable3 FROM 'v:\temp\2008_BSA_Carrier_L ine_Items_PUF_1.csv' USING DELIMITERS ',','\n','"' NULL AS '' ; mapi_query_part:141:COPY 9676440 offset 2 records INTO mytable3 FROM 'v:\temp\20 08_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n','"' NULL AS '' ;
fetch next block: start at:270 got next block: length:14 text:&2 9676439 -1
got complete block: text:&2 9676439 -1
read_line:&2 9676439 -1 allocating new result set fetch next block: start at:284 got next block: length:0 text: got complete block: text:☺
read_line:☺ 9676439 affected rows (2m 14s) closing result set sql>