I am running MonetDB5 Oct 2012 SP3 on my work Windows 7 desktop computer. My office has what I imagine is a pretty typical mix of networked machines, with a few more powerful servers and larger shared drives that everyone can connect to. (1) When I attempt to import a large CSV file into a MonetDB server on the _local_ disk (C:\My Directory\MonetDB\dbfarm), the importation command runs as expected. (2) When I attempt to import the same CSV file to a MonetDB server stored on a networked drive, the import command fails. (3) Finally, when I *remote desktop* into the server that the networked drive is actually attached to, the importation succeeds again! Here's the error seen in scenario (2) --
GDKerror:!ERROR: GDKunlink(bat\13\1327.tail)
!OS: The process cannot access the file because it is being used by another process.
GDKunlink(bat\13\1327.theap)
!OS: The process cannot access the file because it is being used by another process.
So it seems pretty clear that something about windows networked drives is the culprit. This error does not occur on every file (larger files seem more prone to the error), but if an import command does not work on a file, it _consistently_ does not work on that file. As an example, importing this large file works with methods (1) and (3) but not (2) --
I tried disabling the windows firewall and using different server ports, but I still hit the above error.
I spoke with Hannes about it briefly, he said: "The issue is probably related to the Windows mmap() implementation when
talking to a SMB filesystem. It is far from certain that there is
something we can do about this, but in either case, we'd like to know
about it."
Should I submit an official bug report to
http://bugs.monetdb.org/
? Perhaps we are overlooking something obvious about our windows network's
configuration..
Thanks!!!
Here is my MonetDB server info, followed by the COPY INTO commands run in -X mode for each of the three scenarios:
# MonetDB 5 server v11.13.9 "Oct2012-SP3"
# Serving database 'bsa', using 2 threads
# Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked
# Found 15.873 GiB available main-memory.
# Copyright (c) 1993-July 2008 CWI.
# Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved
# Visit
http://www.monetdb.org/ for further information
# Listening for connection requests on mapi:monetdb://
127.0.0.1:50001/# MonetDB/JAQL module loaded
# MonetDB/SQL module loaded
**************************
(1) - local disk - works -----
fetch next block: start at:196
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
Database: MonetDB v11.13.9 (Oct2012-SP3), 'bsa'
closing result set
Type \q to quit, \? for a list of available commands
auto commit mode: on
mapi_query:46:SET TIME ZONE INTERVAL '-05:00' HOUR TO MINUTE
fetch next block: start at:198
got next block: length:3
text:&3
got complete block:
text:&3
read_line:&3
allocating new result set
fetch next block: start at:201
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
closing result set
sql>create table mytable (
mapi_query_part:23:create table mytable (
fetch next block: start at:203
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:206
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_id VARCHAR(255),
mapi_query_part:27: car_line_id VARCHAR(255),
fetch next block: start at:208
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:211
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> bene_sex_ident_cd INTEGER,
mapi_query_part:28: bene_sex_ident_cd INTEGER,
fetch next block: start at:213
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:216
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> bene_age_cat_cd INTEGER,
mapi_query_part:26: bene_age_cat_cd INTEGER,
fetch next block: start at:218
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:221
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_icd9_dgns_cd VARCHAR(255),
mapi_query_part:37: car_line_icd9_dgns_cd VARCHAR(255),
fetch next block: start at:223
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:226
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_hcpcs_cd VARCHAR(255),
mapi_query_part:33: car_line_hcpcs_cd VARCHAR(255),
fetch next block: start at:228
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:231
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_betos_cd VARCHAR(255),
mapi_query_part:33: car_line_betos_cd VARCHAR(255),
fetch next block: start at:233
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:236
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_srvc_cnt INTEGER,
mapi_query_part:28: car_line_srvc_cnt INTEGER,
fetch next block: start at:238
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:241
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_prvdr_type_cd INTEGER,
mapi_query_part:33: car_line_prvdr_type_cd INTEGER,
fetch next block: start at:243
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:246
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_cms_type_srvc_cd VARCHAR(255),
mapi_query_part:41: car_line_cms_type_srvc_cd VARCHAR(255),
fetch next block: start at:248
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:251
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_place_of_srvc_cd INTEGER,
mapi_query_part:36: car_line_place_of_srvc_cd INTEGER,
fetch next block: start at:253
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:256
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_hcpcs_pmt_amt INTEGER
mapi_query_part:27: car_hcpcs_pmt_amt INTEGER
fetch next block: start at:258
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:261
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more>) ;
mapi_query_part:4:) ;
fetch next block: start at:263
got next block: length:3
text:&3
got complete block:
text:&3
read_line:&3
allocating new result set
fetch next block: start at:266
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
operation successful (118.294ms)
closing result set
sql>
fetch next block: start at:268
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
sql>COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Directory\BSAPUF\2008
\2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n','\"' NULL AS '
' ;
mapi_query_part:160:COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Direc
tory\BSAPUF\2008\2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n
','\"' NULL AS '' ;
fetch next block: start at:270
got next block: length:14
text:&2 9676439 -1
got complete block:
text:&2 9676439 -1
read_line:&2 9676439 -1
allocating new result set
fetch next block: start at:284
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
9676439 affected rows (3m 26s)
closing result set
sql>
**************************
(2) - networked disk - fails -----
read_line:☺
more> car_line_id VARCHAR(255),
mapi_query_part:27: car_line_id VARCHAR(255),
fetch next block: start at:208
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:211
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> bene_sex_ident_cd INTEGER,
mapi_query_part:28: bene_sex_ident_cd INTEGER,
fetch next block: start at:213
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:216
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> bene_age_cat_cd INTEGER,
mapi_query_part:26: bene_age_cat_cd INTEGER,
fetch next block: start at:218
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:221
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_icd9_dgns_cd VARCHAR(255),
mapi_query_part:37: car_line_icd9_dgns_cd VARCHAR(255),
fetch next block: start at:223
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:226
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_hcpcs_cd VARCHAR(255),
mapi_query_part:33: car_line_hcpcs_cd VARCHAR(255),
fetch next block: start at:228
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:231
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_betos_cd VARCHAR(255),
mapi_query_part:33: car_line_betos_cd VARCHAR(255),
fetch next block: start at:233
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:236
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_srvc_cnt INTEGER,
mapi_query_part:28: car_line_srvc_cnt INTEGER,
fetch next block: start at:238
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:241
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_prvdr_type_cd INTEGER,
mapi_query_part:33: car_line_prvdr_type_cd INTEGER,
fetch next block: start at:243
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:246
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_cms_type_srvc_cd VARCHAR(255),
mapi_query_part:41: car_line_cms_type_srvc_cd VARCHAR(255),
fetch next block: start at:248
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:251
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_line_place_of_srvc_cd INTEGER,
mapi_query_part:36: car_line_place_of_srvc_cd INTEGER,
fetch next block: start at:253
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:256
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more> car_hcpcs_pmt_amt INTEGER
mapi_query_part:27: car_hcpcs_pmt_amt INTEGER
fetch next block: start at:258
got next block: length:3
text:☺☻
got complete block:
text:☺☻
read_line:☺☻
fetch next block: start at:261
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
more>) ;
mapi_query_part:4:) ;
fetch next block: start at:263
got next block: length:3
text:&3
got complete block:
text:&3
read_line:&3
allocating new result set
fetch next block: start at:266
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
operation successful (289.145ms)
closing result set
sql>
fetch next block: start at:268
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
sql>COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Directory\BSAPUF\2008
\2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n','\"' NULL AS '
' ;
mapi_query_part:160:COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Direc
tory\BSAPUF\2008\2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n
','\"' NULL AS '' ;
fetch next block: start at:270
got next block: length:252
text:!GDKerror:!ERROR: GDKunlink(bat\13\1327.tail)
!!OS: The process cannot access the file because it is being used by another pro
cess.
!GDKunlink(bat\13\1327.theap)
!!OS: The process cannot access the file because it is being used by another pro
cess.
!
got complete block:
text:!GDKerror:!ERROR: GDKunlink(bat\13\1327.tail)
!!OS: The process cannot access the file because it is being used by another pro
cess.
!GDKunlink(bat\13\1327.theap)
!!OS: The process cannot access the file because it is being used by another pro
cess.
!
read_line:!GDKerror:!ERROR: GDKunlink(bat\13\1327.tail)
allocating new result set
got complete block:
text:!!OS: The process cannot access the file because it is being used by anothe
r process.
!GDKunlink(bat\13\1327.theap)
!!OS: The process cannot access the file because it is being used by another pro
cess.
!
read_line:!!OS: The process cannot access the file because it is being used by a
nother process.
got complete block:
text:!GDKunlink(bat\13\1327.theap)
!!OS: The process cannot access the file because it is being used by another pro
cess.
!
read_line:!GDKunlink(bat\13\1327.theap)
got complete block:
text:!!OS: The process cannot access the file because it is being used by anothe
r process.
!
read_line:!!OS: The process cannot access the file because it is being used by a
nother process.
got complete block:
text:!
read_line:!
fetch next block: start at:522
got next block: length:0
text:
got complete block:
text:☺
read_line:☺
GDKerror:!ERROR: GDKunlink(bat\13\1327.tail)
!OS: The process cannot access the file because it is being used by another proc
ess.
GDKunlink(bat\13\1327.theap)
!OS: The process cannot access the file because it is being used by another proc
ess.
closing result set