Sean,

Use sql select to find out duplicates.

Regards Allen

On 11/12/11 12:01 PM, "Sean McNamara" tank@thursday.org wrote:

I tried that (as both the normal dbuser and as admin). Both fail the same way but don't give an indicator of where it feels the constraint is violated:

$ mclient -u monetdb -s "alter table mduser.testdata add constraint testdata_pkey primary key (foo,epoch);" testdb password: SQLException:assert:UPDATE: PRIMARY KEY constraint 'testdata.testdata_pkey' violated

On Sat, Nov 12, 2011 at 11:42 AM, Allen Zhang azhang@pmci-usa.com wrote:

...

How about add constraint after you import? I think it might tell you which one had problem.

Regards Allen

On 11/12/11 11:19 AM, "Sean McNamara" tank@thursday.org wrote:

...

Brilliant suggestion, and I think it exposed the problem.

$ mclient -u monetdb -s "copy into mduser.testdata from '/tmp/testdata2.csv' using delimiters ',','\n',''';" testdb password: SQLException:importTable:value 'ONE_SECOND' from line 12734 field 3 not inserted, expecting type varchar(8) failed to import table

It looks like the field I created is too small to handle some of the values in the file. After expanding all the varchar fields to 32 and re-running, I get a different error:

$ mclient -u monetdb -s "copy into mduser.testdata from '/tmp/testdata2.csv' using delimiters ',','\n',''';" testdb password: SQLException:assert:INSERT INTO: PRIMARY KEY constraint 'testdata.testdata_pkey' violated

That makes me think I have duplicate data. The PKEY is comprised of fields #1 and #5, so I check for that but it looks fine:

$ cat /tmp/testdata2.csv | cut -d, -f1,5 |uniq -d $

However if I drop and re-add the table without the constraint, it works:

sql>create table testdata ( more> foo varchar(32), more> bar varchar(32), more> baz varchar(32), more> woz varchar(32), more> epoch bigint more>); operation successful (1.477ms) sql>\q $ mclient -u monetdb -s "copy into mduser.testdata from '/tmp/testdata2.csv' using delimiters ',','\n',''';" testdb password: 2519931 affected rows

Thanks Allen! This gets me able to import. Now I'm wondering what might be wrong with the constraint. Any insight there?

On Sat, Nov 12, 2011 at 10:58 AM, Allen Zhang azhang@pmci-usa.com wrote:

...

What about just issue the following:

mclient -s "copy into testdata from '/tmp/testdata2.csv' using delimiters ',','\n',''';"

Regards Allen

On 11/12/11 10:48 AM, "Sean McNamara" tank@thursday.org wrote:

...

Thanks Allen. Red-herring there. My first file had "SPY" in the first column, and the second had "testdata1", so the difference there is just because of the contents. If I convert the original SPY file and re-run, I get the same behavior.

$ mclient -s "copy into testdata from STDIN using delimiters ',','\n',''';" testdb - < /tmp/testdata2.csv syntax error, unexpected STRING in: "'SPY'"

On Sat, Nov 12, 2011 at 10:43 AM, Allen Zhang azhang@pmci-usa.com wrote:

...

Hi Sean,

You should inspect your data quality, it seems in the first 3 imports, it stopped at "spy", I suspect it might be some \t in that supposed to be string, and last import you put single-quote around string, then it stopped at "testvalue1", search your import file, look for that "testvalue1", the problem should around there, maybe an extra field because wrong replace by sed.

Best Regards Allen

On 11/12/11 10:26 AM, "Sean McNamara" tank@thursday.org wrote:

>Thanks Viktor, but no dice. I also tried converting from CSV to >TAB-delim to match what you were doing, but it's still not >working... > >$ mclient -s "copy into testdata from STDIN using delimiters >',','\n','';" testdb - < /tmp/testdata.csv >syntax error, unexpected IDENT in: "spy" > >$ sed 's/,/'$'\t''/g' /tmp/testdata.csv > /tmp/testdata.dat > >$ mclient -s "copy into testdata from STDIN;" testdb - < >/tmp/testdata.dat >syntax error, unexpected IDENT in: "spy" > >$ mclient -s "copy into testdata from STDIN using delimiters >'\t','\n','';" testdb - < /tmp/testdata.dat >syntax error, unexpected IDENT in: "spy" > >I'm understanding the format of the documentation quite a bit better >now after spending a few days with it, but I'm still not clear on >why >this insert is failing. >I went one step further and converted the input file to single-quote >the string fields, but this fails as well: > >$ mclient -s "copy into testdata from STDIN using delimiters >',','\n',''';" testdb - < /tmp/testdata2.csv >syntax error, unexpected STRING in: "'testvalue1'" > > >The problem has to be something trivial, but I have no idea what it >is... > >On Sat, Nov 12, 2011 at 5:30 AM, Viktor Rosenfeld >listuser36@googlemail.com wrote: >> Hi Sean, >> >> Sean McNamara wrote: >> >>> $ mclient -s "copy 2519931 records into testdata from STDIN using >>> delimiters ',';" testdb - < /tmp/testdata.csv >>> syntax error, unexpected IDENT in: "testvalue1" >>> >>> where testvalue1 is one of the actual values in the FOO column >>> (actually the only value in this particular file.) >>> >>> I'm wondering if this is telling me it's expecting the field to >>>be >>> quoted, but that seems unlikely as it would make working with >>>CSVs >>> quite a bit painful. >>> >>> Can you guys kindly help me take another step forward. I really >>> appreciate the guidance. >> >> I'm using >> >> COPY ... USING DELIMITERS '\t','\n',''; >> >> to bulk-load tab-delimited unquoted data. The '\t' is the default >>field >> separator, the '\n' is the default record separator. The final '' >>is >>the >> quote string -- empty in my case. >> >> See: >>http://www.monetdb.org/Documentation/Manuals/SQLreference/CopyInto >> >> Cheers, >> Viktor >> > >-------------------------------------------------------------------- >-- >-- >-- >---- >RSA(R) Conference 2012 >Save $700 by Nov 18 >Register now >http://p.sf.net/sfu/rsa-sfdev2dev1 >_______________________________________________ >MonetDB-users mailing list >MonetDB-users@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/monetdb-users