Hello, I'm pretty new to MonetDB and I'm trying to perform what seems like it should be a relatively straightforward inner join between two tables.
I would like to merge the two Alaska files from the American Community Survey --
http://www2.census.gov/acs2010_1yr/pums/csv_hak.zip contains a CSV file with one record per household http://www2.census.gov/acs2010_1yr/pums/csv_pak.zip contains a CSV file with one record per personThese two files should be merged on the field "serialno"
I am working on Windows 7 x64. Here's the contents of my mserver5.exe window:
# MonetDB 5 server v11.11.11 "Jul2012-SP2" # Serving database 'demo', using 2 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 3.905 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2012 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module loaded
The following commands..
1) create a person-level table
2) create a household-level table
3) read the person-level CSV into the person table
4) read the household-level CSV into the household table
5) attempt to print the first five records with "select * from tablename limit 5" -- sometimes these work, sometimes they don't.
6) merge the two tables together
7) attempt to print the first five records of the merged table
Unfortunately, I'm getting an error that I can't find with web searches.. The only place I can even find this error is http://bugs.monetdb.org/show_bug.cgi?id=2963 -- and that seems both time-related and fixed long ago.. I doubt it's a bug, I'm probably just making a beginner's mistake.
invalid tuple received from server, got 138 columns, expected 476, ignoring +------+------+------+------+-------+------+---------+---------+------+------+ 1 tuple (417.984ms) !466 columns dropped! note: to disable dropping columns and/or truncating fields use \w-1
Even after setting \w to -1, the number of columns returned gets limited, crashing my program.
I am ultimately trying to work with the new R/MonetDB package (which uses Java and is available from http://sqlsurvey.r-forge.r-project.org/), but none of the table interactivity will work until I'm able to return all of the columns properly.
Any advice would be appreciated! Thanks!!!
Anthony Damico Kaiser Family Foundation
Here are the SQL commands:
CREATE TABLE person (RT VARCHAR(255), SERIALNO INT, SPORDER INT, PUMA INT, ST INT, ADJINC INT, PWGTP INT, AGEP INT, CIT INT, CITWP INT, COW INT, DDRS INT, DEAR INT, DEYE INT, DOUT INT, DPHY INT, DRAT INT, DRATX INT, DREM INT, ENG INT, FER INT, GCL INT, GCM INT, GCR INT, HINS1 INT, HINS2 INT, HINS3 INT, HINS4 INT, HINS5 INT, HINS6 INT, HINS7 INT, INTP INT, JWMNP INT, JWRIP INT, JWTR INT, LANX INT, MAR INT, MARHD INT, MARHM INT, MARHT INT, MARHW INT, MARHYP INT, MIG INT, MIL INT, MLPA INT, MLPB INT, MLPC INT, MLPD INT, MLPE INT, MLPF INT, MLPG INT, MLPH INT, MLPI INT, MLPJ INT, MLPK INT, NWAB INT, NWAV INT, NWLA INT, NWLK INT, NWRE INT, OIP INT, PAP INT, RELP INT, RETP INT, SCH INT, SCHG INT, SCHL INT, SEMP INT, SEX INT, SSIP INT, SSP INT, WAGP INT, WKHP INT, WKL INT, WKW INT, WRK INT, YOEP INT, ANC INT, ANC1P INT, ANC2P INT, DECADE INT, DIS INT, DRIVESP INT, ESP INT, ESR INT, FOD1P INT, FOD2P INT, HICOV INT, HISP INT, INDP INT, JWAP INT, JWDP INT, LANP INT, MIGPUMA INT, MIGSP INT, MSP INT, NAICSP VARCHAR(255), NATIVITY INT, NOP INT, OC INT, OCCP INT, PAOC INT, PERNP INT, PINCP INT, POBP INT, POVPIP INT, POWPUMA INT, POWSP INT, PRIVCOV INT, PUBCOV INT, QTRBIR INT, RAC1P INT, RAC2P INT, RAC3P INT, RACAIAN INT, RACASN INT, RACBLK INT, RACNHPI INT, RACNUM INT, RACSOR INT, RACWHT INT, RC INT, SCIENGP INT, SCIENGRLP INT, SFN INT, SFR INT, SOCP VARCHAR(255), VPS INT, WAOB INT, FAGEP INT, FANCP INT, FCITP INT, FCITWP INT, FCOWP INT, FDDRSP INT, FDEARP INT, FDEYEP INT, FDOUTP INT, FDPHYP INT, FDRATP INT, FDRATXP INT, FDREMP INT, FENGP INT, FESRP INT, FFERP INT, FFODP INT, FGCLP INT, FGCMP INT, FGCRP INT, FHINS1P INT, FHINS2P INT, FHINS3C INT, FHINS3P INT, FHINS4C INT, FHINS4P INT, FHINS5C INT, FHINS5P INT, FHINS6P INT, FHINS7P INT, FHISP INT, FINDP INT, FINTP INT, FJWDP INT, FJWMNP INT, FJWRIP INT, FJWTRP INT, FLANP INT, FLANXP INT, FMARHDP INT, FMARHMP INT, FMARHTP INT, FMARHWP INT, FMARHYP INT, FMARP INT, FMIGP INT, FMIGSP INT, FMILPP INT, FMILSP INT, FOCCP INT, FOIP INT, FPAP INT, FPOBP INT, FPOWSP INT, FRACP INT, FRELP INT, FRETP INT, FSCHGP INT, FSCHLP INT, FSCHP INT, FSEMP INT, FSEXP INT, FSSIP INT, FSSP INT, FWAGP INT, FWKHP INT, FWKLP INT, FWKWP INT, FWRKP INT, FYOEP INT, pwgtp1 INT, pwgtp2 INT, pwgtp3 INT, pwgtp4 INT, pwgtp5 INT, pwgtp6 INT, pwgtp7 INT, pwgtp8 INT, pwgtp9 INT, pwgtp10 INT, pwgtp11 INT, pwgtp12 INT, pwgtp13 INT, pwgtp14 INT, pwgtp15 INT, pwgtp16 INT, pwgtp17 INT, pwgtp18 INT, pwgtp19 INT, pwgtp20 INT, pwgtp21 INT, pwgtp22 INT, pwgtp23 INT, pwgtp24 INT, pwgtp25 INT, pwgtp26 INT, pwgtp27 INT, pwgtp28 INT, pwgtp29 INT, pwgtp30 INT, pwgtp31 INT, pwgtp32 INT, pwgtp33 INT, pwgtp34 INT, pwgtp35 INT, pwgtp36 INT, pwgtp37 INT, pwgtp38 INT, pwgtp39 INT, pwgtp40 INT, pwgtp41 INT, pwgtp42 INT, pwgtp43 INT, pwgtp44 INT, pwgtp45 INT, pwgtp46 INT, pwgtp47 INT, pwgtp48 INT, pwgtp49 INT, pwgtp50 INT, pwgtp51 INT, pwgtp52 INT, pwgtp53 INT, pwgtp54 INT, pwgtp55 INT, pwgtp56 INT, pwgtp57 INT, pwgtp58 INT, pwgtp59 INT, pwgtp60 INT, pwgtp61 INT, pwgtp62 INT, pwgtp63 INT, pwgtp64 INT, pwgtp65 INT, pwgtp66 INT, pwgtp67 INT, pwgtp68 INT, pwgtp69 INT, pwgtp70 INT, pwgtp71 INT, pwgtp72 INT, pwgtp73 INT, pwgtp74 INT, pwgtp75 INT, pwgtp76 INT, pwgtp77 INT, pwgtp78 INT, pwgtp79 INT, pwgtp80 INT);
CREATE TABLE household (RT VARCHAR(255), SERIALNO INT, DIVISION INT, PUMA INT, REGION INT, ST INT, ADJHSG INT, ADJINC INT, WGTP INT, NP INT, TYPE INT, ACR INT, AGS INT, BATH INT, BDSP INT, BLD INT, BUS INT, CONP INT, ELEP INT, FS INT, FULP INT, GASP INT, HFL INT, INSP INT, MHP INT, MRGI INT, MRGP INT, MRGT INT, MRGX INT, REFR INT, RMSP INT, RNTM INT, RNTP INT, RWAT INT, SINK INT, SMP INT, STOV INT, TEL INT, TEN INT, TOIL INT, VACS INT, VALP INT, VEH INT, WATP INT, YBL INT, FES INT, FINCP INT, FPARC INT, GRNTP INT, GRPIP INT, HHL INT, HHT INT, HINCP INT, HUGCL INT, HUPAC INT, HUPAOC INT, HUPARC INT, KIT INT, LNGI INT, MULTG INT, MV INT, NOC INT, NPF INT, NPP INT, NR INT, NRC INT, OCPIP INT, PARTNER INT, PLM INT, PSF INT, R18 INT, R60 INT, R65 INT, RESMODE INT, SMOCP INT, SMX INT, SRNT INT, SVAL INT, TAXP INT, WIF INT, WKEXREL INT, WORKSTAT INT, FACRP INT, FAGSP INT, FBATHP INT, FBDSP INT, FBLDP INT, FBUSP INT, FCONP INT, FELEP INT, FFSP INT, FFULP INT, FGASP INT, FHFLP INT, FINSP INT, FKITP INT, FMHP INT, FMRGIP INT, FMRGP INT, FMRGTP INT, FMRGXP INT, FMVP INT, FPLMP INT, FREFRP INT, FRMSP INT, FRNTMP INT, FRNTP INT, FRWATP INT, FSINKP INT, FSMP INT, FSMXHP INT, FSMXSP INT, FSTOVP INT, FTAXP INT, FTELP INT, FTENP INT, FTOILP INT, FVACSP INT, FVALP INT, FVEHP INT, FWATP INT, FYBLP INT, wgtp1 INT, wgtp2 INT, wgtp3 INT, wgtp4 INT, wgtp5 INT, wgtp6 INT, wgtp7 INT, wgtp8 INT, wgtp9 INT, wgtp10 INT, wgtp11 INT, wgtp12 INT, wgtp13 INT, wgtp14 INT, wgtp15 INT, wgtp16 INT, wgtp17 INT, wgtp18 INT, wgtp19 INT, wgtp20 INT, wgtp21 INT, wgtp22 INT, wgtp23 INT, wgtp24 INT, wgtp25 INT, wgtp26 INT, wgtp27 INT, wgtp28 INT, wgtp29 INT, wgtp30 INT, wgtp31 INT, wgtp32 INT, wgtp33 INT, wgtp34 INT, wgtp35 INT, wgtp36 INT, wgtp37 INT, wgtp38 INT, wgtp39 INT, wgtp40 INT, wgtp41 INT, wgtp42 INT, wgtp43 INT, wgtp44 INT, wgtp45 INT, wgtp46 INT, wgtp47 INT, wgtp48 INT, wgtp49 INT, wgtp50 INT, wgtp51 INT, wgtp52 INT, wgtp53 INT, wgtp54 INT, wgtp55 INT, wgtp56 INT, wgtp57 INT, wgtp58 INT, wgtp59 INT, wgtp60 INT, wgtp61 INT, wgtp62 INT, wgtp63 INT, wgtp64 INT, wgtp65 INT, wgtp66 INT, wgtp67 INT, wgtp68 INT, wgtp69 INT, wgtp70 INT, wgtp71 INT, wgtp72 INT, wgtp73 INT, wgtp74 INT, wgtp75 INT, wgtp76 INT, wgtp77 INT, wgtp78 INT, wgtp79 INT, wgtp80 INT);
copy 50000 offset 2 records into person from 'C:\Users\user\desktop\ss10pak.csv' using delimiters ',','\n','"' NULL as '' ;
copy 25000 offset 2 records into household from 'C:\Users\user\desktop\ss10hak.csv' using delimiters ',','\n','"' NULL as '' ;
select * from person limit 5;
select * from household limit 5;
create table merged as select 'M' as rt, a.serialno, a.division, a.puma, a.region, a.st, a.adjhsg, a.adjinc, a.wgtp, a.np, a.type, a.acr, a.ags, a.bath, a.bdsp, a.bld, a.bus, a.conp, a.elep, a.fs, a.fulp, a.gasp, a.hfl, a.insp, a.mhp, a.mrgi, a.mrgp, a.mrgt, a.mrgx, a.refr, a.rmsp, a.rntm, a.rntp, a.rwat, a.sink, a.smp, a.stov, a.tel, a.ten, a.toil, a.vacs, a.valp, a.veh, a.watp, a.ybl, a.fes, a.fincp, a.fparc, a.grntp, a.grpip, a.hhl, a.hht, a.hincp, a.hugcl, a.hupac, a.hupaoc, a.huparc, a.kit, a.lngi, a.multg, a.mv, a.noc, a.npf, a.npp, a.nr, a.nrc, a.ocpip, a.partner, a.plm, a.psf, a.r18, a.r60, a.r65, a.resmode, a.smocp, a.smx, a.srnt, a.sval, a.taxp, a.wif, a.wkexrel, a.workstat, a.facrp, a.fagsp, a.fbathp, a.fbdsp, a.fbldp, a.fbusp, a.fconp, a.felep, a.ffsp, a.ffulp, a.fgasp, a.fhflp, a.finsp, a.fkitp, a.fmhp, a.fmrgip, a.fmrgp, a.fmrgtp, a.fmrgxp, a.fmvp, a.fplmp, a.frefrp, a.frmsp, a.frntmp, a.frntp, a.frwatp, a.fsinkp, a.fsmp, a.fsmxhp, a.fsmxsp, a.fstovp, a.ftaxp, a.ftelp, a.ftenp, a.ftoilp, a.fvacsp, a.fvalp, a.fvehp, a.fwatp, a.fyblp, a.wgtp1, a.wgtp2, a.wgtp3, a.wgtp4, a.wgtp5, a.wgtp6, a.wgtp7, a.wgtp8, a.wgtp9, a.wgtp10, a.wgtp11, a.wgtp12, a.wgtp13, a.wgtp14, a.wgtp15, a.wgtp16, a.wgtp17, a.wgtp18, a.wgtp19, a.wgtp20, a.wgtp21, a.wgtp22, a.wgtp23, a.wgtp24, a.wgtp25, a.wgtp26, a.wgtp27, a.wgtp28, a.wgtp29, a.wgtp30, a.wgtp31, a.wgtp32, a.wgtp33, a.wgtp34, a.wgtp35, a.wgtp36, a.wgtp37, a.wgtp38, a.wgtp39, a.wgtp40, a.wgtp41, a.wgtp42, a.wgtp43, a.wgtp44, a.wgtp45, a.wgtp46, a.wgtp47, a.wgtp48, a.wgtp49, a.wgtp50, a.wgtp51, a.wgtp52, a.wgtp53, a.wgtp54, a.wgtp55, a.wgtp56, a.wgtp57, a.wgtp58, a.wgtp59, a.wgtp60, a.wgtp61, a.wgtp62, a.wgtp63, a.wgtp64, a.wgtp65, a.wgtp66, a.wgtp67, a.wgtp68, a.wgtp69, a.wgtp70, a.wgtp71, a.wgtp72, a.wgtp73, a.wgtp74, a.wgtp75, a.wgtp76, a.wgtp77, a.wgtp78, a.wgtp79, a.wgtp80, sporder, pwgtp, agep, cit, citwp, cow, ddrs, dear, deye, dout, dphy, drat, dratx, drem, eng, fer, gcl, gcm, gcr, hins1, hins2, hins3, hins4, hins5, hins6, hins7, intp, jwmnp, jwrip, jwtr, lanx, mar, marhd, marhm, marht, marhw, marhyp, mig, mil, mlpa, mlpb, mlpc, mlpd, mlpe, mlpf, mlpg, mlph, mlpi, mlpj, mlpk, nwab, nwav, nwla, nwlk, nwre, oip, pap, relp, retp, sch, schg, schl, semp, sex, ssip, ssp, wagp, wkhp, wkl, wkw, wrk, yoep, anc, anc1p, anc2p, decade, dis, drivesp, esp, esr, fod1p, fod2p, hicov, hisp, indp, jwap, jwdp, lanp, migpuma, migsp, msp, naicsp, nativity, nop, oc, occp, paoc, pernp, pincp, pobp, povpip, powpuma, powsp, privcov, pubcov, qtrbir, rac1p, rac2p, rac3p, racaian, racasn, racblk, racnhpi, racnum, racsor, racwht, rc, sciengp, sciengrlp, sfn, sfr, socp, vps, waob, fagep, fancp, fcitp, fcitwp, fcowp, fddrsp, fdearp, fdeyep, fdoutp, fdphyp, fdratp, fdratxp, fdremp, fengp, fesrp, fferp, ffodp, fgclp, fgcmp, fgcrp, fhins1p, fhins2p, fhins3c, fhins3p, fhins4c, fhins4p, fhins5c, fhins5p, fhins6p, fhins7p, fhisp, findp, fintp, fjwdp, fjwmnp, fjwrip, fjwtrp, flanp, flanxp, fmarhdp, fmarhmp, fmarhtp, fmarhwp, fmarhyp, fmarp, fmigp, fmigsp, fmilpp, fmilsp, foccp, foip, fpap, fpobp, fpowsp, fracp, frelp, fretp, fschgp, fschlp, fschp, fsemp, fsexp, fssip, fssp, fwagp, fwkhp, fwklp, fwkwp, fwrkp, fyoep, pwgtp1, pwgtp2, pwgtp3, pwgtp4, pwgtp5, pwgtp6, pwgtp7, pwgtp8, pwgtp9, pwgtp10, pwgtp11, pwgtp12, pwgtp13, pwgtp14, pwgtp15, pwgtp16, pwgtp17, pwgtp18, pwgtp19, pwgtp20, pwgtp21, pwgtp22, pwgtp23, pwgtp24, pwgtp25, pwgtp26, pwgtp27, pwgtp28, pwgtp29, pwgtp30, pwgtp31, pwgtp32, pwgtp33, pwgtp34, pwgtp35, pwgtp36, pwgtp37, pwgtp38, pwgtp39, pwgtp40, pwgtp41, pwgtp42, pwgtp43, pwgtp44, pwgtp45, pwgtp46, pwgtp47, pwgtp48, pwgtp49, pwgtp50, pwgtp51, pwgtp52, pwgtp53, pwgtp54, pwgtp55, pwgtp56, pwgtp57, pwgtp58, pwgtp59, pwgtp60, pwgtp61, pwgtp62, pwgtp63, pwgtp64, pwgtp65, pwgtp66, pwgtp67, pwgtp68, pwgtp69, pwgtp70, pwgtp71, pwgtp72, pwgtp73, pwgtp74, pwgtp75, pwgtp76, pwgtp77, pwgtp78, pwgtp79, pwgtp80 from household as a inner join person as b on a.serialno = b.serialno with data;
select * from merged limit 5;
invalid tuple received from server, got 138 columns, expected 476, ignoring