[MonetDB-users] strPut error with large string tables
Hi,
(June2010 SP1 built on Windows)
I loaded a table with a single string column successfully with approximately 500M string values. Running 'Select * from table limit 10' gives me this error:
'strPut: string heaps gets larger than 15GB.'
What does this mean?
On 2010-09-01 23:27, Alex Bo. wrote:
Hi,
(June2010 SP1 built on Windows)
I loaded a table with a single string column successfully with approximately 500M string values. Running 'Select * from table limit 10' gives me this error:
'strPut: string heaps gets larger than 15GB.'
What does this mean?
The proximate cause is that string heaps are only allowed slightly less than 16 GB. This happens when you use 32-bit OIDs on a 64 bit system (the default for 64-bit Windows). The way strings are stored is by using a fixed-size index into the area where the actual strings are stored (the string heap). These indexes in the above-mentioned combination are 32 bits, shifted left by three positions. This gives a maximum index value of 0x3fffffff8 or 17179869176 (which is 15.999999992549419 GiB, truncated to 15 GiB).
The ultimate cause I don't know yet. Obviously you were able to load the data, so it is not clear (yet) why you need a string heap that is this large when doing that select.
On Thu, Sep 02, 2010 at 02:17:28PM +0200, Sjoerd Mullender wrote:
On 2010-09-01 23:27, Alex Bo. wrote:
Hi,
(June2010 SP1 built on Windows)
I loaded a table with a single string column successfully with approximately 500M string values. Running 'Select * from table limit 10' gives me this error:
'strPut: string heaps gets larger than 15GB.'
What does this mean?
The proximate cause is that string heaps are only allowed slightly less than 16 GB. This happens when you use 32-bit OIDs on a 64 bit system (the default for 64-bit Windows). The way strings are stored is by using a fixed-size index into the area where the actual strings are stored (the string heap). These indexes in the above-mentioned combination are 32 bits, shifted left by three positions. This gives a maximum index value of 0x3fffffff8 or 17179869176 (which is 15.999999992549419 GiB, truncated to 15 GiB).
The ultimate cause I don't know yet. Obviously you were able to load the data, so it is not clear (yet) why you need a string heap that is this large when doing that select.
Alex,
the results of running your query prefixed with "PLAN " and "EXPLAIN " (one at a time) on your data and system might help us to investigate the cause of this problem.
Stefan
-- Sjoerd Mullender
This SF.net Dev2Dev email is sponsored by:
Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Here are the outputs you requested:
PLAN:
top N ( | project ( | | table(myschema.mytable) [ mytable.field1, mytable.field2, mytable.%TID% NOT NULL ] | ) [ mytable.field1, mytable.field2 ] ) [ 10 ]
EXPLAIN:
function user.s1_1{autoCommit=true}():void; _2 := sql.mvc(); _43:bat[:oid,:str] := sql.bind(_2,"myschema","mytable","field1",0,0@0,nil:oid); _7:bat[:oid,:str] := sql.bind(_2,"myschema","mytable","field1",2); _46 := algebra.kdifference(_43,_7); _47 := algebra.semijoin(_7,_43); _48 := algebra.kunion(_46,_47); _11:bat[:oid,:oid] := sql.bind_dbat(_2,"myschema","mytable",1); _12 := bat.reverse(_11); _49 := algebra.kdifference(_48,_12); _9:bat[:oid,:str] := sql.bind(_2,"myschema","mytable","field1",1); _50 := algebra.kdifference(_9,_12); _13 := mat.pack(_49,_50); _14 := algebra.slice(_13,0:wrd,9:wrd); _17:bat[:oid,:int] := sql.bind(_2,"myschema","mytable","field2",1); _55 := algebra.kdifference(_17,_12); _57 := algebra.semijoin(_55,_14); _45:bat[:oid,:int] := sql.bind(_2,"myschema","mytable","field2",0,0@0,nil:oid); _16:bat[:oid,:int] := sql.bind(_2,"myschema","mytable","field2",2); _51 := algebra.kdifference(_45,_16); _52 := algebra.semijoin(_16,_45); _53 := algebra.kunion(_51,_52); _54 := algebra.kdifference(_53,_12); _56 := algebra.semijoin(_54,_14); _19 := sql.resultSet(2,1,_14); sql.rsColumn(_19,"myschema.mytable","field1","varchar",50,0,_14); _18 := mat.pack(_56,_57); sql.rsColumn(_19,"myschema.mytable","field2","int",32,0,_18); _29 := io.stdout(); sql.exportResult(_29,_19); end s1_1;
Stefan Manegold wrote:
On Thu, Sep 02, 2010 at 02:17:28PM +0200, Sjoerd Mullender wrote:
On 2010-09-01 23:27, Alex Bo. wrote:
Hi,
(June2010 SP1 built on Windows)
I loaded a table with a single string column successfully with
approximately
500M string values. Running 'Select * from table limit 10' gives me this error:
'strPut: string heaps gets larger than 15GB.'
What does this mean?
The proximate cause is that string heaps are only allowed slightly less than 16 GB. This happens when you use 32-bit OIDs on a 64 bit system (the default for 64-bit Windows). The way strings are stored is by using a fixed-size index into the area where the actual strings are stored (the string heap). These indexes in the above-mentioned combination are 32 bits, shifted left by three positions. This gives a maximum index value of 0x3fffffff8 or 17179869176 (which is 15.999999992549419 GiB, truncated to 15 GiB).
The ultimate cause I don't know yet. Obviously you were able to load the data, so it is not clear (yet) why you need a string heap that is this large when doing that select.
Alex,
the results of running your query prefixed with "PLAN " and "EXPLAIN " (one at a time) on your data and system might help us to investigate the cause of this problem.
Stefan
-- Sjoerd Mullender
This SF.net Dev2Dev email is sponsored by:
Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd
MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |
This SF.net Dev2Dev email is sponsored by:
Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
participants (3)
-
Alex Bo. -
Sjoerd Mullender -
Stefan Manegold