[MonetDB-users] Counting open BATs

Stefan Manegold Stefan.Manegold at cwi.nl
Thu May 5 13:16:40 CEST 2005


Hi Ed,

sorry for the delayed reply ...

> The pattern of usage:
> 
> Read a row of data. Determine the "partition" to stick the data in
> (generates a number). In one BAT, track all such partitions. If this
> partition does not exist, then create the BAT, rename it, set the base,
> etc. Store the name in the tracking BAT. Insert the data into the data
> partition BAT.  Every 10,000 rows, do a commit.
> 
> From your description, after each row, I do not have a variable that is
> keeping the BAT open (I always use "bat("name").XXX", or keep a ref
> variable in a block. So, as far as I can tell, I should not have 7500 BATs
> open at once.
> 
> Am I misunderstanding?

No, as far as I can see from your discription, everything looks fine.
Unfortunately, I cannot yet see, where your memory problems come from. 

Would it be possbile to provide us with your MIL code --- or at least the
curcial parts?
Maybe, you could even send us a kind of trace or log of actions that lead to
the crash?

> Or is there some underlying component/module keeping the BATs open? 

No, none that I'm aware of.

> Or is this just the amount of memory MonetDB will use? 

Well, could be, but to be sure, I'd need more details of you usage of
MonetDB (see above).

> I do not have a BAT of BATs (as I was sure that this would cause
> problems, and from your description, will indeed).

Good. BATs of BATs are indeed considered "problematic" (if not "evil");
better use BATs of bat-names (I guess, that's what you do with your one BAT
that tracks all the partitions?).

Stefan

-- 
| Dr. Stefan Manegold | mailto:Stefan.Manegold at cwi.nl |
| CWI,  P.O.Box 94079 | http://www.cwi.nl/~manegold/  |
| 1090 GB Amsterdam   | Tel.: +31 (20) 592-4212       |
| The Netherlands     | Fax : +31 (20) 592-4312       |




More information about the users-list mailing list