Hi Ed,
sorry for the delayed reply ...
The pattern of usage:
Read a row of data. Determine the "partition" to stick the data in (generates a number). In one BAT, track all such partitions. If this partition does not exist, then create the BAT, rename it, set the base, etc. Store the name in the tracking BAT. Insert the data into the data partition BAT. Every 10,000 rows, do a commit.
From your description, after each row, I do not have a variable that is keeping the BAT open (I always use "bat("name").XXX", or keep a ref variable in a block. So, as far as I can tell, I should not have 7500 BATs open at once.
Am I misunderstanding?
No, as far as I can see from your discription, everything looks fine. Unfortunately, I cannot yet see, where your memory problems come from.
Would it be possbile to provide us with your MIL code --- or at least the curcial parts? Maybe, you could even send us a kind of trace or log of actions that lead to the crash?
Or is there some underlying component/module keeping the BATs open?
No, none that I'm aware of.
Or is this just the amount of memory MonetDB will use?
Well, could be, but to be sure, I'd need more details of you usage of MonetDB (see above).
I do not have a BAT of BATs (as I was sure that this would cause problems, and from your description, will indeed).
Good. BATs of BATs are indeed considered "problematic" (if not "evil"); better use BATs of bat-names (I guess, that's what you do with your one BAT that tracks all the partitions?).
Stefan