Greetings!

In the architectural overview, I find the text: "Most memory resource problems in MonetDB are easily avoidable by minimizing your open BAT references (as discussed above) and making your access patterns cache friendly (see the radix module)."

Is there anyplace that actually describes when a BAT is considered open?

It's also mentioned in the architectural overview.

Basically, a BAT is considered opne, if the is any active reference, e.g., a MIL variable or from a nested "BAT of BATs", to that BAT.

Is it a scope thing (BATs are closed as soon as they are out of scope)?

Kind of: MIL variables are destroyed at the end of the scope (i.e., {} or {||} MIL block) they were declared in; and thus also the BATs they refer to are closed (provided it's the last reference to that BAT).

Is a BAT open when I append/update? Does it close when I perform a commit?

BATs are active/open as soon / as long as you touch/access/use them, i.e., as long as there is a reference to a BAT (see above). commit() does not remove any reference.

If not, is there a way to specify "I am done, clean it out"?

there are two ways:

on the one hand, you can unload a BAT using the "unload(str)" command (e.g., "unload(bbpname(b));", see `help("unload");` for details.

on the other hand, you can remove references to BATs by setting the respective variable(s) to nil ("b:=nil;"), or by limiting the scope of variables using "{}" MIL blocks.

I am noticing a large amount of open memory (approximately 500MB out of a 1GB system) being used by MonetDB. I have hand partitioned my BATs into groups using a naming convention, but I am wondering if during inserts I am somehow keeping open references to the BATs which may cause issues.

I read from you previus posting over the crashing Mserver, that you have about 7500 (possibly large) BATs open, because you just created them and inserted data into them. This can indeed be a problem, if the total size of all these BATs gets close to the address space limit (2-4GB on your 32-bit system). Do you really need to have all these BATs active at the same time? Can you use them one by one, or at least in smaller groups?

Could you give us a more detailed description of your application/approach/MIL-program? This might help us, to give you some more advice how to use MonetDB more efficiently.

Is there a way to find out where memory is being allocated? Which BATs are open?

Try any of these: print(memory()); mem_printmap(); print(vm_usage()); dir(); ls();

Also, I am setting some of the larger BATs to STORE_MMAP. I notice in the documentation that if I compress the heaps, then these must be set to STORE_MEM. Is there a reason that when uncompressing the heap, that the uncompressed version cannot be STORE_MMAP to avoid memory problems?

Sorry, I cannot answer thisone right now. I'll ask someone who knows...

Regards,

Stefan

Regards! Ed