Dear Colin,
Crash/recovery problems are really hard to analyse. Any hint on what is going on during recovery is highly appreciated. E.g. a stack trace and watching the instructions being executed indicate make all the differences. Also the result of 'top' may indicate the system behavior. I wouldn't be surprised if the cpu load has dropped to <1%, which indicate severe disk page management problems in the OS.
Perhaps Niels should at least add some progress warnings when a large log file is recovered. It is a known fact that if you built a table from scratch, without any clue on the ultimate size, the system may become dreadfully slow. Roughly speaking, if a BAT is full, a new one is allocated in (virtual) memory using 1.2* oldsize; This quickly creates a disk bound setting with all the consequences (IO thrashing). In a SQL setting this is even worse, because the event is triggered on all columns at the same time.
Nevertheless, it has our full attention. In particular, since we are currently scaling a 1Gb database to a 100Gb, directly followed by a jump to 2.7Tb.
The stability in this area between M4/M5 is hard to tell, because we don't know at what level the problem occurs. For M5 we have, however, a program called the stethoscope (in the head branch), which can be attached to any running mserver to inspect what's happening. (We would also like to attach a MAL debugger the same way)
RAM usage in M5 is better then M4, due to a different garbage collection scheme. A TPCH batch run on SF-5 (ca 5 Gb, all queries executed one after the other, system had only 2GB of memory) showed improvements between 2x up to 20x better response time on individual queries over M4.
Colin Foss wrote:
For the second time in a few weeks I've got a corrupted database. Each time the corruption occurs when trying to insert (via COPY into) data into a large table and the row count exceeds 100-110M. Each time the database terminates and a restart never allows another connection to the SQL interface.
I've checked and no column is coming close to the 2G limit as well. The largest column is approximately 900M.
My question is this:
Would a switch to version five improve my results? Is version five more stable w.r.t. large (100M+) table handling?
This stability issue is rather disappointing because MonetDB has done so well in all of my other testing. Query performance compared to other (commercial/non-commercial) products was outstanding for result sets of 5 - 750000 rows.
Also, I wish there were better controls over maximum RAM usage.
Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=D... _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users