April 2004 - developers-list

[Monetdb-developers] Build date in Mserver startup info
by Arjan Scherpenisse 03 May '04

03 May '04

Hi, Maybe it would be a good idea to include some information about the build date on startup of Mserver, just like other programs do that (Linux uname -a, Python, etc). That way you quickly can see if you have the latest version... druif:~/> Mserver # Monet Database Server V4.3.17 (Apr 27 2004, 14:27:43) <----- # Copyright (c) 1993-2004, CWI. All rights reserved. # Compiled for i686-pc-linux-gnu/32bit; dynamically linked. # Visit http://monetdb.cwi.nl for further information. monet> regards -- Arjan Scherpenisse Centrum voor Wiskunde en Informatica, Amsterdam, the Netherlands

4 11

[Monetdb-developers] New feature policy
by Arjan Scherpenisse 02 May '04

02 May '04

Hello, I am writing a MIL module which exports some functions for computation of different forms of string similarities. I would love to add this module to monet CVS (so I won't have to patch any Makefiles and copy source files whenever I do a new checkout), but I'm hesitating because I am not sure what the policy is for adding functions, library bindings, etc to Monet. Should it go in a separate CVS repository, or should it not be included at all? Could someone elaborate on this? regards -- Arjan Scherpenisse Centrum voor Wiskunde en Informatica, Amsterdam, the Netherlands

2 1

[Monetdb-developers] New feature policy
by Arjan Scherpenisse 29 Apr '04

29 Apr '04

Hello, I am writing a MIL module which exports some functions for computation of different forms of string similarities. I would love to add this module to monet CVS (so I won't have to patch any Makefiles and copy source files whenever I do a new checkout), but I'm hesitating because I am not sure what the policy is for adding functions, library bindings, etc to Monet. Should it go in a separate CVS repository, or should it not be included at all? Could someone elaborate on this? regards -- Arjan Scherpenisse Centrum voor Wiskunde en Informatica, Amsterdam, the Netherlands

1 0

[Monetdb-developers] mindmap
by Martin Kersten 28 Apr '04

28 Apr '04

Updated version. (Should find a place in the MonetDB Repository for it) Inclusion of Peter's comments. - Mserver becomes a watchdog for local Mserver Instances (other name needed) Mguardian itself is a process to manage a community of Mservers, who have delegated some control. Please add comments. <map version="0.7.1"> <node TEXT="Mguardian"> <node TEXT="Introduction" POSITION="right"> <node TEXT="Automated crash recovery" FOLDED="true"> <node TEXT="Single Mserver failure survivals"/> <node TEXT="Fail-over in case of a hardware crash"/> </node> <node TEXT="Resource management" FOLDED="true"> <node TEXT="Dbfarm management" FOLDED="true"> <node TEXT="Creation of necessary directories"/> <node TEXT="Assurance of sufficient space"/> </node> <node TEXT="Load distribution on multiple servers" FOLDED="true"> <node TEXT="Forwarding query clients to slave"/> </node> <node TEXT="Automatic backup/recovery management" FOLDED="true"> <node TEXT="Day/weekly schedules"/> <node TEXT="Highwatermark scheduling"/> </node> <node TEXT="Disk volume management" FOLDED="true"> <node TEXT="Multi-hierarchical chkpoint stores"/> <node TEXT="log compression"/> </node> </node> <node TEXT="Monitoring" FOLDED="true"> <node TEXT="Lifeness of all Mserver instances in the convoy"/> <node TEXT="Resource usage" FOLDED="true"> <node TEXT="MonetDB client load"/> <node TEXT="Host resource load"/> </node> </node> <node TEXT="Authorization" FOLDED="true"> <node TEXT="User administration"/> <node TEXT="Granting access to services"/> </node> </node> <node TEXT="Design goals" POSITION="right"> <node TEXT="appearance"> <node TEXT="Transparent to users"> <node TEXT="Becoming a member of a guardian group 
is an option in the Mserver configuration file."/> </node> <node TEXT="Transparent to location"> <node TEXT="Cluster machines are supported"/> <node TEXT="loosely coupled servers through HTTP"/> </node> <node TEXT="Transparent to hw/sw"> <node TEXT="Platform independent code"/> <node TEXT="Python? Java?"/> </node> </node> <node TEXT="Convoys" FOLDED="true"> <node TEXT="The guardian manages a group of database services"/> <node TEXT="The guardian process acts as a servant to those requiring it"/> <node TEXT="It is a free association, i.e. a Mserver can at any time
join/leave the group. Depending on the way this is
realised, Mguardian may initiate an action"/> </node> <node TEXT="State management"> <node TEXT="Driven by XML configuration file and state file"/> <node TEXT="Ascii-based state management at transaction level"/> <node TEXT="Distributed transaction management"/> </node> <node TEXT="Multiple points of failure infrastructure" FOLDED="true"> <node TEXT="Multiple servers in a cluster"/> <node TEXT="Mguardian lifes outside the area of Mservers?"/> </node> <node TEXT="Web-based status monitoring" FOLDED="true"> <node TEXT="At the level of a single server"/> <node TEXT="At the level of managin multiple Mservers"/> </node> <node TEXT="Active warning system" FOLDED="true"> <node TEXT="Email warning to DBA"/> </node> </node> <node TEXT="Architecture" POSITION="right"> <node TEXT="Configuration files" FOLDED="true"> <node TEXT="XML-based structure of Mserver Farms"/> <node TEXT="Database specific management conditions"/> <node TEXT="Disaster recovery rules"/> </node> <node TEXT="Implementation"> <node TEXT="Watchdog function" FOLDED="true"> <node TEXT="A lightweight independent process that keeps that
keeps a specific Mserver instance alive"> <node TEXT="It 'behaves' as the main thread of Mserver"/> <node TEXT="It should maintain information for restart" FOLDED="true"> <node TEXT="how to know this ?"/> <node TEXT="By being part of Mserver thread group"/> </node> </node> <node TEXT="Recover policy is database specific"> <node TEXT="Simply restart Mserver using known script"/> <node TEXT="Rejoin the Mguardian convoy"/> <node TEXT="Set-back the latest chkpoint instance"/> </node> <node TEXT="Monitoring" FOLDED="true"> <node TEXT="Collect session information"/> <node TEXT="Obtain webbased state of a single Mserver"/> </node> <node TEXT="Connection server"> <node TEXT="Should take over Internet thread listener"/> <node TEXT="Authenticates valid connections"/> <node TEXT="Forward request to convoy if needed"/> </node> </node> <node TEXT="Management function" FOLDED="true"> <node TEXT="Independent MonetDB application" FOLDED="true"> <node TEXT="Runs on secure system and keeps overview"/> <node TEXT="Mknife access possible"/> </node> <node TEXT="Manages allocation of connection requests"/> <node TEXT="Revives the watchdogs or call for DBA help"/> <node TEXT="Monitors high-water marks and calls the DBA"/> </node> </node> <node TEXT="Mguardian GUI" FOLDED="true"> <node TEXT="A system wide control centre "> <node TEXT="Gui-based"/> <node TEXT="Controls instances in cluster"/> <node TEXT="Can collect information from autonomous nodes"/> </node> <node TEXT="It receives reports from the watchdogs" FOLDED="true"> <node TEXT="Incremental load levels"/> <node TEXT="Database farm size"/> <node TEXT="log sizes"/> <node TEXT="Number of concurrent clients"/> <node TEXT="Machine load level"/> </node> <node TEXT="supports remotely initiated actions" FOLDED="true"> <node TEXT="Watchdog is the proxy for local system actions"/> </node> </node> </node> <node TEXT="Implementation strategy" POSITION="right"> <node TEXT="Phase 1: Mserver becomes a watchdog"> <node TEXT="Performs rudimentary guardian actions 
before admitting a Mserver Instance"/> <node TEXT="Performs auto reboot of Mserver Instance"/> <node TEXT="Tell users about the language/instance port to use"/> <node TEXT="Keeps log of user sessions"/> </node> <node TEXT="Phase 2: Mserver monitoring"> <node TEXT="Mguardian database design"/> <node TEXT="Webservice to collect data" FOLDED="true"> <node TEXT="HW load"/> <node TEXT="Free log space"/> <node TEXT="Database space"/> </node> <node TEXT="Simple admin control to change properties"/> <node TEXT="Prototyping as Mknife scripts"/> </node> <node TEXT="Phase 3: Mguardian in control"> <node TEXT="Primary portal for client connectivity"/> <node TEXT="start/stop Mservers"/> </node> </node> </node> </map>

2 1

[Monetdb-developers] Python DB API
by Arjan Scherpenisse 26 Apr '04

26 Apr '04

I have written some wrapper classes around Sjoerd's SWIG-generated Mapi and MapiLib modules, which serve as a Python DB-API 2.0 compliant database interface to Monet. If youre interested check out http://homepages.cwi.nl/~scherpen/Monet-python-DBAPI.tar.gz Maybe someone even should put this into Monet CVS cheers, -- Arjan Scherpenisse Centrum voor Wiskunde en Informatica, Amsterdam, the Netherlands

1 0

[Monetdb-developers] monetdb as embedded database ??
by Roberto Saccon 26 Apr '04

26 Apr '04

Hi all I would like to know whether it makes sense to use monetdb as database embedded into uerapplication, comparable to Berkeley DB (non-relational) or sqlite (relational). What is the footprint size ? What I have seen so far, the C-Apis only allow access to the DB by sockets. Or am I wrong ?? Does there iexisst a project where monetdb is embedded ? regards Roberto

1 0

[Monetdb-developers] micro CVS HowTo
by Stefan.Manegold＠cwi.nl 25 Apr '04

25 Apr '04

Dear fellow developers, for your convenience, here's a micro CVS HowTo: - initial checkout the head of the main/development branch: cvs -d<user>@cvs.sf.net:/cvsroot/monetdb co -P MonetDB - initial checkout the head of a release branch: cvs -d<user>@cvs.sf.net:/cvsroot/monetdb co -P -rMonetDB_4-3-16 MonetDB - tranforming a check-out main/development branch into a checked-out release branch: cvs up -dP -rMonetDB_4-3-16 - tranforming a check-out release branch into a checked-out main/development branch: cvs up -dP -A - updating a check-out main/development or release branch: cvs up -dP - checking, whether your current check-out it a main/development or release branch: cat CVS/Tag main/development branch: CVS/Tag does not exist release branch: CVS/Tag contains branch name/tag - checking you own changes (preferably prior to checkin): cvs diff Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] MonetDB Testing & Status-Alert
by Stefan.Manegold＠cwi.nl 25 Apr '04

25 Apr '04

Dear fellow developers, let me first apologize for the fact that I haven't been able to maintain the nightly testing as usual while I was abroad the week before last week (Sa, April 10 - Su, April 18). During this time, problems with SourceForge (that are beyond our control) as well as problem with the Windows testing (that are somehow under our control, see below), caused the automatic testing to fail, and hence could not provide you with the usual feedback about your checkins. Furthermore, I also have to apologize, that I did not manage to get the testing working again, during last week, while being back at work. Well, at least, the SF problems seems to be gone, again. The windows problem is still there: recent changes in the release branch(!) of MonetDB seem to cause "Application Errors" (aka SegFaults) on Windows, and my non-interactive test-environment is unfortunately not able to press the "Ok" button on the respective pop-up window... Well, thanks to some hint from our new friends at SAP, I (hopefully) just managed to avoid these pop-up windows on our Windows test machine. Hence, testing should hang there anymore, even in case of "Application Errors". These problem solved, I had a short(!) look at today's Stable testweb, at what I saw did *scare* me a bit: ! The current "stable" version of MonetDB (4.3.16) seems to be pretty broken ! While many difference are "only" due to "harmless" changes in the output (e.g., caused by changes in the behaviour/output of commands such as "atoms()"), there are some changes that indicate that "join" is broken ! Well, I guess, I'll have to spend tomorrow or Monday on looking at the details and sorting out, what did go wrong with your checkins to the (no longer) "stable" release branch... Admittedly, I have to blame myself, again: Not only did I not keep the automatic testing properly running during the last two week, but I also forgot to give any guidelines, how to handle checkins to the "stable" branch after the release had been made... (No, I don't metion anymore, that Mtest might be run by hand on your favorite development platform, at least before checking-in "bug-fixes" to the "stable" branch...) To see it positively, this observation confirms that we do not "waste" any time or money by having one person spending a major part of his/her (official) working hours on "quality control" --- I just forgot to find/name a stand-in while being abroad myself...). Good night, Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 2

[Monetdb-developers] Mapi blocked mode multi query consideration
by Fabian 22 Apr '04

22 Apr '04

Mapi has two modes, which I refer to as line mode and block mode. The main plus for the client when using the block mode variant is that multi-line queries can be sent directly to the server without any parsing. The line mode variant, on the other hand, separates lines by newline characters, so multiline queries are not possible; one has to replace all newlines by spaces when not inside a string, otherwise by \n. (this requires some parsing of the query before sending) Now the problem I'm dealing with is as follows: If I send a multiline query using blockmode to the server, and that query is syntactically incorrect (according to the server), the server returns that same multiline query. In the result mode, however, newlines are treated as row delimiters. The effect of this, is that only the first row of the query sent is visible, which most of the time does not contain the error. For example, suppose the following query is sent: SELECT * FROM "foobar"; where the table foobar does not exist in the catalog, then the error message will be: !ERROR parse error at token (257) in statement: select * from "foobar" which due to row-style parsing will only result in the error message with query 'select *'. The question here is actually, who is doing what wrong? Any thoughts on this?

2 2

Re: [Monetdb-developers] Python + SQL
by Arjan Scherpenisse 22 Apr '04

22 Apr '04

Martin, I have fixed the Mapi.py thanks to Niels' tips. The result strings still need parsing ofcourse. I could write my own parsing or write wrappers for the existing Mapi.h API. I am willing to implement a Python database wrapper according to the Python Database API specs (http://www.python.org/peps/pep-0249.html). Maybe we can talk about this tomorrow with Peter, and how something like this can fit into my graduation work ;) regards, Arjan Martin Kersten wrote: > Arjan, > > It supprises me, because the Mapi protocol does not constrain the > language. I noticed, though, that MapiClient.py is currently hardwired > for mil indeed. It seems that Sjoerd (the Python expert) can help > you (and make MapiClient.py a little better?) > > The main question to decide upon is the functionality required. > Using ODBC would allow your application also to run on other DBMSs > From the perspective of re-use that is a nice property. Ofcourse, > the application builder has to spent more energy to prepare for this. > Using the ODBC layer of Sjoerd would be a good exercise to challenge > its current functionality. Help is close at hand to make it bearable. > > Perhaps an issue to discuss with Peter (and me). > > regards, Martin > > Arjan Scherpenisse wrote: > >> Hello, >> >> can someone point me to a simple way of using the SQL frontend from >> within Python? The default interface (Mapi.py) does only allow the MIL >> language, as far as I can see. >> >> In the past I did it by using the ODBC module, but that was very >> tiresome. Is there a better way? >> >> cheers, Arjan >> >> >> ------------------------------------------------------- >> This SF.Net email is sponsored by: IBM Linux Tutorials >> Free Linux tutorial presented by Daniel Robbins, President and CEO of >> GenToo technologies. Learn everything from fundamentals to system >> administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click >> _______________________________________________ >> Monetdb-developers mailing list >> Monetdb-developers(a)lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/monetdb-developers > > > -- Arjan Scherpenisse Centrum voor Wiskunde en Informatica, Amsterdam, the Netherlands

2 2