Hi Peter,

I have two small comments to your changes:

1. Is there a reason for including monet_utils.h here? -- In my eyes it introduces an unnecessary dependency to MonetDB (as we want the SQL code generation to compile even without MonetDB).

Index: milprint_summer.c =================================================================== RCS file: /cvsroot/monetdb/pathfinder/compiler/mil/milprint_summer.c,v retrieving revision 1.404 retrieving revision 1.405 diff -u -d -r1.404 -r1.405 --- milprint_summer.c 5 Sep 2007 17:06:54 -0000 1.404 +++ milprint_summer.c 11 Sep 2007 22:15:29 -0000 1.405 @@ -56,6 +56,7 @@ #define USE_DEPRECATED_ACCESS_TO_TYPE_SYSTEM 1

#include "pathfinder.h" +#include "monet_utils.h"

#include <stdio.h> #include <assert.h>

2. I think you should use "#pf" (PFns_pf) as prefix for variables names instead of "pf" (PFns_lib). Namespace PFns_lib can be used in the query which allows a user to create pf:heur* variables that might conflict lateron (whereas PFns_pf can only be created inside pathfinder).

--- NEW FILE: heuristic.c --- ... static PFpnode_t* var_(PFloc_t loc, int varnum) { PFpnode_t *r = p_leaf(p_varref, loc); if (r) { r->sem.qname_raw.prefix = "pf"; r->sem.qname_raw.loc = (char*) PFmalloc(8); if (r->sem.qname_raw.loc) snprintf(r->sem.qname_raw.loc, 8, "heur%03d", varnum); } return r; }

On 09/12/2007 12:15 AM, Peter Boncz wrote with possible deletions:

Update of /cvsroot/monetdb/pathfinder/compiler/semantics In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv24884/compiler/semantics

Modified Files: Makefile.ag xquery_fo.c Added Files: heuristic.c Log Message: The infamous push-selections-down heuristic rewrite ===================================================

the good: equi comparisons on attributes, texts and elements in MXQ can now be done in sublinear time thanks to indices. This involves the "reversing"of multi-step XPath paths if required.

the bad: it is a heuristic so no performance improvement assured, though much of the performance pitfalls are mitigated by a loop-lifted run-time decision whether to use the index plan or the original plan.

the ugly: well, since I can't (or couldn't) stomach XQuery core plan, it is all implemeted on the absssyn level.

there are quite a few comments/explanations in compiler/semantics/heuristic.c

runtime/pathfinder.mx

• major changes to the indexing scheme, supporting the pf:text/pf:attribute functions
• added run-time behavior to vx_lookup: for those iterations where index lookup is not beneficial, it returns a bogus node (1@0, recognizable as a docnode) so the original (non index-pushdown) plan is used for those iterations

runtime/pf_support.mx

• new xquery hash function, with less collisions and the ability to preserve equality semantics for any XQuery type (currently supported..).
• added a new dbl(byte1,...byte8) constructor that allows a IEEE double to be constructed from its 8 binary bytes (per byte to avoid little/big endian issues) without any precision loss.
• optimization for the ll_htordered_thetajoin in case we have constant join columns

compiler/compile.c compiler/include/Makefile.ag compiler/semantics/Makefile.ag

• add heuristic.[ch]
• added heuristic index pushdown opt (currently only in mps, but alg/M4 should follow)

compiler/semantics/xquery_fo.c compiler/mil/milprint_summer.c

• rewrote pf:atttribute and pf:text to give approximate answers
• new pf:supernode function
• pass parsed decimals/doubles using the new byte-wise MIL dbl() constructor (to avoid precision loss in the pf -> MonetDB transition)
• bugfix in empty ipik opt for the for-rule

compiler/debug/abssynprint.c

• PFabssyn_stdout for debugging sanity

compiler/debug/prettyp.c

• margin=6 for better debug output readability