Re: [Monetdb-developers] Monetdb-developers Digest, Vol 22, Issue 4
Hi,
I read the digests of the monetdb mailing list, which in this case took more than a month to send me this discussion. That's pretty useless.. I will switch to immediate mails.
Anyway, Jennie is currently working on getting the repeatable reads transactions working. This means that by passing a request-ID you will be guaranteed that all queries and updates execute on the same database image, i.e. a transaction. In case of update requests in the same transaction, these will be executed atomically using a 2PC protocol.
While this functionality is designed to be triggerered from XQuery queries, you client API could also generate the appropriate messages.
It may not answer you needs regarding performance (you would send a read-request, update-request, prepare-request and commit-request), but it would answer your functional requirements AFAIS.
Peter
Message: 1 Date: Wed, 19 Mar 2008 12:30:20 +0100 From: "Keulen, M. van (Maurice)" m.vankeulen@utwente.nl Subject: Re: [Monetdb-developers] [XQ] Mixing update/docmgmt/query To: monetdb-developers@lists.sourceforge.net Message-ID: 47E0F94C.4000709@utwente.nl Content-Type: text/plain; charset=windows-1252; format=flowed
Ying Zhang wrote:
Jennie: my SOAP-messages do not come from MonetDB/XQuery, but from outside (i.e., some scripting language).
Then, an order can only be guaranteed if you, in your
scripts, send a
message, wait for result, and send the next message. In all other cases, unfortunately, no guarantee, as an xrpc server immediately start handling a request when it's received.
Jennie
I thought so already. This is why I don't like to have to send multiple requests to do one thing (transaction). I'd rather be able to send one request to both do, say, an update and return a result, or add a document and return a result. The system so far doesn't allow me to do this, because I cannot define one query (or function in a module) that does both because of typing restrictions, and I also cannot send two separate queries in one request because of restrictions in the XRPC protocol.
Any thoughts about achieving what I want?
Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
*********************************
Hi Peter,
Does it also mean that if I send a stream of messages to one server that all carry the same request-ID without waiting for a result of a message before sending out the next, that all are handled in the correct order and each is handled after completion of the previous one? In that case, it meets all my requirements. In fact, it gives much more than I ask for :-) I definitely like to follow any developments for this one!
Maurice.
Peter Boncz wrote:
Hi,
I read the digests of the monetdb mailing list, which in this case took more than a month to send me this discussion. That's pretty useless.. I will switch to immediate mails.
Anyway, Jennie is currently working on getting the repeatable reads transactions working. This means that by passing a request-ID you will be guaranteed that all queries and updates execute on the same database image, i.e. a transaction. In case of update requests in the same transaction, these will be executed atomically using a 2PC protocol.
While this functionality is designed to be triggerered from XQuery queries, you client API could also generate the appropriate messages.
It may not answer you needs regarding performance (you would send a read-request, update-request, prepare-request and commit-request), but it would answer your functional requirements AFAIS.
Peter
Message: 1 Date: Wed, 19 Mar 2008 12:30:20 +0100 From: "Keulen, M. van (Maurice)" m.vankeulen@utwente.nl Subject: Re: [Monetdb-developers] [XQ] Mixing update/docmgmt/query To: monetdb-developers@lists.sourceforge.net Message-ID: 47E0F94C.4000709@utwente.nl Content-Type: text/plain; charset=windows-1252; format=flowed
Ying Zhang wrote:
Jennie: my SOAP-messages do not come from MonetDB/XQuery, but from outside (i.e., some scripting language).
Then, an order can only be guaranteed if you, in your
scripts, send a
message, wait for result, and send the next message. In all other cases, unfortunately, no guarantee, as an xrpc server immediately start handling a request when it's received.
Jennie
I thought so already. This is why I don't like to have to send multiple requests to do one thing (transaction). I'd rather be able to send one request to both do, say, an update and return a result, or add a document and return a result. The system so far doesn't allow me to do this, because I cannot define one query (or function in a module) that does both because of typing restrictions, and I also cannot send two separate queries in one request because of restrictions in the XRPC protocol.
Any thoughts about achieving what I want?
Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
Hi Maurice,
Stupid me. I only described how the current XRPC implementation is, but forgot to mention the ongoing work.
On Mar 28, 2008, at 09:35 , Keulen, M. van (Maurice) wrote:
Hi Peter,
Does it also mean that if I send a stream of messages to one server that all carry the same request-ID without waiting for a result of a message before sending out the next, that all are handled in the correct order and each is handled after completion of the previous one? In that case, it meets all my requirements. In fact, it gives much more than I ask for :-) I definitely like to follow any developments for this one!
This can be easily supported by adding a sequence number to each request message, I think.
But, if you have an updating request, and then a read-only request, do you then want to make the updates made by the first request visible to the second request?
Jennie
Maurice.
Peter Boncz wrote:
Hi,
I read the digests of the monetdb mailing list, which in this case took more than a month to send me this discussion. That's pretty useless.. I will switch to immediate mails.
Anyway, Jennie is currently working on getting the repeatable reads transactions working. This means that by passing a request-ID you will be guaranteed that all queries and updates execute on the same database image, i.e. a transaction. In case of update requests in the same transaction, these will be executed atomically using a 2PC protocol. While this functionality is designed to be triggerered from XQuery queries, you client API could also generate the appropriate messages.
It may not answer you needs regarding performance (you would send a read-request, update-request, prepare-request and commit-request), but it would answer your functional requirements AFAIS.
Peter
Message: 1 Date: Wed, 19 Mar 2008 12:30:20 +0100 From: "Keulen, M. van (Maurice)" m.vankeulen@utwente.nl Subject: Re: [Monetdb-developers] [XQ] Mixing update/docmgmt/query To: monetdb-developers@lists.sourceforge.net Message-ID: 47E0F94C.4000709@utwente.nl Content-Type: text/plain; charset=windows-1252; format=flowed
Ying Zhang wrote:
Jennie: my SOAP-messages do not come from MonetDB/XQuery, but from outside (i.e., some scripting language).
Then, an order can only be guaranteed if you, in your
scripts, send a
message, wait for result, and send the next message. In all other cases, unfortunately, no guarantee, as an xrpc server immediately start handling a request when it's received.
Jennie
I thought so already. This is why I don't like to have to send multiple requests to do one thing (transaction). I'd rather be able to send one request to both do, say, an update and return a result, or add a document and return a result. The system so far doesn't allow me to do this, because I cannot define one query (or function in a module) that does both because of typing restrictions, and I also cannot send two separate queries in one request because of restrictions in the XRPC protocol.
Any thoughts about achieving what I want?
Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
Ying Zhang wrote:
Hi Maurice,
Stupid me. I only described how the current XRPC implementation is, but forgot to mention the ongoing work.
On Mar 28, 2008, at 09:35 , Keulen, M. van (Maurice) wrote:
Hi Peter,
Does it also mean that if I send a stream of messages to one server that all carry the same request-ID without waiting for a result of a message before sending out the next, that all are handled in the correct order and each is handled after completion of the previous one? In that case, it meets all my requirements. In fact, it gives much more than I ask for :-) I definitely like to follow any developments for this one!
This can be easily supported by adding a sequence number to each request message, I think.
But, if you have an updating request, and then a read-only request, do you then want to make the updates made by the first request visible to the second request?
Comparing it to the behaviour of a relational DBMS doing 'begin transaction ; update ; select ; commit' then, yes, I expect the effect of the update to be visible for the select, but not visible to select's done in other concurrently running transactions (the I of isolation in ACID).
Maurice.
Hi Maurice,
That is not possible.. without adding extensions such as proposed in XQueryP or XQuery-Bang.
Why is it that running the update followed by a query is not acceptable?
Another idea is to use fn:put() in the update, and then just a HTTP GET to get the XML result out. That is still a single query.
Peter
-----Original Message----- From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] Sent: Friday, March 28, 2008 11:29 AM To: Ying Zhang Cc: P.Boncz@cwi.nl; monetdb-developers@lists.sourceforge.net Subject: Re: Monetdb-developers Digest, Vol 22, Issue 4
Ying Zhang wrote:
Hi Maurice,
Stupid me. I only described how the current XRPC
implementation is,
but forgot to mention the ongoing work.
On Mar 28, 2008, at 09:35 , Keulen, M. van (Maurice) wrote:
Hi Peter,
Does it also mean that if I send a stream of messages to
one server
that all carry the same request-ID without waiting for a
result of a
message before sending out the next, that all are handled in the correct order and each is handled after completion of the previous one? In that case, it meets all my requirements. In fact, it gives much more than I ask for :-) I definitely like to follow any developments for this one!
This can be easily supported by adding a sequence number to each request message, I think.
But, if you have an updating request, and then a read-only
request, do
you then want to make the updates made by the first request
visible to
the second request?
Comparing it to the behaviour of a relational DBMS doing 'begin transaction ; update ; select ; commit' then, yes, I expect the effect of the update to be visible for the select, but not visible to select's done in other concurrently running transactions (the I of isolation in ACID).
Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
On Mar 28, 2008, at 11:53 , Peter Boncz wrote:
Hi Maurice,
That is not possible.. without adding extensions such as proposed in XQueryP or XQuery-Bang.
Why is it that running the update followed by a query is not acceptable?
Another idea is to use fn:put() in the update, and then just a HTTP GET to get the XML result out. That is still a single query.
Hi Peter,
Do we treat fn:put() as an updating function? If not, this won't work, as fn:put() will be executed before the update list is applied, I think.
I was thinking that what Maurice asks is very similar with what X-Hive does in a transaction. I'm wondering if it might be possible to support?
Jennie
Peter
-----Original Message----- From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] Sent: Friday, March 28, 2008 11:29 AM To: Ying Zhang Cc: P.Boncz@cwi.nl; monetdb-developers@lists.sourceforge.net Subject: Re: Monetdb-developers Digest, Vol 22, Issue 4
Ying Zhang wrote:
Hi Maurice,
Stupid me. I only described how the current XRPC
implementation is,
but forgot to mention the ongoing work.
On Mar 28, 2008, at 09:35 , Keulen, M. van (Maurice) wrote:
Hi Peter,
Does it also mean that if I send a stream of messages to
one server
that all carry the same request-ID without waiting for a
result of a
message before sending out the next, that all are handled in the correct order and each is handled after completion of the previous one? In that case, it meets all my requirements. In fact, it gives much more than I ask for :-) I definitely like to follow any developments for this one!
This can be easily supported by adding a sequence number to each request message, I think.
But, if you have an updating request, and then a read-only
request, do
you then want to make the updates made by the first request
visible to
the second request?
Comparing it to the behaviour of a relational DBMS doing 'begin transaction ; update ; select ; commit' then, yes, I expect the effect of the update to be visible for the select, but not visible to select's done in other concurrently running transactions (the I of isolation in ACID).
Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
Peter Boncz wrote:
Hi Maurice,
That is not possible.. without adding extensions such as proposed in XQueryP or XQuery-Bang.
Why is it that running the update followed by a query is not acceptable?
Running an update followed by a query is acceptable. My point is that the update and the query are both "requests" ... *asynchronous* requests if you use XRPC as communication mechanism. I don't mind sending the update and the query in separated requests, but it seems to me inefficient and cumbersome for a developer that he needs to program it in such a way that he sends the update, necessarily waits for it to be completed, then sends the query and waits for the result. The developer is not interested in the response to the update. Hence, it would be IMO better if this developer can simply send the update, and without waiting for it to be completed, send the query and wait for the result of the query. In this way, he assumes however that the query 'sees' the result of the update ... and I am not convinced that this is what he will get, because: it is not guaranteed that the update request is received by the server before the query request, and that the update request is finished before the query request starts. Like I said, I'm only mimicking an application sending "UPDATE;SELECT" to a relational database.
Another idea is to use fn:put() in the update, and then just a HTTP GET to get the XML result out. That is still a single query.
Why is this a single query? I need to send one XRPC message that calls a function in a module that does the fn:put(), and another request with HTTP GET to get the result out ... and the application has to wait for the response of the XRPC message, before doing the HTTP GET, right?
We are now discussing full-fledged transactions and what I would expect from MXQ if it would support full-fledged transactions. My original question was much simpler. I want the result of a query to be both stored and sent back to the application. This requires 2 requests that need to be executed serially: fn:put(query,"doc.xml") ; doc("doc.xml") Wouldn't it be very helpful, whatever restrictions or visibility constraints, if this simple thing could be done in one function, hence one XRPC-request? I don't see any technical objections to allow the XQuery expression below (i.e., remove the typing constraints) and say that the semantics is that it results in (1) an update tape applied afterwards, and (2) a query result (everything in the sequence that is not an update), and hence that updates done in a query are not visible to the query itself.
let $answer := query return (fn:put($answer,"doc"), $answer)
This would already make live much easier for a developer in many simple cases.
Maurice.
Hi Maurice,
- what you want *really* is XQueryP, but we do not have resources to give you that. The fact that we use the result sequence to pass on the update tape actually makes it difficult to give back a result. - if you send two asynchronous requests, there is no guarantee that one is executed after the other. - fn:put() is currently still an updating function, but this is erroneous. I will check in a modification that makes it a readonly one (soon).
Do I ready you correctly that this would already be enough? Notice that fn:put() does not add a document to the database. It just creates some XML file on the server.
A very ugly hack that comes to mind is to support a special URI in fn:put(), e.g. "xrpc://user-data" that works only in a XRPC request, and causes the node to be included in XRPC response message. Not as the result sequence, but in some optional user-data section.
A more elaborate hack would be to allow multiple function calls inside a single XRPC-exchange. We could define a number of modes of execution, most notably: - snapshot (in the same snapshot, sharing XML fragments). We could even allow a special notation to use the result of one function as a parameter to next function (assuming identical number of iterations). This allows some more decompositions and allows us to do distributed code motion on the result of a function. - sequential (next call sees results of the previous). Here we would just sequentially execute a number of queries. That is the one Maurice wants.
Peter
-----Original Message----- From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] Sent: Friday, March 28, 2008 12:40 PM To: P.Boncz@cwi.nl Cc: 'Ying Zhang'; monetdb-developers@lists.sourceforge.net Subject: Re: Monetdb-developers Digest, Vol 22, Issue 4
Peter Boncz wrote:
Hi Maurice,
That is not possible.. without adding extensions such as
proposed in XQueryP or
XQuery-Bang.
Why is it that running the update followed by a query is
not acceptable?
Running an update followed by a query is acceptable. My point is that the update and the query are both "requests" ... *asynchronous* requests if you use XRPC as communication mechanism. I don't mind sending the update and the query in separated requests, but it seems to me inefficient and cumbersome for a developer that he needs to program it in such a way that he sends the update, necessarily waits for it to be completed, then sends the query and waits for the result. The developer is not interested in the response to the update. Hence, it would be IMO better if this developer can simply send the update, and without waiting for it to be completed, send the query and wait for the result of the query. In this way, he assumes however that the query 'sees' the result of the update ... and I am not convinced that this is what he will get, because: it is not guaranteed that the update request is received by the server before the query request, and that the update request is finished before the query request starts. Like I said, I'm only mimicking an application sending "UPDATE;SELECT" to a relational database.
Another idea is to use fn:put() in the update, and then
just a HTTP GET to get
the XML result out. That is still a single query.
Why is this a single query? I need to send one XRPC message that calls a function in a module that does the fn:put(), and another request with HTTP GET to get the result out ... and the application has to wait for the response of the XRPC message, before doing the HTTP GET, right?
We are now discussing full-fledged transactions and what I would expect from MXQ if it would support full-fledged transactions. My original question was much simpler. I want the result of a query to be both stored and sent back to the application. This requires 2 requests that need to be executed serially: fn:put(query,"doc.xml") ; doc("doc.xml") Wouldn't it be very helpful, whatever restrictions or visibility constraints, if this simple thing could be done in one function, hence one XRPC-request? I don't see any technical objections to allow the XQuery expression below (i.e., remove the typing constraints) and say that the semantics is that it results in (1) an update tape applied afterwards, and (2) a query result (everything in the sequence that is not an update), and hence that updates done in a query are not visible to the query itself.
let $answer := query return (fn:put($answer,"doc"), $answer)
This would already make live much easier for a developer in many simple cases.
Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
On Mar 28, 2008, at 14:01 , Peter Boncz wrote:
Hi Maurice,
- what you want *really* is XQueryP, but we do not have resources to
give you that. The fact that we use the result sequence to pass on the update tape actually makes it difficult to give back a result.
- if you send two asynchronous requests, there is no guarantee that
one is executed after the other.
- fn:put() is currently still an updating function, but this is
erroneous. I will check in a modification that makes it a readonly one (soon).
Hi Peter,
Before you change it, please have a look at the latest version of XQUF (http://www.w3.org/TR/xquery-update-10/). In earlier draft version, fn:put() was (explicitly) specified as a read-only function, but this has been removed in the latest version and it *seems* to have become an updating function.
Do I ready you correctly that this would already be enough? Notice that fn:put() does not add a document to the database. It just creates some XML file on the server.
A very ugly hack that comes to mind is to support a special URI in fn:put(), e.g. "xrpc://user-data" that works only in a XRPC request, and causes the node to be included in XRPC response message. Not as the result sequence, but in some optional user-data section.
A more elaborate hack would be to allow multiple function calls inside a single XRPC-exchange. We could define a number of modes of execution, most notably:
- snapshot (in the same snapshot, sharing XML fragments). We could
even allow a special notation to use the result of one function as a parameter to next function (assuming identical number of iterations). This allows some more decompositions and allows us to do distributed code motion on the result of a function.
- sequential (next call sees results of the previous). Here we would
just sequentially execute a number of queries. That is the one Maurice wants.
I would vote for the multiple function calls solution. With the 'sequential' one, we can already solve the problem that we cannot decompose subqueries in a sequence construction in some cases.
Jennie
Peter
-----Original Message----- From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] Sent: Friday, March 28, 2008 12:40 PM To: P.Boncz@cwi.nl Cc: 'Ying Zhang'; monetdb-developers@lists.sourceforge.net Subject: Re: Monetdb-developers Digest, Vol 22, Issue 4
Peter Boncz wrote:
Hi Maurice,
That is not possible.. without adding extensions such as
proposed in XQueryP or
XQuery-Bang.
Why is it that running the update followed by a query is
not acceptable?
Running an update followed by a query is acceptable. My point is that the update and the query are both "requests" ... *asynchronous* requests if you use XRPC as communication mechanism. I don't mind sending the update and the query in separated requests, but it seems to me inefficient and cumbersome for a developer that he needs to program it in such a way that he sends the update, necessarily waits for it to be completed, then sends the query and waits for the result. The developer is not interested in the response to the update. Hence, it would be IMO better if this developer can simply send the update, and without waiting for it to be completed, send the query and wait for the result of the query. In this way, he assumes however that the query 'sees' the result of the update ... and I am not convinced that this is what he will get, because: it is not guaranteed that the update request is received by the server before the query request, and that the update request is finished before the query request starts. Like I said, I'm only mimicking an application sending "UPDATE;SELECT" to a relational database.
Another idea is to use fn:put() in the update, and then
just a HTTP GET to get
the XML result out. That is still a single query.
Why is this a single query? I need to send one XRPC message that calls a function in a module that does the fn:put(), and another request with HTTP GET to get the result out ... and the application has to wait for the response of the XRPC message, before doing the HTTP GET, right?
We are now discussing full-fledged transactions and what I would expect from MXQ if it would support full-fledged transactions. My original question was much simpler. I want the result of a query to be both stored and sent back to the application. This requires 2 requests that need to be executed serially: fn:put(query,"doc.xml") ; doc("doc.xml") Wouldn't it be very helpful, whatever restrictions or visibility constraints, if this simple thing could be done in one function, hence one XRPC-request? I don't see any technical objections to allow the XQuery expression below (i.e., remove the typing constraints) and say that the semantics is that it results in (1) an update tape applied afterwards, and (2) a query result (everything in the sequence that is not an update), and hence that updates done in a query are not visible to the query itself.
let $answer := query return (fn:put($answer,"doc"), $answer)
This would already make live much easier for a developer in many simple cases.
Maurice.
--
Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen
participants (3)
-
Keulen, M. van (Maurice) -
Peter Boncz -
Ying Zhang