[Monetdb-developers] Mapi slowdown with Java + Linux (4.6.2!)
I have been struggling with this problem for a couple of weeks. We are using Monet 4.6.2 (I know, I know!) to power a web service written in Java. When Monet is running on OS X, I can handle 200+ requests per second, but when I install it on a (much faster) Linux machine, the app only gives me 20 requests per second.
The difference happens on the TCP/IP connection. I have the simplest test:
var a:=bat(int,int).insert(1,1); a.print();
This takes 4ms on OS X, but 40ms on Linux. I've traced down the system calls, and what happens is that when the server is running on OS X my client reads back the results in one or two calls to recv, whereas when the server is on Linux the client only gets a couple of bytes back from the server and then blocks until the rest arrives. The following is a section of strace's output on the Java application:
$ strace -f -ttt java -cp test.java test
Monet running on OS X: ----------------------
send(5, "a.print();\n", 12, 0) = 12 recv(5, "#------------\n# h ....", 8192, 0) = 50 ioctl(5, FIONREAD, [45]) = recv(5, " # type ....", 8192, 0) = 45
Monet running on Linux: -----------------------
send(5, "a.print();\n", 12, 0) = 12 recv(5, "#-", 8192, 0) ioctl(5, FIONREAD, [0]) = recv(5, <unfinished...> ==> blocks [... 40ms...] <recv resumed "----------#\n# h ...", 8192, 0) = 93
The sockets are opened in the same way in both cases, just like the MapiClient opens them:
connect(5, {sa_family=AF_INET, sin_port=htons(45678), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 setsockopt(5, SOL_SOCKET, SO_KEEPALIVE, [0], 4) = 0 setsockopt(5, SOL_IP, IP_TOS, [8], 4) = 0 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
This problem does not happen when I use Monet's MapiClient: the response times on OS X and Linux are similar.
While inspecting the calls between the server and the MapiClient, then, I realized that the handshake is different from the one used by the Mapi.java class that comes with 4.6.2: the C client sets the 'blocked' flag on.
Turning the 'blocked' flag to true in the MapiClient.java class doesn't work. Switching the C client's default mode to be non-blocked (in src/mapi/clients/C/Mapi.mx) does indeed make the connection much slower, as every single character is sent to the client in its own packet (due to TCP_NODELAY?):
write(3, "print(a);\n", 10) = 10 read(3, "#", 1) = 1 read(3, "-", 1) = 1 read(3, "-", 1) = 1 ... read(3, "]", 1) = 1 read(3, "\n", 1) = 1
My question is then: is it possible to write a Java client that communicates with the server using the blocked protocol? Do you think that would solve my performance issues? If so, is there any documentation/example code I can follow (since the actual protocol seems to be different in both cases). I've looked at the latest CVS checkins for the the Mapi protocol, and I see that the only option now is the 'blocked' mode, so I guess it's possible to use it from Java....
FYI, I am using - Monet 4.6.2 - Java 5 - OS X 10.4.9 - Linux 2.6.9, 2.6.15, 2.6.17
Thank you very much,
-- Agustin
On Fri, Mar 23, 2007 at 04:25:50PM -0400, Agustin Schapira wrote:
I have been struggling with this problem for a couple of weeks. We are using Monet 4.6.2 (I know, I know!) to power a web service written in Java. When Monet is running on OS X, I can handle 200+ requests per second, but when I install it on a (much faster) Linux machine, the app only gives me 20 requests per second.
The difference happens on the TCP/IP connection. I have the simplest test:
var a:=bat(int,int).insert(1,1); a.print();
This takes 4ms on OS X, but 40ms on Linux. I've traced down the system calls, and what happens is that when the server is running on OS X my client reads back the results in one or two calls to recv, whereas when the server is on Linux the client only gets a couple of bytes back from the server and then blocks until the rest arrives. The following is a section of strace's output on the Java application:
$ strace -f -ttt java -cp test.java test
Monet running on OS X:
send(5, "a.print();\n", 12, 0) = 12 recv(5, "#------------\n# h ....", 8192, 0) = 50 ioctl(5, FIONREAD, [45]) = recv(5, " # type ....", 8192, 0) = 45
Monet running on Linux:
send(5, "a.print();\n", 12, 0) = 12 recv(5, "#-", 8192, 0) ioctl(5, FIONREAD, [0]) = recv(5, <unfinished...> ==> blocks [... 40ms...] <recv resumed "----------#\n# h ...", 8192, 0) = 93
The sockets are opened in the same way in both cases, just like the MapiClient opens them:
connect(5, {sa_family=AF_INET, sin_port=htons(45678), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 setsockopt(5, SOL_SOCKET, SO_KEEPALIVE, [0], 4) = 0 setsockopt(5, SOL_IP, IP_TOS, [8], 4) = 0 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
This problem does not happen when I use Monet's MapiClient: the response times on OS X and Linux are similar.
While inspecting the calls between the server and the MapiClient, then, I realized that the handshake is different from the one used by the Mapi.java class that comes with 4.6.2: the C client sets the 'blocked' flag on.
Turning the 'blocked' flag to true in the MapiClient.java class doesn't work. Switching the C client's default mode to be non-blocked (in src/mapi/clients/C/Mapi.mx) does indeed make the connection much slower, as every single character is sent to the client in its own packet (due to TCP_NODELAY?):
write(3, "print(a);\n", 10) = 10 read(3, "#", 1) = 1 read(3, "-", 1) = 1 read(3, "-", 1) = 1 ... read(3, "]", 1) = 1 read(3, "\n", 1) = 1
My question is then: is it possible to write a Java client that communicates with the server using the blocked protocol? Do you think that would solve my performance issues? If so, is there any documentation/example code I can follow (since the actual protocol seems to be different in both cases). I've looked at the latest CVS checkins for the the Mapi protocol, and I see that the only option now is the 'blocked' mode, so I guess it's possible to use it from Java....
FYI, I am using
- Monet 4.6.2
- Java 5
- OS X 10.4.9
- Linux 2.6.9, 2.6.15, 2.6.17
Yes its possible to use the blocked mode in java. Its currently used by the 'jdbc' driver. Problem afcourse is that the current jdbc driver doesn't work with your ancient monetdb and mil.
Niels
Thank you very much,
-- Agustin
Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=D... _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
Dear Agustin,
The protocol has been changed in autum 2005 precisely for the reasons you now stumble upon. At that time we were engaged with the C'tMagazine benchmark, which called for a web-server application. The new protocol has put us ahead of Mysql/jdbc on that application.
This exercise took 2 months total, with probably also 2 full manmonths of careful design and experimentation.
You may inspect the archives for background information, e.g. http://sourceforge.net/mailarchive/message.php?msg_id=14228009
Perhaps you should consider spending a few weeks in Amsterdam to leapfrog Proximity to the new platform setting, e.g. M5.
regards, Martin Niels Nes wrote:
On Fri, Mar 23, 2007 at 04:25:50PM -0400, Agustin Schapira wrote:
I have been struggling with this problem for a couple of weeks. We are using Monet 4.6.2 (I know, I know!) to power a web service written in Java. When Monet is running on OS X, I can handle 200+ requests per second, but when I install it on a (much faster) Linux machine, the app only gives me 20 requests per second.
The difference happens on the TCP/IP connection. I have the simplest test:
var a:=bat(int,int).insert(1,1); a.print();
This takes 4ms on OS X, but 40ms on Linux. I've traced down the system calls, and what happens is that when the server is running on OS X my client reads back the results in one or two calls to recv, whereas when the server is on Linux the client only gets a couple of bytes back from the server and then blocks until the rest arrives. The following is a section of strace's output on the Java application:
$ strace -f -ttt java -cp test.java test
Monet running on OS X:
send(5, "a.print();\n", 12, 0) = 12 recv(5, "#------------\n# h ....", 8192, 0) = 50 ioctl(5, FIONREAD, [45]) = recv(5, " # type ....", 8192, 0) = 45
Monet running on Linux:
send(5, "a.print();\n", 12, 0) = 12 recv(5, "#-", 8192, 0) ioctl(5, FIONREAD, [0]) = recv(5, <unfinished...> ==> blocks [... 40ms...] <recv resumed "----------#\n# h ...", 8192, 0) = 93
The sockets are opened in the same way in both cases, just like the MapiClient opens them:
connect(5, {sa_family=AF_INET, sin_port=htons(45678), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 setsockopt(5, SOL_SOCKET, SO_KEEPALIVE, [0], 4) = 0 setsockopt(5, SOL_IP, IP_TOS, [8], 4) = 0 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
This problem does not happen when I use Monet's MapiClient: the response times on OS X and Linux are similar.
While inspecting the calls between the server and the MapiClient, then, I realized that the handshake is different from the one used by the Mapi.java class that comes with 4.6.2: the C client sets the 'blocked' flag on.
Turning the 'blocked' flag to true in the MapiClient.java class doesn't work. Switching the C client's default mode to be non-blocked (in src/mapi/clients/C/Mapi.mx) does indeed make the connection much slower, as every single character is sent to the client in its own packet (due to TCP_NODELAY?):
write(3, "print(a);\n", 10) = 10 read(3, "#", 1) = 1 read(3, "-", 1) = 1 read(3, "-", 1) = 1 ... read(3, "]", 1) = 1 read(3, "\n", 1) = 1
My question is then: is it possible to write a Java client that communicates with the server using the blocked protocol? Do you think that would solve my performance issues? If so, is there any documentation/example code I can follow (since the actual protocol seems to be different in both cases). I've looked at the latest CVS checkins for the the Mapi protocol, and I see that the only option now is the 'blocked' mode, so I guess it's possible to use it from Java....
FYI, I am using
- Monet 4.6.2
- Java 5
- OS X 10.4.9
- Linux 2.6.9, 2.6.15, 2.6.17
Yes its possible to use the blocked mode in java. Its currently used by the 'jdbc' driver. Problem afcourse is that the current jdbc driver doesn't work with your ancient monetdb and mil.
Niels
Thank you very much,
-- Agustin
Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=D... _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
Dear Martin,
Thank you very much for your response. It's good to know that this problem is solved in the new versions of the MAPI protocol.
I downloaded and installed the latest release, and wrote a basic Java utility to send MIL commands to the new 4 server, 4.16. It does indeed work wonderfully now, in blocked mode and giving the same speed for Linux and OS X. Thank you very much.
The next step will be to follow the code in the JDBC driver and use some of its ideas to parse the results of the MIL commands that Proximity sends to the server. I read about a much improved protocol with special header lines that indicate the rowcount of the result, columncount, etc. Is that only available for SQL, or does it work with plain MIL as well?
The offer to go to Amsterdam sounds very tempting, thanks :-) It might be possible to arrange this is in the near future (end of the summer, maybe?), when we get ready to say good-bye to MIL and finally embrace the new MAL. For the time being upgrading to 4.16 seems to be enough for our needs, but we are already thinking about moving to M5.
Again, that you for your help. Regards from Amherst,
-- Agustin
PS: The installation from the new sources worked without a problem on three different platforms, with a single command, and very fast. It was very impressive!
On Mar 23, 2007, at 6:19 PM, Martin Kersten wrote:
Dear Agustin,
The protocol has been changed in autum 2005 precisely for the reasons you now stumble upon. At that time we were engaged with the C'tMagazine benchmark, which called for a web-server application. The new protocol has put us ahead of Mysql/jdbc on that application.
This exercise took 2 months total, with probably also 2 full manmonths of careful design and experimentation.
You may inspect the archives for background information, e.g. http://sourceforge.net/mailarchive/message.php?msg_id=14228009
Perhaps you should consider spending a few weeks in Amsterdam to leapfrog Proximity to the new platform setting, e.g. M5.
regards, Martin Niels Nes wrote:
On Fri, Mar 23, 2007 at 04:25:50PM -0400, Agustin Schapira wrote:
I have been struggling with this problem for a couple of weeks. We are using Monet 4.6.2 (I know, I know!) to power a web service written in Java. When Monet is running on OS X, I can handle 200+ requests per second, but when I install it on a (much faster) Linux machine, the app only gives me 20 requests per second.
The difference happens on the TCP/IP connection. I have the simplest test:
var a:=bat(int,int).insert(1,1); a.print();
This takes 4ms on OS X, but 40ms on Linux. I've traced down the system calls, and what happens is that when the server is running on OS X my client reads back the results in one or two calls to recv, whereas when the server is on Linux the client only gets a couple of bytes back from the server and then blocks until the rest arrives. The following is a section of strace's output on the Java application:
$ strace -f -ttt java -cp test.java test
Monet running on OS X:
send(5, "a.print();\n", 12, 0) = 12 recv(5, "#------------\n# h ....", 8192, 0) = 50 ioctl(5, FIONREAD, [45]) = recv(5, " # type ....", 8192, 0) = 45
Monet running on Linux:
send(5, "a.print();\n", 12, 0) = 12 recv(5, "#-", 8192, 0) ioctl(5, FIONREAD, [0]) = recv(5, <unfinished...> ==> blocks [... 40ms...] <recv resumed "----------#\n# h ...", 8192, 0) = 93
The sockets are opened in the same way in both cases, just like the MapiClient opens them:
connect(5, {sa_family=AF_INET, sin_port=htons(45678), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 setsockopt(5, SOL_SOCKET, SO_KEEPALIVE, [0], 4) = 0 setsockopt(5, SOL_IP, IP_TOS, [8], 4) = 0 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
This problem does not happen when I use Monet's MapiClient: the response times on OS X and Linux are similar.
While inspecting the calls between the server and the MapiClient, then, I realized that the handshake is different from the one used by the Mapi.java class that comes with 4.6.2: the C client sets the 'blocked' flag on.
Turning the 'blocked' flag to true in the MapiClient.java class doesn't work. Switching the C client's default mode to be non- blocked (in src/mapi/clients/C/Mapi.mx) does indeed make the connection much slower, as every single character is sent to the client in its own packet (due to TCP_NODELAY?):
write(3, "print(a);\n", 10) = 10 read(3, "#", 1) = 1 read(3, "-", 1) = 1 read(3, "-", 1) = 1 ... read(3, "]", 1) = 1 read(3, "\n", 1) = 1
My question is then: is it possible to write a Java client that communicates with the server using the blocked protocol? Do you think that would solve my performance issues? If so, is there any documentation/example code I can follow (since the actual protocol seems to be different in both cases). I've looked at the latest CVS checkins for the the Mapi protocol, and I see that the only option now is the 'blocked' mode, so I guess it's possible to use it from Java....
FYI, I am using
- Monet 4.6.2
- Java 5
- OS X 10.4.9
- Linux 2.6.9, 2.6.15, 2.6.17
Yes its possible to use the blocked mode in java. Its currently used by the 'jdbc' driver. Problem afcourse is that the current jdbc driver doesn't work with your ancient monetdb and mil. Niels
Thank you very much,
-- Agustin
Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php? page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
Hi Augustin!
On 26-03-2007 14:49:32 -0400, Agustin Schapira wrote:
The next step will be to follow the code in the JDBC driver and use some of its ideas to parse the results of the MIL commands that Proximity sends to the server. I read about a much improved protocol with special header lines that indicate the rowcount of the result, columncount, etc. Is that only available for SQL, or does it work with plain MIL as well?
I'm working on re-factoring the JDBC code, to turn it into a library of utility classes and the JDBC driver that uses those utilities to encode the JDBC specific logic on top of the communication protocol. It might be worth for you to keep this in mind when looking somewhat closer at it. I cannot say anything about when it will be finished, but it probably gives you the tools to simply do what you want without hacking too mych protocol specific code yourself. That would also guarantee you to have a painless migration to newer protocol versions as the APIs hopefully stay the same.
PS: The installation from the new sources worked without a problem on three different platforms, with a single command, and very fast. It was very impressive!
I suppose you used the monetdb-install.sh script or similar when you refer to "a single command"? Are there any platforms there that we don't test on? It is always good to know what platforms it is working on.
participants (4)
-
Agustin Schapira -
Fabian Groffen -
Martin Kersten -
Niels Nes