From manuel@visokio.com Wed Jun 20 19:25:47 2018
From: Manuel Pagliai <manuel@visokio.com>
To: users-list@monetdb.org
Subject: Performance issues when streaming records from table.
Date: Wed, 20 Jun 2018 18:19:45 +0100
Message-ID:
 <CAO3t3aYR3wsqOpEojDCEm2KE8J1td=0MnDUdc9bp7bmMDgLbzg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============5921454568466726293=="

--===============5921454568466726293==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

We are interested in adding support to import records from Monetdb tables
in the ETL we are developing. We have performance issues when executing
queries like:

select col0, col1, ... coln from <table>

on tables with a large number of records. What I am concerned about is the
time spent upfront, before the first record is retrieved:  once records
become available we can start processing them.

For example, on my machine, executing a executing a select * from <table>
query on a 68 million records table takes around 24 seconds for the query
to execute, before it is actually possible to start retrieving records from
the server response.

Queries are submitted via the JDBC driver.

Is there a more efficient way of streaming records from a table ?

Thanks,

Best regards,

*Manuel Pagliai*
Visokio Ltd
www.visokio.com
+44 20 7351 4646



--===============5921454568466726293==
Content-Type: text/html
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="attachment.html"
MIME-Version: 1.0

PGRpdiBkaXI9Imx0ciI+PGRpdj5XZSBhcmUgaW50ZXJlc3RlZCBpbiBhZGRpbmcgc3VwcG9ydCB0
byBpbXBvcnQgcmVjb3JkcyBmcm9tIE1vbmV0ZGIgdGFibGVzIGluIHRoZSBFVEwgd2UgYXJlIGRl
dmVsb3BpbmcuIFdlIGhhdmUgcGVyZm9ybWFuY2UgaXNzdWVzIHdoZW4gZXhlY3V0aW5nIHF1ZXJp
ZXMgbGlrZTo8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pjxmb250IGZhY2U9Im1vbm9zcGFjZSwg
bW9ub3NwYWNlIj5zZWxlY3QgY29sMCwgY29sMSwgLi4uIGNvbG4gZnJvbSAmbHQ7dGFibGUmZ3Q7
PC9mb250PjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+b24gdGFibGVzIHdpdGggYSBsYXJnZSBu
dW1iZXIgb2YgcmVjb3Jkcy4gV2hhdCBJIGFtIGNvbmNlcm5lZCBhYm91dCBpcyB0aGUgdGltZSBz
cGVudCB1cGZyb250LCBiZWZvcmUgdGhlIGZpcnN0IHJlY29yZCBpcyByZXRyaWV2ZWQ6wqDCoG88
c3BhbiBzdHlsZT0iZm9udC1zaXplOnNtYWxsO2JhY2tncm91bmQtY29sb3I6cmdiKDI1NSwyNTUs
MjU1KTt0ZXh0LWRlY29yYXRpb24tc3R5bGU6aW5pdGlhbDt0ZXh0LWRlY29yYXRpb24tY29sb3I6
aW5pdGlhbDtmbG9hdDpub25lO2Rpc3BsYXk6aW5saW5lIj5uY2UgcmVjb3JkcyBiZWNvbWUgYXZh
aWxhYmxlIHdlIGNhbiBzdGFydCBwcm9jZXNzaW5nIHRoZW0uPC9zcGFuPjwvZGl2PjxkaXY+PGJy
PjwvZGl2PjxkaXY+Rm9yIGV4YW1wbGUsIG9uIG15IG1hY2hpbmUsIGV4ZWN1dGluZyBhIGV4ZWN1
dGluZyBhPGZvbnQgZmFjZT0ibW9ub3NwYWNlLCBtb25vc3BhY2UiPiBzZWxlY3QgKiBmcm9tICZs
dDt0YWJsZSZndDs8L2ZvbnQ+wqAgcXVlcnkgb24gYSA2OCBtaWxsaW9uIHJlY29yZHMgdGFibGUg
dGFrZXMgYXJvdW5kIDI0IHNlY29uZHMgZm9yIHRoZSBxdWVyeSB0byBleGVjdXRlLCBiZWZvcmUg
aXQgaXMgYWN0dWFsbHkgcG9zc2libGUgdG8gc3RhcnQgcmV0cmlldmluZyByZWNvcmRzIGZyb20g
dGhlIHNlcnZlciByZXNwb25zZS48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PgoKPHNwYW4gc3R5
bGU9ImZvbnQtc2l6ZTpzbWFsbDtiYWNrZ3JvdW5kLWNvbG9yOnJnYigyNTUsMjU1LDI1NSk7dGV4
dC1kZWNvcmF0aW9uLXN0eWxlOmluaXRpYWw7dGV4dC1kZWNvcmF0aW9uLWNvbG9yOmluaXRpYWw7
ZmxvYXQ6bm9uZTtkaXNwbGF5OmlubGluZSI+UXVlcmllcyBhcmUgc3VibWl0dGVkIHZpYSB0aGUg
SkRCQyBkcml2ZXIuPHNwYW4+wqA8L3NwYW4+PC9zcGFuPgoKPGJyPjwvZGl2PjxkaXY+PGJyPjwv
ZGl2PjxkaXY+SXMgdGhlcmUgYSBtb3JlIGVmZmljaWVudCB3YXkgb2Ygc3RyZWFtaW5nIHJlY29y
ZHMgZnJvbSBhIHRhYmxlID88YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5UaGFua3MsPC9k
aXY+PGRpdj48YnIgY2xlYXI9ImFsbCI+PGRpdj48ZGl2IGNsYXNzPSJnbWFpbF9zaWduYXR1cmUi
PjxkaXYgZGlyPSJsdHIiPjxkaXY+PGRpdiBkaXI9Imx0ciI+PGRpdj48ZGl2IGRpcj0ibHRyIj48
ZGl2IGRpcj0ibHRyIj48ZGl2IHN0eWxlPSJmb250LXNpemU6MTIuOHB4Ij48ZGl2IHN0eWxlPSJj
b2xvcjpyZ2IoMTM2LDEzNiwxMzYpO2ZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNl
cmlmIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEyLjhweDtmb250LWZhbWlseTphcmlhbCxzYW5z
LXNlcmlmO2NvbG9yOnJnYigzNCwzNCwzNCkiPkJlc3QgcmVnYXJkczwvc3Bhbj48c3BhbiBzdHls
ZT0iY29sb3I6cmdiKDAsMCwwKTtmb250LWZhbWlseTphcmlhbDtmb250LXNpemU6MTMuMzMzM3B4
Ij4sPC9zcGFuPjxicj48L2Rpdj48c3BhbiBzdHlsZT0iY29sb3I6cmdiKDEzNiwxMzYsMTM2KTtm
b250LWZhbWlseTphcmlhbCI+PGJyIHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDApIj48L3NwYW4+PGZv
bnQgc2l6ZT0iMiIgc3R5bGU9ImZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlm
Ij48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6YXJpYWwiPjxmb250IGNvbG9yPSIjNTc3MDhmIj48
Yj5NYW51ZWwgUGFnbGlhaTwvYj48L2ZvbnQ+PGJyIHN0eWxlPSJmb250LXdlaWdodDpib2xkO2Nv
bG9yOnJnYig4NywxMTIsMTQzKSI+PGZvbnQgc2l6ZT0iMSIgc3R5bGU9ImNvbG9yOnJnYigxMzYs
MTM2LDEzNikiPjxzcGFuIHN0eWxlPSJjb2xvcjpyZ2IoODcsMTEyLDE0MykiPlZpc29raW8gTHRk
PC9zcGFuPjxiciBzdHlsZT0iY29sb3I6cmdiKDg3LDExMiwxNDMpIj48c3BhbiBzdHlsZT0iY29s
b3I6cmdiKDg3LDExMiwxNDMpIj48YSBocmVmPSJodHRwOi8vd3d3LnZpc29raW8uY29tLyIgc3R5
bGU9ImNvbG9yOnJnYigxNyw4NSwyMDQpIiB0YXJnZXQ9Il9ibGFuayI+d3d3LnZpc29raW8uY29t
PC9hPsKgPC9zcGFuPjwvZm9udD48L3NwYW4+PGZvbnQgc2l6ZT0iMSIgc3R5bGU9ImNvbG9yOnJn
YigxMzYsMTM2LDEzNikiPjxiciBzdHlsZT0iZm9udC1mYW1pbHk6YXJpYWw7Y29sb3I6cmdiKDg3
LDExMiwxNDMpIj48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6YXJpYWw7Y29sb3I6cmdiKDg3LDEx
MiwxNDMpIj48YSB2YWx1ZT0iKzQ0MjA3MzUxNDY0NiIgc3R5bGU9ImNvbG9yOnJnYigzNCwzNCwz
NCkiPis0NCAyMCA3MzUxIDQ2NDY8L2E+PC9zcGFuPjwvZm9udD48L2ZvbnQ+PC9kaXY+PC9kaXY+
PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+CjwvZGl2PjwvZGl2Pgo=

--===============5921454568466726293==--