[Monetdb-developers] How to create a binary load files with string values?
Hi,
I've tried quite a few things (creating the files/strings with 0 at the end, with 0 and \n at the end, fixed length, etc.) but I still get an error
sql>copy into sku from ('/home/mark/MonetDbData/myfile); SQLException:sql:failed to attach file /home/mark/MonetDbData/myfile
I know that I can load integer values, because that works, so the problem must be in how I am formatting strings for MonetDB binary load. Are there any examples or any dumps to look at?
Thank you, Mark
On 2011-10-05 20:21, Mark Kerzner wrote:
Hi,
I've tried quite a few things (creating the files/strings with 0 at the end, with 0 and \n at the end, fixed length, etc.) but I still get an error
sql>copy into sku from ('/home/mark/MonetDbData/myfile); SQLException:sql:failed to attach file /home/mark/MonetDbData/myfile
I know that I can load integer values, because that works, so the problem must be in how I am formatting strings for MonetDB binary load. Are there any examples or any dumps to look at?
Attaching strings is not supported. In other words, there is no way you can do this with strings.
The reason that it is not supported is that the layout of strings is much more complex.
Sjoerd,
I have a limited task at hand: I need to binary-load fixed-length strings, or byte arrays, or anything that is not a number. Is there workaround?
Thank you, Mark
On Wed, Oct 5, 2011 at 2:14 PM, Sjoerd Mullender sjoerd@acm.org wrote:
On 2011-10-05 20:21, Mark Kerzner wrote:
Hi,
I've tried quite a few things (creating the files/strings with 0 at the end, with 0 and \n at the end, fixed length, etc.) but I still get an
error
sql>copy into sku from ('/home/mark/MonetDbData/myfile); SQLException:sql:failed to attach file /home/mark/MonetDbData/myfile
I know that I can load integer values, because that works, so the problem must be in how I am formatting strings for MonetDB binary load. Are there any examples or any dumps to look at?
Attaching strings is not supported. In other words, there is no way you can do this with strings.
The reason that it is not supported is that the layout of strings is much more complex.
-- Sjoerd Mullender
All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
Dear Mark,
Overall, the attachment was put in place for exceptional cases e.g. a TB-size table coming from a scientific instrument. In most situations we have seen that a csv load is fast enough, it runs in parallel.
However, i am working on a completion of the missing functionality.
As said the storage layout for strings does not allow for an easy binary attachment. Instead, what we can do is to handle it identical to a simple, single column CSV file (record terminator \n, no quotes) import.
Binary attachment for time and timestamp has to be considered too.
regards, Martin On 10/7/11 5:59 AM, Mark Kerzner wrote:
Sjoerd,
I have a limited task at hand: I need to binary-load fixed-length strings, or byte arrays, or anything that is not a number. Is there workaround?
Thank you, Mark
On Wed, Oct 5, 2011 at 2:14 PM, Sjoerd Mullender <sjoerd@acm.org mailto:sjoerd@acm.org> wrote:
On 2011-10-05 20:21, Mark Kerzner wrote: > Hi, > > I've tried quite a few things (creating the files/strings with 0 at the > end, with 0 and \n at the end, fixed length, etc.) but I still get an error > > sql>copy into sku from ('/home/mark/MonetDbData/myfile); > SQLException:sql:failed to attach file /home/mark/MonetDbData/myfile > > I know that I can load integer values, because that works, so the > problem must be in how I am formatting strings for MonetDB binary load. > Are there any examples or any dumps to look at? Attaching strings is not supported. In other words, there is no way you can do this with strings. The reason that it is not supported is that the layout of strings is much more complex. -- Sjoerd Mullender ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net <mailto:Monetdb-developers@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2dcopy2
Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
On 2011-10-07 05:59, Mark Kerzner wrote:
Sjoerd,
I have a limited task at hand: I need to binary-load fixed-length strings, or byte arrays, or anything that is not a number. Is there workaround?
I'm afraid I can't think of anything. You'll just have to use COPY INTO from a CSV file (fastest) or INSERT.
The way strings are stored internally is quite complex so therefore there is no quick-and-dirty way to get them into the server (like there is with integers).
If the strings are all 4 or 8 bytes long, you could of course treat them as integers, but then you can't do string-like operations on them, and converting them to strings within the server will be hard.
Thank you, Mark
On Wed, Oct 5, 2011 at 2:14 PM, Sjoerd Mullender <sjoerd@acm.org mailto:sjoerd@acm.org> wrote:
On 2011-10-05 20:21, Mark Kerzner wrote: > Hi, > > I've tried quite a few things (creating the files/strings with 0 at the > end, with 0 and \n at the end, fixed length, etc.) but I still get an error > > sql>copy into sku from ('/home/mark/MonetDbData/myfile); > SQLException:sql:failed to attach file /home/mark/MonetDbData/myfile > > I know that I can load integer values, because that works, so the > problem must be in how I am formatting strings for MonetDB binary load. > Are there any examples or any dumps to look at? Attaching strings is not supported. In other words, there is no way you can do this with strings. The reason that it is not supported is that the layout of strings is much more complex. -- Sjoerd Mullender ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net <mailto:Monetdb-developers@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/monetdb-developers
participants (3)
-
Mark Kerzner -
Martin Kersten -
Sjoerd Mullender