Fwd: Bulk-loading in monetdb with primary key constraint
I am trying to bulk-load a list of objects into a one-column (primary key) db. The only reason is to remove duplicates. I can't load the list in memory, because the file size is way greater than my memory size.
I use monetdb's COPY-INTOhttp://www.monetdb.org/Documentation/Manuals/SQLreference/CopyIntocommand, but I don't want it to fail when there is a duplicate. I want it to add everything that is not a duplicate and skip the duplicates.
Is there any way to do that with monetdb? Any other way?
Thank you in advance!
Hi
Consider loading the objects into a column without KEY constraint first and then select/delete duplicates using a query and, finally, alter the constraint
On 23/12/13 10:17, Vasilis Efthymiou wrote:
I am trying to bulk-load a list of objects into a one-column (primary key) db. The only reason is to remove duplicates. I can't load the list in memory, because the file size is way greater than my memory size.
I use monetdb's COPY-INTO http://www.monetdb.org/Documentation/Manuals/SQLreference/CopyInto command, but I don't want it to fail when there is a duplicate. I want it to add everything that is not a duplicate and skip the duplicates.
Is there any way to do that with monetdb? Any other way?
Thank you in advance!
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Hi,
Does monetdb support recursive CTE? Does monetdb has a group_concat similar function? if no how to do a similar operation? Does monetdb support execution of sql string? if no how to do dynamic pivot tables?
Thank you.
On Mon, Dec 23, 2013 at 11:21 AM, Martin Kersten Martin.Kersten@cwi.nlwrote:
Hi
Consider loading the objects into a column without KEY constraint first and then select/delete duplicates using a query and, finally, alter the constraint
On 23/12/13 10:17, Vasilis Efthymiou wrote:
I am trying to bulk-load a list of objects into a one-column (primary key) db. The only reason is to remove duplicates. I can't load the list in memory, because the file size is way greater than my memory size.
I use monetdb's COPY-INTO http://www.monetdb.org/Documentation/Manuals/SQLreference/CopyInto
command, but I don't want it to fail when there is a duplicate. I want it to add everything that is not a duplicate and skip the duplicates.
Is there any way to do that with monetdb? Any other way?
Thank you in advance!
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Hi,
I am implementing group_concat in MAL language, this is what i have sofar, CREATE AGGREGATE group_concat(val STRING) returns STRING external name "udf"."group_concat";
module udf; function group_concat(b:bat[:oid,:str]):str; r:=""; length:=aggr.count(b); barrier (h,t):= iterator.new(b); r:=r+t; redo (h,t) := iterator.next(b); exit (h,t); return r; end group_concat;
when i use group_concat in group by, i get this error: TypeException:user.s1_1[105]:'udf.subgroup_concat' undefined in: _168:bat[:oid,: str] := udf.subgroup_concat(_167:bat[:oid,:str], _153:bat[:oid,:oid], r1_153:bat [:oid,:oid], _169:bit) program contains errors
any help or reference to implement an aggregate function? Thank you.
On Mon, Dec 23, 2013 at 11:46 AM, imad hajj chahine < imad.hajj.chahine@gmail.com> wrote:
Hi,
Does monetdb support recursive CTE? Does monetdb has a group_concat similar function? if no how to do a similar operation? Does monetdb support execution of sql string? if no how to do dynamic pivot tables?
Thank you.
On Mon, Dec 23, 2013 at 11:21 AM, Martin Kersten Martin.Kersten@cwi.nlwrote:
Hi
Consider loading the objects into a column without KEY constraint first and then select/delete duplicates using a query and, finally, alter the constraint
On 23/12/13 10:17, Vasilis Efthymiou wrote:
I am trying to bulk-load a list of objects into a one-column (primary key) db. The only reason is to remove duplicates. I can't load the list in memory, because the file size is way greater than my memory size.
I use monetdb's COPY-INTO http://www.monetdb.org/Documentation/Manuals/SQLreference/CopyInto
command, but I don't want it to fail when there is a duplicate. I want it to add everything that is not a duplicate and skip the duplicates.
Is there any way to do that with monetdb? Any other way?
Thank you in advance!
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
participants (3)
-
imad hajj chahine -
Martin Kersten -
Vasilis Efthymiou