Adding data types (maybe to core), probably BUG

Lefteris lsidir at gmail.com
Fri Mar 8 17:21:04 CET 2013


Extending core data types is not going to happen, I am not using the
word never though:)

For the cases you describe you will have to do wrappers around the
types that monetdb's kernel (aka gdk) provides you with.

To give you an example, imagine gdk being your c compiler and monetdb5
modules being your c programs. You would not ask the c compiler to
support a new data type, rather you would create a struct in your c
program that wraps that new type you need. Of course there are
performance penalties, but it is a trade-off: if we made all types
part of gdk and then again you would have performance issues.

Now there are many ways to do things, and many times if it is done
correctly you would see small to zero performance problems. For
example if you need a 16 byte type, maybe you need to define 2 columns
of 8 bytes each, and create monetdb5 procedures that understand this
semantics by using the primitives provided by the gdk.

As for your uint_32, maybe allowing a regular sorting of signed 32 bit
ints and then flipping (in the monetdb5 wrapper procedure) the
negatives at the end of the column would not be the optimal solution
but close to it, since a flip of this kind would not cost too much,
especially if the memory allocation is down correctly from the start.

Just some quick thoughts, maybe I am missing something.

lefteris

On Fri, Mar 8, 2013 at 4:53 PM, Mike Gomez <mail_server at lavabit.com> wrote:
> Hi lefteris,
>        Thanks for your advice.
>
> i get your point. i have confirmed that OIDs arithmetic is unsigned
> type. but that again opens up a can of worms.
>
> x32 bit is passing out and x64 bit is in.
> but considering each uint32 will now take 64 bit, that's double the
> space.It might not count much for 1 or 10 million rows, but when the
> row count increases, the system will hit the memory wall 50% earlier,
> (considering only this change).
>
> also, your advice mentions years of stuff to get GDK atoms in place SO i
> will fall back to external atoms (monetdb5/modules/atoms).
> but suppose i want a datatype of say 12 or 16 bytes or even more, then
> how we do we get around it. Limited understanding of mine tells, it not
> into possible.so how do i put around that limitation or is it not
> possible !!
>
> also,i have a question about external atoms.
> say i cast lng into few bit fields, and later on i want to sort the
> table according to bit fields. since inherent type is lng 9 as you have
> mentioned, arithmetic is done on lng), so again this is impossible!
> right?
>
> is there work going to extend core data types ?
>
> more pointers ??
>
> Thanks.
> Mike
>
>
>
>>Hi Mike,
>>
>>your second solution through the kernel is definitely not the right
>>way to go. Data types in the kernel are first class citizens and used
>>to optimize the code in many different ways. Adding a type there is
>>inefficient, next to impossible if you do not have years of experience
>>in monet, and 99.9% of the cases I can think of unnecessary.
>>
>>The correct way to do it is through monetdb5/modules/atoms
>>
>>But you have to expose the data type to the sql frontend too (I assume
>>that is what you are using).
>>
>>Your uint must be mapped to a datatype that monetdb kernel knows and
>>has the same behaviour. That is oid's for this case (even if oid's may
>>be larger than uints).
>>
>>MonetDB does not use function pointers for comparisons in the kernel,
>>so you can not just add another comparison function and expect
>>everything to work out of the box. Rather you map your new type in
>>monetdb5 to a type in the kernel that subsumes it.
>>
>>Hope this helps.
>>
>>cheers,
>>
>>lefteris
>>
>>On Fri, Mar 8, 2013 at 2:24 PM, Mike Gomez <mail_server at lavabit.com>
>>wrote:
>>> Hi,
>>>    i am working on monetdb so as it can be used efficiently by more
>>> users(myself included).However, the issue i seem to face ( as maybe
>>> others) is the data types.
>>>
>>> for many weeks now,I have been going through the source code,
>>> understanding and learning it.Data types in monetdb can be extended
>>> in 2 types, external and core GDK.
>>>
>>> external method problem ( maybe bug in monetdb)
>>> ========================
>>>
>>> I have created new data type, say unsigned int 32 bit, using
>>> extensions (such as inet,color and so on).However, the bug is monetdb
>>> system doesn't use my comparison operation, instead uses the default
>>> inbuilt operators.
>>>
>>> eg.  in the uint mal file( MonetDB/monetdb5/modules/atoms/uint.mal),
>>> i
>>> have declared
>>>
>>> --------------
>>> atom uint:int;
>>> ..
>>> ..............skipped source code
>>>
>>> however, when i sort the table, the system still places negative
>>> values ahead, means instead of using my declared comparison algos,
>>> it's using internal algos.
>>>
>>> my question is how do i override that ( or is it even possible to do
>>> it).
>>>
>>>
>>> core GDK method
>>> ===============
>>> After failing that, i thought lets add the core data type uint in the
>>> gdk_atoms.c. i modified the parsor(& everything else) and the system
>>> recognizes my data type.
>>>
>>> however,when i run simple insert query such as below, i get
>>> insert into test1 values('2');
>>> TypeException:user.s1_1[3]:'calc.uint' undefined in: _5:any :=
>>> calc.uint(_4:str)
>>>
>>> however, i have got completely stuck in the MAL operators and
>>> language. so, how do i get the algebra, operations and everything
>>> right, so it works?
>>>
>>> Any pointers will help.
>>> let me know if something more is required.
>>>
>>> i am working on this out of my interest,so eventually if i get it to
>>> work somehow, the code goes back to project.
>>>
>>> Thanks for help.
>>> mike
>>>
>
> _______________________________________________
> developers-list mailing list
> developers-list at monetdb.org
> http://mail.monetdb.org/mailman/listinfo/developers-list



More information about the developers-list mailing list