Thanks!
The point to MAL plan really helps me out!

Best,
Guodong
On Sep 21, 2020, 06:00 -0400, users-list-request@monetdb.org, wrote:
Send users-list mailing list submissions to
users-list@monetdb.org

To subscribe or unsubscribe via the World Wide Web, visit
https://www.monetdb.org/mailman/listinfo/users-list
or, via email, send a message with subject or body 'help' to
users-list-request@monetdb.org

You can reach the person managing the list at
users-list-owner@monetdb.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of users-list digest..."


Today's Topics:

1. Internals of intermediate results in MonetDB (Guodong Jin)
2. Re: Internals of intermediate results in MonetDB (Martin Kersten)


----------------------------------------------------------------------

Message: 1
Date: Sun, 20 Sep 2020 23:31:50 -0400
From: Guodong Jin <guod.jin@gmail.com>
To: users-list@monetdb.org
Subject: Internals of intermediate results in MonetDB
Message-ID: <932d01d5-a742-4f57-8e19-ba7599667e42@Spark>
Content-Type: text/plain; charset="utf-8"

Hi all,

Thanks for your hard works on this great open source project.
I?m a student newly working on MonetDB to implement some research ideas, and I want to understand internals of how MonetDB chooses to materialize and how it represents intermediate results.
As I understand, table data and intermediate results in MonetDB are all stored as BATs, but I?m not sure if there are some special optimizations on BATs, rather than just array(s) of atoms. Here are two simple examples to elaborate my questions a little bit:

?? ?1.Given two tables,?Items(id, order, price, tax), and?Orders(id, discount).
?? ? ? ? ?Items
?? ? ? ? ? ? ? ? ?[I1, I2, I3, I4, I5, I6, I7, I8, I9, I10, I11...]
?? ? ? ? ? ? ? ? ?[O3, O3, O3, O3, O3, O3, O3, O3, O3, O3, O4...]
?? ? ? ? ? ? ? ? ?[10, 11, 20, 15, 110, 80, 90, 12, 13, 88, 30...]
?? ? ? ? ? ? ? ? ?[0.1, 0.15, 0, 0.1, 0.1, 0.15, 0.11, 0.18, 0.10, 0.15, 0.20, ...]
?? ? ? ? ?Orders
?? ? ? ? ? ? ? ? ?[O1, O2, O3]
?? ? ? ? ? ? ? ? ?[0.8, 0.9, 0.95]

?? ? ? ? ?When we perform a join between these two tables on the field `order` (one-to-many relationship represented by the primary-foreign key, O3 matches with [I1, I2, ?, I10]),?in the output intermediate BATs of the join, will MonetDB duplicate `O3` 10 times? Are there any optimizations currently in MonetDB to remove/reduce the duplication?

?? ?2. Given a table?item(id, price, tax), when MonetDB performs the filter `price<10` on the table,?will MonetDB actually copy all matched tuples into intermediate BATs? Or just keep a list of OIDs of matched tuples as references?

Thanks!

Best,
Guodong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.monetdb.org/pipermail/users-list/attachments/20200920/19b6b41c/attachment-0001.htm>

------------------------------

Message: 2
Date: Mon, 21 Sep 2020 07:40:37 +0200
From: Martin Kersten <martin.kersten@cwi.nl>
To: users-list@monetdb.org
Subject: Re: Internals of intermediate results in MonetDB
Message-ID: <8c640600-9f14-a752-422e-3ec920e7fcb0@cwi.nl>
Content-Type: text/plain; charset=windows-1252; format=flowed

Hi

On 21/09/2020 05:31, Guodong Jin wrote:
Hi all,

Thanks for your hard works on this great open source project.
I?m a student newly working on MonetDB to implement some research ideas,

For a PhD? or semester task?

and I want to understand internals of how MonetDB chooses to materialize and how it represents intermediate results.
As I understand, table data and intermediate results in MonetDB are all stored as BATs, but I?m not sure if there are some special optimizations on BATs, rather than just array(s) of atoms. Here are two simple examples to elaborate my questions a little bit:

?? ?1.Given two tables, /Items(id, order, price, tax)/, and /Orders(id, discount)/.
?? ? ? ? ?Items
?? ? ? ? ? ? ? ? ?[I1, I2, I3, I4, I5, I6, I7, I8, I9, I10, I11...]
?? ? ? ? ? ? ? ? ?[O3, O3, O3, O3, O3, O3, O3, O3, O3, O3, O4...]
?? ? ? ? ? ? ? ? ?[10, 11, 20, 15, 110, 80, 90, 12, 13, 88, 30...]
?? ? ? ? ? ? ? ? ?[0.1, 0.15, 0, 0.1, 0.1, 0.15, 0.11, 0.18, 0.10, 0.15, 0.20, ...]
?? ? ? ? ?Orders
?? ? ? ? ? ? ? ? ?[O1, O2, O3]
?? ? ? ? ? ? ? ? ?[0.8, 0.9, 0.95]

When we perform a join between these two tables on the field `/order/` (one-to-many relationship represented by the primary-foreign key, O3 matches with [I1, I2, ?, I10]), *in the output intermediate BATs of the join, will MonetDB duplicate `O3` 10 times?
Have a look at the result of EXPLAIN SELECT * FROM Items JOIN Orders to view the MAL program to learn
the intermediate types.
The join result is a collection of OID-pairs

Are there any optimizations currently in MonetDB to remove/reduce the duplication?*

?? ?2. Given a table /item(id, price, tax)/, when MonetDB performs the filter `price<10` on the table, *will MonetDB actually copy all matched tuples into intermediate BATs? Or just keep a list of OIDs of matched tuples as references?*
See MAL plan, a list of oids


Thanks!

Best,
Guodong

_______________________________________________
users-list mailing list
users-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/users-list



------------------------------

Subject: Digest Footer

_______________________________________________
users-list mailing list
users-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/users-list


------------------------------

End of users-list Digest, Vol 97, Issue 11
******************************************