Hi all:
Monet version 11.35.19 (Nov2019-SP3)
We realized that json columns in any table produces bigger stored disk, no matter how many rows has big data. It seems like depends on rows with different values in same json column: if many rows has different values, then the space on disk increases exponentially. So we can't use jaon columns in big tables, since we run out of space disk in short time. (also tested with 11.43.5 -Jan2022, same issue)
Is there any solution to this problem? Tanks! Alberto
Alberto
Do you have some script which replicates this issue? And could you post the issue on the github issues?
Niels
On Thu, Sep 22, 2022 at 07:57:07AM -0300, Alberto Ferrari wrote:
Hi all:
Monet version 11.35.19 (Nov2019-SP3)
We realized that json columns in any table produces bigger stored disk, no matter how many rows has big data. It seems like depends on rows with different values in same json column: if many rows has different values, then the space on disk increases exponentially. So we can't use jaon columns in big tables, since we run out of space disk in short time. (also tested with 11.43.5 -Jan2022, same issue)
Is there any solution to this problem? Tanks! Alberto _______________________________________________ users-list mailing list -- users-list@monetdb.org To unsubscribe send an email to users-list-leave@monetdb.org
Niels, thanks for reply. I will try to create an example. regards
El jue, 22 sept 2022 a la(s) 10:35, Niels Nes (niels.nes@monetdbsolutions.com) escribió:
Alberto
Do you have some script which replicates this issue? And could you post the issue on the github issues?
Niels
On Thu, Sep 22, 2022 at 07:57:07AM -0300, Alberto Ferrari wrote:
Hi all:
Monet version 11.35.19 (Nov2019-SP3)
We realized that json columns in any table produces bigger stored disk, no matter how many rows has big data. It seems like depends on rows with different values in same json column: if many rows has different values, then the space on disk increases exponentially. So we can't use jaon columns in big tables, since we run out of space disk in short time. (also tested with 11.43.5 -Jan2022, same issue)
Is there any solution to this problem? Tanks! Alberto _______________________________________________ users-list mailing list -- users-list@monetdb.org To unsubscribe send an email to users-list-leave@monetdb.org
users-list mailing list -- users-list@monetdb.org To unsubscribe send an email to users-list-leave@monetdb.org
Niels, It's hard to me to create a demo script, but at leats I'd want to know if it's a known "issue" (maybe it's not an issue, just as Monet works internally)
I can tell you that a table with 100 columns (ints, varchars, dates, etc) with 30 million records, may occupy 30 Gb of disk. When I merge some of these columns into one new json column -(i.e. 5 single columns into one object like {col1:value, col2:value...}- and dropped the 5 old columns (already merged in json), the table now grows upto 100 Gb of disk.
Thanks
El jue, 22 sept 2022 a la(s) 10:35, Niels Nes (niels.nes@monetdbsolutions.com) escribió:
Alberto
Do you have some script which replicates this issue? And could you post the issue on the github issues?
Niels
On Thu, Sep 22, 2022 at 07:57:07AM -0300, Alberto Ferrari wrote:
Hi all:
Monet version 11.35.19 (Nov2019-SP3)
We realized that json columns in any table produces bigger stored disk, no matter how many rows has big data. It seems like depends on rows with different values in same json column: if many rows has different values, then the space on disk increases exponentially. So we can't use jaon columns in big tables, since we run out of space disk in short time. (also tested with 11.43.5 -Jan2022, same issue)
Is there any solution to this problem? Tanks! Alberto _______________________________________________ users-list mailing list -- users-list@monetdb.org To unsubscribe send an email to users-list-leave@monetdb.org
users-list mailing list -- users-list@monetdb.org To unsubscribe send an email to users-list-leave@monetdb.org
On Wed, Sep 28, 2022 at 03:17:49PM -0300, Alberto Ferrari wrote:
Niels, It's hard to me to create a demo script, but at leats I'd want to know if it's a known "issue" (maybe it's not an issue, just as Monet works internally)
I can tell you that a table with 100 columns (ints, varchars, dates, etc) with 30 million records, may occupy 30 Gb of disk. When I merge some of these columns into one new json column -(i.e. 5 single columns into one object like {col1:value, col2:value...}- and dropped the 5 old columns (already merged in json), the table now grows upto 100 Gb of disk.
So you convert some columns into a single json column which are encode using the format above. With 30gb growing to 100GB, you have 70GB extra (not even counting that the original columns are gone). So for 70Gb for 30M rows, gives about 2333 bytes per value. Even for your json explosion, that sounds very big.
Niels
Thanks
El jue, 22 sept 2022 a la(s) 10:35, Niels Nes (niels.nes@monetdbsolutions.com) escribió:
Alberto
Do you have some script which replicates this issue? And could you post the issue on the github issues?
Niels
On Thu, Sep 22, 2022 at 07:57:07AM -0300, Alberto Ferrari wrote:
Hi all:
Monet version 11.35.19 (Nov2019-SP3)
We realized that json columns in any table produces bigger stored disk, no matter how many rows has big data. It seems like depends on rows with different values in same json column: if many rows has different values, then the space on disk increases exponentially. So we can't use jaon columns in big tables, since we run out of space disk in short time. (also tested with 11.43.5 -Jan2022, same issue)
Is there any solution to this problem? Tanks! Alberto _______________________________________________ users-list mailing list -- users-list@monetdb.org To unsubscribe send an email to users-list-leave@monetdb.org
users-list mailing list -- users-list@monetdb.org To unsubscribe send an email to users-list-leave@monetdb.org
users-list mailing list -- users-list@monetdb.org To unsubscribe send an email to users-list-leave@monetdb.org
participants (2)
-
Alberto Ferrari -
Niels Nes