If you want to model an NxM matrix of elements of type MYTYPE in a DBMS
(not just MonetDB), you would typically have a table with 3 columns:
CREATE TABLE my_matrix (
"row" INTEGER NOT NULL,
"col" INTEGER NOT NULL,
"cell_value" MYTYPE NOT NULL,
PRIMARY KEY ("row", "col")
);
of length N x M.
Anyways, I'm not the expert on Arrays in our group so I'll let others
chime in.
Eyal
On 10/10/2018 13:01, Oxana Dolmatova wrote:
> Hello,
>
> Thank you for all the replies and information!
> The version is 11.23, plus I'm using the cloud platform with 4cpu-16ram-hpc.
> The number of rows was just 1K for 15K attributes in a table.
> I understand that MonetDB was made with no intention to create and
> process tables with that many columns. But I'm interested in matrix
> operations integration, so I would like to know what is the main
> bottleneck to process such data (matrix-like sized) in the system.
>
> Best regards,
> Oksana
>
>
> On Tue, 9 Oct 2018 at 21:02, Eyal Rozenberg <eyalroz@technion.ac.il
> <mailto:eyalroz@technion.ac.il>> wrote:
>
> I'll add two observations to what Zhang Ying said.
>
> 1. Internally in MonetDB, the index of a column (or a BAT) in MonetDB's
> BBP (BAT Buffer Pool) is a signed 32-bit integer (on most platforms), so
> there isn't a theoretical problem having more than 15,000 columns; but
> MonetDB was not designed with that many tables in mind. The "long
> dimension" is expected to be the length of columns.
>
> 2. When you have as many as 15,000 columns, you're not simply finding
> more attributes of the same items. So, for example, a person has a first
> name, last name, year of birth, etc. etc. - but after several dozens of
> these you're probably adding columns such as "data relevant to this
> person at point X", "data relevant to this person at point Y" and so on.
>
> In your case, I'm guessing you have a lot of points and hence a lot of
> columns with data for your different items. What you should consider
> doing instead is having an extra "coordinate" column. Thus, in the
> example I gave, I would drop most of the columns, in favor of a second
> table named "person_data". This table's columns would be "person_id" (or
> some other key columns), "sample_point" and "value". Thus instead of M
> columns and N records you would have N x M records with 3 columns.
>
> Eyal
>
>
>
> On 09/10/2018 16:50, Ying Zhang wrote:
> > Hai Oksana,
> >
> > A.f.a.I.k., there is no explicit limit on the #columns you can put
> in a relation (did you mean “table”?). The reason that mserver5
> stopped responding is probably because it is busy with I/O.
> >
> > A long time ago, I once tried to create 1M tables, each with 3
> columns. After about a day or something, MonetDB has created ~400K
> tables, and I could hear my HDD being constantly busy. MonetDB was
> still working, but that was already a sufficient evidence that that
> idea wasn’t going to work.
> >
> > So the question is actually what do you want to achieve?
> >
> > Regards,
> > Jennie
> >
> >
> >> On 9 Oct 2018, at 13:43, Oxana Dolmatova
> <oxana.dolmatova@gmail.com <mailto:oxana.dolmatova@gmail.com>> wrote:
> >>
> >> Hello,
> >>
> >> Could you tell the limitation for number of columns in a relation
> and the main reason for that limitation. So far I could create
> relations with up to 15000 columns. When the number of columns is
> bigger mserver5 stops responding.
> >>
> >> Thank you and best regards,
> >> Oksana Dolmatova
> >> _______________________________________________
> >> users-list mailing list
> >> users-list@monetdb.org <mailto:users-list@monetdb.org>
> >> https://www.monetdb.org/mailman/listinfo/users-list
> >
> > _______________________________________________
> > users-list mailing list
> > users-list@monetdb.org <mailto:users-list@monetdb.org>
> > https://www.monetdb.org/mailman/listinfo/users-list
> >
>
>
>
> --
> Best regards,
> Oksana Dolmatova
>
> _______________________________________________
> users-list mailing list
> users-list@monetdb.org
> https://www.monetdb.org/mailman/listinfo/users-list
>