Failed to import table 'tier3_score_201807_part53', Failed to extend the BAT, perhaps disk full
Hello,
We are hitting the below error while loading data into table, There is no space crunch or memory (resource crunch). Please let us know if there are any restrictions in loading data.
sql>COPY INTO "DM_POS_PROSCORES_MRT_P1".tier3_score_201807_part53 FROM '/backup8/schema_backups/tier3_score_201807_part53'; Failed to import table 'tier3_score_201807_part53', Failed to extend the BAT, perhaps disk full
[monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd version MonetDB Database Server v1.7 (Jul2017-SP4) You have mail in /var/spool/mail/monetadmin [monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdb version MonetDB Database Server Toolkit v1.1 (Jul2017-SP4) [monetadmin@lnx1538 monet_data01]$ mclient -version mclient, the MonetDB interactive terminal (Jul2017-SP4) support for command-line editing compiled-in character encoding: UTF-8 [monetadmin@lnx1538 monet_data01]$ df -hP /monet_data01 Filesystem Size Used Avail Use% Mounted on /dev/mapper/encrmonet_data01lv 12T 7.0T 4.4T 62% /monet_data01 [monetadmin@lnx1538 monet_data01]$ ps -ef|grep -i monetdbd monetad+ 331928 1 0 Nov02 ? 00:00:14 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_TEST01 monetad+ 333365 332892 0 10:57 pts/9 00:00:00 grep --color=auto -i monetdbd monetad+ 405353 1 0 Oct31 ? 00:00:30 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_PROSCORE_P [monetadmin@lnx1538 monet_data01]$
[monetadmin@lnx1538 monet_data01]$ free -g total used free shared buff/cache available Mem: 1007 18 302 7 685 978 Swap: 31 0 31 [monetadmin@lnx1538 monet_data01]$
Thank You, Gautham
Hello, There is no limitation on copy into other than the underlying system resources. How many records are you planning to COPY into and what is the expected final size of the table?
On 4 Nov 2018, at 18:00, Murthy, Gautham Gautham.Murthy@harman.com wrote:
Hello,
We are hitting the below error while loading data into table, There is no space crunch or memory (resource crunch). Please let us know if there are any restrictions in loading data.
sql>COPY INTO "DM_POS_PROSCORES_MRT_P1".tier3_score_201807_part53 FROM '/backup8/schema_backups/tier3_score_201807_part53'; Failed to import table 'tier3_score_201807_part53', Failed to extend the BAT, perhaps disk full
[monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd version MonetDB Database Server v1.7 (Jul2017-SP4) You have mail in /var/spool/mail/monetadmin [monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdb version MonetDB Database Server Toolkit v1.1 (Jul2017-SP4) [monetadmin@lnx1538 monet_data01]$ mclient -version mclient, the MonetDB interactive terminal (Jul2017-SP4) support for command-line editing compiled-in character encoding: UTF-8 [monetadmin@lnx1538 monet_data01]$ df -hP /monet_data01 Filesystem Size Used Avail Use% Mounted on /dev/mapper/encrmonet_data01lv 12T 7.0T 4.4T 62% /monet_data01 [monetadmin@lnx1538 monet_data01]$ ps -ef|grep -i monetdbd monetad+ 331928 1 0 Nov02 ? 00:00:14 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_TEST01 monetad+ 333365 332892 0 10:57 pts/9 00:00:00 grep --color=auto -i monetdbd monetad+ 405353 1 0 Oct31 ? 00:00:30 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_PROSCORE_P [monetadmin@lnx1538 monet_data01]$
[monetadmin@lnx1538 monet_data01]$ free -g total used free shared buff/cache available Mem: 1007 18 302 7 685 978 Swap: 31 0 31 [monetadmin@lnx1538 monet_data01]$
Thank You, Gautham _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Hello Martin, Rows – 122520330, size 46GB – We were able to load it on a different database. User has been loading close to 7TB of data from past 2 days, he has loaded 600+ tables and he was left with 6 more tables to be loaded when he started hitting the issue. Do we need to try restarting the database ? Are there any other files which are not visible on the file system. Memory mapped files ? df reports we still have enough space.
+---------------------------+-----------+-------+------+ | table | count | mb | gb | +===========================+===========+=======+======+ | tier3_score_201807_part53 | 122520330 | 47672 | 46 | +---------------------------+-----------+-------+------+
Thank You, Gautham
From: users-list users-list-bounces+gautham.murthy=harman.com@monetdb.org On Behalf Of Martin Kersten Sent: Sunday, 04 November, 2018 11:02 PM To: Communication channel for MonetDB users users-list@monetdb.org Subject: [EXTERNAL] Re: Failed to import table 'tier3_score_201807_part53', Failed to extend the BAT, perhaps disk full
Hello, There is no limitation on copy into other than the underlying system resources. How many records are you planning to COPY into and what is the expected final size of the table?
On 4 Nov 2018, at 18:00, Murthy, Gautham <Gautham.Murthy@harman.commailto:Gautham.Murthy@harman.com> wrote: Hello,
We are hitting the below error while loading data into table, There is no space crunch or memory (resource crunch). Please let us know if there are any restrictions in loading data.
sql>COPY INTO "DM_POS_PROSCORES_MRT_P1".tier3_score_201807_part53 FROM '/backup8/schema_backups/tier3_score_201807_part53'; Failed to import table 'tier3_score_201807_part53', Failed to extend the BAT, perhaps disk full
[monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd version MonetDB Database Server v1.7 (Jul2017-SP4) You have mail in /var/spool/mail/monetadmin [monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdb version MonetDB Database Server Toolkit v1.1 (Jul2017-SP4) [monetadmin@lnx1538 monet_data01]$ mclient -version mclient, the MonetDB interactive terminal (Jul2017-SP4) support for command-line editing compiled-in character encoding: UTF-8 [monetadmin@lnx1538 monet_data01]$ df -hP /monet_data01 Filesystem Size Used Avail Use% Mounted on /dev/mapper/encrmonet_data01lv 12T 7.0T 4.4T 62% /monet_data01 [monetadmin@lnx1538 monet_data01]$ ps -ef|grep -i monetdbd monetad+ 331928 1 0 Nov02 ? 00:00:14 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_TEST01 monetad+ 333365 332892 0 10:57 pts/9 00:00:00 grep --color=auto -i monetdbd monetad+ 405353 1 0 Oct31 ? 00:00:30 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_PROSCORE_P [monetadmin@lnx1538 monet_data01]$
[monetadmin@lnx1538 monet_data01]$ free -g total used free shared buff/cache available Mem: 1007 18 302 7 685 978 Swap: 31 0 31 [monetadmin@lnx1538 monet_data01]$
Thank You, Gautham _______________________________________________ users-list mailing list users-list@monetdb.orgmailto:users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-listhttps://clicktime.symantec.com/a/1/MRRSRGwdDf-g_rXhFl1YoywULgEXfLcgeqNMc_sdFpg=?d=cRUZwZH_7LuUPJN6PBcnluQfdskJ98DAum0-j35rh2dCX4-MBXV-evtn0yIfkF12nP-PfjYGyR4_LofdHKzZOdaoGgT1vzim7oMVsKz_3IcGFWzTs2Oqc3tbYql3W0tjkRimcnNIigAs_BKxD72YqjFbUwHXWI60XJpAFeq_0Z0YT5rnj_OYE801qnPwlOREtvYrFIrf5VsyUOimsHbFXLeRzcVsjve5o6jsoVZjlemPS6bxDYHbZjVosZuxRj3nN0ptmmDuL9a-OwVxkSeJrGox1CTlW09WTqp-mJxKgdQJ9_Nqnkn8tVgjKWUBsdW1r2wpAp9LNpi1DkV8kvaDD-ufgjvzbdxO6MkRzzNcb4lR48Dpr_Pf7kCmltdiHwOFuhC_hmllIemleNL1eoM4tZyn1aSIwnpeVvpvmqojb0CV9WUp0c7yfqxcTrlAD8vNAfDACyaWt-T_ki2c6VAcAV99nYWNT00E4nOJK2LgtYHcYibHDlk_ByTM1IE%3D&u=https%3A%2F%2Fwww.monetdb.org%2Fmailman%2Flistinfo%2Fusers-list
If you know the number of records, then that should (!) appear in the COPY INTO command, otherwise MonetDB can not guess and will repeatedbly grow the files, possibly by requesting a larger one. You should also use the LOCKED mode to speedup.
On 04/11/2018 18:41, Murthy, Gautham wrote:
Hello Martin,
Rows – 122520330, size 46GB – We were able to load it on a different database.
User has been loading close to 7TB of data from past 2 days, he has loaded 600+ tables and he was left with 6 more tables to be loaded when he started hitting the issue.
Do we need to try restarting the database ?
Are there any other files which are not visible on the file system. Memory mapped files ?
df reports we still have enough space.
+---------------------------+-----------+-------+------+
| table | count | mb | gb |
+===========================+===========+=======+======+
| tier3_score_201807_part53 | 122520330 | 47672 | 46 |
+---------------------------+-----------+-------+------+
Thank You,
Gautham
*From:* users-list users-list-bounces+gautham.murthy=harman.com@monetdb.org *On Behalf Of *Martin Kersten *Sent:* Sunday, 04 November, 2018 11:02 PM *To:* Communication channel for MonetDB users users-list@monetdb.org *Subject:* [EXTERNAL] Re: Failed to import table 'tier3_score_201807_part53', Failed to extend the BAT, perhaps disk full
Hello,
There is no limitation on copy into other than the underlying system resources. How many records are you planning to COPY into and what is the expected final size of the table?
On 4 Nov 2018, at 18:00, Murthy, Gautham <Gautham.Murthy@harman.com mailto:Gautham.Murthy@harman.com> wrote:
Hello, We are hitting the below error while loading data into table, There is no space crunch or memory (resource crunch). Please let us know if there are any restrictions in loading data. sql>COPY INTO "DM_POS_PROSCORES_MRT_P1".tier3_score_201807_part53 FROM '/backup8/schema_backups/tier3_score_201807_part53'; Failed to import table 'tier3_score_201807_part53', Failed to extend the BAT, perhaps disk full [monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd version MonetDB Database Server v1.7 (Jul2017-SP4) You have mail in /var/spool/mail/monetadmin [monetadmin@lnx1538 monet_data01]$ /monet_binaries/MonetDB-11.27.13_PY/bin/monetdb version MonetDB Database Server Toolkit v1.1 (Jul2017-SP4) [monetadmin@lnx1538 monet_data01]$ mclient -version mclient, the MonetDB interactive terminal (Jul2017-SP4) support for command-line editing compiled-in character encoding: UTF-8 [monetadmin@lnx1538 monet_data01]$ df -hP /monet_data01 Filesystem Size Used Avail Use% Mounted on /dev/mapper/encrmonet_data01lv 12T 7.0T 4.4T 62% /monet_data01 [monetadmin@lnx1538 monet_data01]$ ps -ef|grep -i monetdbd monetad+ 331928 1 0 Nov02 ? 00:00:14 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_TEST01 monetad+ 333365 332892 0 10:57 pts/9 00:00:00 grep --color=auto -i monetdbd monetad+ 405353 1 0 Oct31 ? 00:00:30 /monet_binaries/MonetDB-11.27.13_PY/bin/monetdbd start /monet_data01/DBFARM_PROSCORE_P [monetadmin@lnx1538 monet_data01]$ [monetadmin@lnx1538 monet_data01]$ free -g total used free shared buff/cache available Mem: 1007 18 302 7 685 978 Swap: 31 0 31 [monetadmin@lnx1538 monet_data01]$ Thank You, Gautham _______________________________________________ users-list mailing list users-list@monetdb.org <mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list <https://clicktime.symantec.com/a/1/MRRSRGwdDf-g_rXhFl1YoywULgEXfLcgeqNMc_sdFpg=?d=cRUZwZH_7LuUPJN6PBcnluQfdskJ98DAum0-j35rh2dCX4-MBXV-evtn0yIfkF12nP-PfjYGyR4_LofdHKzZOdaoGgT1vzim7oMVsKz_3IcGFWzTs2Oqc3tbYql3W0tjkRimcnNIigAs_BKxD72YqjFbUwHXWI60XJpAFeq_0Z0YT5rnj_OYE801qnPwlOREtvYrFIrf5VsyUOimsHbFXLeRzcVsjve5o6jsoVZjlemPS6bxDYHbZjVosZuxRj3nN0ptmmDuL9a-OwVxkSeJrGox1CTlW09WTqp-mJxKgdQJ9_Nqnkn8tVgjKWUBsdW1r2wpAp9LNpi1DkV8kvaDD-ufgjvzbdxO6MkRzzNcb4lR48Dpr_Pf7kCmltdiHwOFuhC_hmllIemleNL1eoM4tZyn1aSIwnpeVvpvmqojb0CV9WUp0c7yfqxcTrlAD8vNAfDACyaWt-T_ki2c6VAcAV99nYWNT00E4nOJK2LgtYHcYibHDlk_ByTM1IE%3D&u=https%3A%2F%2Fwww.monetdb.org%2Fmailman%2Flistinfo%2Fusers-list>
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
participants (3)
-
Martin Kersten -
Martin Kersten -
Murthy, Gautham