Connection Reset(mserver still alive?)
Hello,
I have recently started using MonetDB for a BI project, and it's doing pretty well. However, I receive a connection reset error each time I run an ETL process (I use Pentaho PDI) to populate my dimensional schema. This error is reported by the MonetDB JDBC driver
*Connection reset(mserver still alive?)*
This usually happens after a random amount of time, 30 min, 45 min, 1 hour, etc.
Any pointers related to this would be very helpful.
Specs ------- Ubuntu 12.04 8 GB Ram 500 GB Space Pentaho PDI (community edition) Monetdb5 (installed using apt-get)
Thank you.
Are you using the table output step or the MonetDB bulk loader step?
We experienced stability problems with MonetDB from time to time, especially when bulk loading to a MonetDB server that runs for days on end without being stopped and started again. That's in combination of dropping large tables, recreating and bulk loading or delete and reloads via bulk loading. It just seemed like the best in practice use for MonetDB was starting clean, creating tables, doing a large single load and making sure you restart the server at some point each day. Otherwise the ETL would lock up randomly or the server would stop responding to queries or annoyingly stop accepting connections from any new clients.
I'm interested in what the developers suggest as a way to nail down what's happening. It's not as simple as running a command and getting the wrong result, at least in my case. Hopefully you find a more easily reproducible problem.
Sent from my iPhone
On Jul 23, 2014, at 4:34 PM, Ali Salem sarafandee@gmail.com wrote:
Hello,
I have recently started using MonetDB for a BI project, and it's doing pretty well. However, I receive a connection reset error each time I run an ETL process (I use Pentaho PDI) to populate my dimensional schema. This error is reported by the MonetDB JDBC driver
Connection reset(mserver still alive?)
This usually happens after a random amount of time, 30 min, 45 min, 1 hour, etc.
Any pointers related to this would be very helpful.
Specs
Ubuntu 12.04 8 GB Ram 500 GB Space Pentaho PDI (community edition) Monetdb5 (installed using apt-get)
Thank you. _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Hello Brandon,
Thank you for your reply. I am using the Table Output step. I restarted the process many times, fresh install, fresh load, but the problem persisted. If I use mclient to load the data or copy into, I face no issues. Could this be a JDBC driver issue?
From: users-list [mailto:users-list-bounces+sarafandee=gmail.com@monetdb.org] On Behalf Of Brandon Jackson Sent: Wednesday, July 23, 2014 11:53 PM To: Communication channel for MonetDB users Subject: Re: Connection Reset(mserver still alive?)
Are you using the table output step or the MonetDB bulk loader step?
We experienced stability problems with MonetDB from time to time, especially when bulk loading to a MonetDB server that runs for days on end without being stopped and started again. That's in combination of dropping large tables, recreating and bulk loading or delete and reloads via bulk loading. It just seemed like the best in practice use for MonetDB was starting clean, creating tables, doing a large single load and making sure you restart the server at some point each day. Otherwise the ETL would lock up randomly or the server would stop responding to queries or annoyingly stop accepting connections from any new clients.
I'm interested in what the developers suggest as a way to nail down what's happening. It's not as simple as running a command and getting the wrong result, at least in my case. Hopefully you find a more easily reproducible problem.
Sent from my iPhone
On Jul 23, 2014, at 4:34 PM, Ali Salem <sarafandee@gmail.com mailto:sarafandee@gmail.com > wrote:
Hello,
I have recently started using MonetDB for a BI project, and it's doing pretty well. However, I receive a connection reset error each time I run an ETL process (I use Pentaho PDI) to populate my dimensional schema. This error is reported by the MonetDB JDBC driver
Connection reset(mserver still alive?)
This usually happens after a random amount of time, 30 min, 45 min, 1 hour, etc.
Any pointers related to this would be very helpful.
Specs
-------
Ubuntu 12.04
8 GB Ram
500 GB Space
Pentaho PDI (community edition)
Monetdb5 (installed using apt-get)
Thank you.
_______________________________________________ users-list mailing list users-list@monetdb.org mailto:users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
First i would recommend against using the table output step because it uses normal inserts to populate data which will be extremely slow on any Column store database in general.
The bulk loader is the way to go and amazingly fast.
The JDBC driver I would check to get the latest version. Not sure which version of PDI you have or the driver version that shipped with it. Pentaho historically has not kept up to dat with JDBC jars. So yes, it's possible that you have an old or bad jar file giving trouble.
Sent from my iPhone
On Jul 24, 2014, at 12:21 AM, "Juan Saloom" sarafandee@gmail.com wrote:
Hello Brandon,
Thank you for your reply. I am using the Table Output step. I restarted the process many times, fresh install, fresh load, but the problem persisted. If I use mclient to load the data or copy into, I face no issues. Could this be a JDBC driver issue?
From: users-list [mailto:users-list-bounces+sarafandee=gmail.com@monetdb.org] On Behalf Of Brandon Jackson Sent: Wednesday, July 23, 2014 11:53 PM To: Communication channel for MonetDB users Subject: Re: Connection Reset(mserver still alive?)
Are you using the table output step or the MonetDB bulk loader step?
We experienced stability problems with MonetDB from time to time, especially when bulk loading to a MonetDB server that runs for days on end without being stopped and started again. That's in combination of dropping large tables, recreating and bulk loading or delete and reloads via bulk loading. It just seemed like the best in practice use for MonetDB was starting clean, creating tables, doing a large single load and making sure you restart the server at some point each day. Otherwise the ETL would lock up randomly or the server would stop responding to queries or annoyingly stop accepting connections from any new clients.
I'm interested in what the developers suggest as a way to nail down what's happening. It's not as simple as running a command and getting the wrong result, at least in my case. Hopefully you find a more easily reproducible problem.
Sent from my iPhone
On Jul 23, 2014, at 4:34 PM, Ali Salem sarafandee@gmail.com wrote:
Hello,
I have recently started using MonetDB for a BI project, and it's doing pretty well. However, I receive a connection reset error each time I run an ETL process (I use Pentaho PDI) to populate my dimensional schema. This error is reported by the MonetDB JDBC driver
Connection reset(mserver still alive?)
This usually happens after a random amount of time, 30 min, 45 min, 1 hour, etc.
Any pointers related to this would be very helpful.
Specs
Ubuntu 12.04 8 GB Ram 500 GB Space Pentaho PDI (community edition) Monetdb5 (installed using apt-get)
Thank you. _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Hello Brandon,
I am using COPY INTO, and it's loading the data very quickly. However, it is always reporting with this error at the end:
*Incomplete record at end of file*
There is nothing wrong with the CSV file as I see it, since I tried exporting random number of records each time. Any idea what might be the problem?
Thank you.
On Thu, Jul 24, 2014 at 8:37 AM, Brandon Jackson usbrandon@gmail.com wrote:
First i would recommend against using the table output step because it uses normal inserts to populate data which will be extremely slow on any Column store database in general.
The bulk loader is the way to go and amazingly fast.
The JDBC driver I would check to get the latest version. Not sure which version of PDI you have or the driver version that shipped with it. Pentaho historically has not kept up to dat with JDBC jars. So yes, it's possible that you have an old or bad jar file giving trouble.
Sent from my iPhone
On Jul 24, 2014, at 12:21 AM, "Juan Saloom" sarafandee@gmail.com wrote:
Hello Brandon,
Thank you for your reply. I am using the Table Output step. I restarted the process many times, fresh install, fresh load, but the problem persisted. If I use *mclient* to load the data or *copy into*, I face no issues. Could this be a JDBC driver issue?
*From:* users-list [ mailto:users-list-bounces+sarafandee=gmail.com@monetdb.org users-list-bounces+sarafandee=gmail.com@monetdb.org] *On Behalf Of *Brandon Jackson *Sent:* Wednesday, July 23, 2014 11:53 PM *To:* Communication channel for MonetDB users *Subject:* Re: Connection Reset(mserver still alive?)
Are you using the table output step or the MonetDB bulk loader step?
We experienced stability problems with MonetDB from time to time, especially when bulk loading to a MonetDB server that runs for days on end without being stopped and started again. That's in combination of dropping large tables, recreating and bulk loading or delete and reloads via bulk loading. It just seemed like the best in practice use for MonetDB was starting clean, creating tables, doing a large single load and making sure you restart the server at some point each day. Otherwise the ETL would lock up randomly or the server would stop responding to queries or annoyingly stop accepting connections from any new clients.
I'm interested in what the developers suggest as a way to nail down what's happening. It's not as simple as running a command and getting the wrong result, at least in my case. Hopefully you find a more easily reproducible problem.
Sent from my iPhone
On Jul 23, 2014, at 4:34 PM, Ali Salem sarafandee@gmail.com wrote:
Hello,
I have recently started using MonetDB for a BI project, and it's doing pretty well. However, I receive a connection reset error each time I run an ETL process (I use Pentaho PDI) to populate my dimensional schema. This error is reported by the MonetDB JDBC driver
*Connection reset(mserver still alive?)*
This usually happens after a random amount of time, 30 min, 45 min, 1 hour, etc.
Any pointers related to this would be very helpful.
Specs
Ubuntu 12.04
8 GB Ram
500 GB Space
Pentaho PDI (community edition)
Monetdb5 (installed using apt-get)
Thank you.
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Perhaps your last line is not properly ended with a record separator (\n)
On 28/07/14 19:37, Ali Salem wrote:
Hello Brandon,
I am using COPY INTO, and it's loading the data very quickly. However, it is always reporting with this error at the end:
*Incomplete record at end of file*
There is nothing wrong with the CSV file as I see it, since I tried exporting random number of records each time. Any idea what might be the problem?
Thank you.
On Thu, Jul 24, 2014 at 8:37 AM, Brandon Jackson <usbrandon@gmail.com mailto:usbrandon@gmail.com> wrote:
First i would recommend against using the table output step because it uses normal inserts to populate data which will be extremely slow on any Column store database in general. The bulk loader is the way to go and amazingly fast. The JDBC driver I would check to get the latest version. Not sure which version of PDI you have or the driver version that shipped with it. Pentaho historically has not kept up to dat with JDBC jars. So yes, it's possible that you have an old or bad jar file giving trouble. Sent from my iPhone On Jul 24, 2014, at 12:21 AM, "Juan Saloom" <sarafandee@gmail.com <mailto:sarafandee@gmail.com>> wrote:Hello Brandon,____ __ __ Thank you for your reply. I am using the Table Output step. I restarted the process many times, fresh install, fresh load, but the problem persisted. If I use *mclient* to load the data or *copy into*, I face no issues. Could this be a JDBC driver issue?____ __ __ __ __ __ __ *From:*users-list [mailto:users-list-bounces+sarafandee=gmail.com@monetdb.org] *On Behalf Of *Brandon Jackson *Sent:* Wednesday, July 23, 2014 11:53 PM *To:* Communication channel for MonetDB users *Subject:* Re: Connection Reset(mserver still alive?)____ __ __ Are you using the table output step or the MonetDB bulk loader step?____ __ __ __ __ We experienced stability problems with MonetDB from time to time, especially when bulk loading to a MonetDB server that runs for days on end without being stopped and started again. That's in combination of dropping large tables, recreating and bulk loading or delete and reloads via bulk loading. It just seemed like the best in practice use for MonetDB was starting clean, creating tables, doing a large single load and making sure you restart the server at some point each day. Otherwise the ETL would lock up randomly or the server would stop responding to queries or annoyingly stop accepting connections from any new clients.____ __ __ I'm interested in what the developers suggest as a way to nail down what's happening. It's not as simple as running a command and getting the wrong result, at least in my case. Hopefully you find a more easily reproducible problem. Sent from my iPhone____ On Jul 23, 2014, at 4:34 PM, Ali Salem <sarafandee@gmail.com <mailto:sarafandee@gmail.com>> wrote:____ Hello,____ __ __ I have recently started using MonetDB for a BI project, and it's doing pretty well. However, I receive a connection reset error each time I run an ETL process (I use Pentaho PDI) to populate my dimensional schema. This error is reported by the MonetDB JDBC driver____ __ __ *Connection reset(mserver still alive?)*____ __ __ This usually happens after a random amount of time, 30 min, 45 min, 1 hour, etc.____ __ __ Any pointers related to this would be very helpful.____ __ __ __ __ Specs____ -------____ Ubuntu 12.04____ 8 GB Ram____ 500 GB Space____ Pentaho PDI (community edition)____ Monetdb5 (installed using apt-get)____ __ __ Thank you.____ _______________________________________________ users-list mailing list users-list@monetdb.org <mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list____ _______________________________________________ users-list mailing list users-list@monetdb.org <mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list_______________________________________________ users-list mailing list users-list@monetdb.org <mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Hello Martin,
Import is working fine now, I believe I had a problem with the export utility in the first place. For the previous error I was receiving "mserver still alive!", the only way I was able to solve it is by setting the forward parameter to redirect instead of proxy. Now although my MonetDB is installed locally, this seems to have played the trick. I just finished importing 500M records with no problems. Any idea why proxy was causing the process to die during import?
This is a wonderful DB, with a lot to offer. Thank you for your help.
On Mon, Jul 28, 2014 at 2:31 PM, Martin Kersten Martin.Kersten@cwi.nl wrote:
Perhaps your last line is not properly ended with a record separator (\n)
On 28/07/14 19:37, Ali Salem wrote:
Hello Brandon,
I am using COPY INTO, and it's loading the data very quickly. However, it is always reporting with this error at the end:
*Incomplete record at end of file*
There is nothing wrong with the CSV file as I see it, since I tried exporting random number of records each time. Any idea what might be the problem?
Thank you.
On Thu, Jul 24, 2014 at 8:37 AM, Brandon Jackson <usbrandon@gmail.com mailto:usbrandon@gmail.com> wrote:
First i would recommend against using the table output step because it uses normal inserts to populate data which will be extremely slow on any Column store database in general. The bulk loader is the way to go and amazingly fast. The JDBC driver I would check to get the latest version. Not sure which version of PDI you have or the driver version that shipped with it. Pentaho historically has not kept up to dat with JDBC jars. So yes, it's possible that you have an old or bad jar file giving trouble. Sent from my iPhone On Jul 24, 2014, at 12:21 AM, "Juan Saloom" <sarafandee@gmail.com <mailto:sarafandee@gmail.com>> wrote: Hello Brandon,______ __ Thank you for your reply. I am using the Table Output step. I restarted the process many times, fresh install, fresh load, but the problem persisted. If I use *mclient* to load the data or *copy into*, I face no issues. Could this be a JDBC driver issue?____ __ __ __ __ __ __ *From:*users-list [mailto:users-list-bounces+sarafandee=gmail.com@monetdb.org] *On Behalf Of *Brandon Jackson *Sent:* Wednesday, July 23, 2014 11:53 PM *To:* Communication channel for MonetDB users *Subject:* Re: Connection Reset(mserver still alive?)____ __ __ Are you using the table output step or the MonetDB bulk loader step?____ __ __ __ __ We experienced stability problems with MonetDB from time to time, especially when bulk loading to a MonetDB server that runs for days on end without being stopped and started again. That's in combination of dropping large tables, recreating and bulk loading or delete and reloads via bulk loading. It just seemed like the best in practice use for MonetDB was starting clean, creating tables, doing a large single load and making sure you restart the server at some point each day. Otherwise the ETL would lock up randomly or the server would stop responding to queries or annoyingly stop accepting connections from any new clients.____ __ __ I'm interested in what the developers suggest as a way to nail down what's happening. It's not as simple as running a command and getting the wrong result, at least in my case. Hopefully you find a more easily reproducible problem. Sent from my iPhone____ On Jul 23, 2014, at 4:34 PM, Ali Salem <sarafandee@gmail.com <mailto:sarafandee@gmail.com>> wrote:____ Hello,____ __ __ I have recently started using MonetDB for a BI project, and it's doing pretty well. However, I receive a connection reset error each time I run an ETL process (I use Pentaho PDI) to populate my dimensional schema. This error is reported by the MonetDB JDBC driver____ __ __ *Connection reset(mserver still alive?)*____ __ __ This usually happens after a random amount of time, 30 min, 45 min, 1 hour, etc.____ __ __ Any pointers related to this would be very helpful.____ __ __ __ __ Specs____ -------____ Ubuntu 12.04____ 8 GB Ram____ 500 GB Space____ Pentaho PDI (community edition)____ Monetdb5 (installed using apt-get)____ __ __ Thank you.____ _______________________________________________ users-list mailing list users-list@monetdb.org <mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list____ _______________________________________________ users-list mailing list users-list@monetdb.org <mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list_______________________________________________ users-list mailing list users-list@monetdb.org <mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
participants (4)
-
Ali Salem -
Brandon Jackson -
Juan Saloom
-
Martin Kersten