[MonetDB-users] underlying storage
What is the recommended storage for MonetDB? We are planning to buy a Sun storage appliance and were wondering if it would work with MonetDB
On Fri, 31 Jul 2009, Mag Gam wrote:
What is the recommended storage for MonetDB? We are planning to buy a Sun storage appliance and were wondering if it would work with MonetDB
Yup done that :) If you go for the iSCSI based storage think very careful about the choose filesystem. It really matters.
Since your network cable mostlikely will be your bottleneck and if more clients are an issue. Think about some efficient way to get the appliance to the switch if you go beyond the 'standard' 4xGe
Stefan
Interesting, what filesystem have you used?
We are planning to use a simple NFS filesystem. Would that have any problems?
On Fri, Jul 31, 2009 at 8:03 PM, Stefan de Koninkstefan@konink.de wrote:
On Fri, 31 Jul 2009, Mag Gam wrote:
What is the recommended storage for MonetDB? We are planning to buy a Sun storage appliance and were wondering if it would work with MonetDB
Yup done that :) If you go for the iSCSI based storage think very careful about the choose filesystem. It really matters.
Since your network cable mostlikely will be your bottleneck and if more clients are an issue. Think about some efficient way to get the appliance to the switch if you go beyond the 'standard' 4xGe
Stefan
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Fri, 31 Jul 2009, Mag Gam wrote:
Interesting, what filesystem have you used?
For benchmarking I know that EXT3 is not an option; lets say the OpenStorage sequential output hit so badly that it was 10 times as slow as the input. With 128GB of RAM that is 'kinda' odd.
We are planning to use a simple NFS filesystem. Would that have any problems?
I would only use NFS in a shared storage, envirionment not to support databases, serious locking issues and mmaping (like MonetDB does) from NFS is asking for out of memory conditions on Linux (Xen does something in the same way for images, this is not what you want).
If you are just going to support your database from storage and you do not share it. I would just create a volume export that over iSCSI and go XFS over it on Linux. That would also allow snapshotting in an easier way than NFS would do.
Stefan
Can you explain to me more about this mmaping? I was really going for the NFS solution.
On Sat, Aug 1, 2009 at 1:24 AM, Stefan de Koninkstefan@konink.de wrote:
On Fri, 31 Jul 2009, Mag Gam wrote:
Interesting, what filesystem have you used?
For benchmarking I know that EXT3 is not an option; lets say the OpenStorage sequential output hit so badly that it was 10 times as slow as the input. With 128GB of RAM that is 'kinda' odd.
We are planning to use a simple NFS filesystem. Would that have any problems?
I would only use NFS in a shared storage, envirionment not to support databases, serious locking issues and mmaping (like MonetDB does) from NFS is asking for out of memory conditions on Linux (Xen does something in the same way for images, this is not what you want).
If you are just going to support your database from storage and you do not share it. I would just create a volume export that over iSCSI and go XFS over it on Linux. That would also allow snapshotting in an easier way than NFS would do.
Stefan
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Mag Gam wrote:
Can you explain to me more about this mmaping? I was really going for the NFS solution.
http://en.wikipedia.org/wiki/Memory_mapped_files
On Sat, Aug 1, 2009 at 1:24 AM, Stefan de Koninkstefan@konink.de wrote:
On Fri, 31 Jul 2009, Mag Gam wrote:
Interesting, what filesystem have you used?
For benchmarking I know that EXT3 is not an option; lets say the OpenStorage sequential output hit so badly that it was 10 times as slow as the input. With 128GB of RAM that is 'kinda' odd.
We are planning to use a simple NFS filesystem. Would that have any problems?
I would only use NFS in a shared storage, envirionment not to support databases, serious locking issues and mmaping (like MonetDB does) from NFS is asking for out of memory conditions on Linux (Xen does something in the same way for images, this is not what you want).
If you are just going to support your database from storage and you do not share it. I would just create a volume export that over iSCSI and go XFS over it on Linux. That would also allow snapshotting in an easier way than NFS would do.
Stefan
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On 01-08-2009 13:12:51 +0200, Martin Kersten wrote:
Mag Gam wrote:
Can you explain to me more about this mmaping? I was really going for the NFS solution.
also, google results like these may be interesting for you: http://osdir.com/ml/linux.nfs/2002-12/msg00062.html
On Sat, 1 Aug 2009, Martin Kersten wrote:
Mag Gam wrote:
Can you explain to me more about this mmaping? I was really going for the NFS solution.
Linux overcommits memory in a way that you can get an out of memory situation while *caching* big files from NFS. Memory mapping would talk to this cache aswell. If for what reason what so ever you get bigger files than your total amount of memory in your client system, this goes boom.
So please test it for yourself :) this was my experience with Xen over NFS with direct mmaping :)
Stefan
participants (4)
-
Fabian Groffen -
Mag Gam -
Martin Kersten -
Stefan de Konink