Difference between revisions of "Main Page"

From MonetDB
Jump to navigationJump to search
(should use reliable paths; homes are under /home)
(examples are (even) more helpful, if they are correct ...)
Line 116: Line 116:
and directory locations involved, e.g. while on bricks09 you can move the file data to bricks10
and directory locations involved, e.g. while on bricks09 you can move the file data to bricks10
  scp data bricks10/scratch/USER
  scp data bricks10:/scratch/USER
You can also use the rsync command to clone your environment easily on multiple machines.
You can also use the rsync command to clone your environment easily on multiple machines.

Revision as of 15:50, 19 October 2015

What is this "wiki" thing?

If you're new to Wiki's, please read a brief explanation of what they are. You might then want to continue to this short introduction video.

We use this Wiki for easly-updatable, easily-expandable and user-editable MonetDB-related content, as well as for some coordination within our group.

Consult the MediaWiki User Guides page for detailed guides on using various aspects of MediaWiki (the platform on which this Wiki is built).

Getting started with MonetDB


Software development

Project related


SciLens cluster

The SciLens cluster has been acquired to provide a sizable and flexible experimentation platform for the Database Architectures group at CWI. Access is granted to members of the DA group.

SciLens policy

In case you are cooperating with non-DA-members that are (temporarily) granted access to our SciLens cluster (there are no non-DA people that have access to our SciLens cluster without cooperation with us!), please make sure to (1) inform them about our usage (reporting/claiming) policies (see above) and (2) introduce them and their work to the group

If you plan to use machines in our SciLens cluster please make sure you report your usage (plans) and claim machines the usual way via the webpages below. Please do not forget to release them, again, once you're done with using them! Do not hesitate to ask, in case you have any questions!

SciLens backup

The data on the SciLens machines is not backed-up by ITF. To ensure a continual use of your data, you either have to find a backup storage place within the cluster, your desktop, external, or be able to re-generate your settings. In case of doubt or advice where and how to make your backup, contact Arjen.

SciLens login

Your first login

Logging into the cluster is regulated through the scilens-ssh (virtual) machine. Initially, you _cannot_ access it; please ask our system administrator, Arjen de Rijke for access. With his help you'll obtain a special license file for use with the cluster, which you can place at $HOME/.ssh/id_rsa_scilens on your machine.

Thereafter you can SSH to the scilense-ssh gateway machine:

$ ssh -A -i $HOME/.ssh/id_rsa_scilens scilens-ssh.ins.cwi.nl 

and subsequently move to the specific machine you desire to use, e.g.

$ ssh bricks09

Automation of proxying

After first making sure you can ssh into the cluster via the gateway machine, consider adding the following to $HOME/.ssh/config:

Host scilens-ssh.ins.cwi.nl
        User your_username_here
Host bricks* rocks* pebbles* diamonds* stones*
        ProxyCommand ssh scilens-ssh.ins.cwi.nl -W %h:%p

You will now be able to issue an SSH command directly:

$ ssh bricks09

and the connection will be automatically proxied over an SSH connection to the gateway machine.


  • Remember to make $HOME/.ssh/config readable only by you if you've just created it.
  • This might conflict with other hostnames beginning with bricks, rocks or pebbles.

Disk usage on the cluster

When you login to one of the SciLens machines, you land in your home directory on that machine. But - try to avoid using this directory: It is mounted on a small disk, shared by all users, which is deleted when the operating system is reinstalled. Restrict yourself in using it to configuration settings files, symlinks etc.

For most practical cases, you should make yourself a directory in /scratch or /data where there should be ample of disk space.

SciLens data transport

The easiest way is to PULL small files using scp into the selected Scilens machine. For example,

scp vienna.ins.cwi.nl:/ufs/mk/.monetdb .

[Arjen update next] This requires the ssh configuration file for the scilens-ssh machine on your deskop to be set to work as a forwarding device.

scp vienna.ins.cwi.nl:/ufs/mk/.monetdb .

For example, .../.ssh/config looks like:

Host scilens-ssh.ins.cwi.nl
        User <USER>

Scilens local data transport

Copying files within the cluster is straightforward using the scp command identifying the machine and directory locations involved, e.g. while on bricks09 you can move the file data to bricks10

scp data bricks10:/scratch/USER

You can also use the rsync command to clone your environment easily on multiple machines. It requires a .rsync file in your homedirectory with directives. Create a rsync daemon configurationfile in your /scratch/USER directory containing:

port = 2873
use chroot = no
path = /scratch/USER

The port number is free to choose above 1000, otherwise conflicts with others users might occur. Let's assume again you are on brick09 and you want a synchronized copy on bricks10. On bricks09 create the above configuration file and start the rsync daemon with the command:

rsync --daemon --config=/scratch/USER/rsyncd.conf

Then login onto bricks10 in your /scratch/USER directory execute the command:

rsync -aH bricks09:/scratch/USER/* /scratch/USER/

For further details see the rsync manual or contact the expert. After the rsync is complete you should terminate the rsync daemon on bricks09.

Returning small files from the SciLens machine to your desktop can be performed using scp. For big ones contact the expert. Importing large amounts of data into the cluster, contact the expert.

Export data from the cluster

For small files on any of the machines, you first copy it to the scilens-ssh machine, whereafter you can pick it up from your desktop using scp.

For large exports we need to design a better solution.

Accessing the internet

You can use wget to access any web-source from within the cluster.

You have to contact the expert if you want to run a web-client/server setup on the machine. For this you need a ssh-tunnel setup.

Root specific features

Some tools have been added to make your life easier wrt performance analyse. They have to be called with the sudo command, example

sudo iotop ...

Pre-installed user requested packages

External software that requires root permissions can only be installed from source in your local environment. For example, postgresql, mysql and friends.

Non-installed libraries available within the Fedora distributions can be enabled by contacting the expert.

Specific application frameworks, e.g. java, can be installed from the Fedora repository, but due to versioning issues we advice to use a local copy as much as possible. When in doubt follow the expert route.

Work with multiple machines

With the command 'clush', one can execute the same command on multiple SciLens machines simultaneously. Assume you are logged in scilens-ssh, then run:

clush -w bricks[01-16]

This will give you a prompt. Any command you type in here, e.g., 'df -h /scratch', will be run on the machines bricks01, ..., bricks16. Try it out to see the results produced by the selected bricks machines.

The '-w' option allows you to pass a list of machines you want to work on.

In theorie, you can address all rocks machines with one clush command: clush -w rocks[001-144]. However, in practice, it is more advisable to work with smaller groups of machines, say 20. This also makes it easier to cancel a command if one of the machines freeze.

With 'quit' you can stop clush. For more information, please see the man page of clush.

SciLens cluster hardware

  • Non-standard hardware
    • diamonds
    • stones
    • bricks
    • rocks
    • pebbles

SciLens cluster use

We use below pages to register usage claims of SciLens cluster machines.

There is one page per month per machine class / tier.

When a new month starts, we create a new set of pages for this month, one per machine class / tier, from the respective template pages.

We can use the template pages to claim machines for usages that last more than a month.

Please do always claim machines before you start using them!

Please recall to release machines (also from the templates!) once you're done using them!

(In case you cannot access this wiki yourself, please ask your "host" or any DA member for help!)

Additional resources for messing around with this wiki