Skip to main content

MonetDB embraces Node.js

As the Node.js community is growing stronger, and we have started to experiment with the framework ourselves, we deem it useful for Node.js users to have a native MonetDB connector. Using MonetDB as the data store for your analytical web app can improve your user appreciation significantly.

This is the current list of Node.js modules developed by the MonetDB team:

  • monetdb: The MonetDB driver for Node.js. The package defines a MAPI (the MonetDB communication protocol) connection, which allows easy communication between a Node.js application and a running MonetDB server.  It comes with native Q support, which allows calls to be wrapped in a promise. This way errors or slow responses can be handled in a non-blocking fashion. The MonetDB Node.js driver lets you use the native connector and Q-based APIs interchangeably.(installer & documentation: monetdb @ NPM, source code: monetdb @ hg)
  • connect-monetdb: A straightforward MonetDB session store for Connect/Express. Designed to run on top of a connection to a MonetDB server established with the Node.js driver. (Installer & documentation: connect-monetdb @ NPM, source code: connect-monetdb @ GitHub)
  • monetdb-import: This module provides a straightforward API for loading data files into MonetDB. Given a delimited text file, the module figures out its details, creates appropriate tables for it inside your MonetDB database and then loads the data. It literally does everything for you. Alternatively, it can be used for interactive data loading. You can call a sniffing function iteratively, until you are happy with the result and then use the import function to finish the data loading process. (Installer & documentation: monetdb-import @ NPM, source code: monetdb-import @ GitHub)
  • csv-sniffer: A delimited file sniffer. This module takes a sample of CSV text and tries to guess newline string, column delimiter, quote character, and whether or not the first row in the file contains labels. It was used in the monetdb-import module, now available for everybody as a separate module. (Installer & documentation: csv-sniffer @ NPM, source code: csv-sniffer @ GitHub)

Why use Node.js ?

Using a bit of JavaScript is unavoidable for most modern, dynamic websites. Node.js allows for writing client and server code in a single language - JavaScript.  You spend less time switching programming paradigms. Moreover,  client and server code can be easily shared, reducing code duplication and improving code maintenance.

Contrary to the usual multi-threaded programming model, Node.js builds on an event-driven architecture with a non-blocking I/O API. It is the perfect solution for web apps with multiple task running simultaneously in your app, such as real-time and data streaming applications.

Last, but not least, the extensive Node.js community provides a software ecosystem with a great selection of high-quality modules. Since the introduction of npm (the package manager for Node.js) and npmjs.org, package handling and dependency management are even more easier. This all makes working with Node.js a pleasant experience, even for programmers new to the platform.

Stream processing in the Cloud

On AWS you can do even more with Node.js. For example, you can stream-process data on Elastic MapReduce with a bit of Node.js and the Hadoop Streaming parallel processing architecture [1]. The new AWS Lambda service makes it even easier to build applications that respond quickly to new data [2]. The MonetDB/Node.js combination is a perfect starting point for such a setup, where continuous queries are sent to the database. A read-optimised analytical database management system delivers great query performance in these scenarios [3], as very little of the data that passes through the pipeline is actually persistent. To help you with such setups, MonetDB will soon be available as an AMI in the AWS Marketplace.

Availability

The Node.js connector is the latest in the set of language bindings for MonetDB. We have previously made available ODBC/JDBC, Python, Ruby, Perl, PHP and R connectors. All of our current MonetDB modules are available on npmjs.org, contributing back to the community. The source code of all modules is openly available under the MonetDB Public Licence terms. Since the Node.js driver is currently part of the MonetDB core packages, its source is hosted on our own open-access Mercurial repository. You can find all other modules on GitHub. Each module comes with instructions on npmjs.org and a small example on how to use it.

Showcase

We are looking forward to your application and how that can challenge our database system. We are building a new showcase for applications and extensions for MonetDB, not contributed by the MonetDB core team. If you are interested in being featured on our showcase, drop us a line. Even if your applications are not written in Node.js, as long as they contribute to the MonetDB ecosystem.

Links

[1] Node.js Streaming MapReduce with Amazon EMR

[2] AWS Lambda FAQs

[3] MonetDB/DataCell demo