An open source solution provides a stepping stone for others to extend its kernel functionality with specific types and functions. Experience shows that the need for those are fairly limited. Often the use of the built-in data types, the MAL algebra and functional abstraction, provide the necessary toolkit to achieve your goal.
In the few cases where the MonetDB kernel and SQL runtime system needs extensions, it calls for access to the source code of MonetDB and proficiency in C-programming, compilation and debugging. The openess of MonetDB means that extensions are not sand-boxed; they run within the system address space. Moreover, the multi-layered architecture means you have to make the functions written in C known to the MAL interpreter, before they can be made known to the SQL compiler. The current setup makes this a little more cumbersome, but the added benefit is that both simple scalar functions and columnar operations can be introduced.
The following link, https://www.monetdb.org/hg/MonetDB-extend, leads to a Mercurial repository that contains detailed explanations, multiple READMEs and also code that already implements some functions, e.g., string reverse. Examples range from scalar functions to bulk and/or polymorphic operations.
sql> select 'hello',reverse('hello');
+---------------+
| hello | olleh |
+---------------+
As a last resort you can contact us on the mailing lists for further advice.