Martin Kersten received “The Wildest Idea” award at the Gong Show of the Conference on Innovative Data Systems Research (CIDR) 2015. The award is given for his thought provoking “Big Data Space Fungus”, which addresses data rotting in modern data management pipelines.
Every 1.5 years the amount of data and processing power doubles. We neglected to put into place the core technology to let data decay over time. Overdue data should be subject to a data rotting process, i.e. the database becomes a substrate for data fungi. Most fungi should silently purge information from your persistent store. Some fungi may use the substrate to grow delicious fungi as local summaries. Or even more drastic, every record looked upon in a query will be purged after a short period, leaving the user to keep enriched data instead. In other words, rather then creating bigger “fridges” or “silos” for storing raw data, we should develop better “fungi farming” schemes, to distil data into useful knowledge for the users or get rid of it.
Hannes Mühleisen’s work on “Capturing the Laws of (Data) Nature”, describes how statistical models can be used to improve data management. These models could allow a data management system to automatically gain a deeper insight into the stored data. The MonetDB/R integration is a required first step on this way, but much future work is required. If successful, this could lead to improvements in data compression and approximate query answering.
For more information on “Big Data Space Fungus” check out the paper and presentation. For full details on “Capturing the Laws of (Data) Nature” read Hannes’ paper and presentation.
If you are still craving for more, visit the CIDR website for more innovative data systems research (pun intended).