With the growing volumes of digital content, app developers — and the businesses they work for — need smarter storage platforms. Object storage is a relatively new method of storing information, and is much better than traditional file systems in dealing with the massive volumes of unstructured content that most businesses are collecting, storing, and accessing.
Swift Storlets take object storage to the next level. A storlet is code that runs in the object store. The code runs close to where the data is stored and therefore minimizes costs of data transfer when operating in the cloud. Storlets are typically used to transform the data, filter the data, or analyze the data, all within the object store.
Our storlet engine is an extension of OpenStack Swift, which is the most popular open source object-storage project. We have designed the storlet engine so that it is easy to write, deploy, and run storlets. We have demonstrated the business value of storlets for media and entertainment use cases, in collaboration with RAI, Italy’s national public broadcasting company. We are also working on integrating Apache Spark with the storlet engine for analytics use cases.
Why should I contribute?
You’ll be working in with a key technology and learning about a fast-growing project with massive potential. The ability to dynamically load code that runs inside the object store opens up an exciting new and uncharted territory of possibilities which we have only started to explore.
Storlets can be used for media and entertainment use cases, as shown in our collaboration with RAI. We use storlets for digital preservation as part of the ForgetIT European project. Storlets are also part of the data analytics infrastructure for the COSMOS European project, which deals with IoT systems.
If you need to run computations on data, and if you want to learn and play a part in a fast-growing and highly visible project, Swift Storlets could be exactly the coding challenge you’re looking for.
What technology will I use and what technology problem will I help solve?
Swift Storlets are easy to write, deploy, and run. Writing a storlet is as easy as implementing a Java function. Deploying is as easy as uploading a jar file to an object store. And running a storlet is as easy as adding a single header to a REST request.
Invoking a storlet on a data object is done in an isolated manner so that the only data accessible by the computation is the object’s data and its user metadata. Moreover, the computation has no access to disks, the network, or the Swift request environment. The sandboxing technology we use for the storlet engine is Docker.
Storlets are best suited for stream processing. Although they can currently only be written in Java, it should be easy to extend the storlet engine to support any other programming language.
How will the project help my business?
Swift Storlets will conserve your resources, save you time, and perform computations securely. Running computation through storlets is relatively quick and saves network bandwidth and CPU resources. If you need to run a computation that analyzes a large object and extracts only a small amount of data, rather than transferring all the data to your own servers and processing it there, you can use storlets to process the data on the OpenStack Swift cluster, and transfer only the computation result. Storlets can also be used to improve security by filtering out sensitive information at the source.
That’s efficiency, that’s security, and that’s good business.