q Data Marketplace or simply “q”, a young startup from Israel, developed a platform for data providers and consumers to share raw and aggregate data. They applied for IBM Alpha Zone Accelerator with the mission to build a robust data exchange solution. As their mission is to support exchange of sensitive data between providers and consumers they were looking for an architecture that will be able to exchange the data in a secure way. q’s development team wanted to focus on rapidly writing their code without the hassle of setting up, configuring, or maintaining servers. As a startup, it was also important for them that the infrastructure will have minimal costs and that they will need to pay only for their exact demand. After they were accepted to the IBM Alpha Zone Accelerator, we helped q build a secured, scalable solution based on IBM Cloud Functions and containers, which could address exactly what they were looking for.  In this Blog post I will describe the architecture that we designed for q.    

The Process flow

Step 1, step 2 and step 3: Initiate a transaction q’s market place connects between providers (sellers) who offer to sell data and consumers (buyers) that want to buy data. When transaction terms are agreed, the buyer transfers the agreed fee to q’s market place and the q data exchange process begins.

Step 4: initiate a q data container that will hold the exchanged data. q’s engine, triggers an IBM Cloud Function action that starts  a q data exchange container, this container will serve as a “real container” to hold the data during the exchange process .

Step 5: Give control to the seller q’s engine notifies the seller that the container is ready to be used and sends the seller the container’s IP address.

Step 6: Seller encrypts and uploads the data using q client The seller uploads the data using q client .The data is encrypted and then uploaded to the container.  At this phase the seller has a secured and exclusive ownership of the container and access to the data.

Step 7: Transfer control to the buyer Once the data upload is completed, q’s engine calls another IBM cloud function that transfers the container ownership and symmetric key from the seller to the buyer, providing the buyer with the full secured, exclusive ownership of the container and the purchased data.

Step 8: Buyer downloads and decrypts the data using q client The buyer downloads the data and decrypts it using q client using the symmetric key.

Step 9: The Buyer approves the data exchange process and the container can be deleted Once the buyer has downloaded and verified the data, he approves that the data exchange process is done. The seller then receives the payment and q engine calls an IBM cloud function that deletes the container.


Why did we choose this architecture?                                    

  • Implementing data exchange using containers solved q’s issue of storing and maintaining sensitive data, with this solution data is encrypted and ownership is transferred securely between the provider and consumer. Data is not persisted on the server side during the data transfer process. This concept aligns with the HIPAA regulation for exchange of health sensitive records.
  • Choosing IBM Cloud Functions released q form the need to take care of high availability and scaling in order to be ready for peak performance.
  • Another major benefit that is especially important for young startups is the fact that with IBM Cloud Functions you need to pay only for actual usage. You do not pay anything for idle time or for the time the system is not being used.
  • Using IBM Cloud functions enabled q to focus on the proactive engine logic and benefit from all the IBM Cloud Functions built in capabilities like events and periodic execution

Image: q Data Marketplace

Source: www.q-dx.com

When building this architecture we referred to Serverless Code Patterns , this is a great source of information that helps you quickly start developing and coding your solution.

About q

q Data Marketplace is a platform for data providers and consumers to share raw and aggregate data. We supply you with a self-service exchange where you can both distribute and trade your own data as well as find the data you need.As a cross-industry exchange our mission is to make all big data in the world accessible through an open platform where data providers and consumers from any industry transact without barriers using machine friendly interfaces.

About IBM Alpha Zone Accelerator

IBM Alpha Zone Accelerator is a 20-week program, helping startups to build leading solutions for the enterprise market. The program focuses on post Seed & Round A funded startups with aim to create long term technology and business partnership with IBM worldwide.

Join The Discussion

Your email address will not be published. Required fields are marked *