Update: After the Quarks project was announced, it was renamed to Edgent. All references to Quarks below refer to the Apache Edgent project.
Stream processing systems are commonly used to process data from edge devices and there is a need to push some of the streaming analytics to the edge to reduce communication costs, react locally and offload processing from the central systems. Today, IBM announced an open source project called Quarks to build a community for accelerating analytics at the edge. Quarks provides the ability to react locally to events while also allowing central streaming analytic systems like IBM Streams to deliver a hierarchy of analytic processing across devices.
Quarks was developed by IBM as an entirely new project to provide an SDK and lightweight embeddable runtime for streaming analytics at the edge. Quarks can be integrated with centralized analytic systems like Apache Storm or IBM Streams to provide edge to center analytics. Quarks allows devices to react locally to events and help reduce communication costs by sending data to central systems when an event of interest occurs. This allows all relevant data to be delivered the moment an event of interest occurs – rather than just relying on a periodic heartbeat of information, which may not include the signals necessary or might arrive too late to take action on. Device hubs like Watson IoT platform provide device management and message brokers for communication. Central Analytic systems like IBM Streams provide full featured streaming analytics with additional context & state, correlating across devices and access to data-of-record systems. The integration of Quarks and central systems through a device hub enables automatic application connectivity and central job management and health summaries. The consistent streaming programming model enables applications to control the edge devices based upon analytics across devices and the central system.
Quarks applications are developed using a functional flow api to define operations on data streams that is executed as a graph of “oplets” in a lightweight embeddable runtime. The SDK provides capabilities like windowing, aggregation and connectors with an extensible model for the community to expand its capabilities. Quarks provides a consistent and easy to use programming model to allow application developers to focus on their application rather than issues like device connectivity and threading. Quarks focuses on IoT and edge computing, with attributes such as small footprint and modularity to allow deployments to only include what is needed for that specific device and/or application.
Quarks addresses requirements for analytics at the edge for IoT use cases that were not addressed by central analytic solutions. We believe that these capabilities will be useful to many organizations and that the diverse nature of edge devices and use cases is best addressed by an open community. IBM has submitted a proposal to Apache Software Foundation, requesting incubation of Quarks.
Update: Edgent is now officially incubating at Apache!
Quarks aims to build a broad community and foster collaboration on common analytics and connectors across a broad ecosystem of devices. We encourage companies and individuals to join the Quarks community to enable edge analytics and innovate around the Internet of Things. You can find out more and get involved at https://edgent.apache.org/