StreamSets, the startup that develops software for managing the performance of data pipelines, has extended the capabilities of its product portfolio with a new product for managing the flow of data from "edge" devices, and a new cloud service that helps businesses gain more control over multiple data pipelines.
The additions extend the range of the company's Streamsets Data Operations Platform portfolio for managing the performance of data flows to and from big data applications.
"A good metaphor for us is air traffic control for data," said Rick Bilodeau, StreamSets marketing vice president, in an interview with CRN. "We think of that as our role in the data world."
[Related: The 10 Coolest Big Data Products Of 2017]
Managing the lifecycle of "data in motion" from operational applications, Internet of Things networks, industrial sensors, real-time analytical applications and cybersecurity endpoint systems is a challenge for businesses. While custom-coded software has been the traditional remedy, Bilodeau said such systems are difficult to maintain and update.
StreamSets, founded in 2014 and based in San Francisco, debuted its StreamSet Data Collector software in 2015 and StreamSets Dataflow Performance Manager in 2016.
More businesses are building pipelines to fuel their next-generation analytics projects, said Clarke Patterson, StreamSets' product marketing executive, making business intelligence more pervasive throughout their organizations.
The nature of such analytics projects is shifting from batch mode to streaming data analysis, doing more lightweight analysis at the edge devices rather than just collecting data, and mixing different analytical workloads – including "hardcore machine learning and artificial intelligence," Patterson said.
"We see a lot of opportunity moving forward," Patterson said.
The new Streamsets Data Collector Edge (SDC Edge) is an open-source, ultralight version of the StreamSets Data Collector software that provides data collection and ingestion from edge systems such as Internet of Things and cybersecurity devices.
The software takes up only 5MB, making it ideal for resource- and connectivity-constrained systems, according to the company. Today data ingestion logic for such applications is often custom-coded for specific devices. Some packaged software, such as Splunk and General Electric's Predix, have their own data ingestion capabilities.