AMMP Technologies runs monitoring for energy systems, usually off mini-grids in Africa. The company uses Grafana to monitor interface with physical objects that are not servers or containers. “It’s interesting how a toolkit for visualizing essentially internet/computer/server metrics is so well-suited to working with real-life streaming data,” AMMP Cofounder Svet Bajlekov said during his talk at GrafanaCon L.A.
In fact, AMMP built its whole stack, not just the visualization part, to be open source, and Bajlekov shared his vision of how other industrial IoT organizations can do the same.
AMMP works with electrical substations, and the SCADA systems that are used for running and monitoring them were built a couple of decades ago. “They were not initially built with the Internet in mind,” Bajlekov said. “Today, we live in a place where most things are getting connected, and with that, it’s no longer simply safe to assume that a sensitive piece of infrastructure is going to be safe anymore.”
Bajlekov gave an example from a couple of years ago, when Russian hackers allegedly got into the control center of the utility in Kiev, tripped the switches on 60 substations, and took the power out for 250,000 people. “Everything is getting connected, and it’s pretty clear that the attack vectors, the attack surface, and the vulnerabilities are growing with that,” he said. And according to one survey, 51% of companies don’t feel prepared to deal with those vulnerabilities.
A Vision for the IIoT Stack
“Now we’ve got the picture of critical equipment that’s being operated by pretty ancient software that was built behind closed doors and for operations that happen behind closed doors, which is neither secure nor particularly flexible,” Bajlekov said. Everything is essentially airgapped, by necessity.
His goal is a secure, open, and extensible technology stack – and it’s within reach.
“We do have more devices coming online, which is challenging, but they also nowadays have a lot more processing power than they once did, and there’s a lot more bandwidth than we once had,” he said. “We can be a little less kind of frugal with the resources that we use for this communication. So why don’t we embrace the internet and the standards and the protocols and best practices on which it has been successfully built over the past decades?”
Adapting those best practices means assuming that everything that you work with is online and figuring out how to encapsulate everything the right way. Bajlekov offered this diagram of a possible architecture for industrial IoT:
The diagram shows:
- A real-world system that you want to talk to some industrial product.
- An edge gateway device.
- Following internet best practices, an MQTT or HTTPS API is used to connect the gateway to some endpoint.
- A data store of time series for metrics.
- Something for managing devices.
- Analytics and visualization.
“That endpoint is going to be working with those data stores to provision the edge devices,” he said, “and then from there you’re going to probably want to do some analytics and some visualization on all of that.”
To accomplish all of this, he said, “it obviously makes sense to go to open source building blocks, just the way that the rest of the plumbing on the internet has been built up out of systems like that, with the open interfaces and fully interoperable components.”
That’s all possible, he adds, because over the past few years, enterprises have increasingly embraced open source. “It’s actually a way for large companies to de-risk their vendor dependencies and to ensure the continuity of their operations if something goes wrong with a particular vendor,” he said. “Mature open source projects actually have a pretty high level ability for security requirements because basically it’s all out in the open. People tend to look at the code base a lot more than they would for a closed-source system… And if you have a good project, you probably have interfaces that are well-designed, well-documented, and have flexibility and robustness in mind.”
The Open Source IIoT Stack
Bajlekov then offered his suggestions of what the stack could look like:
“I think that Grafana really stands out in terms of its best practices for extensibility, interoperability,” he said. “You know, it can do everything via an API, and the way that Grafana approaches the ecosystem is what I feel that ought to be replicated across this chain.”
LoudML, which is next to Kapacitor for analytics, applies machine learning on time series data for anomaly detection.
“You’ve got all this data streaming in, and you want to make sure that you’re making sense of it,” he said. “We’re getting tools that are fully open source that just literally plug into everything else that’s going on here and can get you these great insights off the shelf.
EdgeX Foundry, a Linux Foundation project, is designed to be a fully-interoperable, vendor-neutral microservices framework for IoT edge computing. “The ethos here basically is API first,” he said. “Everything interconnects with everything else via very well-documented interfaces, and you can plug and play different microservices that get you the functionality that you need and allows you to interface with your actual physical objects.”
Bajlekov mentioned that EdgeX is looking into having an implementation that runs on PLCs so that data can be pushed to the cloud directly rather than going through the edge devices.
Where Do We Go From Here?
The barriers to adopting this model are real; utility companies tend to be fairly conservative. Plus, not all of the functionality has been fully worked out. “While a lot of this stack is great for pulling metrics out of a system, processing and visualizing them, there’s not really been an established industry standard for doing end-to-end configuration and control of these remote devices,” he said.
Still, he concluded, “The ecosystem shaping up gives me a lot of hope that things are going to move in a direction where we have a more open, more interoperable, kind of value chain of different components that work together.”
For more from GrafanaCon 2019, check out all the talks on YouTube.