How MIT Open Learning unified observability with Grafana Cloud to scale toward a billion learners

When millions of people around the world open a free MIT course, they rarely think about the systems behind the screen. Tobias Macey, Senior Associate Director of Platform Engineering and DevOps at MIT, thinks about little else. As the head of platform engineering and DevOps at MIT Open Learning, he leads the small team that builds and runs the platforms powering all of MIT’s non-degree learning.

MIT Open Learning is a department of MIT that helps bring the university’s teaching approaches and educational resources to people around the world. It recently set an ambitious goal: reach a billion learners over the next 10 years. Today it already serves millions of distinct learners and tens of thousands of concurrent ones across multiple installations of the Open edX platform, a leading open source online learning platform that itself began as a collaboration between MIT and Harvard.

To hit their goal, Tobias knew his team needed full visibility into what’s happening across their stack. “The move to Grafana Cloud was our effort to bring everything under one roof,” he says.

A decade of accumulated tooling

Tobias’s engineering org has grown from around 10 to 15 people when he joined a decade ago to around 35 today, with a platform and DevOps team of five.

Over those ten years, MIT Open Learning’s observability practice had grown into what Tobias calls “a little bit of a hodgepodge.” New Relic came along with the team’s Heroku deployments. Datadog covered the infrastructure side. Amazon CloudWatch handled some monitoring and alerting, and Fluentd collected logs. Each tool did a job, but together they made it hard to get a  unified view.

This fragmentation collided with a larger architectural change. The team was consolidating disparate applications into MIT Learn—the new core platform and single entry point for all of MIT’s non-degree material—and moving from multiple monoliths to a service-oriented architecture. It also migrated everything onto Kubernetes over the past year. With requests now crossing many service boundaries, stitching together a coherent picture from separate tools was, in Tobias’s words, “just too difficult.”

Consolidating on open standards

Wanting to lean into the open standards the broader industry had converged on, the team decided to unify its signals in Grafana Cloud. 

“Just investing in the standardization that the industry has gone through of using Prometheus and OpenTelemetry really helped simplify our stack a lot,” Tobias says.

First, the team moved logs into Grafana Cloud Logs and then adopted Grafana Cloud Metrics. Over the years, the team had jumped between different log and metric shippers before settling on Alloy, the open source OpenTelemetry collector with built-in Prometheus pipelines, which made it easier to get up and running and to pick up out-of-the-box dashboards. 

Once the Kubernetes migration completed, the Grafana Helm chart for Alloy meant the team could, as Tobias puts it, “just install the Helm chart, use most of the defaults, and then tune things where we want to.”

As applications moved to a service-oriented architecture, the team needed complete traces, not fragments. They adopted tail sampling in Grafana Cloud Traces so that every trace ending in an error was captured and surfaced as a high-signal event worth acting on — waiting until all spans were collected before deciding what to keep, rather than sampling blindly at the start. “Having the enablement from the Grafana Labs team to point us in that direction was really helpful,” Tobias says.

To round it out, the team standardized logging across its Django and React applications on structlog, so a traceback lands as a single structured record instead of something to stitch back together.

Cost as a feature, not an afterthought

In the education space, cost efficiency is non-negotiable, and it shaped the team’s choice in an observability platform, as well as the configuration. They didn’t want the vendor lock-in associated with some cloud providers’ monitoring tools, and CloudWatch was both expensive and, in Tobias’s words, less ergonomic.

With Grafana Cloud, Adaptive Logs and Adaptive Metrics turned cost control into something the team could act on without losing signal. “The savings were immediate,” Tobias says. “As soon as we applied the adaptive logging filters, it immediately cut down on the level of noise and the level of wasted storage that was being dedicated to some of the logs that we didn’t need.” 

There was an operational bonus, too: suggestions from Adaptive Logs periodically surface the sources of high-volume, low-signal logs, which sometimes point at a silent failure or a noisy error worth fixing.

From a service desk to a self-serve substrate

The clearest change wasn’t a metric—it was a way of working. For a long time the platform team operated as a service desk, taking requests from developers to provision capabilities. Unifying observability changed that dynamic. “Now we’re much more in an enablement model. We’re building the substrate that the developers can self-serve and get the information that they need without necessarily having to block on the availability from somebody on the platform team,” Tobias says.

A single observability and alerting layer also made the team’s alerting smarter. It built a more layered view of its systems, which led to a better status page and more proactive detection of degraded latencies. It even wired the data back into operations: Prometheus metrics in Grafana Cloud now act as a scaling signal to the team’s KEDA autoscalers in Kubernetes, scaling out when latency crosses a threshold rather than relying on the coarse CPU and memory metrics Kubernetes provides by default.

Running on real data, not guesswork

For Tobias, observability is what makes the billion-learner ambition tractable: running with as small a footprint as possible while scaling out to meet demand, grounded in real usage data rather than guesswork about which service to split next. The roadmap is more of the same discipline: additional OpenTelemetry integrations, more explicit instrumentation where granularity is needed, and default integrations baked into application code so developers don’t have to think about what to instrument. He calls it “falling into the pit of success” for platform engineering.

And in true MIT fashion, none of it is a secret. “Everything that we do and that we talked about is fully open source, BSD 3-Clause licensed,” Tobias notes. The instrumentation and deployment code lives in the team’s mitodl/ol-infrastructure repository on Github, with the logging and OpenTelemetry hooks for its Django apps.

His advice to other teams, especially in higher education, is to reframe the whole conversation around observability.

“Observability is not always easy to get budget for because it’s seen as a cost center rather than a value add,” Tobias says. “But once you actually have it in place, it can reduce waste and act as a cost-savings."

MIT Open Learning cut observability costs immediately after applying Adaptive Logs filters. See what it looks like in your stack.