Implementing Grafana Play privacy policies with Grafana k6: A behind-the-scenes look

Grafana Play is a free and publicly accessible sandbox environment that allows users to explore and learn Grafana without setting up their own instance. Grafana Play comes preloaded with ready-made sample dashboards, and showcases how to work with different data sources, create visualizations, and use advanced Grafana features. 

Over the last decade or so, Grafana Play — which is hosted on Grafana Cloud — has become increasingly popular in the community, amassing thousands of public dashboards and users worldwide. With this growth came a great responsibility to ensure proper safeguards are in place to protect users’ personal information.

As part of these security and privacy efforts, we announced plans earlier this year to delete inactive Grafana Play accounts. We know that data minimization is a cornerstone of protecting user privacy, and believe in only retaining data that serves a valid purpose to keep Play running for you. 

In this follow-up post, we’ll provide a behind-the-scenes look at these efforts, detailing how we removed over 21,000 inactive users, as well as old and broken dashboards, in Grafana Play using basic Python, JavaScript, and Grafana k6. All code examples in this blog can also be found in this public GitHub repository.

Decluttering dashboards

With thousands of publicly available dashboards in Grafana Play, we would sometimes hear from community members who encountered a broken dashboard when they searched for a particular topic. These dashboards were often broken because of old data sources, or because they had not been generally maintained. 

In most cases where we couldn’t contact the dashboard owner, or find a clear owner internally, we decided that fixing these dashboards would not be a viable solution. Instead, we deprecated them and hid them from public view. 

NOTE: If you notice your dashboard is not publicly available in Play anymore, you want it back up, and can assist in fixing it, we’re happy to work with you. Please reach out on our community Slack and in the #grafana-play channel, provide us with a link to your dashboard, and we’ll help get it fixed.

The role of Grafana k6

From a tooling perspective, you might have heard of k6, our open source load testing tool. Grafana k6 supports various use cases that extend beyond load testing, including browser testing. While the Grafana API could provide us with a list of all dashboards on Play, there was no way for us to know which dashboards contained a broken panel on a protocol level. Since the broken panel element is rendered on the front end, we needed to use the k6 browser module to gain access to different browser-level APIs to interact with browsers and simulate user actions like navigating, clicking, and getting various DOM (Document Object Model) elements.

Here’s a look at the k6 script we used.

JavaScript Copy

import { check, sleep } from 'k6';
import { browser } from 'k6/browser';
import { Gauge } from 'k6/metrics';
import exec from 'k6/execution';
import http from 'k6/http';

First, we imported various k6 modules, such as the browser module, Gauge (a custom metric that you can create for tracking numeric values), the exec module to get information about the current test execution state inside the test script, and the http module to fetch dashboards from Play on a protocol level.

In our k6 setup function, we fetched dashboards from Play using an HTTP GET request, excluding folders as needed, and then returning the filtered dashboards.

JavaScript Copy

export async function setup() {
   const dashboards = await http.get('https://play.grafana.org/api/search?type=dash-db&limit=5000');
   const urls = dashboards.json().filter(d => {
       // Exclude dashboards in folders that are used to replicate GitHub issues
       // These tend to naturally break after the issue is fixed, but teams have requirements
       // to keep them around to prevent them from littering GitHub issues with broken links.
       return (!(d.folderTitle === 'Bug reports reproduction') &&
               !(d.folderTitle === 'Replicating issues') &&
               !(d.folderTitle === 'dev-dashboards'))
   }).map(dashboard => ({
       url: `https://play.grafana.org/d/${dashboard.uid}`,
       title: dashboard.title,
       folderTitle: dashboard.folderTitle,
       uid: dashboard.uid,
   }))

   console.log("A total of", urls.length, "dashboards will be checked")
   return { dashboards: urls };
}

Outside the setup function, we created a custom metric to record the number of broken panels per dashboard.

JavaScript Copy

const brokenPanelsGauge = new Gauge('broken_panels');

Then, for each dashboard, we leveraged the k6 browser testing functionality to find all broken panel elements in the DOM.

JavaScript Copy

export async function checkForBrokenPanels({ dashboards }) {
    const page = await browser.newPage();
    
    // See https://grafana.com/docs/k6/latest/javascript-api/k6-execution/#scenario
    if (exec.scenario.iterationInTest >= dashboards.length) {
        console.log("No more dashboards to check")
        return
    }


    // Pick the dashboard for this iteration
    const dashboard = dashboards[exec.scenario.iterationInTest];
    const url = dashboard.url;

    try {
        await page.goto(url);

        // Wait for the time picker to confirm page load
        await page.waitForSelector('[aria-controls="TimePickerContent"]');

        sleep(2)
        
        // Look for broken panel elements in the DOM
        const brokenPanels = await page.evaluate(() => {
        const brokenPanels = document.querySelectorAll('button[data-testid="data-testid Panel status error"]');
            return brokenPanels.length;
        });

// Log result and record custom metric       
console.log(`ResultMeasurement,${dashboard.title},${dashboard.folderTitle},${dashboard.uid},${url},${brokenPanels}`)
        brokenPanelsGauge.add(brokenPanels, { url: url });

        check(brokenPanels, {
            'Dashboard contains no broken panels': count => count === 0
        });
    } catch (error) {
        console.error("Error:", error.message);
        console.error(error.stack);
    } finally {
        await page.close();
    }
}

The real magic here is looking for button[data-testid="data-testid Panel status error"] in the DOM that comes back when the browser gets a copy of the dashboard. In the example of the broken dashboard we gave earlier, this button state indicates that the user would see a red icon indicating that the panel is broken.

k6 can output custom metrics, so we were able to generate actual metrics and telemetry about the thousands of dashboards running on Play.  

For our initial cleanup effort, though, we exported the results to a CSV file and sorted the logs to deprecate the dashboards with the highest amount of broken panels. A similar test could easily be set up in Grafana Cloud k6 — the fully managed performance testing platform powered by k6 OSS — to monitor your dashboards on an ongoing basis.

Removing inactive users

Our next step was to remove inactive users on Grafana Play. As mentioned above, and in a previous blog post, we wanted to do this so we don’t store personal identifiable information unnecessarily. We defined inactive users as those who met two criteria: they have not logged in for one year or more and they have never created dashboards or apps on Play.

This would be a lot of work to do by hand, so we wrote various scripts. To create our “active users list” — meaning a list of users we did not want to remove — we wrote scripts that enumerated every dashboard on Play, checked its edit history, and noted every user ID that has ever made a modification. 

Let’s break it down below.

Identifying active users 

To build the active user list, we used Jupyter Notebook and Google Colab to quickly run our code. Jupyter Notebook is an interactive, web-based environment used primarily for writing and running code, especially in Python. Google Colab is a cloud-hosted version of Jupyter Notebook provided by Google. 

We first created a script that queried the Grafana /api/search endpoint to retrieve the dashboards’ unique identifier and stored it in a variable called uids. The token came from creating a service account, which was used to authenticate against the Grafana API. 

python Copy

from google.colab import userdata

import requests
import json

tok = userdata.get('token')

def grafana_search():
 url = "https://play.grafana.org/api/search/"
 params = {
     "type": "dash-db",
     "limit": 5000
 }
 try:
   response = requests.get(url, params=params, headers={"Authorization": "Bearer %s" % tok})
   response.raise_for_status() # Raise an exception for bad status codes
   return response.json()
 except requests.exceptions.RequestException as e:
   print(f"An error occurred: {e}")
   return None

results = grafana_search()
uids = [r['uid'] for r in results]

Once we had all the unique identifiers for all dashboards, the next step was to check which users edited a dashboard, how often they made edits, and when they last edited it to give us an idea of who our most active users are. This was achieved with the following code.

python Copy

from google.colab import userdata
tok = userdata.get('token')

import pandas as pd

frame = {
    "author": [],
    "versions": [],
    "latest": []
}

headers = {
    "Authorization": "Bearer " + tok
}

cache = {}
limit = 2000
x=0

for uid in uids:
  x = x + 1
  if x >= limit: break

  if x % 10 == 0:
    print("Request %d" % x)
  response = requests.get(f"https://play.grafana.org/api/dashboards/uid/{uid}/versions", headers=headers)

  try:
    j = response.json()
    versions = j['versions']
  except KeyError:
    print("Failed versions on %s" % json.dumps(j))
    versions = []

  for version in versions:
    createdBy = version['createdBy']
    created = version['created']
    if not createdBy in cache:
      cache[createdBy] = { "latest": created, "versions": 1 }
    else:
      cache[createdBy]['versions'] = cache[createdBy]['versions'] + 1
      if cache[createdBy]['latest'] < created:
        cache[createdBy]['latest'] = created

for author in cache:
  frame["author"].append(author)
  frame["versions"].append(cache[author]['versions'])
  frame["latest"].append(cache[author]['latest'])

df = pd.DataFrame(frame)
df.to_csv(path + "/dashboard-authors-summarized-stats.csv", index=False)

Essentially, we iterated over the dashboard identifiers and fetched the version history. For each saved version of the dashboard, we extracted the createdBy (the user who saved the dashboard) and the created (when they saved it) fields. We then added the new user information to the cache. If the user already existed in the cache, we incremented the version count. Finally, we saved the results as a CSV file.

Identifying inactive users

The next step for us was to build a list of inactive users by comparing our previous list with all the Play users. Again, an “inactive user” is one who has not logged in for one year or more and has never created dashboards or apps on Play.

We wanted to be sure not to delete any account that had created a dashboard, since this would destroy the attribution metadata on the dashboard. 

• The active list: all user IDs who had ever contributed or modified a dashboard. We don’t want to delete these!
• The full list: all Grafana Play users, period.
• The inactive list: (full list) - (active list)

Once we had the list of inactive users, the next step was to delete them. Here is an example code snippet that read the user IDs one by one from an Excel sheet, called the relevant Grafana API endpoint for the user, and invoked a delete action:

JavaScript Copy

const baseUrl = "https://play.grafana.org/api/org/users/";
const token = process.env.TOKEN;

 for (const row of data) {
   const userId = row.userId;
   if (userId) {
     const url = `${baseUrl}${encodeURIComponent(userId)}`;
     console.log(url);

     try {
       const response = await axios.delete(baseUrl, {
         headers: {
           'Authorization': `Bearer ${token}`,
         }
       });
       console.log(`Deleted user ${userId}: ${response.status}`);
     } catch (error) {
       console.error(`Error deleting user ${userId}: ${error.message}`);
     }
   }
 }

Before doing the mass deletion, we liaised with multiple teams to ensure we’d taken all necessary precautions. For example, we did thorough checks to ensure that any users who were removed can still log back in to Play in the future if they wish, and also to check that they can access other Grafana instances.

Today, we have successfully removed over 21,000 inactive users from Play. This should not impact viewing permissions or compromise anyone’s access to anything, as Play remains a public playground to learn about Grafana, no login required. 

Wrapping up — and how to contribute

The code examples above are basic scripts that you can convert to any programming language (apart from the k6 script). Here at Grafana Labs, these scripts helped us declutter our Grafana Play instance and clear out unnecessary data, making it more secure. If you want to try this with your Grafana instance, you can familiarize yourself with the Grafana API, Grafana k6 browser module, and service accounts. 

In addition, if you want to become an active contributor to Grafana Play, we hope to hear from you. We love to host community dashboards that demonstrate fun and unique ways to use Grafana, whether it’s playing video games or monitoring the US electrical grid. We encourage you to share your dashboard ideas, and would love to discuss showcasing them on Play.

Lastly, we’ve created a dedicated channel within our community Slack, #grafana-play, for discussions specifically about Play. Please join the conversation to share feedback, ask questions, or clarify anything related to our new policy. 

Thank you for being a part of the Grafana Play community and helping us maintain this valuable resource for learning and sharing dashboard expertise!