Integrating a custom web app with the Elastic (ELK) stack using SAML

We've been working with a financial services client to help them migrate from self-hosted Elastic (ELK) stacks to Elastic Cloud.

One of the problems they faced as part of the migration was losing access to the on-premises LDAP service from Elasticsearch, because a browser-based custom web app they use (which consumes logging data and generates reports) relied on users being able to authenticate with Elasticsearch using LDAP.

In this blog post I'll show you how we solved that problem.

tl;dr - https://github.com/frontierhq/elastic-custom-app-saml-starter 

Custom web app before (with LDAP)

Here was the user experience using the custom app against the existing self-hosted stack:

1. User selected the report they want to view.
2. App prompted the user for their directory username and password.
3. User entered credentials and clicked submit.
4. App displayed generated report to the user.

Under the covers, here's what was happening:

1. User selects the report they want to view.
2. App prompts the user for their directory username and password.
3. App backend generates a Basic authentication header and then uses it to request data from Elasticsearch using the GET /_search API.
4. Elasticsearch authenticates the user using the username and password using the LDAP service.
5. Elasticsearch returns data to the app, which then generates a report and displays it to the user.

The authentication context being created in this flow was user-scoped - i.e. Elasticsearch knew the identity of the person (rather than the app) trying to access the data - and so features like field level security worked as planned. It was a nice enough user experience - manual username and password entry wasn't ideal, but a browser password manager with autofill functionality would have helped with that.

We replaced LDAP with SAML

As part of the migration from self-hosted Elastic (ELK) stacks to Elastic Cloud deployments, we replaced the LDAP realm with a SAML realm, using Microsoft Entra as the identity provider.

What is SAML?

SAML is an acronym used to describe the Security Assertion Markup Language (SAML). Its primary role in online security is that it enables you to access multiple web applications using one set of login credentials. It works by passing authentication information in a particular format between two parties, usually an identity provider (idP) and a web application.

https://www.onelogin.com/learn/saml

Although understanding the user experience and application flow of Kibana login after switching to SAML isn't strictly required, the configuration is relevant to the overall solution, so I've included it for completeness (but feel free to skip ahead).

The user experience:

1. User clicks "Log in with Microsoft Entra" on the Kibana login screen.
2. Kibana redirects the user to Microsoft Entra.
3. User authenticates, and is then redirected back to Kibana, logged in.
4. When the user is finished, they click "Log out".

And under the covers:

1. User clicks "Log in with Microsoft Entra" on the Kibana login screen.
2. Kibana redirects the user to Microsoft Entra with a SAML request.
3. Microsoft Entra authenticates the user and returns a SAML response HTML form.
4. User's browser POSTs the SAML response form to Kibana.
5. Kibana validates the SAML response, creates a session and redirects the user back to Kibana.

The new config in Elastic Cloud's equivalent of elasticsearch.yml looks something like this:

xpack.security.authc.realms.saml:
  saml1:
    order: 2
    attributes.principal: nameid
    attributes.groups: "http://schemas.microsoft.com/ws/2008/06/identity/claims/groups"
    idp.metadata.path: "https://login.microsoftonline.com/[microsoft-entra-tenant-id]/federationmetadata/2007-06/federationmetadata.xml?appid=[microsoft-entra-ent-app-id]"
    idp.entity_id: "https://sts.windows.net/[microsoft-entra-tenant-id]/"
    sp.entity_id: "https://my-deployment.kb.eu-west-2.aws.cloud.es.io/"
    sp.acs: "https://my-deployment.kb.eu-west-2.aws.cloud.es.io/api/security/saml/callback"
    sp.logout: "https://my-deployment.kb.eu-west-2.aws.cloud.es.io/logout"

And in Elastic Cloud's equivalent of kibana.yml, something like this:

xpack.security.authc.providers:
  saml.saml1:
    order: 2
    realm: saml1
    description: "Log in with Microsoft Entra ID"

When users open Kibana and click Log in with Microsoft Entra ID, an SP-initiated SAML SSO flow starts. It's a great user experience in this instance because Microsoft Entra Connect is providing pass-through authentication to the on-premises directory, meaning in many cases users don't have to re-authenticate - it's a single click login.

Custom web app authentication broken

Unfortunately though, switching out LDAP for SAML broke the custom web app because it no longer had anything it could provide to Elasticsearch on behalf of the user in order to authenticate. Elasticsearch trusts the identity provider - in this case Microsoft Entra - to assert identity rather than handling a username and password itself.

One of the solutions to this is API keys. Using API keys would enable the custom web app to continue to provide a Basic authentication header - this time containing a token rather than username and password - that Elasticsearch would then use to establish identity outside of the SAML mechanism.

One problem with this, though, is that it's not a great user experience. It forces the user to have to first log into Kibana and follow a step-by-step process in there, something like:

1. Log in to Kibana.
2. Browse to Stack Management > Security > API Keys > Create API key.
3. Create an API key of the right type, with the right privileges and the right expiry.
4. Open the custom web app and select the report to view.
5. Provide the API key when prompted.

This approach also introduces the risk of one user sharing a token with another.

Fortunately, we found a better solution

First, let's talk about the user experience we've enabled for users of the custom app:

1. User selects the report they want to view.
2. App redirects the user to Microsoft Entra.
3. User authenticates, and is then redirected back to the app generated report, logged in.
4. When the user is finished, they click "Log out".

It's identical to the Kibana user experience I described above, and feels like this in practice:

Under the covers, here's what's happening:

1. User selects the report they want to view.
2. Custom app uses the POST /_security/saml/prepare API to generate a SAML request and redirects the user to the identity provider specified in the API response from Elasticsearch (in this case Microsoft Entra).
3. Microsoft Entra authenticates the user and returns a SAML response HTML form.
4. User's browser POSTs the SAML response form to the custom app.
5. Custom app relays the SAML response to Elasticsearch using the POST /_security/saml/authenticate API, which authenticates the user and returns access and refresh tokens back to the custom app. Custom app creates a session to cache the tokens.
   
6. Custom app generates a Bearer token authentication header using the access token from the session and then uses it to fetch data using the GET /_search API.
7. Elasticsearch returns data to the custom web app, which then generates a report and displays it to the user.
   
8. User selects "Log out" on the custom web app, which calls POST /_security/saml/logout to invalidate the access and refresh tokens, and then destroys the session.

Here's how we made it work

Elasticsearch provides a set of APIs to integrate a custom web app using SAML. The important ones are:

To demonstrate how to use them I've written a simple Express web app called elastic-custom-app-saml-starter that I've made available on GitHub. The code examples following are all taken from there.

Step 1: Configure Elasticsearch

Enable the custom app to use the APIs by adding an additional SAML realm to Elasticsearch configuration (in either elasticsearch.yml or Elastic Cloud's equivalent of elasticsearch.yml):

xpack.security.authc.realms.saml:
  saml1:
    (Kibana config, omitted for brevity)
  saml2:
    order: 3
    attributes.principal: nameid
    attributes.groups: "http://schemas.microsoft.com/ws/2008/06/identity/claims/groups"
    idp.metadata.path: "https://login.microsoftonline.com/[microsoft-entra-tenant-id]/federationmetadata/2007-06/federationmetadata.xml?appid=[microsoft-entra-ent-app-id]"
    idp.entity_id: "https://sts.windows.net/[microsoft-entra-tenant-id]/"
    sp.entity_id: "http://localhost:3000/"
    sp.acs: "http://localhost:3000/saml/callback"
    sp.logout: "http://localhost:3000/logout"

The differences here vs the saml1 realm for Kibana are the three sp fields (where http://localhost:3000 is the URL our custom web app is running on in this example).

Step 2: Prepare the SAML request

When login is requested, call the POST /_security/saml/prepare API, passing the realm name to use. Encode the realm name and request ID (required later) as relay state and append it to the redirect URL generated by Elasticsearch, and then redirect to the identity provider:

async login(req: Request, res: Response) {
  let redirectUrl;
  try {
    const samlPrepareResponse = await this.client.security.samlPrepareAuthentication({
      realm: 'saml2',
    })
    
    redirectUrl = new URL(samlPrepareResponse.redirect);
    const relayState = {
      id: samlPrepareResponse.id,
      realm: samlPrepareResponse.realm,
    }
    redirectUrl.searchParams.set('relayState', Buffer.from(JSON.stringify(relayState)).toString('base64'));
    
    logger.info(`${this.constructor.name}: redirecting to identity provider:`, { data: redirectUrl.toString() });
    return res.redirect(redirectUrl.toString())
  } catch (error) {
    logger.error(`${this.constructor.name}: unable to prepare saml request:`, { error });
    return res.status(500).send(error)
  }
}

See controllers/saml.ts.

Step 3: Handle the response

Validate that the request contains both a SAML response and relay state, and then pass them to the POST /_security/saml/authenticate API. Elasticsearch will respond with a JSON body that includes access and refresh tokens. Store them as session data, and then redirect back to the app homepage:

async callback(req: Request, res: Response) {
  if (!req.body.SAMLResponse) {
    return res.status(400).send('SAMLResponse not found in request body');
  }

  if (!req.body.RelayState) {
    return res.status(400).send('RelayState not found in request body');
  }

  const relayState = JSON.parse(Buffer.from(req.body.RelayState, 'base64').toString('utf-8'));

  try {
    const authenticateResponse = await this.client.security.samlAuthenticate({
      content: req.body.SAMLResponse,
      ids: [relayState.id],
      realm: relayState.realm,
    });

    req.session.user = authenticateResponse;
    
    return res.redirect('/');
  } catch (error) {
    logger.error(`${this.constructor.name}: unable to authenticate to elasticsearch:`, { error });
    return res.status(500).send(error)
  }
}

See controllers/saml.ts.

Step 4: Authenticate to Elasticsearch using the token

Create a new client using the access token returned earlier (available from the session in the starter app), and then call the relevant Elasticsearch API (in this case just the cluster info API):

async index(req: Request, res: Response) {
  if (!req.session || !req.session.user) {
    return res.render('home/index');
  }

  const client = new Client(
    {
      node: process.env.ELASTIC_ENDPOINT,
      auth: {
        bearer: req.session.user.access_token,
      }
    }
  );

  const authenticateResponse = await client.info()

  return res.render('home/index', { data: JSON.stringify(authenticateResponse, null, 2) });
}

See controllers/home.ts.

Render the user session data and raw API response (just for the sake of demonstration):

<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-QWTKZyjpPEjISv5WaRU9OFeRpok6YctnYmDr5pNlyT2bRjXh0JMhjY6hW+ALEwIH" crossorigin="anonymous">
  <link rel="stylesheet" href="css/main.css">
  <title>Elastic Custom App SAML Starter</title>
</head>

<body>
  <div class="container">
    <h1>Elastic Custom App SAML Starter</h1>
    <% if(user) { %>
      <p class="green">User session detected.</p>
      <h2>Session data</h2>
      <pre><%= JSON.stringify(user, null, 2) %></pre>
      <h2>Elasticsearch response (using access token)</h2>
      <pre><%= data %></pre>
      <a href="/logout">
        <button class="btn btn-primary" type="button">Log out</button>
      </a>
    <% } else { %>
      <p class="red">No user session detected.</p>
      <a href="/saml/login">
        <button class="btn btn-primary" type="button">Log in</button>
      </a>
    <% } %>
  </div>

  <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/js/bootstrap.min.js" integrity="sha384-0pUGZvbkm6XF6gxjEnlmuGrJXVbNuzT9qBBavbLwCsOGabYfZo0T0to5eqruptLy" crossorigin="anonymous"></script>
  <script src="js/main.js"></script>
</body>

</html>

See views/home/index.ejs.

Lets wrap this up

What I've shown here is how you can use a set of APIs made available by Elasticsearch to integrate a custom web app with the Elastic (ELK) stack using SAML. All the code I've used in this post (and more) is available at https://github.com/frontierhq/elastic-custom-app-saml-starter, and obviously the same approach is possible across different languages and ecosystems - check Elasticsearch clients to get started.

Thanks for reading.