Integrating Kibana in Elastic Cloud with Okta – Step by Step

Implementing SSO is very useful when teams grow. Okta is well known as Identity provider and in specific for SSO. Elastic is also well known for their great products including Elasticsearch and Kibana! Elastic started its hosted service (Elastic Cloud) and they added nice features such as Hot/Warm deployments which made it popular. They both have good documentation but when it comes to this specific integration, things are not clear. I spent some time and communicated with support on both sides and in this post I will show how to integrate Kibana hosted by Elastic Cloud with Okta as IdP, Step by Step:

  • First step is to configure Okta side to get the Assertion XML. Go to Okta Admin page and Add an application. Choose a SAML 2.0 App. Then you have to specify some basic information such as App name and Logo. Next is SAML Settings which is the important part. In specific the following parameters should be defined:
    1. Single sign on URL: for Elastic cloud the format is:
      https://YOUR_CLUSTER_ADDRESS:9243/api/security/v1/saml
      please note that /api/security/v1/saml is fixed (at least by the time this post is written)
    2. Audience URI (SP Entity ID): This is exactly the URL of your Kibana in Elastic Cloud but please don’t forget / at the end:
      https://YOUR_CLUSTER_ADDRESS:9243/
    3. Name ID Format: depends on your Okta usernames. In my case it’s EmailAddress

      SAML setting
    4. Group Attribute Statements: This is very important for granular Access management and role mapping in Kibana.
      For the Name specify groups. This is important and then for better management you can specify a filter to map groups that contain kibana (as an example). It will filter groups that you created in Okta directory and will help in mapping with Kibana/Elasticsearch x-pack roles. for example you can create a group in Okta with a name like kibana_admins and add Okta users that you want to have superuser privilege in Elasticsearch to this group. We will come back to this mapping later.
      group attribute
    5. It’s almost done now at Okta side. You can review and check the guide which is given by Okta about how to introduce Assertion and Metadata to service provider (Kibana/Elasticsearch)
  • in Next step is configuring Elasticsearch. The main guide to do this on Elastic Cloud is the following:
    https://www.elastic.co/guide/en/cloud/current/ec-securing-clusters-SAML.html
    You can edit elasticsearch.yml by going to Cloud Console and choosing your deployment and then Edit option and then `User Setting Override`:
    edit elasticsearch

    The following values are important:
    1. attributes.principal: you can see the explanation what this is but in case of Okta, apparently its value should be nameid
    2. attributes.groups: It should be in line with item 4 of previous step (Okta) but I recommend to use exactly groups as value
    3. idp.metadata.path: It is something like the following:
      https://YOURCOMPANY.okta.com/app/OKTA_APP_ID/sso/saml/metadata
      but you can get it by visiting Sign On configuration in Okta. In the following picture see the link pointing to Metadata in blue at the bottom.
      metadata link
    4. idp.entity_id: If you check Setup Instructions in Sign On configuration (picture above), it’s mentioned but it looks like the following:
      http://www.okta.com/OKTA_APP_ID
    5. sp.entity_id: As the guide says it should be:
      "KIBANA_ENDPOINT_URL/" but keep in mind that it should align with item 2 of previous step in Okta and again, don’t forget / 🙂
    6. The rest is straight forward and you won’t miss them
  • Tips:
    • This is very important and took me and Elastic support a lot of time to troubleshoot: If you have Hot and Warm nodes, you should apply the configuration to both type of nodes and there are separate elasticsearch.yml files on Cloud Console
    • The value for attributes.principal should be exactly nameid and nameid:persistent won’t work.
    • You must use the SAML realm name cloud-saml (mentioned in the guide)
  • Next step is to do role mapping. You can read about it here:
    https://www.elastic.co/guide/en/elastic-stack-overview/7.3/saml-role-mapping.html
    So far we setup Okta to send some metadata along with the auth response and using these API’s we have to map the groups in Okta with Roles in ElasticSearch. For example I have 2 groups in Okta named: kibana_operators and kibana_admins. using the following mappings I map them to Monitor and superuser roles in ElasticSearch:
######## Role Mapping 1 - Operators #######
PUT /_security/role_mapping/saml-kibana-operators
{
  "roles": [ "Monitor" ],
  "enabled": true,
  "rules": { "all" : [
      { "field": { "realm.name": "cloud-saml" } },
      { "field": { "groups": "kibana_operators" } }
  ]},
  "metadata": { "version": 1 }
}

######## Role Mapping 2 - Admins #######
PUT /_security/role_mapping/saml-kibana-admins
{
   "enabled": true,
    "roles": [ "superuser" ],
    "rules": { "all" : [
        { "field": { "realm.name": "cloud-saml" } },
        { "field": { "groups": "kibana_admins" } }
    ]},
    "metadata": { "version": 1 }
}
  • Finally, you have to configure Kibana to use SAML as authentication mechanism. This step is straightforward as mentioned as Step 6 of this guide. Just Edit kibana.yml by using User setting Overrides in Elastic Cloud Console and specify the values accordingly.

And that’s it! Now you should be able to create users in Okta and add them to appropriate group to enable them to access Kibana!

ElasticSearch snapshot on S3

If you use ElasticSearch for Log analysis, you probably need to have backup and retirement strategy. It’s very handy to store a backup on a S3 bucket and configure lifecycle on that S3 bucket. I know there is a plugin (curator) that can do this but I preferred to use another approach and use ElasticSearch REST API’s. Here is a step to step guide about how to achieve this:

1) install AWS plugin:

https://www.elastic.co/guide/en/elasticsearch/plugins/current/cloud-aws.html

2) create repository in your Elasticsearch cluster:

curl -XPUT 'localhost:9200/_snapshot/backup_s3_repository?pretty' -d'
{
"type": "s3",
"settings": {
"bucket": "BUCKETNAME",
"region": "REGION",
"base_path": "DIRECTORY_NAME WITHIN BUCKET"
}
}'

Notes

  • AWS plugin should be installed on all nodes and services should be restarted to recognize plugin; otherwise you will get this error:

“Unknown [repository] type [s3]”

3) create snapshot:

curl -k -XPUT ‘https://localhost:9200/_snapshot/backup_s3_repository/snapshot_name?pretty?wait_for_completion=true’

4) create a cron job for taking snapshots (for step 3). You can skip `wait_for_completion=true` in cron job

5) Configure Lifecycle for that S3 bucket.