Getting Started with Kubernetes Secrets

1. Secrets in Kubernetes: An Introduction

Kubernetes is known for its containerization abilities, it separates sensitive data from the application's code and configuration, reducing the risk of unauthorized access and accidental exposure.

Secrets are the objects that store the sensitive data that your applications need. They are a core feature in the world of containerization and orchestration, a standard for sharing critical data without any leaks.

In this blog post, we will dive deep into Kubernetes Secrets, explore purpose, how they work behind the scenes, and best practices for effectively managing secrets within your applications.

Examples of how secrets are used in Kubernetes:

1. You have a web application that needs to connect to a database. You could create a secret that contains the password for the database, and then reference that secret from your application's code. This way, the password is not stored in your application's code, and it is more secure.
2. You have a server that needs to access a cloud storage service. You could create a secret that contains the API credentials for the cloud storage service, and then reference that secret from your server's code. This way, the credentials are not hard-coded in your server's code, you can easily update them and it is more secure.
3. In a microservices architecture, you may have multiple services running in Kubernetes that require HTTPS communication. You can use Kubernetes secrets to store the SSL/TLS certificates and private keys needed for encryption and decryption. Pods can then reference these secrets to enable secure communication between services.

2. Creating and Implementing Secrets in Kubernetes

In Kubernetes, secrets are typically created from plain text files or through command-line input. Kubernetes then encodes and encrypts this data before storing it within the cluster

Secrets can be created using the following methods:

Imperatively Using Kubectl Command Line:

You can easily create secrets using the kubectl create secret command, providing the necessary data and specifying the secret type.

kubectl create secret generic secret-credentials --from-literal=username=admin --from-literal=password=secretpassword

Declarative Using YAML Manifest Files:

Secrets can also be pre-defined in YAML files and applied using kubectl apply -f <filename>.

apiVersion: v1
kind: Secret
metadata:
  name: secret-credentials
type: Generic
data:
  username: YWRtaW4=  # Base64-encoded value for 'admin'
  password: c2VjcmV0cGFzc3dvcmQ=  # Base64-encoded value for 'secretpassword'

3. Constraints on Secret names and data

One must follow the following rules when using secrets in a Kubernetes environment

Object Name Limitations

1. The name of a Secret object must be a valid DNS subdomain name.
2. The name must start with a lowercase letter or digit and can contain only lowercase letters, digits, hyphens (``), underscores (_), and periods (.).
3. The name must be unique within the namespace.

Examples of Valid Secret names:

ck-secretck-secret-123ck-secret-with-dashes-and-underscores

Examples of Invalid Secret names:

MySecret (starts with an uppercase letter)secret withspace (contains a space)

secret.with.dots (contains two periods)

Secret data Constraints

1. The data field in a Secret object can contain arbitrary key-value pairs.
2. The keys in the data field must be alphanumeric strings, hyphens (``), underscores (_), and periods (.).
3. The values in the data field must be base64-encoded strings.
4. The total size of the data field must not exceed 1MB.

Valid Secret data:

key1: value1key2: value2-with-dashkey3: value3_with_underscore

Invalid Secret data:

key1: value1 2 (contains a space)key2: value2 with space (contains a space)key3: value3 with / (contains a forward slash)

4. Types of Secrets in Kubernetes

Kubernetes ships out of the box with support for several inbuilt types of secrets, each varying in terms of validations performed for common use case scenarios

Factors to consider when choosing a secret type:

• The size of the data: Some secret types, such as opaque secrets, can store arbitrary amounts of data. Other secret types, such as Docker config secrets, have a maximum size limit.
• The frequency of access: Some secret types, such as opaque secrets, can be accessed frequently. Other secret types, such as SSH authentication secrets, are typically only accessed once when the pod is first started.
• The compatibility with your application: Some secret types are more compatible with certain types of applications than others. For example, Docker config secrets are only compatible with applications that use Docker.

Additionally, you can define and use your own Secret type by assigning any non-empty string as the type value for a Secret object (an empty string defaults to an Opaque type).

If you are defining a secret type meant for public use, structure the secret type to have your domain name before the name, separated by a /. For example: commandk.example.dev/secret-credentials

5. Using Secrets in a Kubernetes Environment

Secrets can be viewed using kubectl get secrets

Accessing Secrets during Runtime

Now that you have successfully created a secret object, it can be accessed by pods or other Kubernetes resources in several ways:

Environment Variables:

Secrets can be exposed as environment variables within the pod's container. In this approach, the secret values stored are exposed directly to the application running in the pod

apiVersion: v1
kind: Pod
metadata:
  name: ck-pod
spec:
  containers:
  - name: ck-container
    image: ck-image
    ports:
    - containerPort: 8080
    env:
    - name: USERNAME
      valueFrom:
        secretKeyRef:
          name: ck-secret
          key: username
    - name: PASSWORD
      valueFrom:
        secretKeyRef:
          name: ck-secret
          key: password

In this example, the application running in the Pod can access the secret's data through the environment variables USERNAME and PASSWORD to check this type echo $USERNAME $PASSWORD.

⚠️ Environment variables can appear in error reports and application log at startup, Although Kubernetes provides the option to expose secrets in this manner, it may not be the most secure approach due to the inherent risks associated with child processes inheriting all environment variables from their parent process

Mounted Volumes:

To mount a secret as a volume in a Pod, you need to define the volume and its mount path in the Pod's configuration. This approach provides greater security than environment variables by restricting direct access to the secret data

Consider the following example of a Pod definition that mounts the ck-secret secret into the /etc/secrets directory:

apiVersion: v1
kind: Pod
metadata:
  name: ck-pod
spec:
  containers:
  - name: ck-container
    image: ck-image
    ports:
    - containerPort: 8080
    volumeMounts:
    - name: ck-secret-volume
      mountPath: /etc/secrets
  volumes:
  - name: ck-secret-volume
    secret:
      secretName: ck-secret

In this example, the ck-secret secret is mounted as a volume at the path /etc/secrets within the ck-container container. Any data that is stored in the ck-secret will be accessible to the pod's containers at the path /etc/secrets.

The contents of the secret, such as username and password, are made available as files within the mounted directory. The exact filenames depend on how the secret was created and the format it was created in. For example, if the secret was created using a key-value pair, you might have a file named after the key. If it's a username and password pair, you might have files named "username" and "password" inside the "/etc/secrets" directory.

Your application running inside the container can then read these files to access the secret data. For instance, if your application expects a username and password, it would read the "username" and "password" files from the "/etc/secrets" directory.

Updating Secrets

In a Kubernetes environment, unless a Kubernetes secret is defined as immutable, it can be edited using the following methods

Using Kubectl

To update a secret using the command line, you can use the kubectl edit secret command. For example, to update the ck-secret secret, you would run the following command:

kubectl edit secret ck-secret

This will open the secret object in your default editor. You can then make the changes to the secret object and save the file.

Manifest YAML config file

To update a secret using a manifest file, you can create a new manifest file that contains the updated secret object. You can then use the kubectl apply command to apply the manifest file to the cluster.

For example, the following manifest file would update the ck-secret secret to contain the new value for the password key:

apiVersion: v1
kind: Secret
metadata:
  name: ck-secret
data:
  password: new-password

You can then run the following command to apply the manifest file to the cluster:

kubectl apply -f ck-secret.yaml

6. Best Practices for Kubernetes Secrets Management

Now that we understand how to create and use Secrets in Kubernetes, let's explore some best practices to enhance the security and management of Secrets within your cluster.

To ensure the security and proper management of secrets in Kubernetes, it's important to follow these best practices:

1. Use Specific Secret Types: Whenever possible, use specific secret types (e.g., docker-registry, tls, etc.) for commonly-used secret data. This allows Kubernetes to handle certain types of secrets more efficiently and ensures better integration with other Kubernetes components. For instance, Kubernetes can automatically mount the docker-registry secrets as volumes for your pods, or use tls secrets to secure the communication between your services.
2. Avoid Plain Text Secrets: Never store plain text secrets in Kubernetes, even if they are base64-encoded. Instead, use tools or libraries to handle encryption and decryption of secrets at runtime. For example:
3. Cloud KMS: A service that uses a key to encrypt data at the application layer.
4. Sealed Secrets: A controller and tool that encrypts secrets using asymmetric encryption.
5. Helm Secret Plugin: A plugin that uses Mozilla SOPS to encrypt and decrypt secrets using key management services.
6. Sekret: A CLI tool that encrypts and edits secrets using symmetric encryption.
7. Limit Access to Secrets: Restrict access to secrets by using Kubernetes RBAC (Role-Based Access Control). Only grant access to the necessary users, services, or pods that require the secret.
8. Regularly Rotate Secrets: Implement a practice of regularly rotating your secrets, especially when there are changes in personnel or when secrets have been exposed to an unintended audience.
9. Avoid Committing Secrets to Version Control: Never commit secrets to version control systems, even if they are base64-encoded. Utilize Kubernetes secrets or external secret management tools instead.

7. Managing Secrets with External Systems

While Kubernetes Secrets provide a convenient way to store and manage sensitive data within your cluster, you may also need to integrate with external secret management solutions. For example, you may have existing secrets stored in a secret manager such as AWS Secrets Manager or Google Cloud Secret Manager that you want to leverage within your Kubernetes applications.

The process of accessing secret data typically involves the following steps:

1. The application communicates with the external secrets manager using an API or SDK.
2. The application provides appropriate authentication credentials (such as an API key or token) to the secrets manager.
3. The secrets manager validates the provided credentials and checks the access control policies to determine whether the application is authorized to access the requested secret.
4. If authorized, the secrets manager retrieves the requested secret data from the secure storage.
5. The secrets manager returns the secret data to the application, which can then use the retrieved information to connect to external services or perform other tasks.

By using an external secrets manager, organizations can centralize the management of sensitive information, enhance security through proper authentication and access control, and reduce the risk of secrets exposure in their applications.

Secure your Kubernetes Secrets with CommandK

CommandK helps you create, consume, manage, and rotate secrets across all your environments in one place.

The best part? You can set up CommandK in less than 5 minutes with 0 code changes.

Sign up and try out CommandK here