1. Secrets in Kubernetes: An Introduction
Kubernetes is known for its containerization abilities, it separates sensitive data from the application's code and configuration, reducing the risk of unauthorized access and accidental exposure.
Secrets are the objects that store the sensitive data that your applications need. They are a core feature in the world of containerization and orchestration, a standard for sharing critical data without any leaks.
In this blog post, we will dive deep into Kubernetes Secrets, explore purpose, how they work behind the scenes, and best practices for effectively managing secrets within your applications.
Examples of how secrets are used in Kubernetes:
- You have a web application that needs to connect to a database. You could create a secret that contains the password for the database, and then reference that secret from your application's code. This way, the password is not stored in your application's code, and it is more secure.
- You have a server that needs to access a cloud storage service. You could create a secret that contains the API credentials for the cloud storage service, and then reference that secret from your server's code. This way, the credentials are not hard-coded in your server's code, you can easily update them and it is more secure.
- In a microservices architecture, you may have multiple services running in Kubernetes that require HTTPS communication. You can use Kubernetes secrets to store the SSL/TLS certificates and private keys needed for encryption and decryption. Pods can then reference these secrets to enable secure communication between services.
2. Creating and Implementing Secrets in Kubernetes
In Kubernetes, secrets are typically created from plain text files or through command-line input. Kubernetes then encodes and encrypts this data before storing it within the cluster
Secrets can be created using the following methods:
Imperatively Using Kubectl Command Line:
You can easily create secrets using the
kubectl create secret command, providing the necessary data and specifying the secret type.
kubectl create secret generic secret-credentials --from-literal=username=admin --from-literal=password=secretpassword
Declarative Using YAML Manifest Files:
Secrets can also be pre-defined in YAML files and applied using
kubectl apply -f <filename>.
apiVersion: v1 kind: Secret metadata: name: secret-credentials type: Generic data: username: YWRtaW4= # Base64-encoded value for 'admin' password: c2VjcmV0cGFzc3dvcmQ= # Base64-encoded value for 'secretpassword'
3. Constraints on Secret names and data
One must follow the following rules when using secrets in a Kubernetes environment
Object Name Limitations
- The name of a Secret object must be a valid DNS subdomain name.
- The name must start with a lowercase letter or digit and can contain only lowercase letters, digits, hyphens (``), underscores (
_), and periods (
- The name must be unique within the namespace.
Examples of Valid Secret names:
Examples of Invalid Secret names:
MySecret (starts with an uppercase letter)
secret withspace (contains a space)
secret.with.dots (contains two periods)
Secret data Constraints
- The data field in a Secret object can contain arbitrary key-value pairs.
- The keys in the data field must be alphanumeric strings, hyphens (``), underscores (
_), and periods (
- The values in the data field must be base64-encoded strings.
- The total size of the data field must not exceed 1MB.
Valid Secret data:
key1: value1key2: value2-with-dashkey3: value3_with_underscore
Invalid Secret data:
key1: value1 2 (contains a space)
key2: value2 with space (contains a space)
key3: value3 with / (contains a forward slash)
4. Types of Secrets in Kubernetes
Kubernetes ships out of the box with support for several inbuilt types of secrets, each varying in terms of validations performed for common use case scenarios
|Inbuilt Secret Type||Description||Use Cases|
|Opaque Secrets||The default type of secret if omitted from a secret configuration file. Can store any type of data.||Storing arbitrary data, such as passwords, API keys, and encryption keys.|
|Service Account Secrets||Created when you create a service account in Kubernetes. They are used to authenticate the service account with the API server.||Authenticating and authorizing pods to access Kubernetes resources.|
|TLS Secrets||Specifically designed to store TLS certificates and private keys. They consist of two parts: the certificate and the key.||Configuring encryption for Ingress controllers, HTTP endpoints.|
|Dockercfg Secrets||Used to store Docker registry authentication information. They are primarily used by the Kubernetes image puller to authenticate and pull images from private Docker registries.||When you need to store Docker registry credentials that need to be accessed by pods|
|Basic Authentication Secrets||Used to store credentials for basic authentication. They consist of two key-value pairs: username and password.||Configuring basic authentication for HTTP endpoints.|
|SSH Authentication Secrets|
They consist of one key-value pair: ssh-privatekey.
The value of the ssh-privatekey key is the contents of the SSH private key.
|Used to store credentials for SSH authentication.|
|External Secrets||Stores secrets in an external secret management system outside the cluster|
Storing secrets that are too large, frequently updated or sensitive to store in the cluster
eg: database credentials
Factors to consider when choosing a secret type:
- The size of the data: Some secret types, such as opaque secrets, can store arbitrary amounts of data. Other secret types, such as Docker config secrets, have a maximum size limit.
- The frequency of access: Some secret types, such as opaque secrets, can be accessed frequently. Other secret types, such as SSH authentication secrets, are typically only accessed once when the pod is first started.
- The compatibility with your application: Some secret types are more compatible with certain types of applications than others. For example, Docker config secrets are only compatible with applications that use Docker.
Additionally, you can define and use your own Secret type by assigning any non-empty string as the
type value for a Secret object (an empty string defaults to an
If you are defining a secret type meant for public use, structure the secret type to have your domain name before the name, separated by a
/. For example:
5. Using Secrets in a Kubernetes Environment
Secrets can be viewed using
kubectl get secrets
Accessing Secrets during Runtime
Now that you have successfully created a secret object, it can be accessed by pods or other Kubernetes resources in several ways:
Secrets can be exposed as environment variables within the pod's container. In this approach, the secret values stored are exposed directly to the application running in the pod
apiVersion: v1 kind: Pod metadata: name: ck-pod spec: containers: - name: ck-container image: ck-image ports: - containerPort: 8080 env: - name: USERNAME valueFrom: secretKeyRef: name: ck-secret key: username - name: PASSWORD valueFrom: secretKeyRef: name: ck-secret key: password
In this example, the application running in the Pod can access the secret's data through the environment variables
PASSWORD to check this type
echo $USERNAME $PASSWORD.
⚠️ Environment variables can appear in error reports and application log at startup, Although Kubernetes provides the option to expose secrets in this manner, it may not be the most secure approach due to the inherent risks associated with child processes inheriting all environment variables from their parent process
To mount a secret as a volume in a Pod, you need to define the volume and its mount path in the Pod's configuration. This approach provides greater security than environment variables by restricting direct access to the secret data
Consider the following example of a Pod definition that mounts the
ck-secret secret into the
apiVersion: v1 kind: Pod metadata: name: ck-pod spec: containers: - name: ck-container image: ck-image ports: - containerPort: 8080 volumeMounts: - name: ck-secret-volume mountPath: /etc/secrets volumes: - name: ck-secret-volume secret: secretName: ck-secret
In this example, the
ck-secret secret is mounted as a volume at the path
/etc/secrets within the
ck-container container. Any data that is stored in the
ck-secret will be accessible to the pod's containers at the path
The contents of the secret, such as username and password, are made available as files within the mounted directory. The exact filenames depend on how the secret was created and the format it was created in. For example, if the secret was created using a key-value pair, you might have a file named after the key. If it's a username and password pair, you might have files named "username" and "password" inside the "/etc/secrets" directory.
Your application running inside the container can then read these files to access the secret data. For instance, if your application expects a username and password, it would read the "username" and "password" files from the "/etc/secrets" directory.
In a Kubernetes environment, unless a Kubernetes secret is defined as immutable, it can be edited using the following methods
To update a secret using the command line, you can use the
kubectl edit secret command. For example, to update the
ck-secret secret, you would run the following command:
kubectl edit secret ck-secret
This will open the secret object in your default editor. You can then make the changes to the secret object and save the file.
Manifest YAML config file
To update a secret using a manifest file, you can create a new manifest file that contains the updated secret object. You can then use the
kubectl apply command to apply the manifest file to the cluster.
For example, the following manifest file would update the
ck-secret secret to contain the new value for the
apiVersion: v1 kind: Secret metadata: name: ck-secret data: password: new-password
You can then run the following command to apply the manifest file to the cluster:
kubectl apply -f ck-secret.yaml
6. Best Practices for Kubernetes Secrets Management
Now that we understand how to create and use Secrets in Kubernetes, let's explore some best practices to enhance the security and management of Secrets within your cluster.
To ensure the security and proper management of secrets in Kubernetes, it's important to follow these best practices:
- Use Specific Secret Types: Whenever possible, use specific secret types (e.g.,
tls, etc.) for commonly-used secret data. This allows Kubernetes to handle certain types of secrets more efficiently and ensures better integration with other Kubernetes components. For instance, Kubernetes can automatically mount the docker-registry secrets as volumes for your pods, or use tls secrets to secure the communication between your services.
- Avoid Plain Text Secrets: Never store plain text secrets in Kubernetes, even if they are base64-encoded. Instead, use tools or libraries to handle encryption and decryption of secrets at runtime. For example:
- Cloud KMS: A service that uses a key to encrypt data at the application layer.
- Sealed Secrets: A controller and tool that encrypts secrets using asymmetric encryption.
- Helm Secret Plugin: A plugin that uses Mozilla SOPS to encrypt and decrypt secrets using key management services.
- Sekret: A CLI tool that encrypts and edits secrets using symmetric encryption.
- Limit Access to Secrets: Restrict access to secrets by using Kubernetes RBAC (Role-Based Access Control). Only grant access to the necessary users, services, or pods that require the secret.
- Regularly Rotate Secrets: Implement a practice of regularly rotating your secrets, especially when there are changes in personnel or when secrets have been exposed to an unintended audience.
- Avoid Committing Secrets to Version Control: Never commit secrets to version control systems, even if they are base64-encoded. Utilize Kubernetes secrets or external secret management tools instead.
7. Managing Secrets with External Systems
While Kubernetes Secrets provide a convenient way to store and manage sensitive data within your cluster, you may also need to integrate with external secret management solutions. For example, you may have existing secrets stored in a secret manager such as AWS Secrets Manager or Google Cloud Secret Manager that you want to leverage within your Kubernetes applications.
The process of accessing secret data typically involves the following steps:
- The application communicates with the external secrets manager using an API or SDK.
- The application provides appropriate authentication credentials (such as an API key or token) to the secrets manager.
- The secrets manager validates the provided credentials and checks the access control policies to determine whether the application is authorized to access the requested secret.
- If authorized, the secrets manager retrieves the requested secret data from the secure storage.
- The secrets manager returns the secret data to the application, which can then use the retrieved information to connect to external services or perform other tasks.
By using an external secrets manager, organizations can centralize the management of sensitive information, enhance security through proper authentication and access control, and reduce the risk of secrets exposure in their applications.
Secure your Kubernetes Secrets with CommandK
CommandK helps you create, consume, manage, and rotate secrets across all your environments in one place.
The best part? You can set up CommandK in less than 5 minutes with 0 code changes.
Sign up and try out CommandK here