Stop Doing Kubernetes Secrets Wrong: Proper ways to manage secrets in Kubernetes

Secret management is a serious security challenge for organizations running Kubernetes workloads in production environments. While native Kubernetes Secrets offers a basic solution, its limitations have led to more advanced alternatives: the Secrets Store CSI Driver and External Secrets Operator (ESO).

This blog post compares these three approaches, examining how each balances security, complexity, and integration with external secret stores. We'll analyse their strengths and weaknesses, giving you the basic knowledge you need to select the right solution for your environment. At the end of the blog post, we’ve provided a decision tree to choose the best solution for your needs.

In a future blog post, we'll build on this knowledge with practical implementation examples and ready-to-use Kubernetes resources that demonstrate how to deploy each solution in Azure.

Native Kubernetes Secrets: The Foundation

How Kubernetes Secrets Work and Their Limitations

Kubernetes Secrets are built-in objects designed to store sensitive data like passwords, tokens, and SSH keys. They decouple secrets from application code by storing them in the cluster's etcd database. Secrets can be mounted as volumes or exposed as environment variables to pods, reducing the risk of hardcoding credentials in container images or pod definitions.

While Kubernetes Secrets are simple to implement and may be suitable for development and sandbox environments, they are not considered production-ready. With the robust secret management alternatives now available, it's highly recommended to use more secure options for production workloads.

There are several security concerns with native Kubernetes Secrets, for example:

• Not encrypted, only base64 encoded.
• Requires manual configuration of etcd encryption at rest.
• Vulnerable to unauthorized access through kubectl if RBAC is improperly configured.
• Requires TLS/mTLS implementation for secure inter-cluster communication.
• Incompatible with GitOps methodology as secrets cannot be safely stored in Git.
• Lacks built-in secret rotation capabilities.

While these security limitations of native Kubernetes Secrets are significant, they can be hardened to be secure. Let's explore how proper hardening techniques can mitigate many of these risks when alternative solutions aren't suitable.

Hardening Native Kubernetes Secrets

Encryption at Rest for etcd

By default, Kubernetes Secrets are stored unencrypted in etcd. To secure sensitive data, it at least needs to be encrypted at rest. This ensures that even if etcd is compromised, the attacker cannot access the plaintext secrets.

Implementing RBAC Controls

Role-Based Access Control (RBAC) is critical for limiting who can access and manage secrets. Configure precise RBAC policies that grant minimal privileges necessary for operation, ensuring that only authorized users and services can view or modify secrets. RBAC should be implemented at the namespace level rather than cluster-wide whenever possible, following the principle of least privilege.

Securing Communications with TLS/SSL

All communication channels involving secrets should be encrypted. This includes:

• API server to etcd communication
• Peer-to-peer communication between etcd nodes
• Client-to-API server connections

Configure TLS/SSL for all these communication paths and consider implementing mTLS for enhanced security. It is recommended to use encrypted communication channels for all operations of retrieving secrets to prevent man-in-the-middle attacks.

File encryption for git

As GitOps workflows are widely used nowadays, they bring some challenges with them.

It is also a fundamental security challenge for Kubernetes secret management, as storing sensitive credentials in Git repositories creates significant exposure risks. While encryption solutions like Sealed Secrets and SOPS offer pre-commit protection mechanisms, they introduce operational overhead. These tools require complex encryption key rotation and management protocols and don't fully mitigate the risk of accidentally committing unencrypted secrets. Ultimately, the encrypted-at-rest Git approach only relocates the security vulnerability surface rather than providing complete protection.

Beyond Native Secrets: The 2025 Best Practice

Now that we've seen the security problems and limitations of regular Kubernetes Secrets, let's look at some newer, better ways to handle sensitive information. These modern approaches – like SSCSI Driver and ESO – keep your secrets in safer places and just reference them when needed, giving you both better security and keeping things running smoothly while also introducing several useful features.

Beyond Native Secrets: Reference-Based Approaches for Enhanced Security

Secrets Store CSI Driver

The Secrets Store CSI (Container Storage Interface) Driver is a Kubernetes solution that integrates external secret management systems directly with your cluster. It creates a bridge between Kubernetes workloads and secret management services like Azure Key Vault, AWS Secrets Manager, HashiCorp Vault, and GCP Secret Manager.

The CSI Driver operates by mounting secrets as volumes directly into pods, completely bypassing the need to store sensitive information in Kubernetes etcd. This architecture significantly enhances security by ensuring secrets are never stored within the Kubernetes cluster itself.

Features:

• Multiple Provider Support: Works with various external secret stores, including AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, and HashiCorp Vault.
• SecretProviderClass CRD: This custom resource defines which secrets from the external store should be mounted in pods, supporting pod portability across environments.
• Cross-Platform Support: Works with both Linux and Windows containers.
• Native AKS Integration: Azure Kubernetes Service offers the Secrets Store CSI Driver with Azure Key Vault Provider as a built-in add-on, enabling one-click deployment during cluster creation without manual installation steps.
• Kubernetes Secrets Sync: Optionally, the driver can synchronize external secrets to native Kubernetes Secrets, enabling applications that can't directly use volume mounts to access the secrets.
• Auto-rotation: The driver periodically checks for updates to secrets in the external store (typically every two minutes) and automatically updates the mounted volumes.

Strengths:

• Helps to avoid most security concerns of native Kubernetes Secrets
• Secrets can be mounted directly as volumes into pods without ever being stored in Kubernetes etcd
• Even syncing these secrets to Kubernetes Secrets are available with certain conditions
• Supports automatic secret rotation without requiring pod restarts, though this feature is not enabled by default and remains in alpha status
• Seamlessly activated via a simple toggle during AKS cluster provisioning

Weaknesses:

• Requires a more complex setup compared to native Kubernetes Secrets without hardening, including configuration of provider-specific authentication and SecretProviderClass resources
• Cannot sync to secrets without a volume, which means every pod needs to define and mount a volume even if only using environment variables through synced Kubernetes Secrets
• When using the sync feature to create Kubernetes Secrets for environment variables, secrets are still stored in etcd, negating the security advantage of bypassing etcd storage and requiring etcd encryption-at-rest configuration
• Applications must be specifically designed to detect and reload secrets when files or Kubernetes Secrets change

Notes

Neither of the following features is stable; they’re considered alpha functionalities.

Kubernetes Secret Sync

Please note that when using the Kubernetes Secret sync feature to use these secrets as environment variables, the secrets will also be stored in etcd just like regular Kubernetes Secrets. This means that the concern about etcd encryption isn't solved with this approach –you'll still need to enable encryption-at-rest for your etcd database to properly secure these synced secrets. The best practice when using Secret Sync is to "Enable KMS application wrapping" to ensure that data is encrypted before it is stored in etcd.

Auto-rotation Functionality

The Secrets Store CSI Driver includes auto-rotation capability that isn't enabled by default. When activated using the “--enable-secret-rotation” flag, the driver polls the external secret store every 2 minutes (customizable via “--rotation-poll-interval”) and automatically updates both mounted files and any synced Kubernetes Secrets without requiring pod restarts.

For effective implementation, applications must be designed to detect and load updated secrets by watching for file changes or monitoring Kubernetes Secret updates.

External Secrets Operator

External Secrets Operator (ESO) is a Kubernetes operator designed to seamlessly integrate external secret management systems with your Kubernetes cluster. Unlike the CSI Driver's volume-based approach, ESO synchronizes secrets from external providers directly into native Kubernetes Secret objects by fetching secrets from external APIs and syncing them to your Kubernetes cluster. When secrets change in the external system, ESO automatically reconciles and updates the Kubernetes Secrets accordingly.

ESO introduces custom resources that define where secrets live and how to synchronize them:

• SecretStore/ClusterSecretStore: Defines how to authenticate with the external secrets provider
• ExternalSecret: Specifies which secrets to fetch and how they should be transformed into Kubernetes Secrets

ESO offers a wider range of integration capabilities with a lot of secret management systems, including AWS Secret Manager, Azure Key Vault, Google Secrets Manager, HashiCorp Vault, etc.

Features:

• Multiple Provider Support: Works with most of the external secret providers
• Automatic Synchronization: Continuously polls external APIs to keep Kubernetes Secrets updated when external values change
• Secret Transformation: Allows data to be fetched and transformed before creating Kubernetes Secrets
• Cluster-wide Scope: Supports both namespaced and cluster-wide secret management through different CRDs
• RBAC Integration: Supports detailed permissions control within Kubernetes

Strengths:

• Unlike CSI Driver, doesn't require volume mounts to sync secrets – can create Kubernetes Secrets directly
• Well-suited for legacy applications that expect secrets in the form of standard Kubernetes Secret objects
• Maintains synchronization when secrets are rotated in external systems
• Typically uses fewer resources than the CSI Driver approach (which requires a DaemonSet)
• Compared to the CSI Driver, it’s more compatible with external secret providers

Weaknesses:

• What is an advantage is also a disadvantage. It creates standard Kubernetes Secrets that are stored in etcd, requiring encryption at rest to be secure
• As well as CSI, it adds another component to manage within your Kubernetes infrastructure
• Some implementations (like with Vault) can consume significant memory resources for relatively few secrets
• Applications may need pod restarts to detect secret changes, depending on how they consume secrets

Comparison with SSCSI Driver

Conclusion

While modern Kubernetes secret management solutions offer significant improvements over native Secrets, none of them provides absolute security guarantees. Secrets can still be exposed through logs, debug output, or at the container level without proper hardening. A layered security approach remains essential.

The choice between native Kubernetes Secrets, Secrets Store CSI Driver, and External Secrets Operator depends on your specific requirements for security, operational simplicity, and integration with existing systems. For most production environments, reference-based approaches using external secret stores represent the current best practice.

The following flowchart can help you choose the best option for your needs.

In our next blog post, we’ll provide guidelines for hardening native Kubernetes Secrets, and we’ll provide hands-on examples for implementing configurations with Key Vault, Secrets Store CSI Driver, and External Secrets Operator with practical deployment templates and security recommendations as well.

Sources

Official Kubernetes documentation:

https://kubernetes.io/docs/concepts/configuration/secret/

https://kubernetes.io/docs/concepts/security/secrets-good-practices/

https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

Official Secret Store CSI Driver documentation: https://secrets-store-csi-driver.sigs.k8s.io/

Official External Secrets Operator documentation: https://external-secrets.io/

Official Microsoft Azure documentation on the topic of AKS:

https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver

https://learn.microsoft.com/en-us/azure/aks/aksarc/secrets-store-csi-driver

https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-identity-access?tabs=azure-portal&pivots=access-with-service-connector

https://www.redhat.com/en/blog/a-guide-to-secrets-management-with-gitops-and-kubernetes

https://dev.to/argonaut/secret-management-in-kubernetes-approaches-tools-and-best-practices-30k1?comments_sort=oldest

https://microsoft.github.io/code-with-engineering-playbook/CI-CD/gitops/secret-management/