When our team decided to migrate our MongoDB deployment to Kubernetes, we thought it would be straightforward. After all, Helm charts exist for almost everything, right? What followed was a journey through StatefulSets, persistent volumes, and configuration challenges that taught us valuable lessons about stateful applications in Kubernetes.

The Initial Approach: Helm to the Rescue?

Our first instinct was to use the official MongoDB Helm chart. The installation seemed simple enough:

# Add the MongoDB Helm repository
helm repo add bitnami https://charts.bitnami.com/bitnami

# Install MongoDB with default settings
helm install my-mongodb bitnami/mongodb

But when we ran this command, we immediately hit our first error:

Error: INSTALLATION FAILED: Kubernetes cluster unreachable: Get "https://kubernetes.default.svc/version": dial tcp 10.96.0.1:443: connect: network is unreachable

This was just a basic configuration issue with our kubeconfig. After fixing that, we tried again and got further, but soon faced a more complex problem.

Problem #1: PVC Name Conflicts

Our second attempt appeared successful initially, but when we looked closer, we found that the PersistentVolumeClaim (PVC) was in a pending state:

$ kubectl get pvc
NAME                    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
my-mongodb-data         Pending                                      standard        2m

Further investigation showed that we had multiple PVCs with the same name from previous failed deployments. The Kubernetes PVC provisioner was confused because:

1. PVCs are namespaced resources
2. We kept using the same release name (my-mongodb)
3. Deleting a Helm release doesn't automatically delete PVCs for data safety

After cleaning up the old PVCs, we still faced issues with our storage class configuration.

Problem #2: Understanding StatefulSets

The next revelation came when we realized that MongoDB in Kubernetes is deployed as a StatefulSet rather than a Deployment. This was new territory for our team, which had mostly worked with stateless applications.

StatefulSets have some critical differences:

• Pods are created with predictable, stable network identities
• Each pod gets its own persistent storage
• Pods are created and deleted in order (not in parallel)
• Scale-down operations happen one at a time, in reverse order

This explained why our persistent volume claims were so important and why the naming was causing conflicts.

Problem #3: Readiness Probe Configuration

After fixing our PVC issues and learning about StatefulSets, we managed to get the MongoDB pod running, but it kept restarting:

$ kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
my-mongodb-0     0/1     CrashLoopBackOff   5          10m

Looking at the logs, we found that the pod was failing its readiness probe:

$ kubectl logs my-mongodb-0
{"t":{"$date":"2023-09-09T20:41:35.431+00:00"},"s":"I",  "c":"CONTROL",  "id":20698,   "ctx":"main","msg":"***** SERVER RESTARTED *****"}
{"t":{"$date":"2023-09-09T20:41:35.432+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
...
{"t":{"$date":"2023-09-09T20:42:05.553+00:00"},"s":"E",  "c":"NETWORK",  "id":22944,   "ctx":"main","msg":"Failed to set up listener","attr":{"addressString":"my-mongodb-0.my-mongodb-headless.default.svc.cluster.local:27017","error":{"code":9001,"codeName":"SocketException","errmsg":"Cannot bind to socket: Permission denied"}}}
{"t":{"$date":"2023-09-09T20:42:05.553+00:00"},"s":"F",  "c":"CONTROL",  "id":20574,   "ctx":"main","msg":"Error during startup","attr":{"error":"NonExistentPath: Data directory /bitnami/mongodb/data/db not found. Create the missing directory or specify another path using (1) the --dbpath command line option, or (2) by adding the 'storage.dbPath' option in the configuration file."}}
...

The error message revealed two issues:

1. MongoDB couldn't bind to the configured socket (permission denied)
2. The data directory wasn't found or properly mounted

Solution: Custom Values for the Helm Chart

After digging into the Bitnami MongoDB Helm chart documentation, we created a custom values.yaml file to address our specific issues:

# mongodb-values.yaml
architecture: replicaset
replicaCount: 3

auth:
  enabled: true
  rootPassword: "YourSecureRootPassword"  # In production, use Kubernetes secrets
  username: "appUser"
  password: "YourSecureUserPassword"
  database: "appDatabase"

persistence:
  enabled: true
  storageClass: "standard"
  size: 10Gi
  
podSecurityContext:
  enabled: true
  fsGroup: 1001
  
containerSecurityContext:
  enabled: true
  runAsUser: 1001

resources:
  limits:
    cpu: 1000m
    memory: 1Gi
  requests:
    cpu: 500m
    memory: 512Mi

readinessProbe:
  enabled: true
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 6
  successThreshold: 1

livenessProbe:
  enabled: true
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 6
  successThreshold: 1

With this values file, we reinstalled MongoDB with a more specific release name to avoid conflicts:

# First, delete the old release (but keep PVCs)
helm delete my-mongodb

# Install with custom values
helm install mongodb-production bitnami/mongodb -f mongodb-values.yaml -n database

Problem #4: ConfigMap Path Issues

Despite our progress, we still needed to customize MongoDB configuration parameters. We decided to use a ConfigMap to provide a custom mongod.conf file:

# Create a ConfigMap from our mongod.conf file
kubectl create configmap mongodb-config --from-file=mongod.conf -n database

# Our mongod.conf contained:
storage:
  dbPath: /bitnami/mongodb/data/db
  journal:
    enabled: true
  wiredTiger:
    engineConfig:
      cacheSizeGB: 0.5
systemLog:
  destination: file
  path: /bitnami/mongodb/logs/mongodb.log
  logAppend: true
net:
  port: 27017
  bindIp: 0.0.0.0
replication:
  replSetName: rs0
security:
  authorization: enabled

However, when we tried to mount this ConfigMap in our Helm values, we hit another issue:

extraVolumes:
  - name: config-volume
    configMap:
      name: mongodb-config
extraVolumeMounts:
  - name: config-volume
    mountPath: /opt/bitnami/mongodb/conf/mongod.conf
    subPath: mongod.conf

The pod still failed to start, and when we examined the logs, we found:

Error parsing YAML config file: /opt/bitnami/mongodb/conf/mongod.conf: Bad YAML content

The issue was that we had mounted our ConfigMap as a file directly over the path that was expected to be a file, not as a directory containing the file. After testing different approaches, we found the correct way to handle the configuration:

# In mongodb-values.yaml
extraVolumes:
  - name: config-volume
    configMap:
      name: mongodb-config
extraVolumeMounts:
  - name: config-volume
    mountPath: /bitnami/mongodb/conf/
    
# And modify the command to use this config
command:
  - "/bin/bash"
  - "-ec"
  - |
    exec mongod --config=/bitnami/mongodb/conf/mongod.conf

Final Working Configuration

After solving all these issues, we arrived at a stable MongoDB deployment on Kubernetes. Our final approach included:

1. Using the Bitnami MongoDB Helm chart with custom values
2. Deploying in a dedicated namespace with proper resource limits
3. Setting up appropriate security contexts and storage configurations
4. Adjusting readiness and liveness probes to handle MongoDB startup time
5. Using ConfigMaps for custom MongoDB configuration

The final deployment commands were:

# Create a dedicated namespace
kubectl create namespace database

# Create a secret for MongoDB credentials
kubectl create secret generic mongodb-credentials   --from-literal=mongodb-root-password="YourSecureRootPassword"   --from-literal=mongodb-password="YourSecureUserPassword"   -n database

# Create ConfigMap for MongoDB configuration
kubectl create configmap mongodb-config --from-file=mongod.conf -n database

# Install MongoDB with custom values
helm install mongodb-production bitnami/mongodb   -f mongodb-values.yaml   -n database   --set auth.rootPassword=''   --set auth.password=''   --set auth.existingSecret=mongodb-credentials

Monitoring the Deployment

To ensure our MongoDB deployment was healthy and performing well, we added Prometheus monitoring. We added this configuration to capture MongoDB metrics:

# In mongodb-values.yaml
metrics:
  enabled: true
  serviceMonitor:
    enabled: true
    additionalLabels:
      release: prometheus-operator

This allowed us to create Grafana dashboards to monitor:

• MongoDB operation counts and latencies
• Connection pool utilization
• Cache hit ratios
• Disk space usage
• Replication lag between nodes

Lessons Learned

Our journey deploying MongoDB on Kubernetes taught us several critical lessons:

1. StatefulSets are different: Understand how StatefulSets work before deploying stateful applications like databases.
2. Storage matters: Persistent volume claims and storage classes need careful consideration.
3. Security context is critical: Permissions issues can be subtle and difficult to debug.
4. Resource limits prevent surprises: Always configure appropriate CPU and memory limits.
5. Probes need tuning: Databases often need longer startup times than typical applications.

Should You Run MongoDB on Kubernetes?

After going through this process, we had mixed feelings. On one hand, we succeeded in creating a stable MongoDB deployment on Kubernetes. On the other hand, we discovered that running databases in Kubernetes adds complexity.

For production scenarios with critical data, consider these alternatives:

1. Managed MongoDB services (MongoDB Atlas, AWS DocumentDB)
2. Running MongoDB on dedicated instances outside Kubernetes
3. Using MongoDB Kubernetes Operator for more advanced features

However, if you do need to run MongoDB on Kubernetes (for development, testing, or specific production scenarios), the approach outlined here provides a solid foundation.

Conclusion

Deploying MongoDB on Kubernetes is indeed possible, but requires careful consideration of StatefulSets, persistence, security, and configuration. The challenges we faced taught us a lot about both MongoDB and Kubernetes.

For teams looking to follow a similar path, I recommend starting with a simpler deployment in a non-production environment, thoroughly understanding the concepts involved, and gradually adding complexity as you become more comfortable with the system.

The final reward of a successfully running MongoDB cluster on Kubernetes is worth the effort—you gain flexibility, declarative configuration, and integration with your existing Kubernetes ecosystem.