Seamless Elasticsearch Migration to ECK Operator: Architecture, Pitfalls, and Real-World Best Practices

1. Introduction: Why Teams Are Moving to the ECK Operator

If your Elasticsearch Helm charts are getting harder to maintain with every Kubernetes upgrade, you’re not alone.
Helm is great for packaging manifests, but it wasn’t designed to manage the stateful lifecycle of Elasticsearch clusters rolling upgrades, persistent volumes, security contexts, or TLS certificates.

That’s where Elastic Cloud on Kubernetes (ECK) comes in.
ECK turns complex Helm deployments into declarative Custom Resources (CRDs) managed by an Operator that automatically reconciles health, scaling, and configuration.

At Hyperflex, we’ve helped multiple clients migrate production clusters (8.x and newer) to ECK 2.12, achieving full automation of scaling, TLS, and monitoring-with zero downtime.

2. Helm vs ECK Operator: What Changes Under the Hood

‍

The ECK Operator essentially replaces Helm logic with Kubernetes intelligence you declare intent; the Operator ensures the cluster reaches it.

3. Migration Architecture Overview

This flow represents a clean, iterative migration path that avoids downtime while preserving existing persistent volumes and data integrity.

4. Step-by-Step Migration Strategy

4.1 Prepare Your Cluster

• Backup all indices via Snapshot API (_snapshot/repo/snap1).
  
  
• Note existing PVC names and storage classes.
  
  
• Verify Kubernetes version ≥ 1.25 and CRD support enabled.

4.2 Install ECK Operator

kubectl apply -f https://download.elastic.co/downloads/eck/2.12.0/crds.yaml

kubectl apply -f https://download.elastic.co/downloads/eck/2.12.0/operator.yaml

Confirm Operator health:

kubectl get pods -n elastic-system

4.3 Migrate Elasticsearch Helm Charts → ECK CRD

Elasticsearch CRD (illustrative):

Keep it minimal; the Operator fills in TLS secrets, certs, and services automatically.

4.4 Deploy Kibana via ECK

‍

4.5 Ingress (NGINX Example)

4.6 Migrate PVCs and Snapshots

1. Map old PVCs from Helm releases to the new StatefulSets created by ECK.
   
   
2. If storage class names differ, use volumeClaimTemplates under spec.nodeSets.
   
   
3. Restore snapshots via Kibana > Stack Management > Snapshot and Restore.
   
   

4.7 Validate

Run:

kubectl get elasticsearch,kibana

‍

Ensure the Operator shows health = green.

5. Integrating Observability: Prometheus + Grafana + ECK Metrics

ECK exposes native metrics via /metrics endpoints on each component.

Prometheus scrape config snippet:

- job_name: 'eck'

  kubernetes_sd_configs:

  - role: pod

  relabel_configs:

  - source_labels: [__meta_kubernetes_pod_label_elastic_co_cluster]

    target_label: cluster

  metrics_path: /metrics

  scheme: https

‍

Once metrics are flowing, add Grafana dashboards for:

• JVM heap usage per node
  
• Indexing throughput
  
• Search latency
  
• Cluster health status
  
  

This allows proactive scaling before ingestion backpressure appears.

6. Common Pitfalls and How to Avoid Them

1. PVC Binding Mismatch
   
   • StorageClass names differ between Helm and ECK; use consistent naming or manually rebind PVCs.
     
2. CRD Drift after Operator Upgrade
   
   • Always upgrade ECK CRDs before the Operator.
     
3. RBAC Denied Errors
   
   • Ensure the Operator’s ServiceAccount has cluster-admin or namespace-wide permissions.
     
4. Pod Restart Loops
   
   • Usually caused by JVM heap limits-tune resources.requests to ≥ 2 GiB.
     
5. Ingress TLS Failure
   
   • Confirm certificates under elastic-certificate-pem and elastic-credentials Secrets.
     
6. Operator Not Reconciling
   
   • Check logs: kubectl logs -n elastic-system deploy/elastic-operator.
     

7. Migration Validation Checklist

✅ Backed up all indices with snapshots
✅ Old PVCs mapped to ECK nodeSets
✅ ECK Operator pods healthy
✅ CRDs validated with no errors
✅ Kibana reachable via Ingress URL
✅ Prometheus scraping ECK metrics
✅ No orphaned Helm resources left
✅ Indices green status in Cluster Health

If any of these fail - stop and investigate before scaling up nodes or performing rolling restarts.

8. Final Takeaway + CTA

Migrating from Helm to ECK Operator isn’t just about using new manifests-it’s about embracing Kubernetes-native control for stability, observability, and automation.

At Hyperflex, we’ve guided enterprises through production migrations involving multi-node clusters, secured TLS chains, and Prometheus-backed monitoring-achieving smooth transitions without data loss or downtime.

Hyperflex helps teams migrate Elasticsearch to ECK confidently - zero downtime, full observability, and certified Elastic engineers guiding every step.

📩 Contact us at marketing@hyperflex.co to plan your ECK migration.

‍