Executive Summary
Company introduced Central Helm, also known as MasterHelm, as a principal QUAD 3.0.0 capability to standardize Kubernetes application delivery. The program shifted recurring Helm-template maintenance from individual application teams to the DevOps Container Orchestration team, using a reusable central chart and application-specific values overrides.
Business Challenge
- Fragmented ownership: Each application maintained chart templates and had to react independently to Kubernetes API deprecations and removals.
- Slow upgrade readiness: Kubernetes upgrades required repeated application changes, engineering-pod validation, and sign-off before production rollout.
- High QA cost: Repeated chart validation consumed application QA capacity.
- Deployment coordination gaps: Direct Helm execution on Kubemaster hosts did not provide conflict-aware scheduling.
- Limited visibility: Teams needed consistent status across pre-deploy, deploy, post-deploy, and scheduled execution stages.
Program Context
- Central Helm Chart: A reusable DevOps-managed chart supports common Kubernetes workload, networking, storage, security, and scaling patterns. Applications provide only relevant values overrides.
- Configuration Model: Teams keep one quadfile.yaml, with a separate <release-name>-values-override.yaml for each release and an optional shared common-values-override.yaml.
- Scheduled Deployment: QUAD supports immediate, nightly, and later deployment modes. Scheduled deployments use ten-minute slots to coordinate workload changes and avoid operational conflicts.
- Enterprise CD Subscription: Eligible develop branches can automatically deploy new image tags to DEV pods after subscription onboarding.
Responsibilities
- Portfolio governance: Maintain the MasterHelm onboarding tracker across product families and applications.
- Ownership management: Ensure every application has a named TPM, application-team POC, Jira ticket, status, ETA, and remarks.
- Migration planning: Drive applications toward the mandatory onboarding deadline of March 31, 2026; identify non-Helm workloads as Not Applicable.
- Cross-functional coordination: Coordinate application teams, DevOps Development, DevOps QA, SRE/DSRE, and stakeholders during onboarding.
- Status reporting: Track Done, In Progress, Blocked, Duplicate, and Not Applicable states; surface dependencies and deployment-performance concerns.
- Jira governance: Ensure teams close their corresponding MasterHelm adoption tickets with confirmation and update the tracker.
- Risk and escalation management: Escalate migration blockers, platform dependencies, complex multi-service migrations, and deployment-duration issues.
- Acceptance tracking: Confirm application validation and deployment verification, including Central Helm release and app-version checks in the target namespace.
Execution
Platform Foundation
- Implemented the QUAD Operator to manage Continuous Delivery scheduling and deployment intent.
- Implemented the Helm Operator to manage Helm execution and lifecycle conditions.
- Introduced Continuous Delivery UI visibility and success/failure email notifications.
- Made the Central Helm Chart the reusable template layer for Kubernetes applications.
Application Onboarding
- Create a values override file for every Helm release.
- Create a common override file for shared service settings where required.
- Reference nexus/central-helm-chart and its selected version from quadfile.yaml.
- Separate multiple deployment types into individual releases where needed.
- Sync configuration, run QUAD immediately or schedule it, and validate the release using Helm information in the target namespace.
Deployment Flow
- The Jenkins QUAD job submits deployment input.
- Immediate jobs deploy now; nightly jobs run at scheduled time; later jobs allocate the nearest available requested slot.
- At the scheduled time, QUAD creates a Helm object.
- The Helm Operator creates an execution pod that runs helm install or helm upgrade in the target namespace.
Validation Coverage
- Immediate, nightly, scheduled, and parallel deployments.
- Pre-deploy-only, post-deploy-only, combined, invalid-time, and circular-dependency scenarios.
- Deployment, Service, Ingress, volumes, PVCs, ConfigMaps, Secrets, Jobs, CronJobs, DaemonSets, ReplicaSets, StatefulSets, HPA, and custom metrics.
- Rollback behavior, service-mesh workloads, email notification, and transient execution-pod cleanup.
Outcomes
- Standardized deployment model: One centrally governed Helm chart pattern supports Kubernetes application deployment.
- Reduced application chart maintenance: Teams manage configuration values rather than maintaining common template logic for Kubernetes API changes.
- Controlled release coordination: CD resource scheduling and ten-minute slots support operational conflict avoidance.
- Improved observability: Helm resources, UI, execution-pod status, and email notifications expose deployment state.
- Better configuration isolation: Per-release overrides replace the former shared override-file model.
- Portfolio governance: Tracker records product ownership, Jira work items, and completion, in-progress, duplicate, blocked, and not-applicable states.
- Engineering-pod evidence: Deployment status was recorded as complete for all integration pods.
Lessons Learned
- Standardization requires extensibility: A shared chart must cover common workload, networking, storage, security, and scaling needs through configuration.
- Preserve familiar inputs: Retaining quadfile.yaml reduced migration friction while modernizing execution architecture.
- Scheduling is an architectural capability: In-cluster execution and reserved slots made safe coordination part of the delivery platform.
- Status must be visible end to end: Scheduling state, Helm execution conditions, workload health, UI visibility, and alerts are complementary controls.
- Portfolio migration needs governance: Ownership, dates, exception states, trackers, and escalation paths are required for broad adoption.
- Operational validation exceeds template testing: Invalid input, concurrency, rollback, notification, cleanup, and service-mesh behavior must be tested.
- Performance remains a product requirement: Adoption success should be measured alongside deployment-duration performance and platform improvements.
Architecture
Enterprise Central Helm model
Jenkins starts the QUAD Job using the QUAD file and override values. QUAD CD creates and tracks the deployment request. The Scheduler allocates a slot for scheduled deployments; immediate jobs bypass this path. Helm and the Helm Library create an execution pod and perform the Helm action. The system monitors execution and reports completion or failure. QUAD Subscriptions can independently trigger the same deployment flow when Nexus artifacts change.
Layer 01
CI/CD Entry Layer
Jenkins starts the QUAD Job using the QUAD file and override values.Layer 02
Deployment Orchestration Layer
QUAD CD creates and tracks the deployment request. The Scheduler allocates a slot for scheduled deployments; immediate jobs bypass this path.Layer 03
Helm Execution Layer
The Helm Library create an execution pod and perform the Helm action.Layer 04
Kubernetes Runtime Layer
The system monitors execution and reports completion or failure.Layer 05
Continuous Deployment Subscription Layer
QUAD Subscriptions can independently trigger the same deployment flow when Nexus artifacts change.
Operating model
Stakeholder matrix
| Stakeholder | Engagement |
|---|---|
| DevOps Container Orchestration | wn the Central Helm templates; operate QUAD, and Helm; support onboarding; govern common deployment behavior. |
| Application Teams | Create release-specific values overrides, validate application behavior, provide configuration, and close onboarding tickets. |
| SRE / DSRE | Coordinate maintenance windows, protect operational activity, receive deployment notifications, and support escalations. |
| TPM | Manage ownership, milestones, Jira progress, exceptions, blockers, and cross-team communication. |
| Leadership | Portfolio governance and risk decisions. |
Risk discipline
Risks and responses
Risk
Shared chart cannot accommodate application-specific behavior
Response
Used values-driven support for workload types, ingress, storage, secrets, autoscaling, and custom metrics.
Risk
Deployment duration regression
Response
Captured reported MasterHelm performance concerns as improvement work rather than treating functional onboarding as sufficient.
Risk
Large-portfolio migration delay
Response
Applied a tracker, assigned Jira ownership, dedicated TPM, and supported Not Applicable disposition.