Dev-Branch Deploy — Hermes & Maia

Test a Hermes or Maia stack change in a live lab region before merging to master, using the pipeline's parallel dev lane. One workflow, two stacks — parametrized by stack.

The problem this solves

Concourse pins a git branch: per resource at pipeline-definition time; you can't override it at trigger time. So normally you must merge to master to see a chart/secrets change run in a lab. The dev lane adds a second set of jobs pointed at <stack>-dev-branch, so you deploy and validate in a real lab region with no risk to master or production.

Stack parameters

Everything below is identical between stacks except these values.

Param	hermes	maia
Dev branch	hermes-dev-branch	maia-dev-branch
fly target	ci-monitoring	monitoring
Concourse	https://ci1.eu-de-2.cloud.sap	https://ci.eu-de-2.cloud.sap
Pipeline dir	ci/hermes/ in cc/secrets	ci/maia/ in cc/secrets
set-pipeline vars	—	--load-vars-from vars.yaml
Chart	openstack/hermes	openstack/maia

Constants (both stacks):

• Team: monitoring
• Lab regions the dev lane deploys to: qa-de-1, qa-de-2, qa-de-3, eu-de-3
• Repos the dev branch must exist on: cc/secrets (GHE github.wdf.sap.corp) and sapcc/helm-charts (github.com)
• Local cc/secrets clone: ~/gh/secrets

The two lanes

Lane	Branch	Role
labs	master	Guards production-branch health. Never break it.
dev	<stack>-dev-branch	Sandbox: validate a feature in a real lab before master.

Both deploy to the same lab regions, independently. Production lanes (bronze/silver/gold/global) are untouched by this workflow.

Pipeline wiring (reference)

Dev lane = paired git resources + a dev job group.

• secrets-dev.git → <stack>-dev-branch (twin of secrets.git on master)
• helm-charts-dev.git → <stack>-dev-branch (twin of helm-charts.git on master)
• Shared by both lanes: built image, kube-secrets, credentials.

Dev jobs run the same helm upgrade as prod but input-map the dev resources over prod:

input_mapping:
  helm-charts.git:  helm-charts-dev.git
  secrets.git:      secrets-dev.git

Dev group jobs:

Job	Purpose
deploy-all-dev	Manual fan-out trigger for the whole dev lane
deploy-to-dev-<region>	helm upgrade per lab region, from the dev branch
sync-master-to-dev	Keep the dev branch caught up with master

Preflight — ALWAYS run before using a stack's dev lane

Confirm the dev branch exists on both repos and the pipeline is actually wired. Substitute <stack> = hermes or maia.

STACK=hermes            # or: maia
DEVBRANCH=${STACK}-dev-branch

# 1. Dev branch on internal secrets repo
git -C ~/gh/secrets ls-remote --heads origin "$DEVBRANCH"

# 2. Dev branch on public helm-charts repo
git ls-remote --heads https://github.com/sapcc/helm-charts.git "$DEVBRANCH"

# 3. Is the dev lane present in the generated pipeline? (erb must define the dev group)
grep -n "helm-charts-dev.git\|deploy-all-dev\|${DEVBRANCH}" ~/gh/secrets/ci/${STACK}/pipeline.yaml.erb

Decision logic:

1. Both ls-remote return a SHA and grep shows the dev group → dev lane is ready, proceed.
2. Branch missing on one/both repos → stop. It must be created on both before the lane works. Report exactly which repo is missing it.
3. erb has no dev group → the pipeline change isn't merged/generated yet. Report that; do not proceed.

Hermes is fully wired (branch on both repos, dev group live on master). Maia's dev lane may still be on the unmerged branch add-maia-dev-group, and maia-dev-branch may not exist yet — the preflight catches this.

Developer workflow

1. Cut a feature branch from master in helm-charts and/or secrets as needed.
2. Merge the feature branch into <stack>-dev-branch (merge, never force-push).

   git -C ~/gh/secrets fetch origin
   git -C ~/gh/secrets checkout "$DEVBRANCH"
   git -C ~/gh/secrets merge --no-ff origin/<feature-branch>
   git -C ~/gh/secrets push origin "$DEVBRANCH"
   

3. Concourse auto-picks-up the change on the dev branch and triggers the dev lane.
4. Validate deploy-to-dev-<region> goes green across all lab regions — this is the proof.

   fly -t <target> watch -j ${STACK}/deploy-to-dev-qa-de-1
   

   Or trigger the whole lane manually:

   fly -t <target> trigger-job -j ${STACK}/deploy-all-dev -w
   

5. Merge the feature branch to master as normal (via PR).
6. Confirm the labs lane stays healthy on master.

Keeping the dev branch fresh

Before starting new dev work, sync the dev branch to master so you're testing against current state:

fly -t <target> trigger-job -j ${STACK}/sync-master-to-dev -w

(Or merge origin/master into the dev branch locally and push — never force-push.)

Regenerate & ship the pipeline

The pipeline is ERB-generated, then pushed with fly. Log in first: fly -t <target> login.

Hermes:

cd ~/gh/secrets/ci/hermes
erb pipeline.yaml.erb > pipeline.yaml
fly -t ci-monitoring set-pipeline -p hermes -c pipeline.yaml

Maia:

cd ~/gh/secrets/ci/maia
erb pipeline.yaml.erb > pipeline.yaml
fly -t monitoring set-pipeline -p maia -c pipeline.yaml --load-vars-from vars.yaml

Rules (hard constraints)

• Never break labs — it tracks master only and gates production.
• No force-pushes to any <stack>-dev-branch. It's a shared integration branch.
• Merge features into the dev branch; don't rewrite its history.
• This workflow touches lab regions only. Production lanes are out of scope.

Error handling

Symptom	Cause / fix
ls-remote returns nothing for the dev branch	Branch not created on that repo. Create <stack>-dev-branch on both cc/secrets and sapcc/helm-charts before deploying.
grep finds no deploy-all-dev in erb	Dev-lane pipeline change not merged/generated. Merge it, erb > pipeline.yaml, fly set-pipeline.
deploy-to-dev-* never triggers	Confirm the change landed on <stack>-dev-branch (not just a feature branch); check the *-dev.git resource in Concourse is finding the commit.
fly: unknown target	fly -t <target> login first. Hermes=ci-monitoring, Maia=monitoring.
Dev deploy passes but master breaks after merge	The feature diverged from what was tested on dev — re-sync dev to master and re-validate before re-merging.