Home

Backend & DevOps Blog

Real-world experiences with MongoDB, Docker, Kubernetes and more

Creating Environment-Specific Deployment Workflows with GitHub Actions

When our team decided to formalize our deployment process by separating staging and production environments, GitHub Actions seemed like the perfect solution. What started as a simple workflow setup turned into a deeper exploration of GitHub Actions' environment-specific features and some interesting challenges along the way.

The Initial Problem: Deployment Inconsistency

Before implementing proper CI/CD, our deployment process was manual and prone to errors:

  • Developers often forgot to run tests before deploying
  • Configuration differences between environments caused unexpected bugs
  • Rollbacks were complicated and time-consuming
  • No clear history of who deployed what and when

We needed an automated system that would:

  1. Run our test suite before any deployment
  2. Deploy to staging automatically on main branch updates
  3. Deploy to production only after manual approval
  4. Use environment-specific configuration values
  5. Provide clear deployment logs and easy rollbacks

Setting Up GitHub Environments

Our first step was to configure GitHub Environments with the appropriate protection rules:

# In GitHub repository settings:

# 1. Create 'staging' environment:
# - No protection rules
# - Environment variables:
#   - API_URL: https://api-staging.example.com
#   - DEPLOYMENT_BUCKET: staging-deploys

# 2. Create 'production' environment:
# - Protection rules:
#   - Required reviewers: [tech-leads]
#   - Wait timer: 10 minutes
# - Environment variables:
#   - API_URL: https://api.example.com
#   - DEPLOYMENT_BUCKET: production-deploys
#   - CLOUDFRONT_ID: XYZ123

Creating the Workflow Files

We created two separate workflow files to handle our deployment pipeline:

1. Continuous Integration Workflow (.github/workflows/ci.yml):

name: CI

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main, develop ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '16'
          cache: 'npm'
          
      - name: Install dependencies
        run: npm ci
        
      - name: Lint code
        run: npm run lint
        
      - name: Run tests
        run: npm test
        
      - name: Build application
        run: npm run build

      # Store the build for deployment jobs
      - name: Upload build artifact
        uses: actions/upload-artifact@v3
        with:
          name: build-artifact
          path: .next/

2. Environment-Specific Deployment Workflow (.github/workflows/deploy.yml):

name: Deploy

on:
  workflow_run:
    workflows: ["CI"]
    types:
      - completed
    branches:
      - main

jobs:
  # Deploy to staging automatically after CI passes
  deploy-staging:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    runs-on: ubuntu-latest
    environment: staging
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '16'
          
      - name: Install dependencies
        run: npm ci
        
      - name: Download build artifact
        uses: actions/download-artifact@v3
        with:
          name: build-artifact
          path: .next/
          
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
          
      - name: Deploy to S3
        run: |
          aws s3 sync .next/ s3://${{ env.DEPLOYMENT_BUCKET }}/.next/ --delete
          aws s3 sync public/ s3://${{ env.DEPLOYMENT_BUCKET }}/public/ --delete
          
      - name: Notify deployment
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} -H 'Content-Type: application/json' -d '{
            "text": "🚀 Staging deployment successful! Version: ${{ github.sha }}"
          }'

  # Deploy to production only after manual approval
  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '16'
          
      - name: Install dependencies
        run: npm ci
        
      - name: Download build artifact
        uses: actions/download-artifact@v3
        with:
          name: build-artifact
          path: .next/
          
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
          
      - name: Deploy to S3
        run: |
          aws s3 sync .next/ s3://${{ env.DEPLOYMENT_BUCKET }}/.next/ --delete
          aws s3 sync public/ s3://${{ env.DEPLOYMENT_BUCKET }}/public/ --delete
          
      - name: Invalidate CloudFront
        run: |
          aws cloudfront create-invalidation --distribution-id ${{ env.CLOUDFRONT_ID }} --paths "/*"
          
      - name: Create deployment tag
        run: |
          git tag production-$(date +'%Y%m%d-%H%M%S')
          git push origin production-$(date +'%Y%m%d-%H%M%S')
          
      - name: Notify deployment
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} -H 'Content-Type: application/json' -d '{
            "text": "✅ Production deployment successful! Version: ${{ github.sha }}"
          }'

Problems We Encountered

Our initial setup had several issues that required troubleshooting:

1. Artifact Access Between Workflows

We quickly discovered that artifacts from one workflow aren't automatically available to another workflow. The solution required implementing a more complex artifact sharing strategy:

# Modified artifact handling in deploy.yml
- name: Download build artifact
  uses: dawidd6/action-download-artifact@v2
  with:
    workflow: ci.yml
    workflow_conclusion: success
    name: build-artifact
    path: .next/

2. Environment Variable Scope

We learned that environment variables defined in GitHub Environments aren't automatically exposed to jobs. We needed to explicitly reference them:

# Corrected environment variable usage
- name: Deploy with environment-specific settings
  run: |
    echo "Deploying to ${{ env.API_URL }}"
    npm run deploy -- --api-url=${{ env.API_URL }}

3. Conditional Deployment Logic

We needed different behavior based on the environment. We solved this with environment-specific conditions:

# Dynamic deployment step based on environment
- name: Run deployment steps
  run: |
    if [[ "${{ github.event.deployment.environment }}" == "production" ]]; then
      # Production-specific steps
      npm run deploy:production
      npm run post-deploy:notify-users
    else
      # Staging-specific steps
      npm run deploy:staging
      npm run post-deploy:run-tests
    fi

4. Handling Deployment Failures

We needed a better way to handle and report deployment failures. We implemented a custom error handler:

# Error handling for deployments
- name: Deploy with error handling
  id: deploy
  run: |
    if ! npm run deploy; then
      echo "::set-output name=status::failed"
      exit 1
    else
      echo "::set-output name=status::success"
    fi

- name: Report deployment failure
  if: steps.deploy.outputs.status == 'failed' || failure()
  run: |
    curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} -H 'Content-Type: application/json' -d '{
      "text": "❌ Deployment to ${{ github.event.deployment.environment }} FAILED! <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|See details>"
    }'

Adding Emergency Rollback

After a deployment went wrong, we realized we needed a quick rollback mechanism:

name: Emergency Rollback

on:
  workflow_dispatch:
    inputs:
      version:
        description: 'Tag to roll back to (e.g., production-20230615-123045)'
        required: true
        
jobs:
  rollback:
    runs-on: ubuntu-latest
    environment: production
    
    steps:
      - uses: actions/checkout@v3
        with:
          ref: ${{ github.event.inputs.version }}
      
      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '16'
          
      - name: Install dependencies
        run: npm ci
        
      - name: Build application
        run: npm run build
        
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
          
      - name: Deploy previous version
        run: |
          aws s3 sync .next/ s3://${{ env.DEPLOYMENT_BUCKET }}/.next/ --delete
          aws s3 sync public/ s3://${{ env.DEPLOYMENT_BUCKET }}/public/ --delete
          
      - name: Invalidate CloudFront
        run: |
          aws cloudfront create-invalidation --distribution-id ${{ env.CLOUDFRONT_ID }} --paths "/*"
          
      - name: Notify rollback
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} -H 'Content-Type: application/json' -d '{
            "text": "🔄 EMERGENCY ROLLBACK to ${{ github.event.inputs.version }} completed!"
          }'

Monitoring Deployments

To track our deployments across environments, we built a custom dashboard using GitHub's API. The script pulls deployment history from both environments:

#!/usr/bin/env node
const fetch = require('node-fetch');

async function fetchDeployments() {
  const response = await fetch(
    'https://api.github.com/repos/our-org/our-repo/deployments',
    {
      headers: {
        Authorization: `Bearer ${process.env.GITHUB_TOKEN}`,
      },
    }
  );
  
  const deployments = await response.json();
  
  // Group by environment
  const byEnvironment = deployments.reduce((acc, deployment) => {
    const env = deployment.environment;
    if (!acc[env]) acc[env] = [];
    acc[env].push({
      id: deployment.id,
      sha: deployment.sha.substring(0, 7),
      creator: deployment.creator.login,
      created_at: new Date(deployment.created_at).toLocaleString(),
      status: deployment.statuses?.sort((a, b) => 
        new Date(b.created_at) - new Date(a.created_at)
      )[0]?.state || 'unknown'
    });
    return acc;
  }, {});
  
  console.log('=== DEPLOYMENT HISTORY ===');
  
  for (const [env, deploys] of Object.entries(byEnvironment)) {
    console.log(`\n${env.toUpperCase()} ENVIRONMENT:`);
    console.table(deploys.slice(0, 5)); // Show last 5 deployments
  }
}

fetchDeployments().catch(console.error);

Lessons Learned

Setting up environment-specific workflows taught us several valuable lessons:

  1. Start with clear environment definitions: Define what makes each environment unique before writing workflows.
  2. Test workflows in isolation: Use workflow_dispatch to trigger individual workflows during development.
  3. Plan for failure: Always build notification and rollback mechanisms from the start.
  4. Document everything: Create a deployment guide for the team explaining how the workflows function.
  5. Secret management is critical: Use environment-specific secrets and rotate them regularly.

One unexpected benefit was how this process forced us to standardize our deployment requirements across environments, which improved overall system reliability.

Final Implementation: Enhanced Security

After our initial success, we enhanced security by implementing OpenID Connect (OIDC) for AWS authentication instead of using long-lived access keys:

# Updated AWS credential handling
- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v1
  with:
    role-to-assume: arn:aws:iam::123456789012:role/github-actions-${{ github.event.deployment.environment }}
    aws-region: us-east-1

This approach eliminated the need to store AWS credentials as GitHub secrets, reducing our security exposure.

Conclusion

GitHub Actions environment-specific workflows have transformed our deployment process from a manual, error-prone task to a reliable, automated system. The initial setup required some troubleshooting, but the benefits far outweighed the effort.

Our team now enjoys:

  • Consistent deployments with appropriate approvals
  • Environment-specific configuration without code changes
  • Clear deployment history and audit trails
  • Quick recovery options when issues arise
  • Better security through OIDC and environment isolation

For teams considering a similar approach, I recommend starting with a minimal viable pipeline and iteratively adding features like notifications, rollbacks, and enhanced security. This incremental approach allows you to address issues one at a time and build confidence in your deployment system.

← Back to all posts