CI/CD Pipeline Optimization: Cutting Build Times by 70%

Slow pipelines kill developer productivity. A 30-minute build means 30 minutes of waiting per deployment. Here's how to optimize.

Analyze First

Measure Everything

# GitHub Actions timing
jobs:
  build:
    steps:
      - name: Measure step
        run: echo "::notice::Step completed in $SECONDS seconds"

Identify Bottlenecks

1. Dependency installation - Often 30-50% of build time
2. Test execution - Sequential tests waste parallel capacity
3. Docker builds - Unoptimized layers repeat work
4. Deployment - Sequential environments compound delays

Caching Strategies

Dependency Caching

- uses: actions/cache@v3
  with:
    path: |
      ~/.npm
      node_modules
    key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-npm-

Docker Layer Caching

# Order matters - frequently changing layers last
FROM node:18 AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci

FROM node:18 AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

GitHub Actions Cache for Docker

- uses: docker/setup-buildx-action@v2
- uses: docker/build-push-action@v4
  with:
    cache-from: type=gha
    cache-to: type=gha,mode=max

Parallel Execution

Matrix Strategy

strategy:
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: npm test -- --shard=${{ matrix.shard }}/4

Parallel Jobs

jobs:
  lint:
    runs-on: ubuntu-latest
    steps: [lint steps]

  test-unit:
    runs-on: ubuntu-latest
    steps: [unit test steps]

  test-e2e:
    runs-on: ubuntu-latest
    steps: [e2e steps]

  build:
    needs: [lint, test-unit, test-e2e]
    steps: [build steps]

Test Optimization

Selective Testing

# Only run affected tests
git diff --name-only origin/main | grep '\.ts$' | \
  xargs -I {} npm test -- --findRelatedTests {}

Test Parallelization

# Jest configuration
{
  "maxWorkers": "50%",
  "shard": "1/4"
}

Resource Management

Use Appropriate Runners

# CPU-intensive builds
runs-on: ubuntu-latest-8-cores

# Memory-intensive operations
runs-on: ubuntu-latest-32gb

Timeout Management

jobs:
  build:
    timeout-minutes: 15
    steps:
      - timeout-minutes: 5
        run: npm run build

Real Results

Metric	Before	After	Improvement
Total build time	45 min	12 min	73% faster
npm install	8 min	45 sec	90% faster
Test suite	20 min	6 min	70% faster
Docker build	12 min	3 min	75% faster

Conclusion

Optimize incrementally. Start with caching, add parallelization, then fine-tune. Measure before and after each change.