Pattern: Canary Monitoring

Pattern: Canary Monitoring

Category: Quality Assurance Source: garrytan/gstack (/canary) Status: Cataloged Evaluation: RD-0013

When to Use

After a digital talent or system update is deployed to a client environment. Verifies the deployment is healthy by monitoring for errors, performance regressions, and unexpected behavior in the live environment. Fills the gap between "deploy succeeded" and "deploy is actually working."

How It Works

  • Post-deploy health check: Immediately after deployment, run a verification suite
    • Check for console errors, unhandled exceptions, failed API calls
    • Verify core user flows still function (smoke tests)
    • Compare response times against pre-deploy baselines
  • Performance regression detection: Compare key metrics before and after deploy
    • Response latency (P50, P95, P99)
    • Error rates
    • Resource utilization
    • Core Web Vitals for web-facing deployments
  • Alerting: If regressions exceed configurable thresholds, flag for rollback decision
  • Soak period: Continue monitoring for a defined window (e.g., 30 minutes) after deploy to catch delayed issues

Example

After deploying an updated EA agent to the STM environment, the canary pattern runs the agent through 5 standard analysis scenarios, compares output quality and response times against the previous version's baselines, and monitors for 30 minutes. It detects that one scenario now takes 3x longer due to an unoptimized prompt chain. The regression is flagged before the client notices.

Tradeoffs

Pro Con
Catches deployment issues before users do Requires baseline metrics to compare against
Automated — no manual post-deploy verification Adds time between deploy and "done"
Provides rollback decision data Not all regressions are detectable by automated checks
Builds confidence for continuous delivery Client environment access may be restricted

Factory Usage

  • Diego (Deployment Specialist): Primary owner — run after every client deployment
  • Dana (Delivery Manager): Review canary reports as part of delivery sign-off
  • Extends: Quality Gates pattern (orchestration/quality-gates.md) into the post-deploy phase