Playbook

Automated rollback for a failing ECS Fargate deployment without custom scripting.

→Enable the ECS deployment circuit breaker with rollback on the ECS service.

Why: Native ECS feature that automatically rolls back if new tasks fail to stabilize. Least operational overhead compared to custom CodeBuild polling or complex CodeDeploy setups.

Reference↗

Deploy to a primary region, validate with automated tests, then deploy to other regions in parallel.

→Use a single CodePipeline with sequential stages: (1) Deploy Region A, (2) a CodeBuild test stage that runs validation, (3) a parallel deploy stage for Regions B & C.

Why: CodeBuild acts as an automated, programmatic gate. A single pipeline is simpler than orchestrating multiple pipelines with Step Functions.

A long-running validation script in a CodeDeploy lifecycle hook causes premature deployment success.

→Increase the `timeout` property for the specific lifecycle hook script in the `AppSpec.yml` file.

Why: The timeout is configured per-hook in the AppSpec file, not at the deployment group level. This ensures the validation script has enough time to complete.

Accelerate slow CodeBuild Docker image builds caused by re-downloading dependencies and image layers on every run.

→In the CodeBuild project config, enable `LOCAL_DOCKER_LAYER_CACHE` and configure an S3 cache for dependency directories (e.g., `.m2`, `node_modules`).

Why: Addresses both sources of slowness directly. Docker layer caching reuses unchanged image layers; S3 caching reuses downloaded application dependencies.

Implement a canary deployment for a Lambda function with automated, metric-driven rollback.

→Use AWS SAM with `DeploymentPreference` (e.g., type `Canary10Percent5Minutes`). Add a CloudWatch alarm on the `Errors` metric as a rollback trigger.

Why: SAM natively integrates with CodeDeploy for Lambda, automating alias traffic shifting, monitoring, and rollback without custom scripts.

Reference↗

Configure IAM for a CodePipeline in Account A to deploy resources into Account B.

→Pipeline role (Account A) assumes an action role (Account B). The action role in B trusts the pipeline role and has deploy permissions. The S3 artifact bucket and KMS key in A must have resource policies granting access to the action role in B.

Why: This is the standard, secure cross-account access pattern: role assumption for actions, resource-based policies for data access.

Implement a GitOps workflow for EKS where the cluster state is automatically and continuously reconciled with a Git repository.

→Deploy a GitOps controller (e.g., Flux, ArgoCD) in the EKS cluster. Configure it to monitor the Git repository and apply/reconcile changes.

Why: This is the standard "pull-based" GitOps pattern. The in-cluster controller handles continuous reconciliation and drift detection, which is the core principle of GitOps.

Allow a CodeBuild project in a central tooling account to deploy Kubernetes manifests to EKS clusters in separate workload accounts.

→In each workload account, create a cross-account IAM role trusted by the CodeBuild role. Map this new role to a Kubernetes RBAC group in the EKS cluster's `aws-auth` ConfigMap. The CodeBuild script assumes the role before running `kubectl`.

Why: This is the standard, secure pattern for cross-account EKS access. It follows least privilege by creating a dedicated, trusted role for this purpose.

Perform a complex RDS PostgreSQL or MySQL schema migration with zero or near-zero downtime.

→Use the Amazon RDS Blue/Green Deployments feature. Create a synchronized staging (green) environment, apply schema changes to it, and then switch over to promote it to production.

Why: This is the purpose-built, managed service for safe, zero-downtime RDS updates. It handles cloning, synchronization, and a fast (< 1 min) switchover with built-in guardrails.

Deploy a new version of a single-page application (SPA) to S3/CloudFront and ensure users receive the new version immediately with minimal cache invalidation costs.

→Use content-based hashing for asset filenames (e.g., `app.a1b2c3d4.js`). After deploying new assets, invalidate only the `index.html` file in the CloudFront distribution.

Why: Hashed filenames are unique, so CloudFront treats them as new objects and fetches them from the origin, bypassing the cache. Only the single entry point file (`index.html`) needs invalidation, which is significantly cheaper than a wildcard (`/*`) invalidation.

Implement a CI/CD pipeline for an AWS CDK application that automatically updates itself when the pipeline's own definition changes.

→Use the CDK Pipelines construct (`pipelines.CodePipeline`). This construct creates a pipeline that includes a `SelfMutate` stage by default.

Why: CDK Pipelines is a high-level construct purpose-built for this pattern. The `SelfMutate` stage ensures the pipeline always reflects the latest definition from code before deploying application changes.

Deploy a new application version that requires a backward-compatible database schema change (e.g., adding new columns) with zero downtime.

→Implement an expand-and-contract (or parallel change) pattern. First, deploy the additive, backward-compatible database schema changes. Second, deploy the new application version that uses the new schema. Both old and new application versions can coexist with the updated database.

Why: This pattern decouples the database and application deployments, ensuring the database state is always compatible with both the old and new application versions, thus enabling zero-downtime rollouts.

Gradually roll out a new feature to specific user segments and measure the impact on business metrics (e.g., conversion rate) using A/B testing.

→Use Amazon CloudWatch Evidently. Create a feature with multiple variations, a launch to control the rollout percentage, and an experiment to measure the statistical impact on defined metrics.

Why: Evidently is a purpose-built service for feature flagging and A/B experimentation, providing not just the rollout mechanism but also the statistical analysis engine to measure impact.

Enforce mandatory tags on all EC2 instances at launch time across an AWS Organization.

→Use a Service Control Policy (SCP) that denies `ec2:RunInstances` unless the required tag keys are present in the request.

Why: Preventive control that blocks non-compliant resources from being created. Applies to all accounts and cannot be overridden by local IAM policies.

Reference↗

Manage and rotate secrets (e.g., DB credentials) used by applications in multiple accounts without downtime.

→Use AWS Secrets Manager with automatic rotation enabled. Grant cross-account access using resource-based policies on the secret.

Why: Secrets Manager supports zero-downtime rotation strategies (alternating users) and provides secure, native cross-account sharing.

Automatically deploy baseline security resources to new accounts created via Control Tower Account Factory.

→Use the Control Tower lifecycle event `CreateManagedAccount` via EventBridge to trigger a Lambda function that deploys a CloudFormation StackSet. Alternatively, use Customizations for AWS Control Tower (CfCT).

Why: Event-driven automation is the standard, scalable pattern for extending Control Tower baselines without manual intervention after account creation.

Enable SSM Session Manager access to EC2 instances in a private subnet with no internet access.

→Create VPC interface endpoints (powered by PrivateLink) for the `ssm`, `ssmmessages`, and `ec2messages` services in the VPC.

Why: VPC endpoints allow the SSM agent to communicate with the service entirely within the AWS network, providing the most secure access pattern without needing a NAT or internet gateway.

Centralize logs with long-term retention and protect them from deletion or modification, even by administrators.

→Store logs in an S3 bucket with S3 Object Lock in compliance mode. Enable CloudTrail log file integrity validation.

Why: Object Lock (compliance mode) provides WORM protection that even the root account cannot bypass. Log file integrity validation provides a cryptographic check for post-delivery tampering.

Provide developers a self-service way to provision pre-approved infrastructure patterns without granting them full AWS service permissions.

→Use AWS Service Catalog. Create a portfolio of approved products (defined by CloudFormation templates). Use launch constraints to have Service Catalog provision resources using a privileged IAM role managed by the platform team.

Why: Service Catalog is the purpose-built AWS service for creating curated catalogs of IT services. Launch constraints are the key governance feature, allowing developers to provision complex infrastructure without having the underlying permissions themselves.

Securely provide unique secrets to different microservices running as ECS tasks, ensuring each service can only access its own secrets.

→Create separate AWS Secrets Manager secrets for each service. In the ECS task definition, reference the secret ARNs in the `secrets` property of the container definition. Scope the task execution IAM role policy to only allow `secretsmanager:GetSecretValue` on that service's specific secret ARN.

Why: This enforces the principle of least privilege at multiple layers: the secret itself, the IAM policy, and the ECS task definition. Secrets are injected securely at runtime.

Allow a GitHub Actions workflow to securely access AWS without storing long-lived credentials.

→Configure an IAM OIDC identity provider for GitHub. Create an IAM role with a trust policy restricting the federated principal to the specific GitHub organization, repository, and branch. Use the `aws-actions/configure-aws-credentials` action with OIDC to assume the role.

Why: OIDC federation is the most secure method, providing short-lived credentials scoped to a specific workflow run, eliminating the risk of long-lived credential exposure.

Continuously monitor all IAM policies across an AWS Organization to identify and be alerted about resources shared with external entities.

→Enable IAM Access Analyzer at the organization level, defining the organization as the zone of trust. Use EventBridge to capture new findings and trigger notifications.

Why: IAM Access Analyzer is purpose-built to use automated reasoning to find externally shared resources. Running it at the org level provides a continuous, centralized view without custom scripting.

Reduce the blast radius of failed CloudFormation updates in a monolithic or nested stack architecture.

→Decompose the architecture into independent stacks using cross-stack references (CloudFormation Exports/Fn::ImportValue).

Why: A failure in one stack (e.g., database) will not trigger a rollback of other successfully updated stacks (e.g., networking), isolating failure domains.

Centrally manage cross-account patching with different schedules for production and non-production environments.

→Use AWS Systems Manager Patch Manager with custom patch baselines, separate maintenance windows for each environment, and Systems Manager Explorer for centralized compliance reporting.

Why: Natively supports all requirements: custom patch definitions, flexible scheduling via maintenance windows, and cross-account visibility via Explorer.

Preview infrastructure changes across all target accounts before executing a CloudFormation StackSet update.

→Create and review a CloudFormation change set for the StackSet operation before execution.

Why: Change sets are the native CloudFormation mechanism to preview the exact resource changes (add, modify, delete) that an update will perform.

Ensure CloudFormation waits for an EC2 instance's UserData script to complete successfully before proceeding with stack creation.

→Add a `CreationPolicy` with `ResourceSignal` to the EC2 instance resource. Call the `cfn-signal` helper script from UserData upon successful completion.

Why: This is the native CloudFormation mechanism for coordinating with configuration scripts on a resource. Failure to signal within the timeout automatically triggers a stack rollback.

Detect when manually-made, out-of-band changes cause deployed resources to differ from their CloudFormation template definition.

→Run CloudFormation drift detection on the stack periodically. For continuous detection, use the `cloudformation-stack-drift-detection-check` AWS Config rule.

Why: Drift detection is the native feature for comparing a stack's template with the actual state of its resources. Using the Config rule automates this check.

Protect stateful resources (e.g., an S3 bucket or RDS database) from accidental deletion or replacement via CloudFormation stack operations.

→On the resource, set `DeletionPolicy: Retain` (or `Snapshot` for RDS). On the stack, enable `TerminationProtection`. Apply a `StackPolicy` that denies `Update:Replace` and `Update:Delete` actions on the critical resource.

Why: Provides defense in depth: Termination Protection prevents stack deletion, DeletionPolicy preserves the resource if the stack is deleted, and the Stack Policy prevents destructive updates.

Migrate a CloudFormation StackSet from a complex, self-managed IAM role model to a simpler permission model for an AWS Organization.

→Update the StackSet to use service-managed permissions.

Why: Service-managed permissions leverage Organizations trusted access, eliminating the need to create and manage IAM roles in each target account. It also enables automatic deployment to new accounts added to targeted OUs.

A CloudFormation custom resource needs to manage a task that takes longer than the 15-minute Lambda function timeout.

→Trigger an AWS Step Functions state machine from the custom resource's Lambda function. The state machine handles the long-running task using Wait states or the Task Token pattern and sends the response back to CloudFormation's S3 presigned URL.

Why: Step Functions is designed to orchestrate long-running, multi-step workflows, effectively bypassing the Lambda timeout limitation while maintaining integration with CloudFormation.

Centrally enforce a policy (e.g., all S3 buckets must have versioning) across an entire AWS CDK application, regardless of how developers define their resources.

→Create a CDK Aspect that implements the `IAspect` interface. The Aspect visits all constructs in the application tree, finds all S3 bucket constructs, and applies the required configuration or adds a validation error if it's missing.

Why: Aspects are the official CDK pattern for applying cross-cutting concerns and implementing policy-as-code validations centrally without modifying individual constructs.

Prevent automated operations, like patching via SSM Maintenance Windows, from running during specific, changing time periods (e.g., a quarterly financial blackout).

→Use SSM Change Calendar to define events marking the blackout periods as "closed". Associate the Change Calendar with the Maintenance Window.

Why: Change Calendar acts as a gate for automations. It automatically blocks execution during "closed" periods without requiring manual changes to the Maintenance Window schedule, making it highly efficient for managing dynamic blackout periods.

Centrally manage the installation and versioning of a custom software package (e.g., a monitoring agent) across a fleet of EC2 instances.

→Package the software using SSM Distributor. Use SSM State Manager to create an association that applies the Distributor package to all targeted instances.

Why: Distributor manages the package lifecycle (including versions). State Manager ensures the desired state (e.g., "version 1.2 of agent is installed") is continuously enforced, automatically remediating drift and configuring new instances.

Low RPO (< 1 min) and RTO (< 5 min) disaster recovery for an Aurora database and application tier across regions.

→Use an Aurora Global Database for sub-second database replication. For the app tier, use a "warm standby" with an Auto Scaling group set to 0 desired capacity, to be scaled up via automation on failover.

Why: Aurora Global Database provides sub-second RPO and < 1-minute RTO. The warm standby app tier is cost-effective while still meeting a fast RTO.

Reduce Auto Scaling group scale-out time for instances that have long bootstrap/initialization times.

→Create a pre-baked "golden AMI" with dependencies installed. Configure a warm pool on the Auto Scaling group to keep instances pre-initialized.

Why: A golden AMI minimizes bootstrap time. A warm pool minimizes launch time (start vs. launch). Together, they dramatically reduce the time for a new instance to become ready to serve traffic.

An ECS service scales up its task count, but cannot place new tasks because the underlying EC2 cluster is out of capacity.

→Enable ECS Cluster Auto Scaling by associating a capacity provider with the EC2 Auto Scaling group and the ECS cluster.

Why: Capacity providers link ECS service scaling to EC2 instance scaling. When tasks fail to place due to insufficient cluster resources, the capacity provider automatically scales out the EC2 ASG.

Dynamically scale a fleet of EC2 worker instances based on the number of messages in an SQS queue.

→Use a target tracking Auto Scaling policy based on the custom metric: `ApproximateNumberOfMessagesVisible` / `GroupInServiceInstances` (i.e., backlog per instance).

Why: This is the recommended pattern for SQS-based scaling. It maintains just enough workers to process the backlog within a target time, scaling efficiently with queue depth.

Create application-consistent (not just crash-consistent) snapshots of EBS volumes for stateful applications.

→Use AWS Backup with a backup plan. In the plan, use Systems Manager Run Command to execute pre-snapshot scripts to quiesce the application (or enable VSS for Windows).

Why: AWS Backup orchestrates the entire process. Quiescing the application (flushing I/O buffers to disk) before the snapshot ensures data integrity and a recoverable application state.

Ensure critical events from an EventBridge rule are not lost when a target service (e.g., Lambda) is temporarily unavailable or throttled.

→On the EventBridge rule target, configure a Retry Policy (e.g., 24-hour maximum age) and a Dead-Letter Queue (DLQ) using an SQS queue.

Why: The retry policy handles transient failures automatically. The DLQ acts as a final safety net, capturing events that exhaust all retries so they can be reprocessed later, preventing data loss.

Trigger real-time alerts on specific log patterns and include contextual information (e.g., surrounding log lines) in the notification.

→Use a CloudWatch Logs subscription filter to stream matching log events to a Lambda function. The Lambda function formats and sends a detailed notification (e.g., to SNS or Chime).

Why: Subscription filters provide real-time event streaming. Lambda allows for custom logic to extract and format context, which simple metric filters cannot do.

Identify latency bottlenecks in a distributed, microservices-based application.

→Enable AWS X-Ray tracing on entry points (e.g., API Gateway, ALB) and compute (e.g., Lambda, ECS). Use the X-Ray SDK for downstream calls. Analyze the service map and traces.

Why: X-Ray is the purpose-built AWS service for distributed tracing. The service map visualizes the call chain and highlights services with high latency and error rates.

Create a single, high-level alarm that represents the combined health of a multi-tier application to reduce alert noise.

→Create individual CloudWatch alarms for each tier (e.g., ALB 5xx rate, app CPU, RDS connections). Then, combine them using a CloudWatch composite alarm with OR logic.

Why: Composite alarms are designed to reduce alarm noise by creating a single, logical alarm based on the state of multiple underlying alarms.

Analyze petabytes of logs with complex SQL queries (including joins) and retain them for years in a cost-effective manner.

→Stream logs to Amazon S3 via Kinesis Data Firehose. Catalog the data with AWS Glue. Query with Amazon Athena. Use S3 Lifecycle policies to transition data to Glacier/Deep Archive for long-term retention.

Why: This is the standard serverless data lake architecture. Athena provides powerful SQL capabilities on S3 data, and S3/Glacier offers the most cost-effective long-term storage.

Monitor a metric with predictable cyclical patterns (e.g., daily/weekly spikes) and only trigger an alarm on genuine deviations from the pattern.

→Configure CloudWatch Anomaly Detection on the metric. Create an alarm that triggers when the metric value goes outside the model's expected band.

Why: Anomaly Detection uses machine learning to learn the metric's normal patterns, creating a dynamic threshold band that adapts to cycles. This reduces false positives from predictable spikes and improves signal-to-noise ratio.

Gain comprehensive visibility into container-level CPU, memory, disk, and network metrics for workloads on EKS or ECS without installing and managing third-party tools.

→Enable Amazon CloudWatch Container Insights for the EKS/ECS cluster.

Why: Container Insights is a fully managed service that automatically collects, aggregates, and visualizes detailed performance metrics for containerized workloads, providing deep visibility with minimal operational overhead.

Monitor the availability and performance of an internet-facing application from the perspective of end-users, identifying ISP-level and geographic network issues.

→Enable Amazon CloudWatch Internet Monitor for the application.

Why: Internet Monitor leverages AWS global network data to provide visibility into internet weather that affects your end-users, helping diagnose issues outside of your AWS environment.

Measure the real-user experience of a web application by collecting page load times, JavaScript errors, and other client-side performance metrics.

→Integrate the CloudWatch RUM (Real User Monitoring) JavaScript snippet into the web application.

Why: RUM is a managed service that collects client-side performance and error data directly from user browsers, providing true insight into real-user experience without synthetic tests.

Emit custom application metrics from an AWS Lambda function with high resolution and dimensions, without adding the latency and cost of direct CloudWatch API calls.

→Use the CloudWatch Embedded Metric Format (EMF) by writing specially structured JSON to standard output. A client library can simplify this.

Why: CloudWatch Logs automatically and asynchronously extracts metrics from EMF log entries, incurring no additional latency in the Lambda function and reducing cost by avoiding PutMetricData API calls.

Automatically remediate unencrypted EBS volumes detected by AWS Config, ensuring data consistency during the process.

→Use AWS Config auto-remediation with a Systems Manager Automation document. The runbook stops the instance, creates an encrypted copy of the volume, swaps volumes, and restarts the instance.

Why: SSM Automation provides a robust, multi-step, auditable workflow. Stopping the instance is critical to ensure a data-consistent snapshot before creating the encrypted copy.

Run controlled chaos engineering experiments (e.g., inject network latency) with automatic stop conditions to prevent production impact.

→Use AWS Fault Injection Simulator (FIS) with an experiment template. Define stop conditions based on CloudWatch alarms that monitor key application metrics.

Why: FIS is the purpose-built AWS service for chaos engineering, providing safety guardrails (stop conditions) and a catalog of controlled fault injection actions.

A CloudFormation stack is stuck in the `UPDATE_ROLLBACK_FAILED` state because a resource was deleted or changed during a failed update, preventing a clean rollback.

→Use the `ContinueUpdateRollback` API action, specifying the logical ID of the problematic resource in the `ResourcesToSkip` parameter.

Why: This is the standard recovery procedure to force the rollback to complete by telling CloudFormation to ignore the resource it can no longer manage, returning the stack to a stable state.

Receive notifications within minutes of critical security events occurring, such as a root account login, IAM policy changes, or security group modifications.

→Create Amazon EventBridge rules that match specific CloudTrail management event patterns and route them to an SNS topic for notification.

Why: EventBridge receives CloudTrail management events in near real-time, providing the lowest latency for event-driven security alerting compared to polling or log-based methods.

A high-traffic Lambda function is being throttled and is also exhausting RDS database connections when it scales.

→Request a Lambda concurrent execution limit increase. Implement Amazon RDS Proxy between the Lambda functions and the RDS database.

Why: Increased concurrency solves throttling. RDS Proxy is essential for serverless applications, as it pools and reuses database connections, preventing the database from being overwhelmed by a large number of ephemeral connections.

Implement automated DNS failover between regions and trigger an automated recovery runbook for the failed region.

→Use Route 53 failover routing with associated health checks. Create an EventBridge rule that captures the Route 53 health check status change event and triggers a Systems Manager Automation runbook.

Why: This architecture combines automated traffic failover (Route 53) with event-driven, automated incident response (EventBridge + SSM Automation) for a complete resilience pattern.

Prevent an RDS database from running out of storage and causing an application outage.

→Enable RDS Storage Autoscaling by setting a maximum storage threshold. As a secondary control, create a CloudWatch alarm on the `FreeStorageSpace` metric.

Why: Storage Autoscaling is a proactive, managed feature that automatically increases allocated storage. The CloudWatch alarm provides a safety net for monitoring and alerting.

Need to reprocess a batch of events that were processed incorrectly due to a temporary bug in a consumer.

→Configure an EventBridge Archive on the event bus beforehand. After the bug is fixed, create a Replay to resend events from the specific time window of the incident.

Why: Archive and Replay is the native EventBridge feature for storing and reprocessing historical events, crucial for recovery from transient processing failures.

Automate the entire incident response process: create an incident, engage the on-call team, open a chat channel, and execute a remediation runbook when a critical alarm fires.

→Create an SSM Incident Manager response plan that defines all engagement and remediation steps. Configure the CloudWatch alarm to trigger this response plan as its action.

Why: Response plans provide a single, cohesive configuration to orchestrate all aspects of incident response, reducing manual effort and ensuring consistent procedures.