A stateless compute service (Cloud Run, Cloud Functions) needs to process temporary files.
Use Cloud Storage for all temporary file I/O.
Why: The local filesystem of serverless platforms is ephemeral, in-memory, and not shared. Cloud Storage provides durable, scalable storage accessible by all instances.
Manage environment-specific configuration and secrets for GKE workloads following 12-factor principles.
Use K8s ConfigMaps for non-sensitive config. Use Secret Manager for sensitive values, accessed securely via Workload Identity.
Why: Secret Manager is a more secure, managed, and auditable solution than K8s Secrets. Workload Identity avoids managing and distributing service account keys.
Application has extreme traffic peaks but long idle periods where cost must be minimized.
Use Cloud Run with `min-instances` set to 0.
Why: Cloud Run can scale down to zero, eliminating all compute costs during idle periods. GKE and Compute Engine require minimum running nodes/instances.
Implement retries, circuit breakers, and mTLS consistently across microservices without application code changes.
Deploy a service mesh (Anthos Service Mesh) on GKE.
Why: A service mesh injects resilience, security, and observability at the platform level, keeping application code clean and ensuring consistent behavior.
Expose backend services to external partners or mobile apps with rate limiting, API keys, and usage analytics.
Use API Gateway in front of backend services (e.g., Cloud Run, GKE).
Why: API Gateway provides a fully managed solution for API lifecycle concerns (security, monitoring, versioning), offloading them from the backend service.
Select a durable, scalable, and strongly consistent store for an append-only log of events.
Use Cloud Spanner for the event store.
Why: Spanner provides horizontal scalability with strong global consistency, crucial for maintaining the integrity of an event log at scale.
An API for a long-running job must respond immediately while processing continues in the background.
API endpoint enqueues a task in Pub/Sub or Cloud Tasks and returns a 202 Accepted with a job ID. A separate worker (Cloud Run, Cloud Function) processes the task.
Why: This decouples the user-facing response time from the backend processing time, improving UX and system reliability. Use Cloud Storage for status updates.
Maintain data consistency across multiple microservices without a shared database.
Implement the Saga pattern using an orchestrator (Cloud Workflows) or choreography (Pub/Sub events) with compensating transactions.
Why: Avoids complex and locking-prone two-phase commits, favoring eventual consistency which is a better fit for distributed systems.
Application calls a rate-limited third-party API where data changes infrequently.
Use Memorystore for Redis as a distributed cache. Implement cache-aside pattern with TTL. Use a distributed lock (e.g., Redis SETNX) to prevent cache stampedes.
Why: A distributed cache shares data across all app instances, drastically reducing calls to the external API, improving latency and respecting rate limits.
A development team needs consistent, pre-configured, secure development environments with access to private VPC resources.
Use Cloud Workstations.
Why: Cloud Workstations provides managed, container-based development environments with integrated security and VPC access, solving the "it works on my machine" problem.
A SaaS application requires tenants to have completely isolated data, encryption keys, and data residency.
Use a project-per-tenant model. Manage provisioning and configuration centrally using IaC (Terraform).
Why: Provides the highest level of isolation for IAM, billing, quotas, networking, and data location, often required by enterprise or regulated customers.
Building and testing applications
Enforce that only trusted, scanned container images from an official pipeline can be deployed to production.
Use Cloud Build to generate SLSA provenance, Artifact Registry for vulnerability scanning, and Binary Authorization to enforce deployment policies based on attestations.
Why: Creates a verifiable, un-bypassable cryptographic chain of trust from code to deployment, preventing deployment of compromised or unscanned artifacts.
Deploy a new version of a Cloud Run service, test it without user impact, and switch traffic instantly.
Deploy the new revision with `--no-traffic`. Test using the unique revision URL or a revision tag. Switch 100% of traffic to the new revision once validated.
Why: Cloud Run's native traffic management allows for safe, zero-downtime deployments by validating the new version before it receives any production traffic.
Gradually roll out a new version, automatically analyzing metrics and rolling back on failure.
Use Cloud Deploy with a canary deployment strategy. Integrate with Cloud Monitoring for automated metric analysis and rollback triggers.
Why: Cloud Deploy automates the entire progressive delivery workflow, including metric analysis and safety checks, reducing manual effort and risk.
Manage Kubernetes manifests for dev, staging, and prod environments without duplicating code.
Use Kustomize or Helm. Define a base configuration and create environment-specific overlays or values files to patch in differences.
Why: Follows the DRY principle, making configurations easier to manage and reducing the risk of environment drift.
Reduce container image size for faster deployments and a smaller attack surface.
Use multi-stage builds. A `build` stage uses a full SDK/JDK image; the final stage copies only the compiled artifact into a minimal `distroless` base image.
Why: The final image contains only the application and its runtime dependencies, removing all build tools, shells, and package managers.
Automatically deploy ephemeral environments for each pull request for validation before merging.
Use Cloud Build triggers on PR events to deploy to Cloud Run with a revision tag (e.g., `pr-123`). Use another trigger on PR close to clean up the tagged revision.
Why: Revision tags provide unique, temporary URLs for each PR without the overhead of creating new services, making it cost-effective and easy to automate.
Improve CI/CD build speed and reliability by caching public software dependencies (e.g., from npm, Maven Central).
Use an Artifact Registry remote repository, which acts as a pull-through cache for the public repository.
Why: Improves build performance, insulates builds from public registry outages, and allows for vulnerability scanning on cached artifacts.
Securely store and lock Terraform state for concurrent CI/CD pipeline executions.
Use a Cloud Storage backend for Terraform state, with appropriate IAM for the Cloud Build service account.
Why: Cloud Storage provides a durable, versioned, and lockable backend, preventing state corruption from concurrent builds.
Deploying applications
A Cloud Run or Cloud Function needs to access a resource (e.g., Cloud SQL, Memorystore) on a private VPC network.
Configure a Serverless VPC Access connector.
Why: The connector acts as a network bridge, allowing egress traffic from the serverless environment into the target VPC without exposing resources publicly.
A stateful application on GKE requires stable identity and persistent storage that survives pod/node failure.
Use a StatefulSet with a Headless Service for identity. Use a PersistentVolumeClaim (PVC) with a regional Persistent Disk for storage.
Why: This is the canonical Kubernetes pattern for stateful workloads, ensuring data persistence, high availability, and predictable pod naming/networking.
A pod on GKE needs to access GCP APIs securely without managing static service account keys.
Configure and use Workload Identity.
Why: Workload Identity binds a Kubernetes Service Account to a Google Service Account, allowing pods to use short-lived GCP credentials obtained from the metadata server.
Run a batch job that takes hours to complete (e.g., processing a large file, nightly data aggregation).
Use Cloud Run jobs, triggered by Eventarc or Cloud Scheduler.
Why: Cloud Run jobs are designed for long-running (up to 24h) tasks, scale to zero, and are more cost-effective and simpler than a dedicated GKE cluster or VM for batch workloads.
Deploy a multi-container application on Cloud Run where a main container needs a sidecar for logging, metrics, or as a proxy.
Deploy the Cloud Run service with multiple container images specified, one as the main container and others as sidecars.
Why: Cloud Run's native multi-container support enables the sidecar pattern for serverless workloads without the complexity of GKE.
Database migrations must complete before a new Cloud Run revision receives traffic.
Run migrations during container startup, and use a Cloud Run startup probe that only passes after migrations are successful.
Why: The startup probe delays traffic routing until the container is fully ready, ensuring the database schema is correct before any requests are served.
A GKE application needs to scale based on a custom metric like queue depth from Pub/Sub, not just CPU/memory.
Use the Horizontal Pod Autoscaler (HPA) configured to read custom metrics from Cloud Monitoring.
Why: This allows autoscaling to be driven by business logic or application-specific load indicators, providing more accurate scaling than generic resource metrics.
Integrating Google Cloud services
A message-driven service must process each message exactly once, despite retries and potential duplicate deliveries.
Combine Pub/Sub (with at-least-once or exactly-once delivery) with an idempotent consumer. The consumer tracks processed message IDs in a persistent store (e.g., Firestore, Memorystore).
Why: Pub/Sub guarantees delivery, but the consumer is responsible for idempotency to handle application-level retries and achieve true exactly-once processing.
Events related to the same entity (e.g., a specific user) must be processed in the order they were generated.
Publish messages to Pub/Sub with an `orderingKey`. Enable message ordering on the subscription.
Why: Pub/Sub guarantees messages with the same ordering key are delivered in order, while messages with different keys can be processed in parallel for scalability.
Ensure a task (e.g., generate daily user report) runs only once, even if multiple trigger events arrive.
Use Cloud Tasks. Create tasks with explicit names (e.g., `report-userX-2024-10-26`). Cloud Tasks will deduplicate requests to create a task with an existing name.
Why: This offloads deduplication logic to the queuing service, simplifying application code and preventing redundant work.
A Pub/Sub message consistently fails processing after multiple retries and is blocking the queue.
Configure a dead-letter topic (DLQ) on the Pub/Sub subscription and set a maximum number of delivery attempts.
Why: Pub/Sub automatically moves the "poison" message to the DLQ, allowing other messages to be processed and preserving the failed message for analysis.
A business process involves a sequence of service calls with conditional logic, error handling, and long waits.
Use Cloud Workflows to define and execute the orchestration logic.
Why: Cloud Workflows is a serverless orchestrator that manages state, retries, and long waits, providing better reliability and visibility than manually chained functions.
A Cloud Function or Cloud Run service should only be triggered by specific cloud events (e.g., certain file types in Cloud Storage).
Use an Eventarc trigger with CEL (Common Expression Language) filtering on event attributes.
Why: Filtering happens before the service is invoked, saving costs and compute cycles by not processing irrelevant events.
Rate-limit outgoing calls to a third-party API from a horizontally-scaled service like Cloud Run.
Enqueue API calls as tasks in a Cloud Tasks queue with `rateLimits` configured (e.g., max dispatches per second).
Why: Cloud Tasks provides centralized, serverless rate limiting that works across all scaled instances without requiring complex distributed counters.
Securely authenticate calls between two Cloud Run (or Cloud Functions) services.
Grant the caller service's service account the `roles/run.invoker` IAM role on the callee service. The caller sends a Google-signed ID token with its request.
Why: This is the native, secure, and key-less method for service-to-service authentication, leveraging Google's identity infrastructure.
An event-driven service needs to react to row-level changes in a Cloud SQL database in near real-time.
Use Datastream (CDC) to stream database changes to Pub/Sub. Use Eventarc to trigger a Cloud Run service from the Pub/Sub topic.
Why: This is a decoupled, reliable pattern that avoids database polling and does not require modifying the application writing to the database.
A long-running business process must pause to wait for an external event, like a human clicking an approval link in an email.
Use Cloud Workflows with a callback endpoint. The workflow pauses (for up to a year) until it receives an HTTP request on its unique callback URL.
Why: Callbacks allow workflows to wait for external events without consuming any compute resources, making it ideal for long-running, human-in-the-loop processes.
Managing application performance
A latency-sensitive serverless application experiences slow initial responses after idle periods.
Configure `min-instances` to 1 or more. For unavoidable cold starts, use `startup-cpu-boost`. Also optimize the application (smaller image, faster init).
Why: `min-instances` is the most effective way to eliminate cold starts but incurs cost. `startup-cpu-boost` accelerates the startup process itself.
An application is running slow in production, and it is unclear if the issue is CPU-bound, a memory leak, or I/O.
Use Cloud Profiler to continuously analyze CPU and heap usage in production.
Why: Cloud Profiler identifies code-level performance bottlenecks (hot paths, memory leaks) with very low overhead, without needing to reproduce issues in a test environment.
Move from simple threshold alerts (e.g., "latency > 500ms") to more meaningful alerts based on Service Level Objectives (SLOs).
Define SLIs and SLOs in Cloud Monitoring. Create alerting policies based on the error budget's "burn rate."
Why: Burn rate alerting is more sensitive to significant changes and less noisy than simple threshold alerts, signaling when you are on a trajectory to miss your SLO.
Aggregate application exceptions from multiple services to track frequency, see stack traces, and get notified of new error types.
Use Cloud Error Reporting.
Why: Error Reporting automatically ingests, groups, and analyzes exceptions from structured logs, providing a centralized dashboard for managing application errors.
Monitor, visualize, and alert on custom business metrics (e.g., orders per minute, user signups).
Instrument application code using the OpenTelemetry SDK. Configure the exporter to send metrics to Cloud Monitoring.
Why: This is the modern, vendor-neutral standard for custom instrumentation. It allows tracking any metric and leveraging all of Cloud Monitoring's features for it.