A cloud platform is easy to approve when the demo ends at deployment. The harder question is whether your team can restore service when credentials expire, an upgrade fails, or the vendor’s console is unavailable. I would choose a vendor, library, or framework by the recovery paths it lets an on-call engineer exercise—not by the speed of its first successful release.
A vendor is only supportable if you can rehearse the failure
Make the evaluation unit a production incident, not a feature. A claim such as “managed Kubernetes reduces operations” is too broad to test: the vendor may manage the control plane while your team still owns workloads, admission policies, node configuration, and upgrade timing. Ask the vendor to identify the exact boundary for each failure you intend to rehearse. If a deployment stalls after an API change, who identifies the incompatible manifest, who can roll it back, and who is authorized to act at night?
I treat the maintenance warning Buying Cloud Infrastructure Means Owning the Hidden Maintenance Load as a hypothesis rather than a purchasing rule, because a managed service can remove real work if its remaining failure paths are observable and recoverable. Put that hypothesis into an acceptance sheet. For every critical component, record its owner, the signal that detects failure, the action that restores service, and the evidence that the action worked. “Open a support ticket” is an escalation path, not a recovery action, unless the contracted response fits your recovery objective.
Run the sheet against a small, representative service with its real deployment method and identity model. Include a failed rollout, a revoked application credential, an unavailable dependency, and a restore from backup. A value to tune is a 30-minute recovery objective for the trial service; choose it from your application’s tolerance, then time the drills against it. Record separately when the alert fired, when the engineer understood the fault, and when users recovered. A fast restart can conceal slow detection.
Inspect what happens when the primary interface is unavailable. If recovery requires the vendor console, try the documented CLI and API route with the same restricted role the on-call engineer holds. AWS CloudTrail, Azure Activity Log, and Google Cloud Audit Logs should show who changed the affected resource; your test should confirm the relevant events actually arrive in your logging system. Audit-log availability alone does not establish incident usefulness, because an engineer still needs a resource identifier, timestamp, and correlation to the alert.
I would not accept a vendor-run demonstration as the failure drill, because an engineer with administrator access and advance knowledge of the fault does not reproduce your pager conditions. Have your own on-call engineer execute the runbook with normal permissions while another person observes. Where the vendor must intervene, ask for the support case identifier and timestamps so that an escalation promise can be compared with an observed result.
The shortest deployment path often hides the hardest rollback
Compare platforms through a change that goes wrong after traffic arrives. For the trial, deploy a version that passes startup checks but fails one application request. Then test whether the deployment system stops promotion, whether the previous version is still deployable, and whether the database schema permits rollback. A platform that rolls containers back quickly cannot reverse a destructive migration; separating those responsibilities prevents a misleading success result.
Do not mistake a platform comparison Choosing the Right Cloud Platform: AWS, Azure, or Google Cloud? for an operational acceptance test, because a feature match cannot tell you whether your team can complete this rollback with its actual permissions. Evaluate the framework or deployment library that sits between your repository and the cloud as part of the same path. Check whether it pins versions, exposes the underlying API response, and documents what happens after a partially completed change.
For infrastructure changes managed with Terraform, run a plan using the provider lock file and inspect destructive actions before applying anything. The following gate runs from a checkout containing an initialized infra configuration, a committed .terraform.lock.hcl, supplied input values, valid credentials, and jq. It fails if the plan includes a delete action; replacement plans also contain a delete and therefore fail.
#!/usr/bin/env bash
set -euo pipefail
terraform -chdir=infra init -input=false -lockfile=readonly
terraform -chdir=infra validate -no-color
terraform -chdir=infra plan -input=false -lock-timeout=60s -out=trial.tfplan
terraform -chdir=infra show -json trial.tfplan > trial.json
jq -e '[.resource_changes[]? |
select(.change.actions | index("delete"))] | length == 0' trial.json
rm -f trial.tfplan trial.json
The 60-second lock wait is a trial setting to adjust: too short can turn ordinary contention into noisy failures, while too long can hide a stuck pipeline. This gate is deliberately narrow. It does not prove that an update is safe, detect a destructive SQL migration, or replace a reviewed plan; it prevents a specific class of unnoticed infrastructure deletion. If a framework cannot produce a similarly inspectable change record, count that as an operating cost rather than calling its deployment interface “simpler.”
Measure rollback from the moment an engineer decides to revert until a user-visible probe succeeds. Suppose a pilot records an 11-minute rollback; that measured result is useful only if the probe covers the failed request and the same identity, network, and data paths users take. Preserve the command history and alert timeline with the result. Otherwise, a future team cannot tell whether the figure reflects the platform or an operator’s temporary administrator access.
EKS and Cloud Run win under different operating contracts
Amazon EKS and Google Cloud Run are a useful explicit comparison precisely because neither is the universal “more managed” answer. EKS wins when you must retain Kubernetes scheduling, controllers, and familiar workload APIs across several services. Its cost is ownership of more cluster-level decisions: node or compute configuration, Kubernetes version upgrades, admission behavior, and the interaction between workloads and cluster policies. Cloud Run wins when the service fits its request-driven container model and your team benefits from avoiding cluster administration. Its cost is adapting operational controls to that model and giving up direct control of pods, nodes, and Kubernetes controllers.
Use the same incident scenario for both candidates. For EKS, ask the engineer to diagnose a rollout held by an unhealthy Deployment, inspect events with kubectl, and explain how a policy/v1 PodDisruptionBudget affects a planned disruption. For Cloud Run, ask them to identify a failed revision, shift traffic to a healthy revision, and verify that the request error rate falls. The commands differ, but the acceptance evidence should not: detection time, permission required, customer impact, and verified restoration. Do not award EKS extra credit merely for familiar kubectl commands, because familiarity does not eliminate cluster ownership. Do not award Cloud Run extra credit merely for fewer visible resources, because hidden platform behavior still has to fit the service’s timeout, concurrency, and networking needs.
Identity deserves its own drill. On EKS, an application may obtain AWS permissions through IAM roles for service accounts or EKS Pod Identity; on Cloud Run, it commonly uses a Google service account. Revoke the trial workload’s access to one nonessential dependency, confirm the denial appears in the application signal and cloud audit trail, and restore the narrow permission without granting a project-wide or account-wide role. Set a 15-minute target for the exercise as a threshold to negotiate with the security and on-call teams, not as a vendor guarantee. Record whether token caching delays the denial or restoration, because the timestamp of a policy edit is not necessarily the timestamp the workload experiences it.
For either platform, carry the same OpenTelemetry trace identifier through the request and attach it to a Prometheus error-rate query or equivalent metric. The standard and tools are useful here because they let the engineer test whether telemetry survives a platform change, not because a dashboard proves portability. If the proposed library drops trace context at a queue or gateway boundary, document the gap before procurement: the first production failure is an expensive place to discover that the only cross-service clue vanished.
The winning scorecard records evidence, not vendor assurances
Keep the decision record short enough to review during an incident. For each candidate, list the tested failure, the expected alert, the actual alert, the authorized recovery action, the observed duration, and the unresolved dependency on vendor support. Attach the runbook revision and the exact tool versions used. A score without those artifacts is difficult to defend when a vendor updates an API or a library changes its default behavior.
Set an evaluation window that is long enough to encounter a routine change. A two-week pilot is a planning value, not a reliability statistic: use it to include at least one deployment, one access change, and one restore drill. If a candidate requires a longer procurement process, do not substitute reference calls for the restore drill. References can establish that other teams operate the product; they cannot show that your backup, network route, and permissions work together.
Make observability an acceptance criterion with a concrete query. For a user-facing request, inspect the p95 latency and error rate before, during, and after the injected fault, and check whether PagerDuty—or your actual paging system—receives a page tied to the affected service. Define the alert threshold from your service-level objective rather than copying the vendor’s example dashboard, because the same error rate can represent very different user impact at different traffic levels. Check that the page includes a link to the relevant logs and runbook; an alert that requires a fresh console search adds time when context is scarce.
Ask what happens when the evaluation’s assumptions expire. A provider upgrade, deprecated API, changed identity integration, or new region requirement can invalidate a previously successful drill. Put the supported version range and upgrade owner in the decision record. For a library, verify that you can reproduce the build from pinned dependencies and that release notes identify breaking changes. For a hosted service, ask how you learn of mandatory changes and whether you can test them before production. Prefer the option whose changes your team can detect and rehearse, even if another option produces a cleaner demo, because recoverability is the property your on-call rotation will have to live with.
The first approval task is a failed restore drill
Before approving the shortlist, give an on-call engineer normal credentials and a backup from the trial service. Have them restore it, send a real request through the restored path, and record the elapsed time and missing permissions. If the drill fails, keep the failure in the decision record and ask each candidate how it would remove that specific obstacle. That answer is more useful than another platform demonstration.
