A CTO rarely gets surprised by the cloud bill itself; the surprise is the permanent maintenance payroll created by the cure. My position: most companies should buy the cloud cost and platform-control layer before building it, because the “simple” in-house version becomes a second product with uptime, roadmap, migrations, and angry internal customers.
The expensive part is the policy lifecycle, not the first Terraform module
The seductive version of in-house cloud governance starts with Terraform 1.6, a few AWS Organizations service control policies, and dashboards in Grafana 10. It feels rational because the code is visible and the first pull request is cheap. The burden appears later, because every rule has to survive new instance families, new Kubernetes versions, new pricing models, and new product exceptions.
Cloud Cost Optimization for High Performance Apps understates the recurring labor because every saving rule becomes a production assumption that someone must test against latency, throughput, failover, and customer contracts. Turning off idle nodes is easy; proving that a Karpenter v0.37 consolidation event will not evict a latency-sensitive pod during a flash sale is the work.
Real maintenance hides in small details. Kubernetes 1.29 still defaults terminationGracePeriodSeconds to 30 seconds; that default is fine for many services, but it can be too short for a gRPC server draining long-lived streams because the pod may receive SIGKILL before clients finish. Prometheus 2.49 with a scrape_interval of 15 seconds gives better visibility than a 60-second interval, but it also increases time-series cardinality and storage pressure because every pod label can multiply the number of series. OpenTelemetry 1.32 with parentbased_traceidratio set to 0.05 can cut trace volume, but it can also hide rare tail-latency paths because sampling removes evidence before an incident starts.
That is why I would not build a homegrown FinOps and platform-control stack first, because the team will spend the second year maintaining exceptions, reconciling tags, and defending policy drift instead of improving the product. I would build only the narrow pieces that encode proprietary constraints, such as a scheduler extension for a workload with unusual memory locality, because those constraints cannot be bought cleanly.
The trap is that tools do not remove ownership. Terraform or OpenTofu 1.6 state must be migrated, locked, and recovered. Argo CD 2.10 needs repo conventions, sync windows, project boundaries, and RBAC reviews. Helm 3.14 charts need dependency updates and values schemas. OPA 0.63 or Kyverno 1.12 policies need regression tests, because a policy that blocks privileged containers can also block a vendor agent needed during an outage. Infracost 0.10 estimates are useful in pull requests, but estimates decay when actual usage shifts because reserved pricing, data transfer, and storage I/O rarely match the plan.
Internal platforms turn every exception into a support ticket
Platform teams usually believe they are building paved roads. Product teams experience those roads as a ticket queue when the default does not fit. That disagreement matters because the maintenance burden grows with exception handling, not with the number of clusters alone.
Consider a common setup: EKS, Karpenter, Argo CD, External Secrets Operator, AWS Load Balancer Controller, Prometheus, Grafana, and OpenTelemetry Collector. None of those tools is exotic. Yet each creates a version matrix. Upgrading Kubernetes from 1.28 to 1.29 may require checking deprecated APIs with kubectl convert or pluto; upgrading the AWS Load Balancer Controller can change annotation behavior; changing an IAM role for service accounts can break External Secrets Operator access to AWS Secrets Manager. The work is not glamorous, but it is production engineering because one missed permission can block deploys across multiple teams.
Here is a small command that actually runs and exposes one maintenance smell: pods without CPU or memory requests. It is not a full governance system, which is the point; the moment you turn this into an enforced standard, you own the exceptions forever.
#!/usr/bin/env bash set -euo pipefail kubectl get pods --all-namespaces -o json | jq -r ' .items[] | . as $pod | .spec.containers[] | select((.resources.requests.cpu // "") == "" or (.resources.requests.memory // "") == "") | "\($pod.metadata.namespace)/\($pod.metadata.name) container=\(.name) missing_requests" '
A value I would tune rather than copy is 70% average node allocatable utilization, because higher targets often look efficient on paper but increase eviction risk when memory usage is bursty. A field observation worth measuring in your own environment is p95 API latency during scale-down events; 300 milliseconds may be acceptable for one product and unacceptable for another because user-facing paths and batch paths fail differently. A vendor-published limit that often surprises teams is AWS Lambda’s 900-second maximum timeout, which matters when “serverless cleanup jobs” become long-running operational workflows. A Kubernetes default worth remembering is the 30-second pod termination grace period, because it silently defines how much time your app gets to shut down cleanly unless you override it.
The maintenance backlog also includes naming and chargeback. Tags such as team, service, env, and cost_center look simple until shared clusters, shared NAT gateways, and shared observability bills appear. The FinOps FOCUS 1.0 specification helps normalize billing fields across providers, but it does not decide who pays for a shared Kafka cluster because cost allocation remains a business rule. AWS Cost and Usage Report 2.0 gives line-item depth, but it also creates a data pipeline obligation because the raw files must be cleaned, joined to metadata, and reconciled when tags arrive late.
Buying wins when the problem is operational memory, not intellectual property
The explicit comparison is uncomfortable but useful: Build: Backstage plus Crossplane plus Argo CD plus OpenCost versus Buy: Humanitec Platform Orchestrator plus Datadog Cloud Cost Management plus managed Kubernetes. Build wins when your deployment model is a strategic differentiator, because internal abstractions can encode constraints that commercial products will treat as edge cases. Buy wins when the goal is reliable self-service and cost visibility, because vendors amortize integrations, upgrades, and UI maintenance across many customers.
The cost profile differs sharply. The build option may use open-source licenses, but it costs senior engineering capacity: three experienced platform engineers can easily represent a high six-figure annual loaded cost in many US or Western European hiring markets, because salary, benefits, recruiting, management time, and on-call coverage are part of the bill. The buy option costs subscription fees and integration time; Datadog’s publicly listed infrastructure pricing has historically been per host per month, while Terraform Cloud and similar platforms price by user, resource, or run capacity, so the invoice is visible even when it is annoying. Visible cost is often better for a CTO because it can be renegotiated, capped, or replaced, while sunk internal complexity becomes politically protected.
I would treat Cloud Infrastructure Best Practices for Scalable Apps as a checklist to challenge, not a blueprint to own, because best practices become liabilities when a company copies them without funding the upgrade path. A standard such as “use infrastructure as code” is sound, because manual cloud changes create drift; a mandate to own every layer of the platform is not sound, because ownership includes incident response, security review, documentation, migrations, and developer support.
The “buy” side is not magic. Humanitec, Datadog, CAST AI, Harness, Spacelift, and Terraform Cloud all create vendor coupling, because workflows, policy languages, and dashboards shape how teams operate. Still, vendor coupling can be the better risk when the alternative is a half-staffed internal platform that nobody has time to improve. The decisive question is not whether your engineers can build the platform; they can. The question is whether the company benefits from paying them to maintain policy engines, billing pipelines, and deployment portals for the next five years.
- Backstage plus Crossplane plus Argo CD plus OpenCost wins when you need deep customization, strict data residency for operational metadata, or a platform that becomes part of your product offering; it costs engineering headcount, on-call ownership, internal documentation, and slower feature delivery elsewhere.
- Humanitec plus Datadog plus managed Kubernetes wins when you need consistency, auditability, and faster rollout across product teams; it costs subscription spend, procurement effort, vendor risk review, and some loss of low-level control.
The maintenance nobody budgets is semantic, not technical
The hardest part of scalable cloud cost control is agreeing what words mean. “Idle” sounds objective until a team keeps warm capacity for a contractual batch window. “Owner” sounds obvious until a shared service has one team operating it and six teams consuming it. “Production” sounds binary until preview environments start serving sales demos and load tests. Tools cannot settle these semantics because they reflect accountability, not syntax.
This is where homegrown systems age badly. Someone adds a do-not-delete=true tag after an incident, another team adds temporary=true and leaves it for 14 months, and a third team asks for “just one” exemption from a node-size rule. Cloud Custodian 0.9 can enforce tag policies, and AWS Config can detect noncompliant resources, but neither can decide whether an exception still has a business owner because that requires review discipline. Jira tickets can track approvals, but they become stale when the approver changes roles because ticket systems do not understand live cloud dependency graphs.
Metrics also become political. Cost per request is useful because it connects spend to demand, but it punishes services with low traffic and high reliability requirements. CPU throttling is useful because it exposes bad Kubernetes limits, but it can push teams to over-request CPU if managers treat throttling as a shame metric. p99 latency is useful because it captures tail pain, but it can produce expensive overprovisioning if the team optimizes every rare spike without customer impact analysis. Every metric needs a decision rule, because dashboards without rules create debate rather than action.
Security standards add another maintenance layer. SOC 2 evidence, ISO/IEC 27001 controls, CIS Kubernetes Benchmark checks, and SLSA provenance requirements all interact with infrastructure automation. A policy that requires signed containers with cosign is reasonable because it reduces supply-chain risk, but it will break emergency deploys unless there is a documented exception path. A private cluster is reasonable because it reduces public attack surface, but it can complicate debugging because engineers now need bastions, VPNs, or workload identity flows. A strict egress proxy is reasonable because it limits data exfiltration, but it can slow incident response because vendor endpoints and package mirrors change.
The CTO-level mistake is assuming these are implementation details. They are operating-model commitments. If you buy, you still define ownership and exception rules, but you avoid maintaining every parser, controller, report, and UI. If you build, you gain control, but you inherit the semantic backlog permanently because internal users will expect the platform to understand the company’s special cases.
Start by pricing the second year, not the launch quarter
Before approving an in-house build, ask for a one-page second-year operating budget: named owners, upgrade cadence, on-call model, exception process, data retention, and the top 10 integrations that must keep working. Require a pilot that measures cost per request, p95 latency during autoscaling, policy false positives, and engineer-hours per exception for 30 days. Then buy everything that is merely necessary, and build only what makes your company meaningfully different.
