The bill spike emergency runbook: stopping the bleed in one hour

A cost spike is an incident and should be run like one. The instinct to understand before acting is exactly wrong here: every hour spent on root cause analysis while a runaway process burns $400 an hour costs $400. Contain first, diagnose second.

Minutes 0 to 15: confirm and quantify

First confirm the spike is real rather than a billing artifact. One time annual charges, a commitment purchase, a currency conversion, or a delayed usage record can all look like a spike and need no action. Check whether the increase is in usage quantity or only in charge amount.

Then quantify the burn rate in dollars per hour above the normal baseline. This single number governs everything: at $50 an hour you have time to investigate carefully, at $2,000 an hour you contain within minutes and accept collateral damage. Write it at the top of the incident channel and update it as containment lands.

Minutes 15 to 30: find the delta

Do not browse the bill; compare it. Pull cost grouped by service for the spike period and for the same period a week earlier, and sort by absolute difference. In nearly every case one or two services account for the entire delta. The hourly view then gives you the start time, which you correlate against deployments, merges, configuration changes, and traffic events. The structured version of this is in cost spike root cause analysis.

Minutes 30 to 45: contain

Choose the least destructive action that actually stops the burn. In escalating order: apply a rate limit or request quota; set a hard cap such as a maximum bytes billed on queries or a concurrency limit on functions; scale the resource down or to zero; disable the trigger, schedule, or pipeline; and only as a last resort delete the resource.

The decision rule is the burn rate against the cost of the disruption. A runaway batch job burning $3,000 an hour gets killed immediately even if it means restarting a twelve hour run. A misconfigured log pipeline adding $80 an hour gets a filter applied in the next deploy rather than an emergency change.

Common containments by cause. Runaway compute scaling: set maximum capacity on the group to its normal ceiling. Recursive function invocations, where a function writes to the storage bucket that triggers it: set reserved concurrency to zero, which stops it instantly and reversibly. Query cost explosion: set maximum bytes billed so oversized queries fail rather than charge. Data transfer explosion: identify the flow from logs and block or throttle it. Unauthorised use after credential compromise: rotate credentials immediately, then terminate the resources, and treat it as a security incident first.

Minutes 45 to 60: verify

Confirm containment worked by checking the hourly cost trend, not by assuming. Billing data lags, often by several hours, so also verify at the resource level: instance count returned to normal, function invocation rate dropped, bytes processed fell. If the burn rate has not fallen, your containment addressed a symptom rather than the cause, and you escalate to the next more destructive option.

After the hour: diagnose and decide

With the burn stopped, work out the actual cause and, separately, decide what to restore. Not every spike is waste: a genuine traffic surge from a successful launch is a spike you fund rather than fix. Sort the cause into one of four buckets, because each has a different remedy. A bug, which gets fixed and redeployed. A misconfiguration, which gets corrected and guarded by policy. Legitimate growth, which gets budgeted. Or unauthorised use, which becomes a security incident.

The prevention layer

Every spike incident should end with a control that would have caught it. Four layers, roughly in order of how early they act.

Hard technical limits are the only thing that actually caps spend: maximum autoscaling capacity, function concurrency limits, query byte limits, and storage quotas. Alerts tell you after money is spent; limits stop it being spent. Anomaly detection on cost gives faster notification than a monthly invoice, typically within a day, using the approach in cloud cost anomaly detection. Budget alerts at 50, 80, and 100 percent of expected monthly spend give a coarser but useful signal.

And the earliest layer is design time: many spikes trace to an infrastructure change whose cost profile nobody examined, an autoscaling group with no maximum, a log configuration with no filter, a function triggered by the bucket it writes to. Pricing infrastructure changes before merge against the resource catalog catches a meaningful share of those before they can spike at all.

Write the postmortem

Record the timeline, the total cost of the incident, the detection delay, the containment action, and the control added. The detection delay is the most useful metric: a spike caught in two hours costs a fraction of the same spike caught on the monthly invoice, and reducing that delay is usually worth more than preventing any individual cause.

FAQ

What to do next

Catch the spike before it ships. C3X prices infrastructure changes in the pull request against a live resource catalog. Start with the quickstart.