Vertex AI cost optimization guide: controlling GCP ML spend

Quick answer

Vertex AI cost comes from training jobs (billed for compute, often GPU or TPU, while they run), online prediction endpoints (billed continuously for provisioned nodes whether or not they serve traffic), notebooks (billed while running, often left idle), and pipelines and other services. The biggest drains are always-on prediction endpoints and idle notebooks. Optimize by right-sizing and autoscaling endpoints (or using batch prediction for non-real-time needs), stopping idle notebooks, using preemptible or spot capacity for training, and shutting down what you are not using. The theme, as with any ML platform, is do not pay for idle GPU and endpoint capacity.

Vertex AI is GCP's managed platform for the machine-learning lifecycle, and like other ML platforms it bills each stage separately, with several stages billing continuously. The predictable Vertex AI surprise is the online prediction endpoint provisioned for real-time serving and left running around the clock, plus the notebook someone forgot to stop. Optimization means handling each stage on its own terms.

The main cost sources

Stage	Billing	Common waste
Training jobs	Compute while running, often GPU/TPU	On-demand instead of preemptible
Online prediction	Continuously for provisioned nodes	Over-provisioned, always on
Notebooks	While running	Left on idle
Pipelines / batch	Per run and compute used	Inefficient or repeated runs

Training jobs bill for the compute they use, often expensive GPU or TPU accelerators, for the duration of the run. Online prediction endpoints provision nodes for real-time serving and bill for them continuously, whether or not requests arrive. Notebooks bill for every hour they run, and pipelines bill per run. The always-on stages, endpoints and idle notebooks, are where most waste accumulates.

Training and notebooks

For training, use preemptible or spot capacity where the job tolerates interruption (with checkpointing), which cuts accelerator cost substantially, and right-size the training resources to the job, the same approach as the training side of any ML budget. For notebooks, stop them when not in use and use idle-shutdown settings so a forgotten notebook does not bill overnight and over weekends. Both are simple, high-value fixes.

Prediction endpoints, the quiet drain

Always-on online prediction endpoints are usually the largest recurring Vertex AI cost. Right-size the endpoint nodes to real traffic, enable autoscaling so capacity follows load instead of provisioning for peak, and for non-real-time needs use batch prediction, which runs on demand rather than keeping nodes warm. For spiky traffic, scaling to a low minimum avoids paying for idle capacity, the same logic as SageMaker endpoints on AWS.

Keeping Vertex AI spend honest

Label Vertex AI resources by owner and project so idle notebooks and forgotten endpoints are attributable, review deployed endpoints regularly and delete unused ones, and weigh whether an API-based or self-hosted approach is cheaper for your inference volume, as the model hosting comparison covers. The recurring theme is idle infrastructure: Vertex AI charges for what is provisioned and running. Price your ML infrastructure against the resource catalog so the always-on cost is visible before you deploy.

FAQ

What drives Vertex AI cost?

Training jobs (billed for compute, often GPU or TPU, while they run), online prediction endpoints (billed continuously for provisioned nodes whether or not they serve traffic), notebooks (billed while running, often left idle), and pipelines and other services (per run). The biggest drains are always-on prediction endpoints and idle notebooks, since both charge for capacity and time nobody is actively using.

How do I reduce Vertex AI prediction endpoint cost?

Right-size the endpoint nodes to real traffic, enable autoscaling so capacity follows load instead of provisioning for peak, scale to a low minimum for spiky traffic to avoid idle capacity, and use batch prediction for non-real-time needs since it runs on demand rather than keeping nodes warm. Delete endpoints you no longer use. Always-on endpoints are usually the largest recurring Vertex AI cost.

How do I save on Vertex AI training?

Use preemptible or spot capacity where the training job tolerates interruption with checkpointing, which cuts GPU and TPU cost substantially, and right-size the training resources to the job rather than over-provisioning accelerators. Avoid leaving experimental training runs going. Training concentrates the accelerator-heavy part of the ML budget, so these levers deliver the largest training savings.

Why do Vertex AI notebooks waste money?

Because they bill for every hour the instance runs, and they are frequently left on when idle, overnight, over weekends, or after a task is done. A forgotten notebook keeps billing for time nobody uses it. Stopping notebooks when not in use and enabling idle-shutdown settings so they stop automatically eliminates this common, avoidable waste with no downside.

Should I use Vertex AI endpoints or an alternative for inference?

It depends on your inference volume and steadiness. Vertex AI online prediction endpoints are convenient but bill continuously while provisioned, so for spiky traffic autoscaling to a low minimum or using batch prediction saves, and for high steady volume a self-hosted or API-based approach may be cheaper. Compare the always-on endpoint cost against alternatives for your actual traffic pattern before committing.

How does C3X help with Vertex AI cost?

C3X prices your Vertex AI and related GCP infrastructure from Terraform before you deploy, so the continuous cost of an always-on prediction endpoint or an oversized training resource is visible in the pull request. That helps you right-size and choose autoscaling or batch options at design time, rather than discovering an idle endpoint or forgotten notebook draining money on the monthly GCP bill.

What to do next

Price your Vertex AI infrastructure before you deploy it. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.