Vertex AI cost optimization: training, prediction, and pipelines

Quick answer

Vertex AI cost spans custom training (compute-hours, GPU or CPU), prediction endpoints (always-on serving nodes), pipelines (per step compute), and Workbench notebooks (compute while running). The biggest wastes mirror SageMaker: always-on prediction endpoints for intermittent models and idle Workbench instances. The levers: autoscale endpoints (and set minimum nodes appropriately), stop idle notebooks, use Spot/preemptible VMs for training, and right-size every node. Match serving nodes to real prediction traffic rather than provisioning constantly.

Google Cloud Vertex AI is a managed ML platform whose cost, like other platforms, is instance-hours across training, prediction, pipelines, and notebooks. The convenience comes with the same risk: leaving expensive serving nodes or notebooks running when they are not needed.

Where Vertex AI cost comes from

Component	Cost pattern	Main waste
Custom training	Compute-hours (GPU/CPU) per job	Not using preemptible VMs
Prediction endpoints	Always-on serving nodes	Idle endpoints, high minimum nodes
Pipelines	Per-step compute	Oversized steps
Workbench notebooks	Compute while running	Left running idle

Custom training jobs bill compute-hours and can use preemptible (Spot) VMs for a discount on fault-tolerant training. Prediction endpoints run serving nodes, and the minimum-node setting means an endpoint holds capacity even at low traffic, so a high minimum on an intermittent model wastes money. Pipelines bill their steps' compute, and Workbench notebooks bill while running.

The biggest wastes

As with SageMaker, always-on prediction endpoints and idle notebooks concentrate the overspend. For endpoints, autoscale to traffic and set the minimum node count to the least your latency and availability need (a minimum of one or more nodes bills continuously), and do not keep endpoints up for models queried rarely. For Workbench, stop idle instances (idle-shutdown) so forgotten notebooks do not bill.

Controlling Vertex AI cost

Use preemptible/Spot VMs for interruptible training, autoscale prediction endpoints to traffic and set minimum nodes appropriately (avoiding idle always-on capacity for intermittent models), enable idle-shutdown on Workbench notebooks, right-size training, serving, and pipeline compute, and use committed use discounts for steady usage. Because Vertex is instance-hours, the discipline is matching serving and notebook compute to real need rather than provisioning constantly, the same as inference optimization generally.

FAQ

What drives Vertex AI cost?

Instance-hours across custom training (compute-hours, GPU or CPU), prediction endpoints (always-on serving nodes), pipelines (per-step compute), and Workbench notebooks (compute while running). The biggest wastes are always-on prediction endpoints for intermittent models and idle Workbench instances, since both bill continuously whether or not used.

How do I reduce Vertex AI cost?

Use preemptible/Spot VMs for interruptible training, autoscale prediction endpoints to traffic and set minimum nodes appropriately (avoiding idle always-on capacity), enable idle-shutdown on Workbench notebooks, right-size training, serving, and pipeline compute, and use committed use discounts for steady usage. Matching serving and notebook compute to real need is the main discipline.

Why do Vertex AI prediction endpoints cost so much when idle?

Because they run serving nodes continuously, and the minimum-node setting means an endpoint holds at least that capacity even at low or zero traffic. So a high minimum node count on an intermittent model bills around the clock. Autoscaling to traffic and setting the minimum to the least your latency and availability need avoids this idle cost.

Can Vertex AI training use preemptible VMs?

Yes. Custom training jobs can run on preemptible (Spot) VMs at a significant discount, ideal for fault-tolerant or checkpointed training that can tolerate interruption. For interruptible training, this cuts the compute-hours cost substantially, the same benefit as using Spot for training on other platforms.

How do I stop idle Vertex AI notebooks from costing money?

Enable idle-shutdown on Workbench notebook instances so they stop after a period of inactivity rather than billing while left running. Notebooks bill for their compute whenever running regardless of activity, and forgotten notebooks left on overnight or over weekends are a common waste, so automatic idle-shutdown is a high-impact control.

Does C3X estimate Vertex AI cost?

C3X prices Vertex AI resources (training compute, prediction endpoints, notebook instances) from your Terraform, so the instance-hours cost appears before deploy. Actual cost depends on runtime, autoscaling, minimum nodes, and how long resources stay up, which you model, since the main waste is idle compute.

What to do next

Price your Vertex AI infrastructure before you deploy. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.