Notes from the team building C3X. Cost estimation for Terraform, the economics of cloud infrastructure, and how to ship FinOps tooling without a SaaS gate.
A feature store's cost is the offline store (bulk feature data for training), the online store (low-latency serving for inference), and the compute that materializes features. The online store and materialization drive the bill. Here is the model.
Labeling training data, human annotation, managed labeling services, or model-assisted labeling, is often a large and underestimated ML cost. Active learning, pre-labeling, and scoping the labeled set cut it. Here is how.
A model registry stores versioned model artifacts and metadata; its cost is mostly the storage of those artifacts, which can be large for many versions of big models. Lifecycle policies and pruning old versions cut it. Here is the model.
MLOps cost spans data prep, training, serving, feature stores, registries, monitoring, and orchestration, each a piece of the bill. Serving and training compute usually dominate, but the hidden pieces add up. Here is how to budget the whole lifecycle.
ParallelCluster is a free tool that provisions HPC clusters; the cost is the underlying EC2 compute, storage, and networking it launches. Right-sizing nodes, using Spot, and scaling to zero when idle drive the bill. Here is the model.
Azure Batch is free to use; you pay for the underlying VM compute in the pools it manages, plus storage and networking. Low-priority (Spot) VMs, right-sized pools, and autoscaling to zero drive the bill. Here is the model.
Cloud HPC trades a capital cluster purchase for pay-as-you-go compute, so cost is driven by node type, job runtime, and utilization. Spot, scale-to-zero, and efficient jobs make it economical. Here is how the cost works.
Spot instances (spare capacity at up to 90% off) can be reclaimed with little notice, so they suit fault-tolerant, interruptible, or checkpointed work. Diversifying pools and handling interruption make them safe. Here is the strategy.
GPU instances vary enormously in cost, from older inference GPUs to top training accelerators costing many dollars per hour. Matching GPU class to the workload, and using Spot, drives the bill. This compares the tradeoffs.
Training cost is GPU-hours: accelerator class times count times training time. Spot with checkpointing, right-sized GPUs, efficient data pipelines, and stopping idle instances cut it sharply. Here is how.
Inference cost is serving capacity times uptime: the accelerators or CPUs kept ready to respond. Autoscaling, right-sized hardware, batching, and scaling to zero when idle cut it. Here is how.
GPUs accelerate inference but cost far more per hour; CPUs are cheaper but slower per request. For small models, low throughput, or latency-tolerant workloads, CPU is often cheaper per inference. Here is how to decide.
Serving a model can use a managed endpoint (SageMaker, Vertex, Azure ML) that bundles ops at a premium, or self-hosted serving on your own compute that is cheaper per hour but adds operational burden. Here is the tradeoff.
Real-time inference keeps serving capacity always ready to respond instantly; batch inference processes predictions in bulk on transient compute. When predictions can wait, batch is far cheaper. Here is the tradeoff.
LLM API cost is tokens: input plus output tokens times the per-token rate, which varies by model tier. Choosing the right model per task, trimming prompts, caching, and capping output cut the bill. Here is how.
A RAG system's cost spans a vector database, embedding generation, the LLM API for generation, and the retrieval compute. Each scales differently, and the LLM and vector store usually dominate. Here is how to budget it.
Verified Access bills per application connected per hour plus data processed, giving zero-trust access to internal apps without a VPN. Many connected apps and high traffic drive the bill. Here is the model.
AWS Clean Rooms bills for the compute (CRPU) that runs privacy-preserving queries across partners' data without sharing raw data. Query volume and complexity drive the bill. Here is the model.
Firewall Manager centrally manages Azure Firewall policies and security across many networks; the meaningful cost is the Azure Firewalls themselves plus per-policy charges beyond the first. Here is the model.
Azure DDoS Protection (Network or IP Protection) bills a flat monthly fee covering a set of protected resources, with per-resource charges beyond the included count. It protects against volumetric attacks. Here is the model.
GuardDuty bills by the volume of logs and events it analyzes, CloudTrail events, VPC flow logs, DNS logs, and per feature (S3, EKS, malware, RDS protection). High-traffic accounts and enabled features drive the bill. Here is the model.
AWS Secrets Manager charges per secret per month plus API calls; Azure Key Vault and GCP Secret Manager charge mostly per operation with cheap or free storage; self-hosted Vault trades license or ops cost. This compares the models.
AWS WAF charges per web ACL, per rule, and per million requests; Azure WAF and GCP Cloud Armor bundle differently (policy plus per-rule plus requests). The cheapest depends on rule count and traffic. This compares the models.
AWS Shield Standard is free with Advanced a flat monthly commitment; Azure DDoS Protection is a flat plan or per-IP; GCP relies on always-on network protection plus Cloud Armor. The models differ sharply. This compares them.