32 articles on data — what drives the cost, how it is priced, and where the savings actually are.
BigQuery lets you stream rows in for immediate querying, but streaming ingestion is billed per gigabyte on top of storage and query costs, unlike free batch loads. Here is what streaming costs and how the Storage Write API changes the math.
BigQuery editions let you buy query capacity as slots with autoscaling, so you pay for compute that follows demand instead of a flat commitment or unpredictable on-demand scans. Here is how slot autoscaling is priced and when it beats on-demand.
Dataproc Serverless runs Spark jobs without a standing cluster, billing per compute unit for the duration of each job, which eliminates idle cluster cost. Here is how it is priced and when it beats a managed Dataproc cluster.
Cloud Composer bills environment compute (per-hour, always on) plus the underlying GKE, database, and storage. The always-on environment fee is significant, so consolidating DAGs and right-sizing the environment matter. Here is the breakdown.
Redshift Serverless bills per Redshift Processing Unit (RPU) hour while queries run, plus managed storage, with no cluster to pause. It suits variable, intermittent analytics; steady heavy load is cheaper on a provisioned cluster. Here is how to size it.
A data pipeline's cost is the compute that ingests and transforms data, the storage it lands in, and the data transfer between stages. Processing only what changed, right-sizing compute, and efficient formats cut it. Here is how.
ETL transforms data on separate compute before loading it; ELT loads raw data into the warehouse and transforms there using warehouse compute. Which is cheaper depends on your warehouse pricing and transformation load. Here is the tradeoff.
Streaming processes data continuously for low-latency freshness, keeping compute always running; batch processes in scheduled bulk runs on transient compute. When data does not need to be fresh instantly, batch is far cheaper. Here is the tradeoff.
Data warehouse cost is compute (queries and transformations) plus storage, whether Snowflake credits, BigQuery bytes scanned, or Redshift nodes. Scanning less, right-sizing compute, and auto-suspend cut it. Here is how.
A data lake's cost is object storage (often huge volumes) plus the query engines that read it. Storage tiering, lifecycle policies, compaction, and efficient formats cut it. Here is how to keep a growing lake affordable.
Kafka cost is the broker compute, the storage of retained messages, cross-AZ replication transfer, and (if managed) the service premium. Retention, right-sized brokers, and tiered storage cut it. Here is how.
CDC streams inserts, updates, and deletes from source databases to downstream systems, priced by the volume of change data plus the connector compute. High-change sources and full-table snapshots drive the bill. Here is how to keep it lean.
Orchestration tools (Airflow, Dagster, Prefect, Step Functions) schedule and coordinate data workflows; the orchestrator itself is modest, but the tasks it runs are the real cost. Efficient tasks and right-sized orchestration cut it. Here is how.
Running Airflow costs the always-on scheduler, webserver, and workers plus the metadata database, whether self-hosted or on a managed service (MWAA, Cloud Composer) that adds a premium. Right-sizing and worker autoscaling cut it. Here is the model.
dbt transforms data with SQL in your warehouse; dbt Core is free and dbt Cloud has per-seat pricing, but the real cost is the warehouse compute your dbt models run. Efficient models and incremental builds cut it. Here is the model.
Reverse ETL syncs data from the warehouse into operational tools (CRM, marketing, support), priced by rows synced or per destination, plus the warehouse queries that extract the data. Sync frequency and volume drive the bill. Here is the model.
Data observability tools monitor data quality, freshness, and pipeline health, priced by tables or volume monitored plus the queries they run to check data. Scoping monitoring to critical data and efficient checks cut it. Here is the model.
Labeling training data, human annotation, managed labeling services, or model-assisted labeling, is often a large and underestimated ML cost. Active learning, pre-labeling, and scoping the labeled set cut it. Here is how.
AWS Clean Rooms bills for the compute (CRPU) that runs privacy-preserving queries across partners' data without sharing raw data. Query volume and complexity drive the bill. Here is the model.
Datastream bills per GB of change-data-capture (CDC) data processed from source databases, plus backfill of existing data. High-change-rate sources and large backfills drive the bill. Here is the CDC replication model.
Lake Formation's governance and permissions add no direct charge; you pay for the underlying S3, Glue, and Athena, plus a storage-optimization (compaction) charge for governed tables. The underlying services are the real cost. Here is the model.
Partitioning splits a large table into segments by a key (often date), so queries with a matching filter scan only relevant partitions instead of the whole table. On per-scan and IO-billed systems, that directly cuts cost. Here is the tradeoff.
Dataplex bills per data compute unit (DCU) hour for its processing tasks (discovery, quality, tiering) plus shuffle storage, and its catalog metadata. Running unnecessary tasks and over-provisioning DCUs drive the bill. Here is the model.
A materialized view precomputes an expensive query and stores the result, so reads are fast and cheap, at the cost of storage and refresh compute. For frequently-run expensive queries, the trade pays off. Here is the cost math.