data
34 articles on data — what drives the cost, how it is priced, and where the savings actually are.
BigQuery streaming inserts cost: when real-time ingestion adds up
BigQuery lets you stream rows in for immediate querying, but streaming ingestion is billed per gigabyte on top of storage and query costs, unlike free batch loads. Here is what streaming costs and how the Storage Write API changes the math.
BigQuery slots autoscaling cost: editions capacity without over-committing
BigQuery editions let you buy query capacity as slots with autoscaling, so you pay for compute that follows demand instead of a flat commitment or unpredictable on-demand scans. Here is how slot autoscaling is priced and when it beats on-demand.
GCP Dataproc Serverless cost: paying for Spark without managing clusters
Dataproc Serverless runs Spark jobs without a standing cluster, billing per compute unit for the duration of each job, which eliminates idle cluster cost. Here is how it is priced and when it beats a managed Dataproc cluster.
GCP Cloud Composer cost: a managed Airflow baseline that is always on
Cloud Composer bills environment compute (per-hour, always on) plus the underlying GKE, database, and storage. The always-on environment fee is significant, so consolidating DAGs and right-sizing the environment matter. Here is the breakdown.
AWS Redshift Serverless cost: you pay per RPU-hour of query time
Redshift Serverless bills per Redshift Processing Unit (RPU) hour while queries run, plus managed storage, with no cluster to pause. It suits variable, intermittent analytics; steady heavy load is cheaper on a provisioned cluster. Here is how to size it.
Data pipeline cost optimization: moving and transforming data for less
A data pipeline's cost is the compute that ingests and transforms data, the storage it lands in, and the data transfer between stages. Processing only what changed, right-sizing compute, and efficient formats cut it. Here is how.
ETL vs ELT cost: where you transform changes the bill
ETL transforms data on separate compute before loading it; ELT loads raw data into the warehouse and transforms there using warehouse compute. Which is cheaper depends on your warehouse pricing and transformation load. Here is the tradeoff.
Streaming vs batch data processing: the cost of freshness
Streaming processes data continuously for low-latency freshness, keeping compute always running; batch processes in scheduled bulk runs on transient compute. When data does not need to be fresh instantly, batch is far cheaper. Here is the tradeoff.
Data warehouse cost optimization: compute and storage discipline
Data warehouse cost is compute (queries and transformations) plus storage, whether Snowflake credits, BigQuery bytes scanned, or Redshift nodes. Scanning less, right-sizing compute, and auto-suspend cut it. Here is how.
Data lake cost optimization: storage tiers and query efficiency
A data lake's cost is object storage (often huge volumes) plus the query engines that read it. Storage tiering, lifecycle policies, compaction, and efficient formats cut it. Here is how to keep a growing lake affordable.
Kafka cost optimization: brokers, storage, and managed vs self-hosted
Kafka cost is the broker compute, the storage of retained messages, cross-AZ replication transfer, and (if managed) the service premium. Retention, right-sized brokers, and tiered storage cut it. Here is how.
Change data capture cost: streaming database changes efficiently
CDC streams inserts, updates, and deletes from source databases to downstream systems, priced by the volume of change data plus the connector compute. High-change sources and full-table snapshots drive the bill. Here is how to keep it lean.
Data orchestration cost: the scheduler is cheap, the tasks are not
Orchestration tools (Airflow, Dagster, Prefect, Step Functions) schedule and coordinate data workflows; the orchestrator itself is modest, but the tasks it runs are the real cost. Efficient tasks and right-sized orchestration cut it. Here is how.
Airflow cost in the cloud: managed vs self-hosted
Running Airflow costs the always-on scheduler, webserver, and workers plus the metadata database, whether self-hosted or on a managed service (MWAA, Cloud Composer) that adds a premium. Right-sizing and worker autoscaling cut it. Here is the model.
dbt cost: the tool is cheap, the warehouse compute is not
dbt transforms data with SQL in your warehouse; dbt Core is free and dbt Cloud has per-seat pricing, but the real cost is the warehouse compute your dbt models run. Efficient models and incremental builds cut it. Here is the model.
Reverse ETL cost: syncing warehouse data back to tools
Reverse ETL syncs data from the warehouse into operational tools (CRM, marketing, support), priced by rows synced or per destination, plus the warehouse queries that extract the data. Sync frequency and volume drive the bill. Here is the model.
Data observability cost: monitoring pipelines without overspending
Data observability tools monitor data quality, freshness, and pipeline health, priced by tables or volume monitored plus the queries they run to check data. Scoping monitoring to critical data and efficient checks cut it. Here is the model.
Data labeling cost: budgeting the least glamorous ML expense
Labeling training data, human annotation, managed labeling services, or model-assisted labeling, is often a large and underestimated ML cost. Active learning, pre-labeling, and scoping the labeled set cut it. Here is how.
AWS Clean Rooms cost: per compute-unit of collaboration queries
AWS Clean Rooms bills for the compute (CRPU) that runs privacy-preserving queries across partners' data without sharing raw data. Query volume and complexity drive the bill. Here is the model.
GCP Datastream cost: per GB of change data processed
Datastream bills per GB of change-data-capture (CDC) data processed from source databases, plus backfill of existing data. High-change-rate sources and large backfills drive the bill. Here is the CDC replication model.
AWS Lake Formation cost: free governance, you pay for the storage-optimizer
Lake Formation's governance and permissions add no direct charge; you pay for the underlying S3, Glue, and Athena, plus a storage-optimization (compaction) charge for governed tables. The underlying services are the real cost. Here is the model.
Database partitioning: cutting query cost by scanning less
Partitioning splits a large table into segments by a key (often date), so queries with a matching filter scan only relevant partitions instead of the whole table. On per-scan and IO-billed systems, that directly cuts cost. Here is the tradeoff.
GCP Dataplex cost: data compute units for tasks and management
Dataplex bills per data compute unit (DCU) hour for its processing tasks (discovery, quality, tiering) plus shuffle storage, and its catalog metadata. Running unnecessary tasks and over-provisioning DCUs drive the bill. Here is the model.
Materialized views: trading storage and refresh for query savings
A materialized view precomputes an expensive query and stores the result, so reads are fast and cheap, at the cost of storage and refresh compute. For frequently-run expensive queries, the trade pays off. Here is the cost math.