Self-hosting a cloud pricing API on a €4 VPS

Cloud cost estimation tools all face the same problem: you need accurate, current pricing data for every resource type across every region, refreshed regularly, in a format that an estimator can query in milliseconds. The official AWS, Azure, and GCP pricing endpoints give you the data, but they're not designed for the access pattern a cost estimator needs. So you build a layer between them.

This post walks through the design and operational details of the C3X pricing API: what it scrapes, how it stores it, how we serve it at scale on a €4-7/month VPS, and what we learned running it in production.

Why a separate service?

A cost estimator running on a developer laptop or in CI can't call the AWS Pricing API directly for several reasons.

First, payload size. The AWS bulk EC2 pricing file alone is over 2.5 GB of JSON. Loading and parsing that on every estimate is impossibly slow. Even after region filtering, the per-region files are hundreds of MB. CI runners and developer laptops can't shoulder that.

Second, query shape. AWS's pricing data is a flat list of SKUs with attribute maps. To answer "what's the on-demand hourly rate for an m5.xlarge Linux instance in us-east-1, shared tenancy, no software pre-installed?" you have to filter by six attributes, find the right SKU, then look up the right price dimension on it. That's not a one-line query, especially across three clouds with different attribute names for the same concepts.

Third, air-gapped environments. Federal agencies, regulated industries, and any organization with strict egress policies can't let CI runners hit external pricing APIs on every PR. A self-hosted copy of the data inside their VPC removes that constraint.

What gets scraped

Three official sources, each on a different schedule and shape:

AWS Bulk Pricing API

AWS publishes pricing as JSON files at offers-api.aws/savingsPlan/AWSEC2/current/{region}/index.json and similar URLs per service. The pricing/api.aws bulk download index lists every service. EC2 is the biggest at ~2.5 GB; RDS, Lambda, S3, etc. are smaller. New pricing files are typically published within a week of price changes, which AWS does irregularly (small regional adjustments are common, large repricing events rare).

Our scraper downloads the index, downloads each service's pricing file, streams the JSON parsing (rather than loading into memory), and upserts SKUs into PostgreSQL. For EC2 alone this is about 2.2 million SKU records. Stream parsing is mandatory; loading EC2 pricing into memory at once would OOM a 4 GB VPS.

Azure Retail Prices API

Azure publishes prices through prices.azure.com/api/retail/prices, a paginated REST API with ODATA filters. Each page returns up to 1,000 SKUs. Iterating to the end takes ~4,000 requests for the full global dataset (about 490K SKUs). Significantly easier to consume than AWS's bulk files but slower per scrape.

Pagination tip: use the $skiptoken pagination instead of $skip if the API offers it. Skiptoken is server-side cursor-based; skip is offset-based and gets slower deeper in the result set.

GCP Cloud Billing Catalog

GCP exposes pricing through cloudbilling.googleapis.com with OAuth-authenticated calls to /v1/services and /v1/services/{id}/skus. Cleaner API than AWS or Azure; about 72K SKUs total. Authentication is via service account JSON key.

The catalog has some quirks worth knowing: per-region rates are embedded as "tieredRates" with "geoTaxonomy", not flat attribute filters. Parsing them correctly requires walking the GeoTaxonomy tree.

The schema

After three iterations we settled on this PostgreSQL schema for products and prices:

CREATE TABLE products (
  product_hash    VARCHAR(64) PRIMARY KEY,
  vendor_name     TEXT NOT NULL,
  service         TEXT NOT NULL,
  product_family  TEXT,
  region          TEXT,
  sku             TEXT,
  attributes      JSONB NOT NULL,
  last_scraped_at TIMESTAMPTZ NOT NULL,
  -- indexes by (vendor_name, service, region) and GIN on attributes
);

CREATE TABLE prices (
  price_hash     VARCHAR(64) PRIMARY KEY,
  product_hash   VARCHAR(64) NOT NULL REFERENCES products(product_hash),
  purchase_option TEXT NOT NULL,
  term_length    TEXT,
  unit           TEXT NOT NULL,
  usd            NUMERIC(16, 10) NOT NULL,
  description    TEXT,
  last_scraped_at TIMESTAMPTZ NOT NULL
);

The product_hash is a deterministic SHA-256 of vendor + service + region + sorted attributes. This means a re-scrape of the same SKU produces the same hash, which lets us use INSERT...ON CONFLICT to upsert efficiently. The price_hash is similar but includes the purchaseOption.

Attributes are JSONB with a GIN index. Most queries filter on a few attributes (instanceType, operatingSystem, tenancy), and the GIN index handles those filter combinations efficiently.

The GraphQL API

We exposed a single products query with filter and limit arguments:

{
  products(filter: {
    vendorName: "aws"
    service: "AmazonEC2"
    region: "us-east-1"
    productFamily: "Compute Instance"
    attributeFilters: [
      {key: "instanceType", value: "m5.xlarge"}
      {key: "operatingSystem", value: "Linux"}
      {key: "tenancy", value: "Shared"}
      {key: "preInstalledSw", value: "NA"}
      {key: "capacitystatus", value: "Used"}
    ]
  }) {
    prices(filter: {purchaseOption: "on_demand"}) {
      USD
    }
  }
}

Single response under 500 bytes. Typical query latency from the VPS is 20-40 ms cold, <5 ms with cache hits.

The schema is intentionally minimal. There's no "list all products" or open-ended query. Every realistic cost-estimation query has a known vendor/service/region/attribute set. Constraining the API to that pattern keeps Postgres queries predictable.

The deployment

Everything fits in a docker-compose.yml on a single VPS:

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_DB: c3x_pricing
      POSTGRES_USER: c3x
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready -U c3x -d c3x_pricing
      interval: 5s

  api:
    image: c3xdev/c3x-pricing-api:latest
    ports:
      - "127.0.0.1:4000:4000"
    environment:
      DATABASE_URL: postgres://c3x:$${POSTGRES_PASSWORD}@postgres:5432/c3x_pricing
      SCRAPE_CONCURRENCY: "1"
    depends_on:
      postgres:
        condition: service_healthy

volumes:
  postgres_data:

secrets:
  postgres_password:
    file: ./secrets/postgres_password.txt

Caddy handles TLS termination and reverse-proxies to localhost:4000. Let's Encrypt certificates auto-renew. A daily cron at 03:00 UTC runs docker compose exec api c3x-pricing-api scrape --vendor all, which streams from each cloud's pricing endpoint into PostgreSQL.

The full setup including the Caddyfile, cron schedule, and security hardening is in the self-hosted documentation.

Operational lessons

Six months of production gave us a list of things that mattered more than expected.

Streaming JSON parse is mandatory for AWS

First attempt: load the JSON, parse it into a Go struct, write to Postgres. On a 4 GB VPS, this OOMed parsing EC2 pricing. Switched to a streaming JSON decoder that processes one product at a time. Memory peaked at ~120 MB during scrape.

SCRAPE_CONCURRENCY = 1 in production

Running region scrapes concurrently triggers AWS's rate limiting and uses too much memory on small VPSes. Sequential scraping takes ~45 minutes for full AWS but is reliable. SCRAPE_CONCURRENCY is an env var so you can crank it up on larger hardware.

Statement timeout matters

Default Postgres statement_timeout (no timeout) means a bug in the scraper or a slow disk can hang queries forever, blocking the API. We set statement_timeout = 300s for application queries and 60s for API queries. Forced surfacing of the bug instead of mysterious latency.

HSTS and ReadHeaderTimeout

Public HTTP endpoints need basic hardening. HSTS prevents downgrade attacks. ReadHeaderTimeout on the Go server prevents Slowloris attacks where a client opens a connection and trickles bytes forever to exhaust the server's connection pool. Both are one-liners.

Daily scrape, not hourly

Cloud pricing changes infrequently. AWS publishes updates a few times per quarter on average. Daily scrapes are enough. Hourly scrapes would multiply load 24x for almost no freshness benefit and risk getting rate-limited.

Cost breakdown of the actual setup

Real numbers from our deployment:

• Hetzner CPX21 VPS: €6.05/month
• Domain (pricing.c3x.dev): ~$15/year prorated to $1.25/month
• Bandwidth: included in VPS up to 20 TB/month, we use ~50 GB
• TLS: free via Let's Encrypt
• PostgreSQL: free, runs on the VPS
• Monitoring: free Caddy access logs analyzed by a shell script

Total: about €7.30/month or ~$8/month for the entire infrastructure serving a public pricing API across three clouds with 2.7M SKUs.

FAQ

Where to find the code

The full source is at github.com/c3xdev/c3x-pricing-api. Apache 2.0. PR-friendly. The repo includes the scrapers, the GraphQL server, the database migrations, and a Helm chart for Kubernetes deployments if you'd rather not use docker-compose.

For documentation on self-hosting in your own environment including air-gapped setups, see the self-hosted docs. For connecting C3X CLI to your self-hosted API, set the C3X_PRICING_API_ENDPOINT environment variable as documented in the CLI reference.

If you'd rather just use the public endpoint, no setup is required. The CLI uses pricing.c3x.dev by default. See the quickstart to install C3X and run your first estimate in five minutes.