The Cheapest Way to Centralize Logs at Scale

The Cheapest Way to Centralize Logs at Scale#

The default architecture for logs is to ship everything to one platform and pay whatever that costs. It works, it is easy to set up, and at scale it produces a bill that makes people ask whether logging is worth it. It is, but not in that shape. The cheapest workable architecture separates the pipeline into stages where each stage discards what the next one does not need, and it usually cuts spend by more than half without losing anything anyone actually reads.

The four levers below are ordered by how much they return per hour of work.

Filter at the agent, not at the platform#

Everything you drop before it leaves the node is free. Everything you drop after it arrives has already been billed. Yet the common setup ships the full stream and filters in the platform's UI, which controls what people see rather than what you pay for.

The highest-yield filters are almost always the same three: health check and readiness probe traffic, successful access logs for static assets, and debug output that someone enabled during an incident two years ago and never turned off.

# vector.toml style: drop before egress
[transforms.drop_noise]
type = "filter"
inputs = ["kubernetes_logs"]
condition = '''
  !contains(string!(.message), "/healthz") &&
  !contains(string!(.message), "/readyz") &&
  !(.level == "debug" && .kubernetes.pod_namespace != "staging")
'''

Run this against a day of real traffic before you deploy it. The number that matters is bytes dropped, and on a typical cluster these three rules alone remove thirty to sixty percent of volume.

Sample what is high volume and low information#

Some logs are individually worthless and collectively useful. A successful HTTP 200 tells you nothing on its own; a count of them tells you throughput. For that class, sample aggressively and keep a metric alongside.

The rule that works: sample successes, keep every error. A one-percent sample of 200s and a hundred percent of 5xx preserves every debugging path while removing most of the bytes. The failure mode to avoid is uniform sampling across all severities, which loses the rare events you actually need and keeps noise you do not.

For request-level data specifically, ask whether a trace already covers it. If you run distributed tracing, a large share of your access logs is duplicating information that the trace carries with better structure. Sending both is common and wasteful.

Tier retention instead of picking one number#

Most teams set a single retention period for everything, usually thirty days, chosen by nobody in particular. Actual access patterns are steeply front-loaded: the overwhelming majority of queries hit the last forty-eight hours, and the long tail exists for compliance and the occasional investigation.

That shape maps onto three tiers. Keep a short hot window, seven days or so, indexed and fast. Move the next stretch to cheap object storage in a queryable format, where a search is slower and costs a fraction. Keep whatever compliance requires as compressed archives you can restore if asked.

The saving here is large because indexed storage costs many times what object storage costs, and because the middle tier usually turns out to cover ninety-nine percent of real investigations. The main work is making sure the slow tier is genuinely queryable rather than a bucket nobody knows how to search, because an archive you cannot query during an incident is a compliance artifact, not a log.

Stop sending what another system already holds#

The last lever is structural. Audit what is being shipped and you will generally find duplication: application logs that repeat what metrics already record, Kubernetes events already available through the API, load balancer logs stored twice, and one or two services logging entire request bodies because it helped during a launch.

# Rank namespaces by log volume. The top three usually explain most of the bill.
$ kubectl get ns -o name | sed 's|namespace/||' | while read -r ns; do
    bytes=$(kubectl logs -n "$ns" --since=1h --all-containers --prefix --tail=-1 2>/dev/null | wc -c)
    printf '%12s  %s\n' "$bytes" "$ns"
  done | sort -rn | head

Pair that with query statistics from your platform. A service in the top three for volume and absent from the query logs is the first candidate for aggressive filtering, and the conversation with its owners is usually short once you show them both numbers.

The decision, concretely#

• Have not filtered at the agent? Start there today. It is the only lever that reduces the bill without changing anything downstream, and it typically removes a third of the volume in an afternoon.
• Paying to index everything? Move to a model where retention and indexing are priced separately, whether that means changing tiers on your current platform or changing platform. Our comparison of log management tools covers which pricing model suits which query pattern.
• Keeping thirty days of everything because it is the default? Replace one number with three tiers. Hot for a week, cheap and queryable for a month or two, archived beyond that.
• Running traces and full access logs? Pick one as the primary record for request-level data. Maintaining both is paying twice for the same information.

The call we'd make#

Do the filtering work before you renegotiate anything. Vendors will discount volume commitments, and a discount on data you should not be sending is a worse deal than not sending it. Teams that filter first usually find the renegotiation unnecessary, because the bill lands back in a range nobody is worried about, and the pipeline they built to get there keeps working when volume grows again. If you want the tool-selection side of this, the open-source observability options are worth pricing against your current platform once your volume reflects what you actually read.