Elasticsearch Indexing Slow? Fix Write Backpressure

"Elasticsearch can't keep up with our writes." I hear this sentence in almost every ingest-heavy engagement, and it usually arrives with a stack trace: es_rejected_execution_exception, HTTP 429, retries piling up in Logstash or Kafka consumers. The instinct is to blame hardware. The real story is almost always the segment lifecycle.

Every document you index becomes part of an immutable Lucene segment. Refresh creates small segments constantly; background merges consolidate them. Push segment creation faster than merges and disks can absorb, and Elasticsearch pushes back — first with queueing, then rejections, then throttling. This guide is about reading those signals correctly and fixing the actual bottleneck.

Quick answer: Slow or rejected indexing is backpressure, not a bug. Check the write thread pool for rejections (GET _cat/thread_pool/write?v), check merge stats and logs for "now throttling indexing", then reduce segment churn: raise index.refresh_interval, right-size bulk requests, use auto-generated IDs, drop replicas during initial loads, and make clients retry 429s with backoff. Do not enlarge thread pool queues — that hides the signal and grows heap pressure.

How segments are born — and why they fall behind

By default Elasticsearch refreshes an actively searched shard every second (index.refresh_interval: 1s). Each refresh turns the in-memory indexing buffer into a new searchable segment. Across dozens of shards that means a constant stream of tiny segments, which Lucene's ConcurrentMergeScheduler must merge in the background while bulk traffic keeps arriving.

Every operation is also written to the translog. With the default index.translog.durability: request, Elasticsearch fsyncs the translog before acknowledging each bulk — safe, but your disks sit in the hot path twice: translog fsyncs plus segment writes and merges.

So "segment writing can't keep up" is really three queues competing for the same I/O and CPU: refresh, merge, and translog. The failure shows up in two distinct ways, and they need different fixes.

Reading the rejection: es_rejected_execution_exception

The write thread pool is fixed-size (one thread per allocated processor) with a bounded queue (10000 by default). When the queue is full, Elasticsearch answers with HTTP 429 and es_rejected_execution_exception. Since 7.9 there is a second guard on top: indexing pressure rejects requests when in-flight indexing bytes exceed indexing_pressure.memory.limit (10% of heap by default).

First command, always:

GET _cat/thread_pool/write?v&h=node_name,name,active,queue,rejected,completed

rejected is cumulative since node start — watch whether it is growing, not just non-zero. Then check where the pressure sits:

GET _nodes/stats?filter_path=nodes.*.name,nodes.*.thread_pool.write,nodes.*.indexing_pressure

If one node shows most of the rejections, you have a hot shard or skewed routing, not a cluster-wide capacity problem. If all data nodes reject together, ingest genuinely exceeds cluster capacity and the fix is on the producer side: smaller concurrent bulk load, more nodes, or faster disks.

Two client-side rules before touching the cluster:

1. Treat 429 as backpressure, not failure. Every serious client (Logstash, Elastic Agent, the language clients' bulk helpers) can retry 429 with exponential backoff. A retried 429 is data safely deferred; a dropped 429 is data loss you chose.
2. Never "fix" rejections by raising queue_size. A longer queue is the same throughput with more heap usage and worse latency. The queue is the messenger.

Merge throttling: when Lucene pushes back

The second failure mode is quieter. When merges fall behind, Elasticsearch throttles indexing on that shard down to a single thread and logs a line like now throttling indexing: numMergesInFlight=6, maxNumMerges=5. Bulk latency climbs and nobody gets a 429 — the cluster is deliberately slowing you down so merges can catch up.

Check for it:

GET my-index/_stats/merge?filter_path=indices.*.total.merges
GET _nodes/hot_threads

A growing current count and large total_throttled_time_in_millis in merge stats confirm it; hot threads will show Lucene merge threads dominating. Merge behavior is controlled by the merge scheduler (index.merge.scheduler.max_thread_count, auto-throttled I/O). The defaults are right for SSDs; on spinning disks or throttled cloud volumes the honest fix is storage, not settings. Persistent merge throttling on NVMe usually means you are simply creating too many segments — a refresh and bulk-sizing problem.

Tuning that actually moves the needle

Everything below is from Elastic's own tune for indexing speed guidance, ordered by how often it wins in the field.

Raise refresh_interval

If nobody needs 1-second search visibility on a write-heavy index, stop paying for it:

PUT logs-write/_settings
{
  "index.refresh_interval": "30s"
}

Fewer refreshes → fewer, larger initial segments → dramatically less merge work. For pure bulk loads (reindex, migration, initial import), go further:

PUT bulk-load-index/_settings
{
  "index": {
    "refresh_interval": "-1",
    "number_of_replicas": 0
  }
}

Restore both after the load. Replicas during an initial load mean every document is indexed twice for no benefit — replica recovery from the primary afterwards is cheaper.

Right-size bulk requests

There is no magic bulk size, and I refuse to invent one. Benchmark: start small, double until throughput stops improving, and stay below the point where requests trigger indexing-pressure rejections. Too-small bulks waste round trips; too-large bulks amplify heap pressure and make each 429 more expensive to retry. Use multiple client workers — a single bulk stream rarely saturates a cluster.

Use auto-generated IDs

With external IDs, Elasticsearch must check whether each document already exists — an extra read per write. If you don't need idempotent upserts, let Elasticsearch generate IDs and skip the lookup.

Consider async translog durability — with eyes open

PUT logs-write/_settings
{
  "index.translog.durability": "async",
  "index.translog.sync_interval": "30s"
}

This acknowledges writes before fsync, taking translog fsyncs off the latency path. The trade-off is explicit: a dying node can lose up to sync_interval of acknowledged operations. Fine for replayable logs fed from Kafka; wrong for anything you cannot re-ingest.

What not to do

• Don't force merge a live index. "Defragment your indices" advice floats around the internet; Elasticsearch has no defragmentation. _forcemerge is for read-only indices (e.g., after ILM rollover). On an actively written index it creates oversized segments and steals the I/O your real merges need.
• Don't grow thread pool queues or thread counts. Fixed pools sized to CPU cores are intentional.
• Don't scale coordinating nodes to fix data-node rejections. The bottleneck is where segments are written.

Indexing pressure also compounds instability: a node drowning in merges responds slowly to the master and can drop out, taking you from "slow writes" to unassigned shards. If you are already there, I wrote up that path separately: fixing unassigned shards and RED clusters. And if you want rejections, merge throttling, and thread pool saturation visible before your producers alarm, that is what searchali.com/monitoring watches continuously.

Frequently Asked Questions

What causes es_rejected_execution_exception in Elasticsearch?

The write thread pool queue on a data node filled up: bulk requests arrived faster than the node could index them. Elasticsearch returns HTTP 429 so clients can back off. Check GET _cat/thread_pool/write?v — one hot node points to shard/routing skew, all nodes point to real capacity limits.

Is setting refresh_interval to -1 safe in production?

Safe for durability — documents sit in the translog and are not lost — but they stay invisible to search until the next refresh. Use -1 for bulk loads and migrations, then restore a real interval. For steady-state write-heavy indices, 30s is usually the better trade.

How do I know if merge throttling is slowing my indexing?

Three signals: now throttling indexing in data node logs, growing throttle_time_in_millis in GET <index>/_stats/merge, and Lucene merge threads dominating GET _nodes/hot_threads. If it persists, reduce segment creation (higher refresh_interval, larger bulks) or move to faster storage.

Should I increase the write thread pool queue_size?

No. A bigger queue adds heap usage and latency, not throughput, and delays the backpressure signal your producers need. Keep the default, retry 429 with backoff, and fix the segment-churn or capacity problem underneath.

Key Takeaways

1. 429 is a feature. Retry with backoff at the client; never silently drop rejected bulks.
2. Diagnose before tuning: _cat/thread_pool/write, _nodes/stats indexing pressure, index merge stats, hot threads — in that order.
3. refresh_interval is the highest-leverage knob. 1s visibility on a logging index is a cost, not a requirement.
4. Bulk size is discovered, not copied — benchmark upward until throughput plateaus.
5. Replicas off + refresh off for initial loads; restore both afterwards.
6. Force merge belongs to read-only indices only. There is no "defragmentation" in Elasticsearch.

Fighting indexing backpressure on your own cluster? Reach out at searchali.com.