Keeping Elasticsearch Fresh: Sync, Deletes, and Refresh

I sat down with the founding team of an AI startup building natural-language search on top of CRM data. Python jobs pull records from the CRM API, web crawling is triggered manually, and the product turns a sentence like "find me an Elasticsearch engineer, ignore anything older than a year" into a query. Their words for the main problem: "refreshing the index is a problem."

That single sentence hides two completely different engineering problems, and mixing them up is the most common freshness mistake I see. This guide untangles them, then covers the two other things from that conversation that bite every crawl-plus-CRM stack: deleted records and index-per-tenant design.

Quick answer: "Refreshing the index" means two things. (1) Elasticsearch's own refresh—making just-written documents visible to search, controlled by refresh_interval, near-real-time by design. (2) Data freshness—getting new, changed, and deleted source records into the index at all. The first is a one-line setting. The second is your pipeline's job: schedule the sync, propagate deletes with a last_synced sweep or rebuild behind an alias, and put a recency filter in the query so stale documents stop ranking.

Two meanings of "refreshing the index"

Elasticsearch refresh: segment visibility

Elasticsearch is near-real-time. A document you index is durable immediately, but it becomes searchable only after the next refresh—by default every second on actively-searched indices. That is the refresh_interval:

PUT /candidates/_settings
{
  "index": { "refresh_interval": "30s" }
}

For a CRM/candidate-search workload fed by batch sync jobs, nobody needs 1-second visibility. Raising refresh_interval to 30s or 60s reduces segment churn and indexing overhead for free. If a single write must be visible right away (a user just edited a record and reloads the page), use ?refresh=wait_for on that one request instead of tightening the whole index. Details in Elastic's near-real-time search docs.

If this is your "refresh problem," it is solved in an afternoon. Usually it is not.

Data freshness: your pipeline's job

The real problem in that meeting was the second meaning: the index drifts away from the source. Crawling was triggered manually—which means freshness depends on someone remembering to press a button. The fix is boring and non-negotiable: put the sync on a schedule (cron, Airflow, whatever you already run), make it incremental using the CRM's modified-since cursor, and record a last_synced timestamp on every document you touch. That timestamp is the backbone of everything below.

Handling deleted records

A record deleted in the CRM does not announce itself. Your incremental sync only sees records that exist, so deletions silently accumulate as stale documents—exactly the "deleted records" item on the startup's problem list. Two working patterns:

Sweep by sync timestamp

If every full pass stamps last_synced on live documents, anything the pass didn't touch is dead. Sweep it:

POST /candidates/_delete_by_query
{
  "query": {
    "range": { "last_synced": { "lt": "now-2d" } }
  }
}

This works when you can afford a periodic full pass. Run it after the pass completes, with a margin wider than one sync cycle so a slow job doesn't delete live data.

Rebuild behind an alias

When the dataset is small-to-medium (one tenant's CRM usually is), the cleanest answer is: reindex everything into a fresh index, then swap an alias atomically. Deletes disappear by construction—they were never copied.

POST /_aliases
{
  "actions": [
    { "remove": { "index": "candidates_v41", "alias": "candidates" } },
    { "add":    { "index": "candidates_v42", "alias": "candidates" } }
  ]
}

The application only ever queries candidates. No downtime, no tombstone bookkeeping, and mapping changes ride along for free. This is my default recommendation for crawl-fed indices, because crawlers are even worse than CRMs at reporting deletions.

Index per tenant, one tenant

The team had designed index-per-tenant—and had exactly one tenant. That is the right instinct applied too early. Index-per-tenant gives you clean deletion (drop the index when a tenant leaves), per-tenant mappings, and easy alias-swap rebuilds. But every index costs shards, and shards cost heap and cluster-state size; at hundreds of small tenants you get the classic oversharding problem.

The pragmatic path: keep index-per-tenant while tenants are few and datasets differ, use one primary shard per index until a tenant's data proves it needs more, and hide every physical index behind a per-tenant alias from day one—so you can later merge small tenants into a shared index with a tenant_id filter without touching application code.

From a sentence to a freshness-aware query

The product scenario from the notes: "find me an Elasticsearch engineer — ignore older than 1 year." Whatever LLM or parser produces the query, the recency rule belongs in a filter clause, not in the text matching:

GET /candidates/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "profile": "elasticsearch engineer" } }
      ],
      "filter": [
        { "range": { "updated_at": { "gte": "now-1y/d" } } }
      ]
    }
  }
}

Filters are cacheable and don't distort scoring. And notice the dependency: this query is only as honest as updated_at—which comes straight back to the sync pipeline. A freshness filter on stale data just filters confidently on lies. If you want recency to rank rather than exclude, add a gauss decay function on updated_at instead of the hard cutoff.

One thing I will not do here is quote performance numbers: this engagement pack contains a discovery conversation, not benchmarks, so there are no metrics to report—and I don't invent them.

Key Takeaways

• Split "refresh" into two tickets: refresh_interval (Elasticsearch, one setting) and pipeline freshness (your job, the real work).
• Batch-fed search indices rarely need 1s refresh—set 30s+ and use ?refresh=wait_for for the rare read-your-own-write case.
• Never trigger crawls manually. Schedule incremental syncs and stamp last_synced on every document.
• Deletes don't sync themselves: sweep with _delete_by_query on last_synced, or rebuild and alias-swap.
• Put aliases in front of every physical index from day one—rebuilds and tenancy changes become invisible to the app.
• Recency belongs in a filter clause (now-1y/d): cacheable, score-neutral, and only as truthful as your sync timestamps.

If your search index is drifting away from its source and nobody can say how stale it is, monitoring the cluster and the pipeline together is where I start.

Frequently Asked Questions

Is a lower refresh_interval the fix for a stale index?

No. refresh_interval only controls when already-indexed documents become searchable—seconds, not days. If your index is missing or misrepresenting source records, the problem is the sync pipeline, and no refresh setting will fix it.

How do I remove documents whose source records were deleted?

Either detect absence—stamp last_synced during full passes and _delete_by_query anything untouched—or sidestep the problem by reindexing into a new index and swapping an alias. For crawl-fed data, prefer the alias swap.

Is index-per-tenant a good multi-tenancy model?

With few tenants, yes: isolation, per-tenant mappings, trivial offboarding. It stops scaling when tenant count drives shard count into the thousands. Aliases per tenant keep both doors open.

How do I exclude results older than one year?

A range filter on your freshness field inside bool.filter: { "range": { "updated_at": { "gte": "now-1y/d" } } }. Use a decay function instead if old results should rank lower rather than vanish.