ImportYeti: OpenSearch Health Check at 500M Documents

ImportYeti lets anyone search roughly 70 million U.S. bills of lading—public customs data, obtained via FOIA, turned into a supplier-search product. Under the hood: Amazon OpenSearch Service, about 500 million documents across 4 clusters, and a search page where almost every request fans out into aggregations.

They brought me in for a health check with one clear goal: the best price/performance cluster they can run. The reported bottleneck was aggregations. Instead of guessing, we turned on evidence: slowlogs, application logs, and the actual queries the app sends. Here is what we found—and only the numbers that exist in the evidence pack.

Quick answer: A slowlog threshold of 1 second exposed the real workload: query_string searches with fuzziness plus cardinality and deeply nested terms aggregations over 5.3M+ hit result sets, some taking up to 7 seconds. Application logs added three concrete findings: GC overhead (10.2s collecting in 10.2s on an 8 GB heap), a TooManyBucketsException at the 65,535 default, and query parse failures from a broken escape in the app's query builder. The fixes were unglamorous and effective: grow data nodes vertically toward 64 GB RAM (memory-optimized r7i/r7g families), keep at least 3 nodes, skip dedicated coordinators below ~20 nodes, raise search.max_buckets via AWS support, and fix the query generator app-side.

Challenge

ImportYeti's product is search. A user types "shoes" and gets 567,259 matching shipments, 170,210 companies, and 49,981 suppliers—each tab powered by aggregations over the bill-of-lading index. A filter like "suppliers with more than 2,000 shipments" touches 24 million shipments. This is not a logging cluster where a slow query annoys a dashboard; slow aggregations are the product being slow.

The estate at intake:

• ~500 million documents, 4 clusters, Amazon OpenSearch Service (engine 2.17)
• 6 data nodes on r7g.large—memory-optimized family, but the small end of it
• Ingestion once per month: data is rebuilt in date-suffixed indices behind stable aliases, with refresh_interval: -1 during batch loads
• Goal: best price/performance, not maximum hardware

One thing worth saying early: the index design was already good. dynamic: strict mappings, unused fields disabled, multi-fields only where needed (keyword / snowball / edge_ngram), updateable synonym_graph filters for ports and countries, and eager_global_ordinals on the hot aggregation fields. This was not a "fix the mapping" engagement. The pressure was in the query workload and the hardware envelope.

Diagnosis

Read the slowlog before touching anything

The main BOL index already had search slowlog thresholds at 1000ms for query and fetch phases. That single setting did most of the diagnostic work:

"search": {
  "slowlog": {
    "threshold": {
      "query": { "warn": "1000ms" },
      "fetch": { "warn": "1000ms" }
    }
  }
}

The slowlog told a consistent story. Queries took 1.1s to 7s, and the slow ones shared a shape:

• query_string with fuzziness: AUTO and max_determinized_states: 10000, combined with an arrival_date range
• A cardinality aggregation on company_name.keyword with precision_threshold: 40000, running over result sets reported as total_hits[5332728+ hits]
• Nested terms aggregations: a supplier terms agg of size 500, each bucket carrying six to eight sub-aggregations (top products, weight sums, country, shipment counts)
• A company terms agg of size 403 ordered by a sub-aggregation (weight desc)—ordering by sub-agg forces more work per shard
• Deep pagination in the fetch phase: from: 15000, size: 3000 page walks

The application logs added three findings

GC overhead. JvmGcMonitorService warned that the JVM spent 10.2s collecting in the last 10.2s—the node was doing nothing but garbage collection—with old gen sitting at 1.9gb -> 1.9gb on an 8 GB heap. On r7g.large (16 GB RAM, ~8 GB heap), this workload simply does not fit.

Bucket explosion. One aggregation tried to create 65,536 buckets and hit the default limit:

TooManyBucketsException: Trying to create too many buckets.
Must be less than or equal to: [65535] but was [65536].
This limit can be set by changing the [search.max_buckets] cluster level setting.

On managed OpenSearch Service you don't flip this setting yourself—the recorded workaround was an AWS support ticket to raise search.max_buckets to 100k, per the AWS OpenSearch Service error-handling docs.

Broken queries from the app. QueryShardException traces showed Lucene ParseException: Lexical error ... Encountered: <EOF>—a trailing escaped quote in a generated query_string. That is not a cluster problem; it is a query-builder bug, and the action item went to the application side.

Solution

Grow vertically, keep the node count honest

The first-meeting recommendation held after the log review: with 6 data nodes, dedicated coordinator nodes are premature—guidance was to skip them until around 20 nodes and put the money into data nodes instead. The sizing direction: grow vertically until data nodes reach 64 GB RAM, and never drop below 3 nodes (split-brain protection).

We compared memory-optimized candidates with real list prices ($0.711/hr on-demand for r7g.2xlarge.search at 8 vCPU / 64 GiB, with 1-yr and 3-yr reserved tiers below that) and used the OpenSearch Service dry run analysis to preview configuration changes—3 data nodes on r7i.xlarge.search / r7i.2xlarge.search with EBS GP3 storage scaling from 450 GiB toward 900 GiB—before applying anything to a production domain.

Raise the limits that the workload legitimately needs

The bucket limit was not an abuse case: the product genuinely aggregates wide. So the fix was to raise search.max_buckets through AWS support rather than contorting the queries. Where the aggregation cost was accidental—sub-agg ordering, oversized terms, deep from pagination—those went on the application backlog, because OpenSearch's own search performance guidance points the same direction: page with search_after, keep bucket counts intentional.

Keep what already works

Monthly batch ingestion with refresh_interval: -1, alias-swapped date-suffixed indices, strict mappings, and eager global ordinals on aggregation fields all stayed. A health check is also about knowing what not to touch. This is the same philosophy behind continuous cluster visibility—the reason we built searchali monitoring: the slowlog and GC signals were sitting there; someone just had to read them.

Results

Evidence-only. This engagement's deliverable was diagnosis and a sizing/settings direction—there are no before/after latency percentages in the source pack, and I won't invent any.

Finding / action	Evidence
Slow queries identified: 1.1s–7s, aggregation-heavy	Search slowlog exports, 1000ms WARN threshold
GC overhead confirmed on 8 GB heap nodes	[gc] overhead, spent [10.2s] collecting in the last [10.2s]
Bucket limit hit at default 65,535	TooManyBucketsException; AWS ticket path to 100k
App-side query bug isolated	Lucene ParseException EOF on escaped quote
Sizing direction set: 64 GB RAM data nodes, ≥3 nodes, no dedicated coordinators <20 nodes	Meeting notes + investigation notes
Config changes previewed safely	OpenSearch Service dry run analysis (r7i family, GP3 450→900 GiB)

Key Takeaways

1. Turn on the search slowlog before you buy hardware. A 1-second WARN threshold turned "aggregations are slow" into five named query shapes.
2. GC spending 10.2s out of 10.2s means the heap is too small, full stop. On memory-optimized instances, grow vertically toward 64 GB RAM before adding node count.
3. Don't add dedicated coordinator nodes to a 6-node cluster. Below ~20 nodes, that budget does more in the data tier.
4. search.max_buckets is a cluster setting—and on managed OpenSearch, a support ticket. Raise it when the workload legitimately aggregates wide; fix the query when it doesn't.
5. Some "cluster problems" are application bugs. A single broken escape character in a query builder produced a stream of parse exceptions in the server logs.
6. Use dry run analysis on managed domains. Preview the blue/green impact of instance and storage changes before committing a production search product to them.

Running an aggregation-heavy OpenSearch product and unsure where the time goes? → searchali.com