Autocomplete on Every Field with Elasticsearch edge_ngram

A SOC analyst does not paste a full UUID into a search box. They type the first eight characters of a MITRE technique ID, a hostname fragment, half a username — and expect the case to show up. That was the exact requirement a cybersecurity analytics company brought to me: partial-match search across their case documents, including fields buried three levels deep in nested attack patterns and stages.

The catch: their case documents are wide and deep. A single case carries users, endpoints, techniques, an array of patterns, each pattern holding twenty-plus stages, each stage with its own MITRE ID. Listing every field in a query — or hand-mapping every field for autocomplete — does not scale. Here is the pattern we proved instead, straight from the working PoC.

Quick answer: Define one custom edge_ngram analyzer, attach it to every string field via a dynamic_templates entry with match_mapping_type: "string", and query with multi_match on "fields": ["*"]. Every text field at any nesting depth becomes prefix-searchable with zero per-field mapping work. Use _source includes to slim the response. Watch the trade-offs: index size grows, and set a proper search_analyzer before production.

The requirement: partial search on a nested SOC case

The field list came from the client's own note: uid, users, endpoints, assignedTo, updatedAt, triggersNumber — plus patterns, stages, and steps pending an internal decision. The document shape is a SOC classic: a case with techniques like lateral_movement and privilege_escalation, patterns describing an attack narrative, and stages carrying STIX-style IDs such as attack-pattern--dcaa092b-....

Two things make this hard with default mappings. First, default text analysis tokenizes on word boundaries — typing dcaa matches nothing because no term starts a token boundary match for partial input without ngrams. Second, the interesting fields keep moving: when the schema is still being negotiated ("I need to check with my manager about stages"), per-field mappings are a maintenance trap.

The index: one analyzer, one dynamic template

This is the entire index definition from the PoC, generalized only in name:

PUT soc-cases
{
  "mappings": {
    "dynamic_templates": [
      {
        "all_strings_autocomplete": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "text",
            "analyzer": "auto_complete"
          }
        }
      }
    ]
  },
  "settings": {
    "analysis": {
      "analyzer": {
        "auto_complete": {
          "tokenizer": "ngram_tokenizer",
          "filter": [ "lowercase" ]
        }
      },
      "tokenizer": {
        "ngram_tokenizer": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 10,
          "token_chars": [ "letter", "digit" ]
        }
      }
    }
  }
}

Three deliberate choices in here:

token_chars does the heavy lifting on IDs

With token_chars: ["letter", "digit"], the tokenizer splits on anything else — hyphens included. A value like attack-pattern--dcaa092b-7de9-... breaks into attack, pattern, dcaa092b, 7de9, and so on. Each segment then gets edge ngrams. That is exactly why an analyst can find a case from the middle of a UUID: the fragment is the start of its own token.

min_gram: 2, max_gram: 10

Single-character lookups are noise, so min_gram: 2. And max_gram: 10 caps term explosion — long tokens are only indexed up to their first ten characters. If your users paste longer fragments, either raise max_gram or rely on a search-time analyzer that doesn't ngram the query (more on that below). Elastic's edge_ngram tokenizer docs cover the parameters.

The dynamic template catches every string, at every depth

match_mapping_type: "string" means every string field Elasticsearch discovers — top-level uid, nested patterns.stages.mitreId, anything added next sprint — is mapped as text with the auto_complete analyzer. No mapping updates when the schema debate resolves. Dates and numbers keep their detected types, so updatedAt stays a real date and range queries still work.

The query: multi_match on * plus _source surgery

The verification query searched an 8-character UUID fragment across everything, and trimmed the response to only what the UI needed:

GET soc-cases/_search
{
  "query": {
    "multi_match": {
      "query": "dcaa092b",
      "fields": [ "*" ]
    }
  },
  "_source": [
    "uid", "users", "endpoints", "updatedAt",
    "techniques", "patterns",
    "steps.triggersNumber", "steps.mitreId"
  ]
}

The _source include list is worth copying: steps.triggersNumber and steps.mitreId return only those two fields from every object in the steps array, instead of twenty full objects. On chatty SOC documents that is real payload savings for a search-as-you-type UI.

What the test proved

Evidence from the PoC response, nothing more:

• The fragment dcaa092b — the first 8 characters of one nested stage's MITRE UUID — returned the case: hits.total.value: 1, max_score: 2.079955.
• took: 1 ms — on a single-document test index, so it proves correctness, not performance. I won't dress that up as a benchmark.
• The steps objects in the response contained exactly triggersNumber and mitreId, confirming _source filtering works on nested arrays.

No index-size or latency-at-scale numbers exist in this PoC — those were not in the source pack, and inventing them would be worse than useless.

Trade-offs before you ship this

Set a search_analyzer. The PoC used the ngram analyzer at both index and search time, which works for short inputs but can produce surprising matches: the query itself gets chopped into ngrams, so longer queries match more loosely. In production, index with auto_complete and search with a plain lowercase-style analyzer so the query is matched as typed.

edge_ngram inflates the index. Every token up to 10 characters becomes up to 9 terms. Applying that to every string field is a deliberate storage-for-latency trade. If only a handful of fields need autocomplete, scope the dynamic template with path_match instead of catching everything.

fields: ["*"] is convenient, not free. Query-time field expansion across hundreds of fields costs CPU. It was right for this PoC; for a hot search box, list the field groups explicitly or use copy_to into a single search field. If cluster-level impact is a concern, watch it — this is exactly the query-load pattern we track in Elasticsearch monitoring engagements.

Key Takeaways

1. One dynamic template beats fifty mapping updates. match_mapping_type: "string" gives every current and future text field the autocomplete analyzer automatically.
2. token_chars is your ID-search feature. Letter/digit tokenization splits UUIDs and hyphenated IDs into independently searchable segments.
3. Cap your grams. min_gram: 2 kills single-char noise; max_gram: 10 caps term explosion — know both limits before users report "long queries don't match."
4. Trim nested arrays with _source includes. steps.triggersNumber returns two fields per object, not the whole object.
5. Split index and search analyzers before production. Same-analyzer ngram search is the classic edge_ngram footgun.
6. Only claim what you measured. A 1 ms took on one document is a correctness check, not a benchmark.

Frequently Asked Questions

Why edge_ngram instead of a wildcard query like *dcaa092b*?

Leading-wildcard queries scan term dictionaries at search time and get slower as the index grows. edge_ngram pays the cost once at index time, turning prefix lookups into exact term matches — the right trade for interactive search boxes.

Why not the search_as_you_type field type?

search_as_you_type is a solid built-in, but it shines for phrase-prefix completion on chosen fields. Here the requirement was fragment matching across all fields — including inside hyphenated UUIDs — with no per-field mapping. A custom edge_ngram analyzer plus a dynamic template fits that shape better and keeps token_chars under your control.

Does the dynamic template break dates and numbers?

No. It only intercepts match_mapping_type: "string". Detected dates (like updatedAt) and numerics keep their normal types, so sorting and range filters keep working.

How do I search fragments longer than max_gram?

Either raise max_gram (and accept the index growth) or configure a non-ngram search_analyzer: then a long query is matched as one term against the indexed grams up to the cap, and you can combine it with a prefix query for exact long-form lookups.

Need partial-match search designed — or an Elasticsearch cluster that can afford it? → searchali.com