MetadataWorks: Metadata Quality Scores in Elasticsearch

MetadataWorks is a UK company that builds metadata catalogues for health data. Their dataset documents follow the HDR UK descriptive schema (hdruk@2.0.0): summary, documentation, coverage, provenance, accessibility, enrichment & linkage, observation, structural metadata. Rich schema, real-world reality: many datasets arrive with half the fields missing.

The ask was simple to state and easy to get wrong: give every dataset a quality score that reflects metadata completeness, keep it fresh on every update, and make it explorable per publisher and per organisation in Kibana. No external batch job, no application-side scoring code. Here is how we did it with Elasticsearch ingest pipelines alone.

Quick answer: We built a pipeline-of-pipelines: one ingest pipeline per schema section (8 in total) computes sub-level and high-level completeness scores with Painless containsKey checks, and a final pipeline combines them into a weighted overall_score (weights proportional to each section's field count, 43 total). Documents are indexed with ?pipeline=main-pipeline, so the score is recalculated on every index and update operation. Kibana Lens then charts average quality per publisher and per organisation membership.

Challenge

The requirement from the working sessions was explicit: "create an ingest pipeline to create a quality score per document... for each update/indexing operation the quality score will be re-calculated again." The scoring model had three levels:

1. Sub-level score — e.g. summary.publisher, accessibility.usage
2. High-level score — per section: summary, documentation, accessibility, ...
3. Overall score — one number per document

The agreed field list covered 8 sections and their sub-groups — from summary.title down to accessibility.formatAndStandards.formats. Some fields that exist in the data (summary.publicationDate, the internal origin block) were deliberately excluded from scoring.

There was also a first milestone before any scoring: reproduce an existing Tableau visualization in Kibana. That immediately hit a classic wall — a field mapped without a keyword sub-field cannot be used in terms aggregations, so the dashboard could not be built until the mapping was fixed. Mapping hygiene came before analytics, as usual.

A validity score (is the value correct, not just present) was discussed but parked for a later phase. What shipped is a completeness score — and the notes honestly track that distinction.

Diagnosis

Where should a derived score live? Three options: compute in the application, compute at query time, or compute at ingest time. Application-side scoring means every writer must reimplement the logic. Query-time scoring (runtime fields, scripted aggs) burns CPU on every dashboard refresh. Ingest-time scoring pays the cost once per write and makes the score a plain indexed field you can aggregate, sort, and filter like any other.

We prototyped the idea with a tiny generic pipeline first — a Painless script that counts how many of a configured fieldNames list exist in the document and writes existing_fields. Once that pattern was validated, we scaled it to the full HDR UK field list.

The one design decision worth arguing about: a flat average of the 8 section scores would let a one-field section (observation) count as much as the 15-field accessibility section. So the overall score became a weighted average, with weights matching each section's scored field count.

Solution

One ingest pipeline per schema section

Each section got its own pipeline with a single script processor and a remove cleanup step. Summary, shortened:

PUT _ingest/pipeline/summary-score-pipeline
{
  "processors": [
    {
      "script": {
        "lang": "painless",
        "source": """
          Map summaryScore = new HashMap();
          int countSummarySub = 0;
          int countSummaryPublisher = 0;
          if (ctx.containsKey("summary")) {
            if (ctx.summary.containsKey("title")) { countSummarySub++; }
            if (ctx.summary.containsKey("abstract")) { countSummarySub++; }
            // ... contactPoint, keywords, publisher.name, publisher.contactPoint, publisher.memberOf
          }
          summaryScore.summary = ((countSummarySub + countSummaryPublisher) * 100) / params.totalFieldsSummary;
          summaryScore.sub = countSummarySub * 100 / params.totalFieldsSummarySub;
          summaryScore.publisher = countSummaryPublisher * 100 / params.totalFieldsSummaryPublisher;
          ctx.score_summary = summaryScore;
        """,
        "params": { "totalFieldsSummary": 7, "totalFieldsSummarySub": 4, "totalFieldsSummaryPublisher": 3 }
      }
    }
  ]
}

Details that matter:

• Presence checks are nested and null-safe: ctx.containsKey("provenance") && ctx.provenance.containsKey("temporal") && ... — an ingest pipeline must never throw on a sparse document.
• Field totals are params, not literals. When the schema evolves, you change a parameter, not the script body.
• Sub-level scores come free: score_provenance.origin and score_provenance.temporal are computed in the same pass as score_provenance.provenance.
• Temporary counters are removed with a remove processor, so documents stay clean.

A weighted overall score

The final pipeline reads the 8 section scores from ctx (defaulting to 0 when a section pipeline produced nothing) and combines them:

int weightSummary = 7;   int weightDocumentation = 3;
int weightCoverage = 5;  int weightProvenance = 8;
int weightAccessibility = 15; int weightEnrichment = 3;
int weightObservation = 1;    int weightStructuralMetadata = 1;
// totalWeight = 43
ctx.overall_score = (weighted sum) / totalWeight;

Accessibility carries 15/43 of the overall score because it has 15 scored fields — the weighting mirrors the schema instead of someone's opinion.

Chain everything with pipeline processors

The main-pipeline is nothing but 9 pipeline processors, one per sub-pipeline, ending with overall-score-pipeline:

PUT datasets/_doc/1?pipeline=main-pipeline
{ "summary": { "title": "..." }, "coverage": { ... } }

Because scoring happens in the ingest pipeline, every index and update operation recomputes all scores automatically — the example document in the evidence pack sat at _version: 34 with scores intact. Each sub-pipeline stays small, testable with _simulate, and replaceable on its own. See Elastic's ingest pipeline documentation for the pipeline processor mechanics.

Kibana on top

With overall_score and score_* as plain indexed fields, the dashboards were straightforward Lens charts: average quality score per publisher (with dataset count on the second axis), a breakdown by organisation membership (memberOf: ALLIANCE, HUB, NCS, OTHER) filtered to status: FINALIZED, and a free-text search box on top of the dashboard, so typing a publisher name filters everything live.

Results

Only what the evidence pack supports:

Result	Evidence
8 section pipelines + 1 overall, chained by main-pipeline	Shipped pipeline definitions
3 score levels (sub, high, overall) per document	Scored example document
Example dataset scored: overall_score: 88	_source after pipeline run
Section detail on same doc: summary 100, provenance 85 (origin 100 / temporal 80), accessibility 95, enrichment 33, structural metadata 0	Scored example document
Scores recomputed on every write	?pipeline=main-pipeline indexing; doc at _version: 34
Per-publisher average quality dashboard (values ranging 95 down to 16)	Kibana dashboard screenshots
Per-organisation (memberOf) quality breakdown	Kibana Lens screenshot

Two honest footnotes. First, the 33 for enrichment is 1 of 3 fields present under Painless integer division (1 * 100 / 3) — acceptable here, but know your rounding. Second, no latency, volume, or business KPIs appear above because they are not in the source pack, and we do not invent numbers.

Key Takeaways

1. Compute derived scores at ingest, not at query time. Pay once per write; aggregate for free forever.
2. Split one big script into pipelines per domain section. Each is independently testable with _simulate and swappable via the pipeline processor.
3. Weight by field count, not by gut feeling. A 15-field section should not count the same as a 1-field section in the overall average.
4. Pass totals as params. Schema changes become config changes, not script rewrites.
5. Watch integer division in Painless. 1 * 100 / 3 is 33, not 33.3 — decide whether truncation is acceptable before you ship.
6. Fix mappings before dashboards. No keyword sub-field, no terms aggregation — the first milestone stalled on exactly this.

If you want to turn a compliance-style checklist into a live, per-document score your team can chart and filter, this pattern works on any Elasticsearch or OpenSearch cluster. We also keep an eye on cluster health while pipelines run — see searchali.com/en/monitoring.

Need help building ingest pipelines that carry real business logic? → searchali.com