Skip to content

feat(plugins/enrichment): CSV lookup enrichment plugin#2365

Open
JocLRojas wants to merge 3 commits into
v11from
backlog/enrichment-plugin-filesystem
Open

feat(plugins/enrichment): CSV lookup enrichment plugin#2365
JocLRojas wants to merge 3 commits into
v11from
backlog/enrichment-plugin-filesystem

Conversation

@JocLRojas

Copy link
Copy Markdown
Contributor

What

New plugin com.utmstack.enrichment that enriches events by looking up values against CSV files dropped in the event-processor filesystem (/workdir/pipeline/csv-datasets/).

It is invoked from filter YAMLs via the dynamic step with these params: dataset, source, match_column, output_column, destination.

Example usage in a filter to enrich events:

- dynamic:
    plugin: com.utmstack.enrichment
    params:
      dataset: filename
      source: log.fieldName
      match_column: column_name_to_match
      output_column: column_name_output_data
      destination: log.desiredFieldName

Why

Currently, enriching events using CSV dictionaries is not supported. This configuration requires an external tool. This plugin integrates this functionality directly into the event processor, eliminating reliance on external services.

How it works

  • A sysadmin (or automated process) drops a name.csv file into /workdir/pipeline/csv-datasets/.
  • The plugin polls the directory every 30s and detects added, modified, or deleted files.
  • On changes, the CSV is loaded into an in-memory registry using an atomic-swap pattern (readers always see a consistent snapshot).
  • The plugin looks up the event's source field value in the CSV's match_column. On a hit, it writes the value from the output_column into the event's destination field.

Guarantees

  • Fail-open: any lookup miss (dataset not loaded, column not present, no row match) returns the event unchanged. The plugin never drops an event.
  • Fail-hard: only on missing required params or invalid destination path — indicates a filter config bug.
  • Guardrails: hard limit of 10000 rows per CSV. Oversized files are skipped with an ERROR log and never enter the registry.

Issue
No related issue.

New Go plugin com.utmstack.enrichment that enriches log events by
performing exact-match lookups against CSV files dropped in the
filesystem at /workdir/pipeline/csv-datasets/.

Invoked from filter YAMLs via the dynamic step. Filesystem is the
source of truth: files are picked up automatically within 30s of
being dropped, modified or deleted (polling watcher, no fsnotify to
stay compatible with shared filesystems in multi-node deploys).

Params for the dynamic step: dataset, source, match_column,
output_column, destination. All required. Match values are
normalised (lowercase + whitespace collapsed to underscore) both
when indexing the CSV and when reading the source event field, so
CSV maintainers and event producers can be inconsistent about case
and spacing.

Guardrails: 10000 rows hard limit per CSV (skip with ERROR), 5000
row warn threshold, UTF-8 required. Invalid or oversized files are
skipped without affecting other datasets.

Fail-open on any lookup miss (dataset not registered, column not
present, no row match) so events keep flowing. Log dedup ensures
warnings fire once per dataset id and errors once per (dataset,
column) tuple to avoid log flood.

Concurrency: registry uses atomic-swap pattern (readers hold an
immutable *Dataset pointer). Per-column indices built lazily on
first lookup and cached under a per-dataset RWMutex.
Copies the com.utmstack.enrichment.plugin binary into
/workdir/plugins/utmstack/ so the event-processor supervisor picks
it up on container start and the parsing plugin exposes its Unix
socket for the dynamic step.
Adds the go build step for plugins/enrichment so the binary
produced during the Build Plugins job is available for the
event-processor Docker image COPY.
@JocLRojas
JocLRojas requested a review from a team July 21, 2026 18:25
@github-actions

Copy link
Copy Markdown

🛑 AI review — Engineer review required

This PR touches critical paths or introduces changes the model cannot judge with sufficient confidence. @Kbayero @osmontero please review.

architecture (gemini-3-flash-lite) — clean

Summary: Addition of a new enrichment plugin; follows existing plugin architecture and patterns.

No findings.

bugs (gemini-3-flash-lite) — clean

Summary: New enrichment plugin implementation looks solid; no critical bugs found.

No findings.

🛑 security (gemini-3-flash-lite) — blocking — must fix before merge

Summary: Introduces a new plugin that performs file-based CSV ingestion and dynamic log enrichment, touching security-critical paths.

  • medium plugins/enrichment/internal/storage/disk.go:147 — Potential path traversal vulnerability in file reading. The ID is derived directly from filenames in the directory. Ensure that the ID is strictly validated to prevent accessing files outside the intended CSV directory.
  • medium plugins/enrichment/internal/parselog/parselog.go:34 — The plugin performs dynamic log enrichment using user-provided parameters. While there is some field validation, ensure that the 'destination' field path cannot be used to overwrite critical internal log metadata fields.

@utmstackprapprover utmstackprapprover Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested — AI review found blocking issues (high/critical, or engineer review required). See above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant