Plugin System

Sluice’s pipeline configs are intentionally constrained — the schema is fixed, the field types are enumerated, the DQ check types are an explicit list. That keeps configs readable and reviewable. But every real migration eventually needs something the built-ins don’t cover: a regex pattern that only makes sense for one client’s data, a date format only one ERP uses, a merge strategy with custom precedence rules.

Plugins fill that gap without forcing you to fork the engine. Sluice exposes a three-tier extension model that scales from “I just want a reusable composite rule for this client” to “I want to publish a paid adapter package on npm.”

The canonical author guide for plugins is PLUGINS.md in the repo. This page mirrors it for the docs site; pages may diverge in formatting but never in substance.

The three tiers at a glance

Tier	What it is	Where it lives	Who writes it	Distribution
Tier 1 — Composite YAML rules	A named bundle of built-in DQ checks	YAML files in your project	Anyone — no code	In your repo
Tier 2 — File-based plugins	TypeScript module exporting a RulePlugin, TransformPlugin, or MergeStrategyPlugin	plugins/*.{rule,transform,merge}.ts in your project	Anyone with TypeScript	In your repo
Tier 3 — npm package plugins	A published npm package with a register() function	Anywhere installable via npm install	Plugin authors	npmjs.com (public or private)

You can mix tiers freely — a single pipeline can pull composite rules (Tier 1), a local dev’s file plugin (Tier 2), and an installed npm rule pack (Tier 3) all at once.

Tier 1 — Composite YAML rules

A composite rule is a named bundle of built-in DQ checks that you reference by a single id. Useful when the same combination of checks (e.g. notNull + pattern + maxLength for an internal style number) repeats across fields and pipelines.

Library file

shared/rules.yaml

version: "1.0"

rules:
  - id: ukVatNumber
    description: UK VAT registration number — format only.
    checks:
      - { type: pattern, value: "^GB([0-9]{9}|[0-9]{12}|(GD|HA)[0-9]{3})$", severity: warning }

  - id: positivePrice
    description: Price must be a non-negative number with sensible upper bound.
    checks:
      - { type: notNull, severity: critical }
      - { type: min,     value: 0,        severity: critical }
      - { type: max,     value: 99999.99, severity: warning  }

Use in a pipeline

dq:
  rulesFile: ./shared/rules.yaml   # tells ConfigLoader where to find composite rules
  rules:
    - field: VAT_NUMBER
      checks:
        - { type: ukVatNumber }    # expands to the pattern check above
    - field: COST_PRICE
      checks:
        - { type: positivePrice }  # expands to all three checks

ConfigLoader expands composite-rule references into their underlying built-in checks before Zod validation runs, so the DQ engine only ever sees standard check types.

When to reach for Tier 1

• The same combination of checks repeats across multiple fields or pipelines.
• The combination doesn’t need any custom code — built-in checks suffice.
• You want non-developers (data analysts, project managers) to be able to read and edit the rules.

Constraints

• Composite rule ids must be valid identifiers (^[a-zA-Z][a-zA-Z0-9_-]*$) and must not collide with built-in check type names (notNull, unique, pattern, etc.).
• Composite rules can only contain built-in checks — they cannot reference other composite rules (no nesting).

Tier 2 — File-based plugins

When you need actual logic, write a TypeScript plugin file. Plugins are auto-discovered from a plugins/ directory next to your pipeline YAML (or from any directory passed via --plugins).

File naming

Suffix	Plugin type	Export
*.rule.ts	DQ rule	export const rule: RulePlugin
*.transform.ts	Field transform	export const transform: TransformPlugin
*.merge.ts	Merge strategy	export const mergeStrategy: MergeStrategyPlugin

Example — DQ rule plugin

plugins/ifs-customer-no.rule.ts

import type { RulePlugin } from '@caracal-lynx/sluice';

export const rule: RulePlugin = {
  id: 'ifsCustomerNo',
  description: 'IFS customer number — three uppercase letters followed by 4–7 digits',

  validate(value, config, rowIndex, field) {
    if (typeof value !== 'string') return null;
    if (/^[A-Z]{3}[0-9]{4,7}$/.test(value)) return null;
    return {
      field,
      rowIndex,
      value,
      rule: 'ifsCustomerNo',
      severity: config.severity,
      message: config.message ?? `${value} is not a valid IFS customer number`,
    };
  },
};

Use it in a pipeline:

dq:
  rules:
    - field: CUSTOMER_NO
      checks:
        - { type: ifsCustomerNo, severity: critical }

Example — Transform plugin

plugins/normalise-uk-phone.transform.ts

import type { TransformPlugin } from '@caracal-lynx/sluice';

export const transform: TransformPlugin = {
  id: 'normaliseUkPhone',
  description: 'Strip spaces, hyphens, and country prefix from UK phone numbers',

  apply(input) {
    if (input == null) return null;
    const digits = String(input).replace(/[\s\-()]+/g, '');
    if (digits.startsWith('+44')) return '0' + digits.slice(3);
    return digits;
  },
};

Use it as a type: custom field:

transform:
  fields:
    - from: PHONE
      to: TelephoneNumber
      type: custom
      customOp: normaliseUkPhone

Example — Merge strategy plugin

plugins/most-recent.merge.ts

import type { MergeStrategyPlugin } from '@caracal-lynx/sluice';

export const mergeStrategy: MergeStrategyPlugin = {
  id: 'mostRecent',
  description: 'Pick the value from the source whose updatedAt is latest.',

  async merge(store, sources, config) {
    // Implementation uses store.query() to compute a SQL JOIN that picks
    // the row with MAX(updatedAt) per merge key. See the built-in
    // src/merge/strategies/coalesce.ts for the canonical pattern.
    // ...
    return { rowsMerged: 0, conflicts: 0, unmatched: 0, tableName: 'stg_merged' };
  },
};

Use it in a pipeline:

merge:
  key: STYLE_NO
  strategy: mostRecent

Tier 3 — npm packages

For plugins you want to share across multiple client engagements — or sell as a paid package — publish them as an npm package and register them via sluice.config.yaml:

sluice.config.yaml

plugins:
  - '@caracal-lynx/etl-rules-uk'
  - '@caracal-lynx/sluice-adapter-ifs'
  - '@example/sluice-rules-fashion'

Each package exports a register() function that’s called once at the start of every pipeline run:

// In your published package
import type { PluginPackage } from '@caracal-lynx/sluice';

export const sluicePlugin: PluginPackage = {
  name: '@example/sluice-rules-fashion',
  version: '1.0.0',
  register(registry) {
    registry.rules.add(myCustomRule);
    registry.transforms.add(myCustomTransform);
    registry.mergeStrategies.add(myCustomMergeStrategy);
  },
};

Resolution order

When two plugins register the same id, the later registration wins. The order is:

1. Built-in rules / transforms / merge strategies.
2. Composite rules (Tier 1).
3. File plugins (Tier 2).
4. npm package plugins (Tier 3).

This means an npm package can override a built-in if you really want it to — but the convention is to give plugins distinctive ids and avoid the collision in the first place.

Discovering and inspecting plugins

sluice plugins              # list all loaded rule, transform, and merge plugins
sluice merge list-strategies # list all merge strategies
sluice merge info coalesce  # detail on a specific merge strategy

sluice plugins shows the source of every plugin (built-in, composite, file, or npm package), which is invaluable when debugging “why is my custom rule not being picked up”.

See also

• Extension Points — the registry/discovery internals
• PLUGINS.md — the full author guide in the repo
• Commercial Support — Caracal Lynx’s published Tier 3 plugin packages