Changelog

All notable changes to this project are documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

map source: {"fn": "map", "from": S, "values": {...}, "else": V} looks the text of a path or $ variable up in values and yields that entry’s JSON value, any type; a miss yields else or, without one, is missing so a fallback list continues. lower source: {"fn": "lower", "from": S} lowercases the text. Both work wherever a source does: columns, derive, expose, conditions.
as: "number" and as: "bool" on a column: the first source whose text parses as a JSON number or boolean is written as that number or boolean.

max_pooled_input: the largest input whose per-call state is kept for reuse; the default is 4 MiB, as before. A kept state that has seen one large document is now dropped after 64 consecutive inputs below an eighth of that size, so one burst of large documents no longer pins memory for good.
Tests and benchmarks behind the concurrency and memory claims: TestSharedTransformerMixed (16 goroutines mixing Append, Each and Options on shared Transformers, under the race detector), TestRetainedRecordIsInvalid, BenchmarkEachParallel, BenchmarkVarsShared/BenchmarkVarsPerCall, and the report tests TestColdPoolReport, TestBurstReport, TestMemoryReport, TestKeySetReport. Example_lazyCompile shows sync.OnceValues.
make profile: CPU, memory, mutex and block profiles of the hot benchmarks, top 20 of each.

Key lookups skip the hash when no entry has the key’s length. Append is 16 % faster and the RudderStack preset 14 % faster on the benchmarks, still at 0 allocs/op.

Docs: zero allocations is reached on the third call of a fresh pooled state, not the first. The performance page now carries measured numbers for scaling across CPUs, the cost of an empty pool, and the memory one state holds, including the 8× ratio to the largest input.

jsonflat package: Compile, MustCompile, New, and a Transformer with Append, Transform and Each.
Config sections input (explode, inherit), flatten (separator, arrays, max_depth, drop_nulls, drop_empty_objects), keys (snake_case normalisation, digit prefix, collision policy, segment aliases, aliases, keep, drop), derive, columns, sections, rules (rename, drop, merge, default), outputs, expose and newline.
keys.keep: an output-key whitelist next to keys.drop. A flattened key is written only if it matches keep (when set) and does not match drop.
keys.rest: a column that collects the keys keep left out, as a JSON string of one flat object, so a whitelist loses nothing.
Sources with fallbacks: paths, $now, derived values, Options.Vars, and the clock_skew function.
Zero heap allocations in Append and Each in the steady state, guarded by tests.
Output that is always valid JSON: an RFC 8259 string escaper and number check in internal/jsonenc, because the parser’s own serialiser and number parsing are more lenient than the standard.
presets package with the RudderStack warehouse-schema preset.
example/: a command that flattens an NDJSON event export (gzip or plain) into one flat row per line, merging duplicate keys and duplicate fields through its config, with an optional independent check of every row.
Fuzz targets FuzzAppend and FuzzEach, benchmarks, and examples.