Skip to main content

Writing your own pack

The simplest pack is a plain file, one value per line, addressed by its path (see Overview). From there, a header unlocks weighted lists, external files, and small generators — all without touching engine code, and all safe to share, since a pack is never more than data or a parsed, sandboxed DSL.

The example outputs below are illustrative: exact values depend on the seed and can shift between core versions. What is guaranteed — determinism per seed, and the exact proportions — is called out where it matters.

The header​

Put fields between two --- lines at the top of the file. All are optional:

FieldMeaning
descriptionA human-readable description — "what this is"
addressAn explicit address, overriding the one computed from the path
localeLanguage (en, es, ru…) — and the locale segment a flat path lacks
filePoint at an external data file instead of an inline body
columnWith file: take a named/numbered column from an existing CSV
delimiterWith file CSV, or a weighted body: the separator (default ,)
weightWith file: the frequency column that makes the pack weighted
weightedtrue — the body is value,weight lines
generatortdc — the body is a <gen>, not a list
injectA custom interpolation marker for the generator

Each of these is covered below, with a config and its output.

Weighted packs — frequency from the data​

A plain pack is drawn from uniformly: Smith comes up as often as Zabrowski. Real life isn't like that — over 2.4 million Americans are named Smith. A weighted pack fixes that, and it's exact, laying the frequencies out with the Hamilton (largest-remainder) method — the same guarantee as percent. There are two ways to supply the weights.

Inline body — weighted: true​

Set weighted: true and write each line as value,weight:

---
description: US surnames, weighted (2010 Census)
weighted: true
---
Smith,2442977
Johnson,1932812
Williams,1625252
…

(The real en pack carries the census top 1000; three lines are enough to show the shape.)

You call it like any other pack — nothing changes in the config:

<sequence name="Last">
    <gen type="template" value="person.lastName"/>
</sequence>

Across 100,000 rows, each surname shows up in proportion to its weight — Smith about 2,000 times, matching the 2.02% share its count holds among those thousand names:

./run surnames.tdc (100,000 rows)
Smith      2022
Johnson    1599
Williams   1345

Use this form when the list is short and you want the weights right next to the values.

External CSV — file: + weight:​

When the list is large and already lives in a file of its own, point at it with file: and name the frequency column with weight: (and the value column with column:):

---
description: US surnames, weighted (2010 Census)
file: ../../../sources/us/person/lastName.csv
column: name
weight: count
---

The call is identical; the header change is invisible from the config's side:

<gen type="template" value="person.lastName"/>

Use this form to point at a big census or catalog CSV without copying it into the pack.

The proportions are exact in both engines — the streaming default and mode="memory" alike. A weight is a non-negative integer (a raw count, not a percentage). An empty weight cell (Smith,) is an error, not a silent zero, and a deliberate 0 excludes the value.

Values that contain commas — delimiter:​

If your values are phrases that contain commas of their own (notifications, whole sentences), a comma separator would split them in the wrong place. Set delimiter: to any character, or to one of the aliases tab, semicolon, pipe:

---
weighted: true
delimiter: @
---
Your order, ready for pickup, has shipped@100
New message, marked urgent@50

The line is split on its last delimiter, so commas inside a value survive:

./run notices.tdc (30 rows)
Your order, ready for pickup, has shipped
New message, marked urgent
Your order, ready for pickup, has shipped

The same delimiter: also sets the column separator for an external file: CSV.

Generators in a pack — generator: tdc​

A pack can return a generator instead of a list, so that the address yields a computed value. It's written in TDC's own DSL, so there's nothing new to learn. Set generator: tdc in the header; the body is a <gen>. Here is a US-style license plate — three letters, a dash, four digits:

---
description: US license plate
address: usa.vehicle.plate
generator: tdc
---
<gen type="regex" value="[A-Z]{3}-[0-9]{4}"/>

Call it exactly like a list-backed template:

<gen type="template" value="usa.vehicle.plate"/>
./run plates.tdc
FZY9944
YHZ8189
LRG8608

It runs on the same engine as your config, so every guarantee holds — determinism, and portability to the future Python and Java runtimes — and it's safe even when downloaded, because it's a parsed, limited DSL with no system access. Give generator files a .tdc extension (data files stay .txt) so you can tell them apart at a glance; the file name becomes the last address segment (plate.tdc → …plate).

Assembling from data​

A generator can pull neighboring data lists in by address and build a value out of them. The body is a compound sequence — the individual draws, each with a name you reach through a dot — plus one <data> that says what to return.

Names resolve to English under the default en locale. This example deliberately sets locale: es to show a naming convention English doesn't have: a Spanish full name made of two given names and two surnames.

---
description: Spanish full male name
generator: tdc
locale: es
---
<sequence name="p">
  <distinct>
    <gen name="f1" type="template" value="es.person.male.firstName"/>
    <gen name="f2" type="template" value="es.person.male.firstName"/>
  </distinct>
  <distinct>
    <gen name="l1" type="template" value="es.person.lastName"/>
    <gen name="l2" type="template" value="es.person.lastName"/>
  </distinct>
</sequence>
<data>${{p.f1}} ${{p.f2}} ${{p.l1}} ${{p.l2}}</data>
./run es-fullname.tdc
Antonio Javier García Fernández
Miguel Rodrigo López Romero
Carlos Alejandro Martín Ruiz

The data (firstName, lastName) lives in its own files; the generator only assembles it. The <distinct> tag says that the two draws from a single list must differ within a row — otherwise two independent draws could collide and hand you Juan Juan.

Exact percentages inside a generator — <mix> + percent​

A <mix> with percent works inside a generator, and the split is exact by row count. Say 60% of people get two surnames and 40% get one:

---
description: Spanish surname — 60% double, 40% single
address: es.person.surname
generator: tdc
---
<mix name="s" percent="60,40">
  <case>
    <gen type="template" value="es.person.lastName"/>
    <data> </data>
    <gen type="template" value="es.person.lastName"/>
  </case>
  <case>
    <gen type="template" value="es.person.lastName"/>
  </case>
</mix>
<data>${{s}}</data>
./run es-surname.tdc (100 rows)
García Fernández
López
Martín Romero
Ruiz

Over 100 rows, exactly 60 carry two surnames and 40 carry one — the split is laid out with Hamilton over the whole count rather than left to chance.

Engine note. That share is a quota over the whole column, which no streaming engine can apportion a row at a time, so a config using this pack runs on the in-memory engine and its memory grows with count. A pack without percent= costs nothing. See Which engine runs your config.

Inside a <case>, build the value out of the tags themselves — <gen>, plus <data> for any literal text between them.

A <data> there may also read the row it is on: ${{Name}} inside a case resolves the same way it does in an output line, filters included.

<sequence name="City"><gen type="text" value="Alpha,Beta,Gamma"/></sequence>
<mix name="s" percent="60">
  <case><data>${{City}} North</data></case>
  <case><data>${{City|upper}} South</data></case>
</mix>
<data>${{s}}</data>

That is a different thing from a <gen type="template"> beside it: the generator draws a NEW value, while a reference keeps the record coherent with what the row already holds. A name nobody declared is refused (TDC193) rather than printed literally.

A custom interpolation marker — inject:​

Sometimes a generator's output has to contain a literal ${{ }} — you're generating GitHub Actions workflows, Handlebars, or Go templates. Set your own marker with inject:, using exactly one % to mark where the name goes, and TDC's substitution stops colliding with your text:

---
address: common.ci.deploy_step
generator: tdc
inject: <<%>>
---
<sequence name="s"><gen name="env" type="text" value="prod,staging"/></sequence>
<data>  - run: deploy.sh --token ${{ secrets.TOKEN }} --env <<s.env>></data>

Here <<s.env>> is TDC's substitution, while ${{ secrets.TOKEN }} passes through untouched. The output (shown as a code block, because it literally contains the ${{ }} marker):

  - run: deploy.sh --token ${{ secrets.TOKEN }} --env prod
  - run: deploy.sh --token ${{ secrets.TOKEN }} --env staging

The marker is isolated — it doesn't depend on the main config's inject, so the same generator behaves identically wherever you plug it in. Without inject:, the default stays ${{%}}.

A generator that calls another generator​

A generator can reference another generator, not just a list — a "full name" can draw on a "surname" generator that decides single vs. double on its own. TDC checks at load time that there's no cycle (A → B → A, or a self-reference) and fails with generator reference cycle: … before generation starts, rather than recursing forever.

What's allowed inside a generator

Eight generator types produce a value on their own and are allowed anywhere in a pack body: text, number, regex, advanced_regex, symbol, date, increment and decrement. Inside a <sequence> you may also use template to pull in a data list or another generator by address, along with the <mix> / percent distribution.

Anything else is refused by name — file would resolve a path relative to nothing in particular, and http would put a network call behind an address that looks like a word list:

generator uses <gen type="http"> which is not allowed inside a pack generator

uniq= and order= are refused too, wherever they appear in a pack. Both describe the whole column — which values may repeat across rows, and in what order they come out — and a pack is asked for one value per row, so it has neither the row count nor the other rows to answer with. Declare them on the sequence in the config that draws from the pack. <distinct> is different and stays allowed: it constrains fields against each other within one row, which a pack can decide on its own.

Complex correlations between fields belong in the config, not in a pack generator.

<distinct> — no repeats in one row​

Two independent draws from one list will sometimes collide (James James). Wrap the fields — or whole sequences — that must differ within a row in <distinct>:

<sequence name="pair">
    <distinct>
        <gen name="a" type="template" value="person.male.firstName"/>
        <gen name="b" type="template" value="person.male.firstName"/>
    </distinct>
</sequence>
./run pair.tdc
James and Robert
William and John
Michael and David

James and Robert is fine; James and James never appears. On a collision the engine redraws one of the values, and determinism per seed still holds. It works both inside a <sequence>, wrapping <gen>, and inside <env>, wrapping whole sequences.

Don't confuse it with uniq: uniq keeps a whole row from repeating anywhere in the entire dataset (vertical), while <distinct> keeps fields within one row from matching (horizontal). Both are implemented, and they're independent of each other.

Where custom packs go​

  • The built-in set ships in the repo under data/packs/ and is scanned automatically at startup.
  • Your own folders are added with the CLI flag --data-path <folder> (repeatable) or with the library's dataPaths — see Installing packs.

Errors and ignored files​

  • Two files claiming the same address → TDC170, naming both files. Rename or move one.
  • A file that lands at no address — a header, but no address:, no locale:, and a path whose first segment is no locale, country or common → TDC171, a warning naming the file. It is skipped, so a later value= naming it fails with TDC071.
  • A typo in a path in the config (value="person.lastNam") → TDC071, "unknown template path", raised before generation starts.
  • Hidden files (anything starting with .) and README / LICENSE / CHANGELOG are ignored by the scanner.

Address autocomplete in the editor is driven by those same description: headers, and it ships — see Editor support. Write a description worth reading and it is what the person completing an address will see beside it.

Describing a folder — _pack.json​

A pack folder is all content and no provenance: the lists and generators say what they produce and nothing about where they came from. That is fine while the only packs are the bundled ones, and stops being fine the moment somebody hands you a folder and asks whether the data you built from it may be shipped.

Drop a _pack.json at the top of the folder — beside _locale.json, if there is one:

{
  "name": "Acme internal packs",
  "version": "1.2.0",
  "license": "MIT",
  "author": "Acme Data Team",
  "homepage": "https://acme.example/packs",
  "description": "Product codes and internal identifiers."
}

Every field is optional, and nothing here reaches the generated data. A manifest cannot change a single value: the same seed gives the same bytes whether it is there or not.

Read it back with tdcv2 pack info:

tdcv2 pack info
Packs that describe themselves:

mypacks
  name:        Acme internal packs
  version:     1.2.0
  license:     MIT
  author:      Acme Data Team
  homepage:    https://acme.example/packs
  description: Product codes and internal identifiers.

mypacks/en
  license:     CC-BY-4.0
  description: English lists, from the 2019 open census extract.

Where it is looked for​

Each configured data path, and each folder one level inside it. That covers the two shapes a manifest has: "this whole folder is my pack" and "this locale inside it came from somewhere of its own".

It stops there deliberately. Walking deeper would invite a manifest per .txt file, and the question this answers — who wrote this data, and under what licence — is not one a single list of city names has its own answer to.

What happens when it is wrong​

tdcv2 pack info refuses a manifest that will not parse, or one whose name, version, license, author, homepage or description is not text. Reading that file is the whole of that command's job, and a licence its author wrote that nobody can read is worse than one nobody wrote.

A run never mentions it. The manifest reaches no value, so stopping a generation over it would be blocking work on a field the work never reads.

Keys TDC does not know are kept quietly, so a folder written for a newer version — or carrying your own tooling's fields beside these — still works here.

See also​