Expressions
The little language inside if= — and inside filter= on a pool,
which reads the same way. It decides whether a <gen>, a <line>, a <case> or a <data>
takes part in a row.
<sequence name="Zone">
<gen if="Country in [US, CA, MX]" type="text" value="NAFTA"/>
<gen if="Country in [FR, DE]" type="text" value="EU"/>
<gen type="text" value="ROW"/>
</sequence>
<sequence name="Handling">
<gen if="Weight > 20" type="text" value="freight"/>
<gen if="_count % 2 == 0" type="text" value="courier-even"/>
<gen type="text" value="parcel"/>
</sequence>
US NAFTA 2kg parcel FR EU 14kg courier-even CA NAFTA 7kg parcel DE EU 30kg freight MX NAFTA 5kg parcel JP ROW 22kg freight
The last <gen> in each sequence has no if=, so it catches whatever the conditions above
did not — the same shape as an else.
Values
| You write | It means |
|---|---|
Country | the value that sequence produced on this row |
Person.Email | a field of a compound sequence |
Gender.Male | "is Gender currently Male?" — the same reading as parent="Gender.Male" |
Male | a bare word: a name that is no sequence is its own text |
42, 1.5 | a number |
'text' | a quoted string, when the text has spaces or looks like a name |
_count, _last | a built-in — the row number, the last-row flag |
A bare word is what lets Gender == Male be written without quotes. It also means a typo
compares against itself and quietly matches nothing — which is why an unknown name on the
right of a dot raises TDC193 rather than passing.
Whole numbers
A double holds every integer up to 2⁵³ and then starts skipping, so an expression built on doubles alone answers this way:
9007199254740993 == 9007199254740992 true 9007199254740993 - 9007199254740992 0
Both wrong, and wrong without a word — which for a data generator is the worst kind of wrong, because the run finishes and the file looks fine. So an operand that IS a whole number is carried as one, and only becomes a double when something asks it to:
| a literal without a point or an exponent | stays whole |
| a column whose value reads as digits | compares as whole against another whole |
+ - * % on two whole numbers | stay whole |
/ | always floating point — division is not closed over the whole numbers |
anything handed to sqrt, log, sin… | becomes a double, because those have no exact answer to give |
The domain is signed 64-bit, the same as the compute layer — the widest integer all five implementations hold natively. Past it the answer is a refusal, in the same words compute uses, rather than a quiet slide back into floating point:
tdcv2: integer overflow: 10000000000000000000 is outside the signed 64-bit range
One edge worth knowing: −2⁶³ can be reached by arithmetic but not written as a literal, since
-9223372036854775808 is a minus sign applied to a magnitude one past the largest positive.
Operators
| Group | Operators |
|---|---|
| comparison | == != === !== < > <= >= |
| logic | && || ! |
| arithmetic | + - * / % |
| membership | in |
| choice | a ? b : c |
% is Euclidean, and that is not what your language does-3 % 2 is 1 here. JavaScript, Java, C# and Rust all answer −1; Python answers 1.
The reason is not taste. The compute layer already had <mod> and already
answered 1, so a % that borrowed the host convention would make one engine give two
different answers to the same question depending on which layer you reached for.
in takes a list on its right and nothing else — a list anywhere else raises
TDC259. Comparison inside it is as loose as ==, so a text column against a
list of numeric words still matches.
<gen if="Country in [US, CA, MX]" .../> <!-- instead of three == joined by || -->
Functions
| Function | Takes | Gives |
|---|---|---|
abs(x) | 1 | magnitude |
ceil(x) floor(x) | 1 | up / down to a whole number |
trunc(x) | 1 | toward zero — trunc(-7.5) is −7, floor is −8 |
round(x) | 1 | nearest, a half away from zero |
min(…) max(…) | 1 or more | smallest / largest |
len(s) | 1 | how many characters |
is_empty(s) | 1 | whether the text is empty |
starts_with(s, p) | 2 | prefix test |
ends_with(s, p) | 2 | suffix test |
contains(s, p) | 2 | substring test |
lower(s) upper(s) | 1 | case |
split(s, sep) | 2 | text cut into a list — see Lists inside one row |
join(list, sep) | 2 | a list back into text |
count(list) | 1 | how many elements |
at(list, i) | 2 | the i-th element, counting from zero |
sum(list) | 1 | the total — stays whole while every element is |
mean(list) median(list) | 1 | average, and middle value |
stddev(list) | 1 | population standard deviation, divided by n |
sqrt(x) | 1 | square root |
pow(x, y) | 2 | x raised to y |
exp(x) | 1 | e raised to x |
log(x) log10(x) | 1 | natural / base-10 logarithm |
sin(x) cos(x) tan(x) | 1 | circular functions, in radians |
asin(x) acos(x) atan(x) | 1 | their inverses, giving radians |
atan2(y, x) | 2 | the angle of the point (x, y), over (−π, π] |
sinh(x) cosh(x) tanh(x) | 1 | hyperbolic functions |
cbrt(x) | 1 | cube root — works on negatives, unlike pow |
expm1(x) log1p(x) | 1 | eˣ−1 and log(1+x), kept accurate near zero |
log2(x) | 1 | base-2 logarithm — exact on a power of two |
asinh(x) acosh(x) atanh(x) | 1 | inverse hyperbolic functions |
hypot(x, y) | 2 | vector length, without overflowing on the way |
sign(x) | 1 | −1, 0 or 1 |
erf(x) erfc(x) | 1 | the error function and its complement |
gamma(x) lgamma(x) | 1 | Γ(x), and log |Γ(x)| for when Γ overflows |
beta(a, b) | 2 | Γ(a)Γ(b)/Γ(a+b) |
digamma(x) | 1 | ψ(x), the derivative of log Γ |
zeta(s) | 1 | the Riemann zeta function, for real s |
degrees(x) radians(x) | 1 | between the two ways of writing an angle |
Everything above the rule is exact — built from comparisons and from the arithmetic IEEE-754 pins down, so the five implementations cannot disagree. Everything below it, TDC computes itself.
round sends a half away from zero. round(0.5) is 1 and round(-0.5) is −1.
JavaScript rounds a half toward +∞, Python rounds to even, Java rounds half up: three hosts,
three answers, none symmetric. TDC states its own so a column of negatives behaves like a
column of positives.
len counts code points. len("😀") is 1, not the 2 that UTF-16 would give — but a
family emoji built from several code points counts as several. Grapheme clusters would be
the human answer and need a Unicode segmentation table that not every implementation can
carry, so the portable unit wins. len("10") is 2: a string function reads its argument as
text, never as a number.
Lists inside one row
A sequence with repeat= puts several values in one field, joined by its separator=.
An expression sees the joined text, because that is what the field holds — so split
is how a list becomes a list, and everything else works on what it hands back.
<sequence name="Prices">
<gen type="number" value="10..200" repeat="3" separator=","/>
</sequence>
<sequence name="Basket">
<gen if="sum(split(Prices, ',')) > 300" type="text" value="large"/>
<gen type="text" value="ordinary"/>
</sequence>
min and max read a list as readily as loose arguments: max(split(Prices, ',')) and
max(1, 9, 4) both work. An empty separator cuts into single characters, the same unit
len counts, so count(split(s, '')) and len(s) never disagree.
at counts from zero, and refuses an index that is not oneat(list, 0) is the first element. Past the end is empty text — deliberately, because
repeat="1..4" makes rows of different lengths on purpose, and asking for the third element
of a two-element row is a real question with an empty answer. Use count(list) to ask
first.
Everything else is refused rather than answered with that same empty string: a negative index, a fractional one, an index that is not a number, and a subject that was never split. That last one is the mistake everybody makes first —
<gen if="at(Prices, 1) > 100" …/> <!-- refused: TDC260 -->
<gen if="at(split(Prices, ','), 1) > 100" …/> <!-- what was meant -->
Prices is the joined text, so the first line asked for the second element of a
one-element list and used to render a blank column while the run reported success.
Written-out mistakes are caught by tdcv2 check before a row exists; an index worked out
at run time — at(list, _count - 1) — is checked as the row is built.
Why TDC computes its own transcendentals
IEEE-754 gives + - * / and sqrt exactly one legal answer each, so every language agrees
about them. It says nothing about sin, cos, exp, log or pow — each libm picks its
own algorithm — and the gap is measurable, not theoretical:
tan(1) | |
|---|---|
| Node | 3ff8eb245cbee3a6 |
| Python | 3ff8eb245cbee3a5 |
Sixteen of seventy-seven sampled values disagree somewhere across the five implementations.
In a timeseries that never shows, because every number is rounded to a decimal string
before it becomes output — the last bit dies on the way out. A comparison has no rounding
step, so that bit becomes a different row, and a different file, on a tool whose whole
promise is that five implementations produce the same bytes.
So TDC computes these itself, the way it already computes its own random numbers rather than trusting each language's. Every one lands within 4 ulp of the true value — the same neighbourhood a libm occupies — and, far more importantly, on the same double in all five. Matching any particular libm is not the goal and could not be: the libms do not match each other.
That 4 is checked rather than claimed, on grids that run to the ends of each function's range. The ends are where it matters: a series truncated two terms early is invisible in the middle of an interval and thirteen ulp out at the edge, which is exactly the bug the check was written after.
pow is the one function with a wider bound, for a reason worth knowing:
| exponent | how it is computed | drift |
|---|---|---|
| whole, or a half | repeated squaring, sqrt for the half | grows with the exponent — ~4 ulp at 3, ~22 at 20 |
| anything else | exp(y · log x) | grows with |y · log x| — ~2 ulp at 1, ~457 at 400 |
Both come from amplification, not from a defect: squaring doubles whatever error it was
handed, and exp turns an absolute error in its argument into a relative one in its answer.
Twelve significant digits survive either way.
<sequence name="Month"><gen type="increment" value="1"/></sequence>
<sequence name="Load">
<gen if="cos(Month / 2) > 0.5" type="text" value="peak"/>
<gen if="cos(Month / 2) < -0.5" type="text" value="trough"/>
<gen type="text" value="normal"/>
</sequence>
<sequence name="Tier">
<gen if="pow(2, Month) > 100" type="text" value="large"/>
<gen type="text" value="small"/>
</sequence>
1 peak small 2 peak small 3 normal small 4 normal small 5 trough small 6 trough small 7 trough large 8 trough large
That file was run through all five implementations and produced those bytes in every one.
Two things follow from computing them rather than borrowing them. pow with a whole-number
exponent goes through repeated squaring, so pow(10, 3) is exactly 1000 rather than
999.9999999999998 — a config comparing against a round number would have noticed. And the
circular functions take radians, with no degree variant: one convention, stated once.
The pair that exists because subtraction loses things
expm1 and log1p are not shorthands for exp(x) - 1 and log(1 + x). They are those
expressions computed so that the answer survives:
expm1(1e-20) 1e-20 exp(1e-20) - 1 0 log1p(1e-20) 1e-20 log(1 + 1e-20) 0
The second column is not a rounding error — it is the whole answer, gone. 1 + 1e-20 IS 1 as a
double, so the logarithm never sees the argument at all; and exp(1e-20) is 1.0000…, so the
subtraction cancels every digit that mattered. asinh and atanh are built on log1p for the
same reason, and inherit the accuracy.
hypot avoids the mirror-image problem at the other end of the range: sqrt(x² + y²) overflows
to infinity for x = 10²⁰⁰, though the answer is perfectly representable. And log2 separates the
exponent before taking any logarithm, so log2(8) is 3 rather than 2.9999999999999996.
Where the bound stops being a number of ulp
Four of these carry a bound that is not simply "within 4 ulp", and saying so is part of the reference rather than a footnote:
| Function | What holds |
|---|---|
erf | within 4 ulp |
erfc | within 8 ulp — it passes through e^(−x²), and that exponential carries the rounding of the square |
gamma | exact on whole numbers to 23, within 7 ulp on all 171 a double can hold; twelve significant digits elsewhere |
lgamma | within 32 ulp away from its zeros; the meaningful bound AT x = 1 and x = 2 is absolute, under 10⁻¹³ |
lgamma is the interesting one. It is zero at 1 and at 2, and no method that
adds up terms of size 1 can be relatively accurate about their cancelling to
nothing — the claim there has to be absolute, and it is. Both zeros come out
exactly zero.
gamma off the whole numbers ends in an exponential, so its drift grows with
log Γ(x) itself: the same amplification pow has, for the same reason. That is
why a whole number takes the factorial path instead.
What is deliberately absent
The mathematics a data generator has no business carrying. besselj, bessely, airy,
elliptic_k, elliptic_e and polygamma are refused by name rather than guessed at:
error[TDC257]: besselj() is not available yet in an if expression
Note what it does NOT say: "did you mean beta?" Edit distance would have offered exactly
that, and the two name entirely different functions. A name on this list is answered with
the reason instead.
Each of these is a project rather than a function, and none has ever plausibly belonged in a row predicate. They stay on the list so that a person who reaches for one gets an answer rather than "unknown function".
Loops and recursion. The engine is chosen from the config before a row is generated,
preflight() estimates memory before the run, and --jobs
splits rows across workers. All three need to know the work per row without doing it. A loop
breaks all three, and what people reach for a loop to express — "is this row even?" — is
%.
Bitwise operators. _count & 1 is _count % 2 written for a machine. They parse, so
the message can name them, and then they are refused.
When an expression is not enough
A <compute> sequence has integer division, remainders, string surgery,
encodings and checksums. It produces a value like any other sequence, and if= then compares
that:
<sequence name="Checksum">
<compute><result><mod><to_number><field name="Account"/></to_number><int v="97"/></mod></result></compute>
</sequence>
<sequence name="Flag">
<gen if="Checksum == 0" type="text" value="divisible"/>
<gen type="text" value="."/>
</sequence>
The config is XML-shaped, but it is not XML
TDC does not expand entities, so < is four literal characters rather than <. Write the
raw character:
<gen if="Weight > 20" .../> <!-- yes -->
<gen if="Weight > 20" .../> <!-- no: TDC100, and the message says why -->