Flat JSON transformation (Tinker curriculum task set 1)

Flat JSON transformation is the first task set in the Tinker curriculum. The model receives a short natural-language instruction and must emit exactly one Python function that transforms a flat input record into a new flat dictionary. The generated code is never executed as arbitrary Python; a narrow AST interpreter accepts only the task-set grammar and evaluates it against hidden test cases.

Task structure

Each task presents a mapping instruction such as "copy required field id unchanged" or "set region from optional input field area; if it is missing, use literal unknown." The model must produce a function with this signature:

def transform(record):
    return {
        "customer_id": record["id"],
        "region": record.get("region", "unknown"),
        "schema_version": 1,
    }

The task set stays inside one flat-record transformation family while combining eight operations:

• Copy a required field unchanged.
• Rename a required field to a new destination key.
• Copy an optional field with a literal default, using record.get(key, default).
• Add a literal field with a constant value.
• Normalize a string by trimming whitespace and applying lower, upper, or title case.
• Concatenate two trimmed fields with a requested separator.
• Apply a rounded affine formula of the form round(value * multiplier + offset, 2).
• Choose a literal from a numeric threshold rule, using a conditional expression value if record[key] >= threshold else alternative.

The eight operations are combined into eight patterns of increasing compositionality, from two-field tasks to four-field tasks that mix string, numeric, and conditional logic.

Evaluation

The evaluator parses the generated code with Python's ast module and enforces a strict grammar contract:

• exactly one function named transform with one parameter named record;
• the function body is a single return of a dictionary literal;
• dictionary keys are string literals and must be unique;
• values may be: string or numeric constants, record[key] subscripts, record.get(key, default) calls, .strip().lower() / .upper() / .title() method chains, + concatenation, * multiplication, round(expr, 2) calls, and value if record[key] >= threshold else alternative conditionals;
• no imports, decorators, annotations, comprehensions, loops, or arbitrary function calls are permitted.

If the code passes the grammar check, the AST interpreter evaluates it against five hidden cases per task. A task passes only if all five cases produce the exact expected output. This means success is behavioral: the generated transformation must produce correct results, not match the reference implementation's text.

The evaluator boundary is intentionally narrower than Python itself:

tree = ast.parse(generated_code)
transform = validate_restricted_transform(tree)

for case in hidden_cases:
    assert interpret(transform, case.input) == case.expected

validate_restricted_transform rejects every syntax form outside the task grammar before the interpreter sees a case. The evaluator never calls exec or imports generated code.

Corpus

The deterministic corpus contains 96 training tasks, 32 development-validation tasks, and 32 sealed promotion tasks. Field vocabularies are disjoint across splits—training, validation, and test use entirely different source and destination field names—so memorizing training field names cannot help on held-out splits. The dataset is generated from fixed seeds and hash-checked for reproducibility.

Training and results

The base model is Qwen3.5-35B-A3B-Base, a raw 35B-parameter mixture-of-experts model with approximately 3B active parameters per token. The choice of a raw base model—rather than an instruction-tuned model that might already solve the task—is deliberate: the point is to teach a compact active model a task through supervised fine-tuning.

An initial projection-only draft was rejected before training. Both a smaller 4B model and the raw 35B base scored 24 out of 24 on the simpler initial task set, indicating the task was too easy to measure improvement. The task set was then expanded within the same flat-transformation family to require compositional string, numeric, and conditional behavior. The rejected draft remains only as an ignored baseline receipt, not as a second task set.

A rank-16 LoRA adapter was trained on task set 1 only. The promotion contract required at least 85% task accuracy and at least five percentage points improvement over the base model on development validation before the sealed test could be opened.

Task-set-1 accuracy. Task accuracy for the untouched base model and rank-16 adapter on development and the sealed promotion set.

Source: Task-set-1 machine-readable evaluation and lineage receipts; 32 tasks and 160 hidden cases per split. Note: The sealed set was opened only after the trained adapter passed the 85% development threshold and five-point improvement gate.

The trained adapter passed the promotion gate with an 81.25 percentage-point improvement on development validation. After the gate passed, the sealed test was opened, and the trained adapter achieved 100% on both tasks and hidden cases.

Promotion gate

The promotion contract for task set 1 is:

1. Evaluate the untouched base model on development validation.
2. Train one LoRA candidate on task set 1 only.
3. Compare the candidate with the base on the same development set.
4. Open the sealed promotion set only if the candidate reaches at least 85% task accuracy and improves by at least five percentage points.
5. Preserve task set 1's frozen promotion set when task set 2 is eventually added.

The gate passed. The frozen test split is preserved for all future task-set additions.

Receipts

The training and evaluation results are recorded as machine-readable JSON receipts. The lineage receipt records the base model, dataset hash, hyperparameters, checkpoint references, validation scores, and test scores. The evaluation receipt records every prompt, model output, semantic score, and usage estimate. Dataset hashes are preserved for reproducibility.

The task-set-1 dataset SHA-256 is 60fb4f205777f7414db175d76b05538cba9fd784b771d836b7060130dd89fc65. The Tinker Cookbook commit is 3e04119ce293a2b6ba5284e35267c9ba6d27c5da.

Evidence limits

Task set 1 is the cleaner of the two task sets. The task is deterministic, the evaluator is mechanical, and the dataset is reproducible from fixed seeds. The main limitation is that the task family is narrow: it tests whether a model can learn to emit a restricted Python grammar that produces correct flat-record transformations, not whether it can write general-purpose code. The 100% test score should not be generalized beyond this grammar.

Sources

• Tinker
• Qwen3.5-35B-A3B-Base on Hugging Face
• LoRA — Low-Rank Adaptation of Large Language Models