Yuku for TSRX

How it works

@tsrx/yuku is TSRX written as a Yuku dialect. Yuku is a JavaScript and TypeScript compiler toolchain in Zig: a parser, a semantic analyzer, a traverser, and a code generator, with no dependencies. A dialect adds syntax to it. You write only the new syntax, in one Zig file, and Yuku parses, analyzes, walks, and prints the rest of the language, and your nodes with it.

One Zig file#

The grammar is src/tsrx.zig, following the TSRX specification. It declares the files it reads, .tsrx as TSX, and its node kinds, each with the type and fields it has in JavaScript. Then it answers at Yuku's positions in the grammar:

Position What TSRX reads there
statement A code block, a directive, or an element at the start of a statement, which needs no ; before a tag
expression @{ }, @if, @for, @switch, @try, or a <style> as a value
operator A tag at the start of a line, or after output, which starts an element
function_body @{ } as the body of a function, method, or arrow function
jsx_child Template text with its comments, directives, and <style>, and the raw bodies of <style> and <script>
jsx_attribute {name}, short for name={name}
jsx_element_name <{tag}>, a tag chosen at runtime
module_specifier A submodule named by an identifier, as in import { load } from server

At each one the dialect parses its syntax or declines, and Yuku's own grammar goes on. A JavaScript expression inside an @if is parsed by Yuku, and TSRX inside that expression is asked for again, so the two nest to any depth. The shapes also tell Yuku's analyzer what each node means: an @for opens a block scope, declares its item there, and is a loop.

Compiled into Yuku#

build.zig is five lines:

const std = @import("std");

pub fn build(b: *std.Build) void {
    const yuku = b.dependency("yuku", .{ .dialect = b.path("src/tsrx.zig") });
    b.installArtifact(@import("yuku").addWasm(b, yuku, "yuku-tsrx"));
}

zig build compiles Yuku with the dialect inside its parser, to one WebAssembly module, yuku-tsrx.wasm: 746 kB, about 221 kB gzipped. Standard code parses exactly as in Yuku, and the dialect costs a standard file one flag check at each position. The module uses WebAssembly SIMD, which is why Node.js needs 16.4.

The JavaScript is a thin layer over it, src/index.ts: parse, analyze, and Analyzer are Yuku's, passed the TSRX module, and generate is Yuku's code generator with printers for TSRX's nodes. The module's bytes are inlined in the package, so it loads with no file system and no fetch. In Cloudflare Workers, which can't compile WebAssembly from bytes, the workerd condition imports the .wasm file instead.

A tree in one buffer#

Every node that crosses from WebAssembly into JavaScript costs time, so Yuku keeps the crossing to one buffer. Its parser stores the tree flat, in arrays of fixed-size records, and the result crosses as one buffer: 48 bytes a node, then child lists, a string pool, comments, tokens, and diagnostics. JavaScript copies the buffer once.

A decoder turns it into ESTree objects. It is generated from Yuku's Zig layouts, so the two can't disagree, and it runs when you first read program, in one pass over the records. Names and literals are sliced straight from your source string, and only text that needs decoding, such as an escaped identifier, comes from the pool. Offsets are mapped from bytes to UTF-16 as it decodes, so a node's start indexes your string directly. The semantic model crosses the same way, as tables that Scope, Binding, and Reference read by index.

Correct#

The test suite is every .tsrx file of the TSRX repository and every case in the tests of @tsrx/core, synced at the release of the installed @tsrx/core by bun run sync: 1,987 sources that must parse and 128 that must fail, as the official parser parses them, except where it accepts what TypeScript rejects. Every source that parses is printed with generate, parsed again, and must print the same code.

Fast#

It parses the test suite 9.2x faster than @tsrx/core, and the pages of tsrx.dev 4.8x faster. Benchmarks has the numbers and how to run them.

Next: CSS trees#

Today a <style> keeps its CSS as one JSXText, as @tsrx/core does. Next, each <style> gets a CSS tree, parsed in the same pass by yuku-css, a CSS parser written in Zig the same way, which builds css-tree's AST. That needs a dialect to hand its own data to JavaScript, which yuku#280 adds to Yuku. It ships once that lands.

Work on it#

Needs Zig 0.17 and Bun. build.zig.zon pins Yuku to the commit of the yuku version in package.json, and zig build fetches it.

bun install
bun run build      # zig build && tsdown
bun run typecheck
bun run test
bun run sync       # syncs test/suite with the TSRX repository at the release of @tsrx/core
bun run bench

To work on Yuku and the dialect together, check Yuku out next to this repository and set .yuku = .{ .path = "../yuku" } in build.zig.zon and "yuku": "file:../yuku/packages/yuku" in package.json. Yuku's AGENTS.md is the rulebook for changes.

Credits#

@tsrx/yuku is maintained by Jack Shelton, on tsrx-org/yuku. Thanks to Arshad Yaseen for his help.