Source Processing

Matchbox turns source text into a checked AST using the lexer, token helpers, parser, symbol tables, and AST node definitions.

Grammar

The expression grammar is organized by precedence. In 0.2, primary expressions are grouped expressions, integer literals, names, and function calls. Prefix expressions are parsed before exponentiation, exponentiation is parsed left to right, and then factor, term, shift, comparison, equality, bitwise, and logical levels are considered. Comparison, equality, and logical operators are recognized but rejected as unsupported when parsed.

Expression  -> BooleanOr
BooleanOr   -> BooleanAnd { "||" BooleanAnd }
BooleanAnd  -> BitwiseOr { "&&" BitwiseOr }
BitwiseOr   -> BitwiseXor { "|" BitwiseXor }
BitwiseXor  -> BitwiseAnd { "^" BitwiseAnd }
BitwiseAnd  -> Equality { "&" Equality }
Equality    -> Comparison { ("==" | "!=") Comparison }
Comparison  -> Shift { (">" | ">=" | "<" | "<=" | "<=>") Shift }
Shift       -> Term { ("<<" | ">>") Term }
Term        -> Factor { ("+" | "-") Factor }
Factor      -> Exponent { ("*" | "/" | "//" | "%") Exponent }
Exponent    -> Prefix { "**" Prefix }
Prefix      -> ("!" | "~" | "-") Prefix | Primary
Primary     -> Integer | Name | Call | "(" Expression ")"

Lexer

The lexer tracks a start position and current position, including line and column, and returns one token at a time. It skips whitespace, single-line comments beginning with #, and multi-line comments delimited with ## and ##. Unterminated strings, characters, and multi-line comments are reported by the lexer before parsing continues.

Numeric tokens include decimal, binary, octal, hexadecimal, and floating spellings. In 0.2, the parser accepts integer, binary, octal, and hexadecimal literals as integer expressions. Underscores may separate digits when followed by another valid digit.

Tokens

The token enum is broader than the implemented language. It includes current tokens such as var, func, return, int, arithmetic operators, bitwise operators, assignment operators, parentheses, braces, commas, identifiers, and integer literal forms. It also includes reserved or future-facing tokens such as uint, float, class, struct, trait, async, comparison operators, logical operators, ranges, safe access, and compound bitwise assignment.

Token helper functions define the supported subset. For example, isTypeToken accepts only int, isPrefixToken accepts !, -, and ~, and isAssignmentToken accepts simple arithmetic assignment forms. This is why the lexer can recognize more syntax than the parser and compiler currently allow.

The lexer emits token kinds from these groups:

Each token stores its type, the source characters that formed it, its length, and the line and column where it began. The lexer does not copy token text; it points back into the source buffer. Later stages use the token span to print diagnostics and to convert literal text into runtime values.

Parser

The parser consumes tokens recursively and builds AST nodes. It reports unexpected tokens, unsupported operators, invalid operand combinations, undefined names, uninitialized reads, redefinitions, invalid function arguments, and invalid variable types. Most diagnostics include the source line and column of the token that caused the error.

Statements are separated by newlines or semicolons. When two statements appear on the same line, the parser expects a semicolon unless the previous token closed a block. Blocks parse until }, and the top level parses until end of file.

Top-Level Statements

At the top level, Matchbox accepts variable definitions, assignments, function definitions, function-calls, and expression statements. Built-in function calls are represented internally as service requests. Top-level variable definitions are stored as globals. Function definitions produce function objects in the module constant pool. Expression statements are compiled and then popped, so their values are not preserved unless assigned or passed to a function or built-in service.

Symbol Table

Each scope owns a symbol table and a local count. The top-level scope contains globals and functions. Function definitions create child scopes for parameters and local variables. Local variable declarations increment the scope's local count, which later becomes part of the function frame size.

Name lookup first checks the current scope, then the top-level scope for variables and calls that are allowed by the parser. A local declaration can shadow a top-level declaration, but redeclaring a name in the same scope is an error.

AST

The AST represents the checked program. Node kinds include assignment, binary expression, compound block, function call, function definition, integer literal, parameter, prefix expression, return statement, service request, variable reference, variable definition, and a none sentinel used for omitted expressions. AST nodes carry enough resolved information for the compiler to emit bytecode without performing name lookup again.

Variable and function nodes store scope and symbol links. Binary nodes store the operator, left expression, right expression, and resulting type. Function definitions store parameters, return type, body, and scope. This makes the AST both a syntax tree and a compact semantic model for the current compiler.