Crust of Rust: Declarative Macros

Declarative macros in Rust—defined via the macro_rules! construct—provide a powerful pattern-matching metaprogramming system that operates directly on Rust syntax trees. This chapter dissects the mechanics of declarative macros by iteratively implementing an optimized, production-ready replacement for the standard library's vec! macro, demonstrating syntax matching, hygiene, repetition specifiers, side-effect avoidance, and compile-time variadic counting.

1. Declarative vs. Procedural Macros

Rust provides two distinct metaprogramming paradigms:

• Declarative Macros (macro_rules!): Syntax-extension mechanisms that match against code structure using pattern matching and perform token substitutions. They are self-contained, lightweight, compile quickly, and require no external compiler plugins or dedicated crates.
• Procedural Macros (proc-macro): Full Rust programs that accept a token stream (TokenStream) from the compiler, execute arbitrary Rust code during compilation, and output a transformed TokenStream. While significantly more expressive, procedural macros require dedicated crates, heavy external dependencies (such as syn and quote), and introduce compilation overhead.

Declarative macros operate at the syntactic boundary: they execute after the compiler lexes tokens and forms initial syntax trees, but before type checking, borrow checking, or trait resolution.

2. Anatomy of macro_rules!: Matchers and Fragment Specifiers

A declarative macro is declared using macro_rules!, followed by the macro's identifier, and a body structured similarly to a match expression:

macro_rules! avek {
    ( $( $pattern:tt )* ) => {
        // Expansion / Replacement code
    };
}

The arguments provided to a macro are not typed runtime variables; they are matched against syntax patterns. Variables inside patterns begin with a dollar sign ($) followed by a fragment specifier.

Grammar Restrictions

Macro inputs must consist of syntactically valid Rust grammar. Even though a macro can define custom, non-standard DSL syntax (such as u32 as Foo), the sequence of tokens cannot violate the underlying Rust tokenization and parsing rules. For example, following a ty fragment specifier with an unparenthesized single arrow -> causes an immediate parsing error because the grammar parser disallows that token sequence.

3. Delimiters, Tooling, and Macro Hygiene

Delimiter Flexibility

When defining or invoking a macro, Rust accepts three pairs of delimiters interchangeably:

• Parentheses: avek!(...)
• Square Brackets: avek![...]
• Curly Braces: avek! { ... }

The macro author cannot force the caller to use a specific delimiter. However, idiomatic Rust conventions dictate:

• vec![...] for array-like collection initialization.
• println!(...) for function-like calls.
• macro_rules! name { ... } for block-like definitions.

Inspection via cargo-expand

Because macros generate code prior to compilation, diagnosing syntax errors directly from compiler output can be challenging. The cargo-expand sub-command leverages the compiler to print the fully expanded AST:

cargo install cargo-expand
cargo expand --lib tests

Hygiene and Lexical Scope Isolation

Rust declarative macros feature partial hygiene. Identifiers declared inside a macro expansion live in a syntax context separate from identifiers declared in the caller's scope.

Consider an attempt to access a local variable defined within a macro:

macro_rules! make_x {
    () => {
        let x = 42;
    };
}

fn test_hygiene() {
    make_x!();
    // println!("{}", x); // COMPILE ERROR: cannot find value `x` in this scope
}

Similarly, code inside the macro cannot implicitly mutate or read variables from the enclosing scope. To bridge this scope isolation, the variable name must be passed explicitly into the macro as an ident:

macro_rules! set_var {
    ($var:ident, $val:expr) => {
        $var = $val;
    };
}

fn test_bridge() {
    let mut x = 0;
    set_var!(x, 42); // Explicitly passing the identifier permits access
    assert_eq!(x, 42);
}

Note: Passing an identifier ($x:ident) passes a reference to the name in the caller's namespace, not an ownership move. Ownership transfers depend entirely on how the expanded code uses that identifier.

4. Step-by-Step Evolution: Empty and Single-Element Vectors

Milestone 1: The Empty Vector

Using Test-Driven Development (TDD), we begin by supporting empty vector creation:

#[macro_export]
macro_rules! avek {
    () => {
        ::std::vec::Vec::new()
    };
}

#[test]
fn empty_vec() {
    let x: Vec<i32> = avek![];
    assert!(x.is_empty());
}

• #[macro_export]: Elevates the macro to crate-level visibility (analogous to pub). Without this attribute, the macro remains private to its defining module.
• Root path qualification (::std::vec::Vec): Guards against scenarios where a caller shadows or redefines the identifier Vec or module std.

Milestone 2: Single-Element Pattern and the Double-Brace Requirement

To support a single element, we match an expression $element:expr:

#[macro_export]
macro_rules! avek {
    () => {
        ::std::vec::Vec::new()
    };
    ($element:expr) => {{
        let mut vs = ::std::vec::Vec::new();
        vs.push($element);
        vs
    }};
}

Why Double Braces {{ ... }}?

In macro_rules!, the outer braces enclose the replacement template. If the replacement code consists of multiple statements (let mut vs = ...; vs.push(...); vs), it cannot be placed directly into an expression context:

// ILLEGAL EXPANSION if outer braces are omitted:
let x = let mut vs = Vec::new(); vs.push(42); vs; // Syntax Error!

Wrapping the body in an inner pair of braces creates an expression block { ... }. In Rust, a block is an expression whose final evaluated statement determines its return value.

5. Handling Variadic Repetitions and Trailing Commas

Repetition Syntax

To handle multiple arguments, declarative macros provide repetition constructs:

• $( ... )*: Zero or more repetitions.
• $( ... )+: One or more repetitions.
• $( ... )?: Zero or one repetition.

A separator token can be inserted immediately before the quantifier. For example, $($element:expr ),+ matches one or more expressions separated by commas.

#[macro_export]
macro_rules! avek {
    () => {
        ::std::vec::Vec::new()
    };
    ( $( $element:expr ),+ ) => {{
        let mut vs = ::std::vec::Vec::new();
        $(
            vs.push($element);
        )+
        vs
    }};
}

During expansion, the repetition block $( vs.push($element); )+ iterates over the matched elements, repeating the inner template once for every token matched by $element.

Handling Trailing Commas

Idiomatic Rust style allows or encourages trailing commas across multi-line declarations. Matching $($element:expr ),+ causes avek![1, 2, 3,] to fail because the final comma indicates another expression should follow.

We resolve this by appending an optional trailing comma pattern $(,)?:

#[macro_export]
macro_rules! avek {
    ( $( $element:expr ),* $(,)? ) => {{
        let mut vs = ::std::vec::Vec::new();
        $(
            vs.push($element);
        )*
        vs
    }};
}

By switching the quantifier to * and allowing an optional comma $(,)?, this single rule elegantly handles:

• Zero elements: avek![]
• Single elements: avek![1] or avek![1,]
• Multiple elements: avek![1, 2, 3] or avek![1, 2, 3,]

6. The Repeated Element Pattern (vec![elem; count])

The standard library also allows initializing a vector with count copies of an element: vec![x; n].

Naive Implementation (Repeated Evaluation Hazard)

A naive attempt loops over the count and pushes the element:

// HAZARDOUS IMPLEMENTATION
($element:expr; $count:expr) => {{
    let mut vs = ::std::vec::Vec::new();
    for _ in 0..$count {
        vs.push($element);
    }
    vs
}};

The Side-Effect Trap

Because macro expansion performs syntactic replacement, passing an expression that mutates state or executes non-idempotent logic will cause runtime bugs:

let mut opt = Some(42);
let vs = avek![opt.take().unwrap(); 2]; // PANIC ON SECOND ITERATION!

This macro expands to:

for _ in 0..2 {
    vs.push(opt.take().unwrap()); // 1st iteration: Some(42) -> 42; 2nd iteration: None -> Panic!
}

Furthermore, if $element is an expensive function call (e.g., parsing a file or making a network request), it runs $count times instead of once.

Production Implementation: Single Evaluation and Cloning

To ensure correct semantics:

1. Evaluate $element exactly once and assign it to an immutable variable.
2. Evaluate $count exactly once to avoid redundant bounds evaluations.
3. Clone the value into the vector using Vec::resize.

($element:expr; $count:expr) => {{
    let count = $count;
    let mut vs = ::std::vec::Vec::new();
    vs.resize(count, $element);
    vs
}};

Note on Trait Bounds: Vec::resize requires that the element type implement Clone. If a non-cloneable type is passed to the macro, the compiler bubbles the error directly to the line calling avek!, noting that the Clone bound is not satisfied.

7. Allocation Pitfalls and Capacity Optimization

When building a vector dynamically via sequential .push() calls, the vector must continually reallocate memory on the heap as its size exceeds its current capacity.

Growth Strategy and Reallocation Overhead

Standard library vectors start with a small capacity (or 0) and double their allocated buffer each time they exhaust capacity:

Constructing an array of 1,024 elements with Vec::new() and repeated push() statements forces roughly 10 distinct heap allocations and copies hundreds of elements across intermediate buffers.

The Optimization: Vec::with_capacity

If the total number of elements is known before pushing, pre-allocating memory with Vec::with_capacity(N) eliminates intermediate reallocations:

let mut vs = ::std::vec::Vec::with_capacity(count);

For the repeated element pattern ($element:expr;$count:expr), pre-allocation is trivial because $count is provided explicitly. For variadic lists ($($element:expr ),*), the challenge is determining the number of elements at compile time without manual specification.

8. The Counting Problem: Determining Variadic Length

In declarative macros, there is no built-in syntax variable like $len to retrieve the count of repetitions in $($element:expr ),*.

Failed Counting Strategies

1. Re-evaluating in an Array Literal:

   RUST

   Copy

   let count = [$($element),*].len();
   

   Failure: Consumes/moves all $element expressions. When the macro subsequently tries to push them into the vector, the compiler emits a "use of moved value" error.

2. Substituting with Empty Unit Tuples Directly:

   RUST

   Copy

   let count = [$([()]),*].len();
   

   Failure: The compiler outputs: error: attempted to repeat an expression containing no syntax variables matched as repeating at this depth. The macro expansion engine relies on the presence of the matched pattern variable ($element) to know how many times to repeat the template.

9. The Zero-Sized Type (ZST) Slice Counting Hack

To repeat a replacement token without evaluating the original expression, we decouple the variable binding from its evaluated output using an internal substitution macro.

Step 1: The Substitution Pattern

We define an internal pattern that accepts any expression and outputs the unit tuple ():

#[doc(hidden)]
#[macro_export]
macro_rules! avek_subst {
    ($element:expr) => { () };
}

When called inside a repetition:

[ $( $crate::avek_subst!($element) ),* ]

1. The compiler tracks $element, determining the exact repetition count.
2. For each element, $crate::avek_subst!($element) discards the actual expression and emits ().
3. An array of unit types [(); N] is constructed.

Step 2: Extracting Length via Slice Reference

Because unit types () are Zero-Sized Types (ZSTs), creating an array of [(); N] costs zero bytes of runtime memory and consumes no stack space.

To determine its length, we coerce the array to a slice and call <[_]>::len:

const COUNT: usize = <[()]>::len(&[
    $( $crate::avek_subst!($element) ),*
]);

Because <[()]>::len is a const fn, modern Rust compilers resolve this expression during compilation, replacing with_capacity directly with an integer constant.

10. Architectural Encapsulation: Internal Rules and Documentation

Exposing internal helper macros like avek_subst! pollutes the public API and crate documentation. To maintain a clean interface:

1. Annotate helper macros with #[doc(hidden)] so they do not appear in cargo doc.
2. Place helper patterns inside dedicated internal macro blocks or private sub-rules using unique token prefixes (such as @count).

Complete, Production-Ready Implementation

/// Helper macro used strictly for compile-time variadic counting.
/// Replaces an input expression with a zero-sized unit type ().
#[doc(hidden)]
#[macro_export]
macro_rules! __avek_subst {
    ($element:expr) => { () };
}

/// Constructs a `Vec` containing the specified elements.
///
/// Supports standard list syntax with optional trailing commas:
/// ```
/// let v = vecmac::avek![1, 2, 3];
/// assert_eq!(v.len(), 3);
/// ```
///
/// As well as repetition syntax:
/// ```
/// let v = vecmac::avek![0; 5];
/// assert_eq!(v, vec![0, 0, 0, 0, 0]);
/// ```
#[macro_export]
macro_rules! avek {
    // Repeated element syntax: avek![elem; count]
    ($element:expr; $count:expr) => {{
        let count = $count;
        let mut vs = ::std::vec::Vec::new();
        vs.resize(count, $element);
        vs
    }};

    // Variadic list syntax: avek![e1, e2, ...]
    ( $( $element:expr ),* $(,)? ) => {{
        const COUNT: usize = <[()]>::len(&[
            $( $crate::__avek_subst!($element) ),*
        ]);
        let mut vs = ::std::vec::Vec::with_capacity(COUNT);
        $(
            vs.push($element);
        )*
        vs
    }};
}

11. How the Standard Library Implements vec!

While our zero-sized slice counting technique works entirely on stable Rust, the standard library uses a different strategy enabled by compiler intrinsics and nightly box syntax:

// Simplified standard library approach
macro_rules! vec {
    ($elem:expr; $n:expr) => {
        ::std::vec::from_elem($elem, $n)
    };
    ($($x:expr),* $(,)?) => {
        <[_]>::into_vec(
            // Nightly-internal heap box allocation of an array:
            $crate::boxed::Box::new([$($x),*])
        )
    };
}

Memory Layout and Conversions

In Rust, a Vec<T> consists of three words on the stack:

• Pointer to heap memory buffer (*mut T)
• Allocated capacity (usize)
• Current length (usize)

A boxed slice (Box<[T]>) consists of two words on the stack:

• Pointer to heap memory buffer (*mut T)
• Length of slice (usize)

The standard library allocates elements directly onto the heap into a boxed array (box [...]), then reinterprets the boxed slice as a Vec<T> with zero allocations or data copying. Because third-party libraries on stable Rust cannot access unstable compiler intrinsics like box [...], the pre-allocation and push pattern (Vec::with_capacity + ZST counting) represents the canonical, high-performance approach.

12. Summary and Key Takeaways

• AST Pattern Matching: Declarative macros (macro_rules!) match against syntax trees, not runtime types. Inputs must be syntactically valid Rust grammar, while outputs must compile as valid Rust code.
• Hygiene Boundaries: Macro identifiers belong to a separate syntactic scope. An identifier declared outside a macro cannot be accessed inside unless explicitly passed into the pattern as an $ident.
• Expression Blocks Requirement: Multiple initialization statements must be wrapped inside double braces {{ ... }} so the expansion resolves to a single expression block.
• Side-Effect Prevention: When implementing repeated patterns ([elem; count]), evaluate all input expressions once into local bindings (let x = $element;) before consuming or cloning them to avoid repeated side effects.
• Allocation Penalties: Constructing collections with naive Vec::new() and sequential .push() calls incurs $O(\log N)$ heap reallocations as capacity grows by powers of two ($0 \to 4 \to 8 \to 16 \dots$).
• The ZST Counting Idiom: Declarative macros cannot directly count repetition tokens. Creating an array of zero-sized unit tuples [(); N] via an internal substitution macro provides compile-time variadic counting without runtime memory allocation.
• Documentation and API Cleanliness: Mark internal helper macros with #[doc(hidden)] and reference them using $crate::__helper! to ensure paths resolve reliably regardless of caller scope.