Build a jq Style JSON Query Tool in Rust Step by Step

jq is the tool people reach for when a JSON document is too big to read and too awkward for grep. It is also a small language: a query like .users[] | select(.role == "dev") | .name is a program that describes a path through the data, filters it and extracts a field. Building a miniature version in Rust teaches a surprising amount in about 250 lines. You write a tokenizer-free recursive descent parser, model a query as an enum, evaluate it with a function that returns many results, and wrap everything in a command line tool that behaves well in a pipeline.

The tool built here is called mjq. It supports field access, array indexing with negative indexes, iteration, pipes, length, keys, map, select with the six comparison operators, and a mode for streams of newline-delimited JSON. It was built and tested on Rust 1.99 (edition 2024) with serde_json 1.0.154 and clap 4.6.7, and every output below is real.

What the language needs

Start from the queries you want to run, because the grammar follows from them.

• .team reads the field team of the input object.
• .users[0].name goes into an array by index, then reads a field.
• .users[-1] picks the last element.
• .users[] yields every element of the array, as separate results.
• .users | length feeds the array into the length builtin.
• .users | map(.role) applies a filter to each element and collects the results.
• .users[] | select(.role == "dev") keeps only the elements that match.

Two ideas run through the list. A filter takes one JSON value and produces zero, one or many values; .users[] produces three, select may produce none. And filters compose with |, which feeds every output of the left into the right. Those two facts decide the design: a filter is an enum, and running one returns a Vec<Value>.

Model the query as data

The parser's output is a tree of Filter values, and the evaluator walks it. Writing the enum first fixes what the language can say:

#[derive(Debug, Clone, PartialEq)]
pub enum Filter {
    /// `.` returns the input unchanged.
    Identity,
    /// `.name`
    Field(String),
    /// `.[3]`
    Index(i64),
    /// `.[]` yields every element of an array or every value of an object.
    Iterate,
    /// `length`
    Length,
    /// `keys`
    Keys,
    /// `select(<filter> <op> <json>)` keeps the input when the comparison holds.
    Select(Box<Filter>, Cmp, Value),
    /// `map(<filter>)` applies the filter to each element and collects the results.
    Map(Box<Filter>),
    /// `a | b` feeds every output of `a` into `b`.
    Pipe(Box<Filter>, Box<Filter>),
}

Notice the Box<Filter> in Select, Map and Pipe: a filter can contain filters, and a type that contains itself needs an indirection to have a finite size. The path .users[0].name does not need its own variant. The parser turns it into Pipe(Field("users"), Pipe(Index(0), Field("name"))), because a path is just a pipeline of steps. That is a common simplification: make the evaluator handle a few primitives and let the parser build composite structure out of them.

Evaluate: one input, many outputs

The evaluator is one function with a match over the enum. Its type tells the whole story: fn run(filter: &Filter, input: &Value) -> Result<Vec<Value>, String>. The Vec carries the "zero, one or many" semantics, and the Result carries failures such as indexing a string.

Filter::Select(inner, op, wanted) => {
    let found = run(inner, input)?;
    Ok(if found.iter().any(|v| op.holds(v, wanted)) { vec![input.clone()] } else { vec![] })
}
Filter::Map(inner) => {
    let items = match input {
        Value::Array(items) => items,
        other => return Err(format!("cannot map over {}", kind(other))),
    };
    let mut out = Vec::new();
    for item in items {
        out.extend(run(inner, item)?);
    }
    Ok(vec![Value::Array(out)])
}
Filter::Pipe(left, right) => {
    let mut out = Vec::new();
    for mid in run(left, input)? {
        out.extend(run(right, &mid)?);
    }
    Ok(out)
}

Pipe is the heart. It runs the left filter, which may produce many values, and for each one runs the right filter, concatenating all the results. That single rule gives you everything jq users find magical: .users[] | .name first expands the array into three values, then takes .name of each, producing three names. Select returns either the input in a one-element vector or an empty vector, which is how filtering falls out of the same shape. Map is Iterate plus a collect, wrapped in an array. Nothing in the evaluator knows about jq specifically; it is three recursion rules and a handful of primitives.

Errors are values. Indexing a string returns Err("cannot index string"), and the first ? stops the whole run. That is stricter than jq, which can continue past some errors, and it is the right default for a small tool: a query that does not match the data should say so.

Parse by recursive descent

The parser reads bytes with a position and builds the tree. A query is a pipeline of paths separated by |, and a path is a chain of steps:

fn pipeline(&mut self) -> Result<Filter, ParseError> {
    let mut left = self.path()?;
    while self.eat(b'|') {
        let right = self.path()?;
        left = Filter::Pipe(Box::new(left), Box::new(right));
    }
    Ok(left)
}

path handles .name, [3], [-1] and [] in a loop, or dispatches to builtin when the text starts with a letter (length, keys, map(...), select(...)). map calls pipeline recursively for its argument, which is why map(.a | .b) works. The helper eat(byte) skips whitespace and consumes a byte if it is next, which keeps the code free of manual index arithmetic. The parser never panics on bad input; each failure becomes a ParseError variant with enough context to write a good message.

The command line

The binary is thin, as every well-structured Rust tool is. clap supplies the arguments, run does the work and returns a Result, and main turns the result into an exit code:

mod query;

use std::io::{self, Read};
use std::path::PathBuf;
use std::process::ExitCode;

use anyhow::{Context, Result};
use clap::Parser;

/// A small jq-style JSON query tool.
#[derive(Parser)]
#[command(name = "mjq", version)]
struct Cli {
    /// The query, for example `.users[].name`.
    query: String,
    /// JSON file to read. Reads standard input when omitted.
    file: Option<PathBuf>,
    /// Print raw strings without quotes.
    #[arg(short, long)]
    raw: bool,
    /// Print compact one-line JSON.
    #[arg(short, long)]
    compact: bool,
    /// Treat the input as a stream of JSON values, one per line, and run the query on each.
    #[arg(short, long)]
    lines: bool,
}

fn run(cli: &Cli) -> Result<()> {
    let text = match &cli.file {
        Some(path) => std::fs::read_to_string(path).with_context(|| format!("reading {}", path.display()))?,
        None => {
            let mut buf = String::new();
            io::stdin().read_to_string(&mut buf).context("reading standard input")?;
            buf
        }
    };
    let filter = query::parse(&cli.query).context("invalid query")?;
    if cli.lines {
        for (n, line) in text.lines().enumerate().filter(|(_, l)| !l.trim().is_empty()) {
            let input: serde_json::Value = serde_json::from_str(line).with_context(|| format!("line {} is not valid JSON", n + 1))?;
            emit(cli, &filter, &input)?;
        }
    } else {
        let input: serde_json::Value = serde_json::from_str(&text).context("the input is not valid JSON")?;
        emit(cli, &filter, &input)?;
    }
    Ok(())
}

fn emit(cli: &Cli, filter: &query::Filter, input: &serde_json::Value) -> Result<()> {
    for value in query::run(filter, input).map_err(anyhow::Error::msg)? {
        match (&value, cli.raw, cli.compact) {
            (serde_json::Value::String(s), true, _) => println!("{s}"),
            (_, _, true) => println!("{value}"),
            _ => println!("{}", serde_json::to_string_pretty(&value)?),
        }
    }
    Ok(())
}

fn main() -> ExitCode {
    match run(&Cli::parse()) {
        Ok(()) => ExitCode::SUCCESS,
        Err(e) => {
            eprintln!("mjq: {e:#}");
            ExitCode::from(2)
        }
    }
}

It reads the JSON from a file or standard input, parses the query, evaluates it and prints each result on its own line. Flags handle output and input: pretty printing is the default, -c prints compact one-line JSON, -r prints strings without quotes so shell scripts can use them, and -l treats the input as one JSON value per line. Errors go to standard error with exit status 2, matching the convention in a command line tool from clap to a shipped release. The with_context calls are what make failures readable: "reading nope.json" tells you which step failed and the underlying operating system message says why.

Running it

Given a small document with a team name and three users:

{
  "team": "platform",
  "users": [
    {"name": "ada", "role": "admin", "langs": ["rust", "c"], "active": true},
    {"name": "grace", "role": "dev", "langs": ["rust", "go"], "active": true},
    {"name": "linus", "role": "dev", "langs": ["c"], "active": false}
  ]
}

These are the queries and their output, with -c for compact results:

mjq '.team'                                      "platform"
mjq '.users[0].name'                             "ada"
mjq '.users[].name'                              "ada" "grace" "linus"
mjq '.users | length'                            3
mjq '.users[1] | keys'                           ["active","langs","name","role"]
mjq '.users | map(.role)'                        ["admin","dev","dev"]
mjq '.users[] | select(.role == "dev") | .name'  "grace" "linus"
mjq '.users[-1].langs[0]'                        "c"
mjq '.missing'                                   null
mjq '.users[0].name.first'                       mjq: cannot read field "first" of string

Each line is worth reading. A missing field gives null, as in jq, and a field of a string is an error. The keys output is sorted alphabetically because serde_json keeps object keys in sorted order by default; enabling its preserve_order feature would keep the file's order. The select query chains three stages: expand the array, keep the developers, take the names.

The failure paths matter as much as the happy ones:

mjq '.users['       mjq: invalid query: the query ended too early
mjq 'users'         mjq: invalid query: unknown function "users" (try length, keys, map, select)
$ echo 'not json' | mjq '.a'
mjq: the input is not valid JSON: expected ident at line 1 column 2

The second message was a bug in an earlier draft: the unknown function users was reported as "not valid JSON" because it reused the literal error variant. Writing the test that exercised every error case (errors_are_values_not_panics below) is what found it. Give each failure its own variant, so each gets its own message.

Comparison operators

The first version of select supported only ==. Real queries need ordering: "people over 30", "names after h". Adding it touches three places and teaches how a small language grows. A Cmp enum names the six operators. The parser gets a comparison method that reads ==, !=, <=, >=, < or >, checking two-character operators before one-character ones so >= is not read as > followed by garbage. And Cmp::holds decides what each operator means for each pair of JSON types:

impl Cmp {
    fn holds(self, left: &Value, right: &Value) -> bool {
        use std::cmp::Ordering;
        let ordering = match (left, right) {
            (Value::Number(a), Value::Number(b)) => a.as_f64().partial_cmp(&b.as_f64()),
            (Value::String(a), Value::String(b)) => Some(a.cmp(b)),
            _ => None,
        };
        match self {
            Cmp::Eq => left == right,
            Cmp::Ne => left != right,
            // Ordering is defined only between two numbers or two strings.
            Cmp::Lt => ordering == Some(Ordering::Less),
            Cmp::Le => matches!(ordering, Some(Ordering::Less | Ordering::Equal)),
            Cmp::Gt => ordering == Some(Ordering::Greater),
            Cmp::Ge => matches!(ordering, Some(Ordering::Greater | Ordering::Equal)),
        }
    }
}

The design choice is in the _ => None arm: ordering a number against a string, or an object against anything, has no answer, so < and > are simply false, while == and != work on any pair. Real jq defines a total order across types; for a small tool, "false when not comparable" is easier to explain. partial_cmp on f64 returns None for NaN, which the same code handles without a special case, since a None ordering makes every ordered comparison false. The test that locks this behavior in was written to the wrong expectation first:

assertion `left == right` failed
  left: [Number(36)]
 right: [Number(41), Number(36)]

The test asked for ages of people whose names are at least "h", and expected two. The program returned one, and the program was right: "grace" starts with g, which sorts before h, so only "linus" qualifies. The expectation was the bug. Failing tests are evidence, not verdicts; read both sides before editing either. With the expectation corrected, five tests pass, and the new queries work:

mjq -c '.people[] | select(.age > 30) | .name'   "grace" "linus"

Streams of values: newline-delimited JSON

Logs and event feeds are often JSON values one per line, not one big array. Reading such a file as a single document fails on the second line. The -l flag processes each line separately, running the same filter and printing results as they arrive, which also keeps memory bounded by the size of one line. The structure is a loop over text.lines(), a filter that skips blank lines, and the same emit function that handled the single-document case, extracted so both paths share it. Applied to a four-line event log:

mjq -c -l 'select(.level == "error") | .ms' events.ndjson   340 95

The two error events had latencies of 340 and 95 milliseconds, and the two info events were filtered out. A malformed line reports its number and stops:

$ printf '{"a":1}\noops\n' | mjq -c -l '.'
{"a":1}
mjq: line 2 is not valid JSON: expected value at line 1 column 1

The first line's output appeared before the failure, which is the behavior you want from a streaming tool: results for good lines are not held hostage by a bad one later. The exit status was 2. For truly unbounded streams, replace read_to_string with a BufReader and lines() so the file is never fully in memory. That is the version to ship if you point this at a log that grows forever.

How fast is it?

A tool like this should be measured against the real thing. The test inputs were 300,000 generated user records: a 25.7 MB single document and the same records as 25.4 MB of newline-delimited JSON. The release build of mjq ran on an Apple silicon laptop, and jq is the version installed by the system package manager. Times are wall-clock seconds from time.

All times are the best of five runs, in wall-clock seconds, for the release build:

• .team (parse the document, read one field): mjq 0.20 s.
• .users | length: mjq 0.34 s, jq 1.6 0.51 s.
• .users[] | select(.role == "ops") | .name, 59,603 matches: mjq 0.57 s, jq 1.6 0.62 s.
• -l 'select(.age > 60) | .id' over the newline-delimited file, 56,657 matches: mjq 0.26 s, one run.

Three observations come out of it. Parsing dominates: just reading the document and returning one field takes 0.20 s, so more than half of the length query's 0.34 s is serde_json building the tree. The remaining 0.14 s is the cost of a design shortcut named earlier: Field clones the whole array so the next stage can own it. On the filtering query, mjq already matches jq 1.6, which is a respectable result for 300 lines of code and no tuning, since both spend most of their time in the parse. And the streaming mode is fast because each line is a small allocation that is dropped before the next, which is friendly to the cache.

Removing the clones with Cow

The shortcut is easy to remove, and removing it is a good lesson in how Rust expresses "borrow when you can, own when you must". The evaluator returns Vec<Value>, so every result is owned. Most filters do not need to own anything: Field, Index, Iterate and Identity return parts of the input, and Select returns the input itself. Only Length, Keys and Map build new values. The type that models "either a borrow or an owned value" is Cow:

pub fn run<'a>(filter: &Filter, input: &'a Value) -> Result<Vec<Cow<'a, Value>>, String> {
    match filter {
        Filter::Identity => Ok(vec![Cow::Borrowed(input)]),
        Filter::Field(name) => match input {
            Value::Object(map) => Ok(vec![Cow::Borrowed(map.get(name).unwrap_or(&NULL))]),
            Value::Null => Ok(vec![Cow::Borrowed(&NULL)]),
            other => Err(format!("cannot read field {name:?} of {}", kind(other))),
        },
        Filter::Iterate => match input {
            Value::Array(items) => Ok(items.iter().map(Cow::Borrowed).collect()),
            Value::Object(map) => Ok(map.values().map(Cow::Borrowed).collect()),
            other => Err(format!("cannot iterate over {}", kind(other))),
        },
        Filter::Length => Ok(vec![Cow::Owned(match input {
            Value::Array(a) => a.len().into(),
            Value::String(s) => s.chars().count().into(),
            _ => 0.into(),
        })]),
        // ... Index, Keys, Select and Map follow the same pattern
        Filter::Pipe(left, right) => {
            let mut out = Vec::new();
            for mid in run(left, input)? {
                match mid {
                    Cow::Borrowed(r) => out.extend(run(right, r)?),
                    Cow::Owned(v) => out.extend(run(right, &v)?.into_iter().map(|c| Cow::Owned(c.into_owned()))),
                }
            }
            Ok(out)
        }
    }
}

Two details carry the design. The lifetime 'a on the input and on the Cow says that borrowed results live as long as the input document, which the compiler checks. A missing field returns a borrow of a static NULL, so even "not found" allocates nothing. And Pipe shows the cost of mixing: if the left stage returned a borrow, the right stage keeps borrowing from the same input; if it returned an owned value, the right stage's results might point into that value, which dies at the end of the arm, so they are copied out with into_owned. The compiler insists on that copy; without it the borrow would outlive its owner, the E0597 family of errors from E0597. Five tests passed unchanged apart from converting results with into_owned in the helper.

The measured effect on the 25.7 MB file, best of five:

• .users | length: 0.34 s before, 0.21 s after, which is the parse cost alone.
• .users[] | select(.role == "ops") | .name: 0.57 s before, 0.29 s after, about twice as fast as jq 1.6 (0.62 s).

Removing the clones took the length query to the floor set by parsing and cut the filter query nearly in half. The code change was one function. The lesson is the order of work: write the simple, cloning evaluator first, get the tests green, measure, and only then thread lifetimes through, with the tests as your safety net.

Tests

The evaluator and parser are pure functions, which makes them cheap to test without a binary:

#[test]
fn iteration_yields_many_outputs() {
    let v = json!([1, 2, 3]);
    assert_eq!(q(".[]", &v), [json!(1), json!(2), json!(3)]);
}

#[test]
fn select_filters_and_map_collects() {
    let v = json!([{"k": 1}, {"k": 2}, {"k": 1}]);
    assert_eq!(q(".[] | select(.k == 1)", &v).len(), 2);
    assert_eq!(q("map(.k)", &v), [json!([1, 2, 1])]);
}

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Four tests cover paths and negative indexes, iteration, select and map, and the error cases. The json! macro builds Values inline, which keeps the test data next to the assertions. Add a test every time you find a bug by hand, and keep the case small.

What this version leaves out, and the costs

It clones aggressively. Each stage copies its inputs into Vec<Value>, which is fine for documents of a few megabytes and wasteful for a 2 GB file. A faster design returns references (Vec<&Value>) for read-only filters and clones only when a filter must construct a new value, using Cow<Value> to keep both cases in one type. It parses the whole document into memory, where a streaming parser would handle files larger than RAM; the serde_json streaming deserializer reads a sequence of values from a reader, which is how line-delimited JSON is processed. And its language is small: no arithmetic, no if, no variables, no // alternatives, no .. recursive descent, no string interpolation. Each of those is a new Filter variant and a few lines in run, which is a pleasant extension path: add + and -, then comparison operators beyond ==, then if, and you have the core of jq.

Where the real jq is the better tool, use it. It has hundreds of builtins, handles streaming and has been tested against every odd corner of JSON. Build your own when you want a query language embedded in your program, a deliberately small and safe subset for users to write in a config file, or the education. The same recursive-descent structure powers the calculator in ten Rust project ideas, and the Result-returning style is the one explained in error handling in Rust.

Where to take it next

Pick extensions by the Rust concept they teach. A --sort-keys flag teaches BTreeMap versus IndexMap and the preserve_order feature. A sort_by(.field) builtin teaches closures, sort_by and stable ordering of JSON values. An --arg name=value option with $name variables teaches environments threaded through the evaluator, which means passing a context struct alongside input. A real tokenizer in front of the parser teaches how to report error positions precisely. And a --stream mode built on a BufReader and serde_json::Deserializer::into_iter teaches iterator-based pipelines that never hold more than one value. Each is an afternoon of work, each ships with its own tests, and together they turn the toy into something you would use. The next project in the same family is a Markdown converter, which swaps JSON for a text format and the filter tree for an event stream.