Async Networking in Rust With Tokio TCP Listener and Stream
The core of every network service is an accept loop that spawns a task per connection. This walks through an echo server and a line protocol with Tokio, then adds limits.
Every network service in Rust starts with the same loop: accept a connection, hand it to a task, go back to accepting. In the benchmark near the end of this post, 2,000 concurrent clients each did 50 round trips against that loop and the server process peaked at 3.8 MB of resident memory. The same job done with one operating system thread per connection peaked at 34.3 MB and was not meaningfully slower. That gap, and what it does and does not buy you, is the reason the loop exists.
This post builds it with Tokio: an echo server, a proof that TCP does not preserve your messages, a line protocol with limits on size and silence, a cap on connections, a graceful shutdown, and the measurement. Everything compiles and runs on Rust 1.99 with edition 2024 and Tokio 1.54.1. Each program is self-contained: it binds to port 0, so the operating system picks a free port, and it includes its own client, so you can run it and see the output without opening a second terminal. The dependency line is the only configuration:
[package]
name = "tcp"
version = "0.1.0"
edition = "2024"
[dependencies]
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "io-util", "time", "sync"] }An echo server, and what read returning zero means
The smallest useful server accepts connections and sends back whatever arrives:
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, TcpStream};
async fn echo(mut stream: TcpStream) -> std::io::Result<()> {
let mut buf = [0u8; 1024];
loop {
let n = stream.read(&mut buf).await?;
if n == 0 {
return Ok(());
}
stream.write_all(&buf[..n]).await?;
}
}
#[tokio::main]
async fn main() -> std::io::Result<()> {
let listener = TcpListener::bind("127.0.0.1:0").await?;
let addr = listener.local_addr()?;
tokio::spawn(async move {
loop {
let Ok((stream, peer)) = listener.accept().await else {
continue;
};
tokio::spawn(async move {
if let Err(e) = echo(stream).await {
eprintln!("{peer}: {e}");
}
});
}
});
let mut client = TcpStream::connect(addr).await?;
client.write_all(b"hello rust\n").await?;
let mut reply = [0u8; 11];
client.read_exact(&mut reply).await?;
println!("echoed {:?}", String::from_utf8_lossy(&reply));
client.shutdown().await?;
let mut rest = Vec::new();
let n = client.read_to_end(&mut rest).await?;
println!("after we closed our side, the server sent {n} more bytes and closed too");
Ok(())
}echoed "hello rust\n"
after we closed our side, the server sent 0 more bytes and closed tooTcpListener::bind takes an address and returns a listener; 127.0.0.1:0 asks the kernel for any free port and local_addr() tells us which one it chose. The accept loop is the shape you will see in every Tokio server. accept().await yields a TcpStream and the peer's address, and tokio::spawn gives that connection its own task, so a slow client holds up nothing but itself. The async move block takes ownership of the stream, which matters because a spawned task must be 'static and cannot borrow from the loop.
Inside echo, stream.read(&mut buf) returns the number of bytes that arrived, and **a return of 0 means the peer closed its side**. It is not an error and not an empty message; it is end of stream, and the loop must exit on it or it will spin. The error handling is placed with care: echo returns an io::Result, the task prints a failure and ends, and the accept loop never sees it. One client that resets its connection must not stop the listener.
The test client at the bottom does something worth copying. It writes eleven bytes, reads exactly eleven back with read_exact, and then calls shutdown(), which sends a FIN on the write half. The server's next read returns 0, echo returns, the stream drops and the connection closes, so the client's read_to_end returns with nothing more to say. That closing handshake is what lets a client and a server agree the conversation is over without either of them guessing from a timeout.
One honest flaw in this version: let Ok(...) = ... else { continue } swallows an accept error and tries again immediately. If the error is persistent, say the process has run out of file descriptors, the loop spins at 100 percent of a core. The line server below sleeps for 100 ms on an accept error instead, which is the minimum a real server needs.
A read is not a message
The echo server looks correct and teaches a bad habit. The mistake is to treat each read as one message, and it works in every test you write on localhost because small writes usually arrive whole. Here is a client that sends the line ping in two pieces, 50 ms apart, against a naive server and a framed one:
use std::time::Duration;
use tokio::io::{AsyncBufReadExt, AsyncReadExt, AsyncWriteExt, BufReader};
use tokio::net::{TcpListener, TcpStream};
use tokio::time::sleep;
// Treats every read as one message. This is the bug.
async fn naive(mut stream: TcpStream) -> std::io::Result<()> {
let mut buf = [0u8; 64];
loop {
let n = stream.read(&mut buf).await?;
if n == 0 {
return Ok(());
}
println!(
"naive server read {n} bytes: {:?}",
String::from_utf8_lossy(&buf[..n])
);
}
}
// Reads until the newline, however many reads that takes.
async fn framed(stream: TcpStream) -> std::io::Result<()> {
let mut lines = BufReader::new(stream).lines();
while let Some(line) = lines.next_line().await? {
println!("framed server got one line: {line:?}");
}
Ok(())
}
async fn send_in_two_pieces(addr: std::net::SocketAddr) -> std::io::Result<()> {
let mut client = TcpStream::connect(addr).await?;
client.set_nodelay(true)?;
client.write_all(b"pi").await?;
sleep(Duration::from_millis(50)).await;
client.write_all(b"ng\n").await?;
client.shutdown().await
}
async fn serve_once<F, Fut>(handler: F) -> std::io::Result<()>
where
F: FnOnce(TcpStream) -> Fut + Send + 'static,
Fut: Future<Output = std::io::Result<()>> + Send + 'static,
{
let listener = TcpListener::bind("127.0.0.1:0").await?;
let addr = listener.local_addr()?;
let server = tokio::spawn(async move {
let (stream, _) = listener.accept().await?;
handler(stream).await
});
send_in_two_pieces(addr).await?;
server.await.expect("server task")
}
#[tokio::main]
async fn main() -> std::io::Result<()> {
serve_once(naive).await?;
serve_once(framed).await
}naive server read 2 bytes: "pi"
naive server read 3 bytes: "ng\n"
framed server got one line: "ping"TCP delivers a stream of bytes. The sender's calls to write are not preserved, and the receiver's read returns whatever has arrived at that moment, which can be less than one message, exactly one, or three and a half. The naive server saw pi and then ng with its newline, and would have tried to interpret two garbage commands. The framed server asked a different question: read until you have a newline. BufReader::new(stream).lines() buffers whatever arrives, finds the delimiter, and yields one complete line no matter how many reads it took. The client sets set_nodelay(true) so that the first two bytes really leave the machine on their own; without it, the kernel's Nagle algorithm may coalesce small writes, which is a reminder that you cannot rely on either splitting or joining.
So a protocol over TCP has to say where a message ends. There are three common answers: a delimiter (newline, as here), a length prefix (four bytes saying how many bytes follow), or a fixed size (the benchmark below uses 64-byte frames and read_exact). For binary protocols the tokio-util crate ships LengthDelimitedCodec, which does the length-prefix version and plugs into a Framed stream. We stay with newlines because the framing is visible in the code.
Two compile errors you will meet
Both of these mistakes are made by nearly everyone who writes their first Tokio server, and the compiler's messages are better than they look.
The first is calling read without importing the trait that provides it:
use tokio::net::TcpStream;
#[tokio::main]
async fn main() -> std::io::Result<()> {
let mut stream = TcpStream::connect("127.0.0.1:7000").await?;
let mut buf = [0u8; 1024];
let n = stream.read(&mut buf).await?;
println!("{n}");
Ok(())
}error[E0599]: no method named `read` found for struct `tokio::net::TcpStream` in the current scope
--> examples/fail_import.rs:7:20
|
7 | let n = stream.read(&mut buf).await?;
| ^^^^
|
::: ~/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54.1/src/io/util/async_read_ext.rs:177:12
|
177 | fn read<'a>(&'a mut self, buf: &'a mut [u8]) -> Read<'a, Self>
| ---- the method is available for `tokio::net::TcpStream` here
|
= help: items from traits can only be used if the trait is in scope
help: trait `AsyncReadExt` which provides `read` is implemented but not in scope; perhaps you want to import it
|
1 + use tokio::io::AsyncReadExt;
|
help: there is a method `ready` with a similar name
|
7 | let n = stream.ready(&mut buf).await?;
| +
For more information about this error, try `rustc --explain E0599`.
error: could not compile `tcp` (example "fail_import") due to 1 previous errorTcpStream does not have an inherent read method. It gets it from AsyncReadExt, an extension trait, and Rust only lets you call a trait's methods when the trait is in scope. The error even spells out the fix: add use tokio::io::AsyncReadExt;. The same applies to write_all (AsyncWriteExt) and read_line (AsyncBufReadExt), so a Tokio file usually starts with a use tokio::io::{...} line that lists two or three of them. Ignore the suggestion about ready: it is a different method that happens to be spelled similarly.
The second is trying to use one stream from two tasks:
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
async fn read_loop(mut stream: TcpStream) {
let mut buf = [0u8; 64];
while let Ok(n) = stream.read(&mut buf).await {
if n == 0 {
break;
}
}
}
async fn heartbeat(mut stream: TcpStream) {
let _ = stream.write_all(b"ping\n").await;
}
#[tokio::main]
async fn main() -> std::io::Result<()> {
let stream = TcpStream::connect("127.0.0.1:7000").await?;
tokio::spawn(read_loop(stream));
tokio::spawn(heartbeat(stream));
Ok(())
}error[E0382]: use of moved value: `stream`
--> examples/fail_move.rs:21:28
|
19 | let stream = TcpStream::connect("127.0.0.1:7000").await?;
| ------ move occurs because `stream` has type `tokio::net::TcpStream`, which does not implement the `Copy` trait
20 | tokio::spawn(read_loop(stream));
| ------ value moved here
21 | tokio::spawn(heartbeat(stream));
| ^^^^^^ value used here after move
|
note: consider changing this parameter type in function `read_loop` to borrow instead if owning the value isn't necessary
--> examples/fail_move.rs:4:32
|
4 | async fn read_loop(mut stream: TcpStream) {
| --------- ^^^^^^^^^ this parameter takes ownership of the value
| |
| in this function
For more information about this error, try `rustc --explain E0382`.
error: could not compile `tcp` (example "fail_move") due to 1 previous errortokio::spawn(read_loop(stream)) moves the stream into the first task, so the second spawn has nothing left to move. A TcpStream is not Clone, because two owners reading from one socket would steal each other's bytes. The fix is stream.into_split(), which returns an OwnedReadHalf and an OwnedWriteHalf, each of which can move into its own task. The line server below uses it. There is also a borrowed split() that costs nothing but keeps both halves in one task; reach for it when reading and writing happen in the same function, and for into_split when they have to live in different tasks.
A line protocol with limits on size and silence
Now a server that parses commands, and defends itself against the two cheapest attacks on a text protocol: a line that never ends, and a client that connects and says nothing.
use std::time::Duration;
use tokio::io::{AsyncBufReadExt, AsyncReadExt, AsyncWriteExt, BufReader};
use tokio::net::{TcpListener, TcpStream};
use tokio::time::{sleep, timeout};
const MAX_LINE: u64 = 1024;
const IDLE: Duration = Duration::from_millis(300);
async fn serve(stream: TcpStream) -> std::io::Result<()> {
let (read_half, mut write_half) = stream.into_split();
let mut reader = BufReader::new(read_half);
let mut line = String::new();
loop {
line.clear();
// take() caps this one line; a client cannot make us buffer more.
let mut one_line = (&mut reader).take(MAX_LINE);
let n = match timeout(IDLE, one_line.read_line(&mut line)).await {
Ok(result) => result?,
Err(_) => {
write_half.write_all(b"error: idle too long\n").await?;
return Ok(());
}
};
if n == 0 {
return Ok(());
}
if !line.ends_with('\n') {
write_half.write_all(b"error: line too long\n").await?;
return Ok(());
}
let reply = match line.trim() {
"ping" => "pong".to_string(),
other if other.starts_with("echo ") => other[5..].to_string(),
_ => "unknown command".to_string(),
};
write_half.write_all(reply.as_bytes()).await?;
write_half.write_all(b"\n").await?;
}
}
async fn accept_loop(listener: TcpListener) {
loop {
match listener.accept().await {
Ok((stream, peer)) => {
tokio::spawn(async move {
if let Err(e) = serve(stream).await {
eprintln!("{peer}: {e}");
}
});
}
Err(e) => {
// EMFILE and friends: back off instead of spinning or dying.
eprintln!("accept failed: {e}");
sleep(Duration::from_millis(100)).await;
}
}
}
}
async fn ask(addr: std::net::SocketAddr, payload: &[u8]) -> std::io::Result<String> {
let mut client = TcpStream::connect(addr).await?;
client.write_all(payload).await?;
let mut reply = String::new();
BufReader::new(client).read_line(&mut reply).await?;
Ok(reply.trim_end().to_string())
}
#[tokio::main]
async fn main() -> std::io::Result<()> {
let listener = TcpListener::bind("127.0.0.1:0").await?;
let addr = listener.local_addr()?;
tokio::spawn(accept_loop(listener));
println!("ping -> {}", ask(addr, b"ping\n").await?);
println!(
"echo hi there -> {}",
ask(addr, b"echo hi there\n").await?
);
println!("dance -> {}", ask(addr, b"dance\n").await?);
println!(
"10,000 bytes, no newline -> {}",
ask(addr, &[b'a'; 10_000]).await?
);
// Connect and say nothing.
let mut quiet = TcpStream::connect(addr).await?;
let mut reply = String::new();
BufReader::new(&mut quiet).read_line(&mut reply).await?;
println!("silent client -> {}", reply.trim_end());
Ok(())
}ping -> pong
echo hi there -> hi there
dance -> unknown command
10,000 bytes, no newline -> error: line too long
silent client -> error: idle too longThe serve function is the part to study. into_split gives us the halves, and BufReader wraps the read half so we do not make a system call per byte. Then the central line: (&mut reader).take(MAX_LINE) limits the reader to 1,024 bytes for this one line, and read_line appends into a String we reuse. This is deliberate. The convenient lines() adapter from the previous section has no length limit: a client that streams gigabytes without a newline makes the server buffer all of it. With take, a line that reaches the limit comes back without a trailing newline, and the newline check (ends_with) turns that into an error reply and a closed connection. The output confirms it: ten thousand a bytes get error: line too long.
A caveat on that reply: when a server closes a socket that still has unread data in its receive buffer, some operating systems send a reset instead of a clean close, and the client may see "connection reset" in place of our message. The error text is a courtesy. The close is the protection.
The second defense is timeout(IDLE, one_line.read_line(&mut line)). It wraps each read, not the whole connection, so a client that keeps talking is never cut off, while one that goes quiet for 300 ms (a demo value; production servers use 30 seconds or more) gets error: idle too long and is closed. A timeout around the entire serve call would measure something else, the total lifetime of a connection, and kill a healthy long-lived client. Both exist and you choose by the question: "how long may a connection be silent" is a per-read timeout, "how long may a connection exist" is a per-connection one. Tokio timeouts covers how timeout cancels the future it wraps and what that means for partially read data.
The accept_loop function has the fix promised earlier. A failed accept is logged and followed by a 100 ms sleep, and the loop continues. It does not use ?, because returning from the loop would stop the server for every client over a problem that is usually temporary.
The command handling itself is intentionally trivial, three arms in a match. The reply is built as a String and written with two write_all calls. In a hot path you would build the line once with its newline and write it in one call, because two small writes can become two packets.
Cap the number of connections before you accept
An accept loop with no limit has a hidden failure mode: under a connection flood it happily accepts until the process runs out of file descriptors, and then every accept fails, including for well-behaved clients. A Semaphore is the right tool, and the interesting question is where to take the permit.
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::{TcpListener, TcpStream};
use tokio::sync::Semaphore;
use tokio::time::sleep;
async fn serve(stream: TcpStream) -> std::io::Result<()> {
let (read_half, mut write_half) = stream.into_split();
let mut request = String::new();
BufReader::new(read_half).read_line(&mut request).await?;
sleep(Duration::from_millis(100)).await; // pretend this is real work
write_half.write_all(b"done\n").await
}
async fn accept_loop(listener: TcpListener, permits: Arc<Semaphore>) {
loop {
// The permit comes first, so a full server stops calling accept().
let permit = Arc::clone(&permits)
.acquire_owned()
.await
.expect("never closed");
let Ok((stream, _)) = listener.accept().await else {
continue;
};
tokio::spawn(async move {
let _permit = permit;
let _ = serve(stream).await;
});
}
}
async fn client(addr: std::net::SocketAddr, started: Instant) -> (Duration, Duration) {
let mut stream = TcpStream::connect(addr).await.expect("connect");
let connected = started.elapsed();
stream.write_all(b"work\n").await.expect("write");
let mut reply = String::new();
BufReader::new(stream)
.read_line(&mut reply)
.await
.expect("read");
(connected, started.elapsed())
}
#[tokio::main]
async fn main() -> std::io::Result<()> {
let listener = TcpListener::bind("127.0.0.1:0").await?;
let addr = listener.local_addr()?;
tokio::spawn(accept_loop(listener, Arc::new(Semaphore::new(2))));
let started = Instant::now();
let clients: Vec<_> = (1..=5)
.map(|_| tokio::spawn(client(addr, started)))
.collect();
let mut results = Vec::new();
for c in clients {
results.push(c.await.expect("client task"));
}
results.sort_by_key(|(_, replied)| *replied);
for (i, (connected, replied)) in results.iter().enumerate() {
println!(
"client {}: connect() returned after {:>3} ms, reply after {:>3} ms",
i + 1,
connected.as_millis(),
replied.as_millis()
);
}
Ok(())
}client 1: connect() returned after 0 ms, reply after 101 ms
client 2: connect() returned after 0 ms, reply after 101 ms
client 3: connect() returned after 0 ms, reply after 204 ms
client 4: connect() returned after 0 ms, reply after 204 ms
client 5: connect() returned after 0 ms, reply after 308 msThe loop acquires a permit before it calls accept. The server is limited to two simultaneous connections and each one takes 100 ms of "work", and five clients arrive at once. Look at the numbers. Every connect() returned in 0 ms, because the kernel completes the TCP handshake for a connection and parks it in the listen backlog whether or not the application has called accept yet. The replies came back at 101, 101, 204, 204 and 308 ms: two clients at a time, in order, as permits were released. Clients 3 to 5 spent their wait inside the kernel's queue, which costs the server one file descriptor and a few kilobytes of kernel memory per waiting connection but no task, no buffers of ours and no String.
If the permit were taken after accept instead, the server would pull all five connections out of the backlog, spawn five tasks and make four of them wait on the semaphore. The behavior looks the same to a client, but the process now holds a file descriptor and a task for every waiter, so the limit protects nothing. Taking the permit first moves the wait to the cheapest place available.
The backlog is finite. When it fills, the kernel drops new SYN packets, and clients see slow connects and retries instead of a refusal. This is acceptable for a brief spike and a reason to keep the limit above your real concurrent demand.
To pick the limit, work from file descriptors, because they run out before memory does. Each connection uses one descriptor, and the process needs a reserve for the listener, logs and outbound calls. With a soft limit of 1,024 (a common default), a limit of about 900 leaves a margin; with ulimit -n 65536 the descriptor budget stops being the constraint and your per-connection memory and CPU take over. Our benchmark below raised the limit to 20,000 for that reason. Note that the 3.8 MB figure there counts only the process's own memory: the kernel's socket buffers are separate and can add up to far more with large windows or slow readers.
Stop accepting, then drain
A deploy sends SIGTERM, and what the server does in the next few seconds decides whether a client sees a clean "goodbye" or a reset in the middle of a response. The pattern has three steps: stop accepting, let in-flight work finish, then exit.
use std::time::{Duration, Instant};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::{TcpListener, TcpStream};
use tokio::sync::watch;
use tokio::task::JoinSet;
use tokio::time::{sleep, timeout};
async fn serve(stream: TcpStream, mut shutdown: watch::Receiver<bool>) -> std::io::Result<()> {
let (read_half, mut write_half) = stream.into_split();
let mut lines = BufReader::new(read_half).lines();
loop {
tokio::select! {
line = lines.next_line() => {
let Some(line) = line? else { return Ok(()) };
if line == "slow" {
sleep(Duration::from_millis(200)).await;
}
write_half.write_all(b"done\n").await?;
}
_ = shutdown.changed() => {
write_half.write_all(b"bye\n").await?;
return Ok(());
}
}
}
}
async fn accept_loop(listener: TcpListener, mut shutdown: watch::Receiver<bool>) {
let mut connections = JoinSet::new();
loop {
tokio::select! {
accepted = listener.accept() => {
if let Ok((stream, _)) = accepted {
connections.spawn(serve(stream, shutdown.clone()));
}
}
_ = shutdown.changed() => break,
Some(_) = connections.join_next(), if !connections.is_empty() => {}
}
}
drop(listener); // new connections are refused from here on
let drained = timeout(Duration::from_secs(5), async {
while connections.join_next().await.is_some() {}
})
.await;
println!("server: drained={}", drained.is_ok());
}
#[tokio::main]
async fn main() -> std::io::Result<()> {
let listener = TcpListener::bind("127.0.0.1:0").await?;
let addr = listener.local_addr()?;
let (stop, shutdown) = watch::channel(false);
let server = tokio::spawn(accept_loop(listener, shutdown));
let started = Instant::now();
let mut client = TcpStream::connect(addr).await?;
client.write_all(b"slow\n").await?;
sleep(Duration::from_millis(50)).await;
stop.send(true).expect("server is listening");
println!(
"{:>3} ms: shutdown signalled while 'slow' is in flight",
started.elapsed().as_millis()
);
sleep(Duration::from_millis(20)).await;
match TcpStream::connect(addr).await {
Ok(_) => println!(
"{:>3} ms: a new client connected",
started.elapsed().as_millis()
),
Err(e) => println!(
"{:>3} ms: a new client was refused: {e}",
started.elapsed().as_millis()
),
}
let mut lines = BufReader::new(client).lines();
while let Some(line) = lines.next_line().await? {
println!(
"{:>3} ms: first client received {line:?}",
started.elapsed().as_millis()
);
}
server.await.expect("server task");
println!(
"{:>3} ms: server task finished",
started.elapsed().as_millis()
);
Ok(())
} 51 ms: shutdown signalled while 'slow' is in flight
73 ms: a new client was refused: Connection refused (os error 61)
server: drained=true
204 ms: first client received "done"
204 ms: first client received "bye"
204 ms: server task finishedThe shutdown signal is a watch channel, which every connection task can hold a receiver of and which remembers its value, so a task that checks late still sees that shutdown happened. The accept loop uses tokio::select! over three branches: a new connection, the shutdown signal, and a finished task from the JoinSet. The last one is guarded with if !connections.is_empty(), because join_next() on an empty set returns None at once and would turn the select into a busy loop. Reaping finished tasks as we go also keeps the set from growing without bound.
On shutdown the loop breaks and drop(listener) closes the listening socket, so new clients are refused immediately, which is the signal a load balancer's health check needs. The timeline shows it: the signal at 51 ms, a new client refused with Connection refused at 73 ms, while the first client's slow command, which takes 200 ms, is allowed to complete. It received done at 204 ms and then bye, and the server task finished in the same millisecond. Each connection's serve selects between the next line and the shutdown signal only at the point where it is waiting for input, so work that has started is never cut in half: a command is either not yet read or answered in full.
next_line is safe to drop in a select! because it keeps any partial line in its buffer. That property is not universal, and [cancellation safety in select!](/library/tokio-select-cancellation-safety) is worth reading about before you put a different method in one of those branches. The drain is wrapped in a five second timeout as the safety net: a client that never finishes should delay the exit, not prevent it. Signal handling and the wider pattern are in graceful shutdown for Tokio services.
The measurement: tasks against threads
The claim in the first paragraph needs evidence. The programs are a Tokio echo server of 64-byte frames (server.rs), the same server with one std::thread per connection (threads.rs) and a load generator that opens N connections, waits until all are up, and then has each do 50 write-then-read round trips. All three run as separate processes on the same machine, an Apple M1 Max with 10 cores, built with --release. This is the Tokio server:
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, TcpStream};
use tokio::runtime::Builder;
const FRAME: usize = 64;
async fn echo_frames(mut stream: TcpStream) -> std::io::Result<()> {
let mut frame = [0u8; FRAME];
loop {
match stream.read_exact(&mut frame).await {
Ok(_) => stream.write_all(&frame).await?,
Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => return Ok(()),
Err(e) => return Err(e),
}
}
}
fn main() -> std::io::Result<()> {
let workers: usize = std::env::args()
.nth(1)
.and_then(|w| w.parse().ok())
.unwrap_or(0);
let runtime = if workers == 1 {
Builder::new_current_thread().enable_all().build()?
} else if workers > 1 {
Builder::new_multi_thread()
.worker_threads(workers)
.enable_all()
.build()?
} else {
Builder::new_multi_thread().enable_all().build()?
};
runtime.block_on(async {
let listener = TcpListener::bind("127.0.0.1:7100").await?;
loop {
let (stream, _) = listener.accept().await?;
tokio::spawn(echo_frames(stream));
}
})
}The first command-line argument picks the runtime: 1 builds a current-thread runtime, a larger number sets the worker count, and the default is one worker per core. These are the numbers from the second of two runs:
| Server | Connections | Round trips/s | p50 | p99 |
|---|---|---|---|---|
| Tokio, multi-thread (10 workers) | 100 | 96,082 | 767 us | 2,607 us |
| Tokio, multi-thread (10 workers) | 2,000 | 121,780 | 14,577 us | 60,516 us |
| Tokio, current-thread | 100 | 95,673 | 996 us | 1,657 us |
| Tokio, current-thread | 2,000 | 100,872 | 18,667 us | 41,350 us |
| Thread per connection | 100 | 85,658 | 1,127 us | 1,599 us |
| Thread per connection | 2,000 | 107,133 | 17,308 us | 35,971 us |
Peak resident memory of the server process, sampled every 100 ms across both runs of each server: 3.8 MB for multi-thread Tokio, 3.5 MB for current-thread Tokio and 34.3 MB for a thread per connection.
Read it for what it is. On throughput the three servers are within noise of each other: multi-thread Tokio at 2,000 connections did 104,307 round trips per second in the first run and 121,780 in the second, a swing larger than the gap between any two servers, and the load generator is a Tokio program on the same ten cores, so it is probably the bottleneck. Latency is not a clean story either. A connection at 2,000 clients waits behind about 2,000 others, so the median is ten milliseconds or more for everybody.
The number that did not move between runs is memory: 3.5 to 3.8 MB for the tasks against 33.5 to 34.3 MB for the threads, roughly a ten times difference at 2,000 connections. Each thread has its own stack, and each Tokio task is a few hundred bytes of state. That is the entire argument for async networking at this scale. It is not faster at 2,000 connections, it is lighter, and the difference grows linearly with the connection count. Whether you care depends on your count. We did not measure 50,000 connections here; macOS caps threads per process well below that, and a Linux box would give different limits. What the lab can tell you is that for a few hundred connections a plain thread-per-connection server is a fine design, and that its cost shows up in memory long before it shows up in speed.
The same table says that a single-threaded runtime handled 100,872 round trips per second, within a few percent of ten workers. If your connections spend their time waiting on the network, one thread is often enough; the multi-thread runtime earns its keep when handlers do real computation. The server's own worker count is worth measuring against your handler and not choosing by habit, and tuning the number of Tokio workers shows how.
Where this loop is the wrong level
Do not write an HTTP server this way. HTTP framing, keep-alive, header limits, chunked bodies and HTTP/2 are a long list of things the loop above does not do, and Hyper without a framework shows the layer that does them, with Axum on top for routing. The accept loop is the foundation under those libraries, and knowing it makes their configuration pages readable: a connection limit, a header read timeout and a graceful shutdown are the same three ideas wearing different names.
It also does not cover encryption (TLS needs a crate such as tokio-rustls wrapped around the stream), datagrams (UDP has message boundaries and no connections, so UdpSocket is a different program) or a server that must push to many clients at once. For that last one, a chat server with broadcast channels is the natural next step, and it brings the question this post left alone: a slow reader on the other end makes your outgoing buffer grow, and something has to decide when to give up on it.
