Yun Sheng's Site

New Domain (Finally)

Thu, 27 Aug 2026 12:31:50 -1000

New Domain Purchased at Hawaii

Ellison Onizuka Kona International Airport

https://shengy.dev will be the new domain, the previous https://www.luckieboutique.com/ would be deprecated Nov 05 2027

async await in asyncio

Sun, 02 Aug 2026 12:30:39 -0400

tldr

An await may suspend the current task, allowing the event loop to run other ready work on the same thread.
async/await within a single thread can still have race conditions
In general, async/await is cooperative scheduling, this means tasks/jobs needs to voluntarily “give up” control
async/await depends on the “execution environment”, the execution environment could be single threaded or multi threaded.
python’s asyncio tasks could be garbage collected, so hold a reference to it

Overview

Since the JavaScript days I don’t think I ever understood async/await to the point where I’d feel comfortable.

We recently got a bug in our code and it was related to async/await usage in python’s asyncio.

I’m also trying to learn tokio to be able to try out Framed

All the above resulted into me vibing with Gemini and Claude again to go over this topic.

What is async/await about?

Instead of thinking async/await as language keywords, I think it’s best to understand them as a programming pattern.

Say I want to do two things this weekend, drink coffee and play games. However I don’t have coffee at home, so I want my kid to go to Starbucks and get me a cup of coffee. I could do one of the following.

Wait at the door until he comes back with my coffee
Before he goes to get coffee, tell him that I’m going to play games and he could shove coffee in my face when he gets it back
Me and my kid work with my wife, she would help coordinate

It’s not hard to tell that (1) above is blocking. (2) is the good old callback function and (3) is async/await where the scheduler is my wife.

Let’s write some code for the above.

The case where I wait for my coffee

import time
from datetime import datetime


def print_with_time(x):
    now = datetime.now()
    print(now, x)


def send_kid_to_get_coffee():
    print_with_time("kid goes to get coffee")
    time.sleep(5)
    print_with_time("kid got coffee after 5 seconds")
    return "coffee"


def dad_plays_games():
    print_with_time("dad starting to play games")
    time.sleep(3)
    print_with_time("dad played for 3 seconds")
    time.sleep(7)
    print_with_time("dad played for 7 seconds")

    print_with_time("dad done playing games")


coffee = send_kid_to_get_coffee()
print_with_time('dad drinks ' + coffee)
dad_plays_games()

The above is an abstraction of scenario (1), I send kid to get coffee but I wait at the door for 5 seconds. I don’t get to play video games until my kid gets back with my coffee.

Running the above would result the following output

shengy@Mac ~/tmp/yo [22:30:09]
> python3 block.py
2026-08-02 22:30:13.506923 kid goes to get coffee
2026-08-02 22:30:18.511882 kid got coffee after 5 seconds
2026-08-02 22:30:18.511983 dad drinks coffee
2026-08-02 22:30:18.511993 dad starting to play games
2026-08-02 22:30:21.515760 dad played for 3 seconds
2026-08-02 22:30:28.520653 dad played for 7 seconds
2026-08-02 22:30:28.520798 dad done playing games

This obviously is a waste of my time, so how about we try out (2). The main idea of (2) is that I do not need to wait for my kid to get my coffee back. Before I sent him to get my coffee, I gave him a “callback” to pass the coffee into when he gets it.

The case where I use a worker thread and callback

Since time.sleep() blocks the thread that calls it, I run send_kid_to_get_coffee() on a separate thread so it can proceed concurrently with dad_plays_games().

import time
import threading
from datetime import datetime


def print_with_time(x):
    now = datetime.now()
    print(now, x)


def send_kid_to_get_coffee(cb):
    print_with_time("kid goes to get coffee")
    time.sleep(5)
    print_with_time("kid got coffee after 5 seconds")
    cb("coffee")


def dad_plays_games():
    print_with_time("dad starting to play games")
    time.sleep(3)
    print_with_time("dad played for 3 seconds")
    time.sleep(7)
    print_with_time("dad played for 7 seconds")

    print_with_time("dad done playing games")


def hand_dad_his_coffee(coffee):
    print_with_time('dad drinks ' + coffee)


kid_thread = threading.Thread(
    target=send_kid_to_get_coffee,
    args=(hand_dad_his_coffee,)
)

kid_thread.start()
dad_plays_games()

kid_thread.join()

This is multi threading in python, I created a new thread to send the kid to get coffee, and provided it a callback function to hand dad the coffee.

shengy@Mac ~/tmp/yo [22:31:50]
> python3 callback.py
2026-08-02 22:32:11.885150 kid goes to get coffee
2026-08-02 22:32:11.885185 dad starting to play games
2026-08-02 22:32:14.890236 dad played for 3 seconds
2026-08-02 22:32:16.890134 kid got coffee after 5 seconds
2026-08-02 22:32:16.890338 dad drinks coffee
2026-08-02 22:32:21.895275 dad played for 7 seconds
2026-08-02 22:32:21.895467 dad done playing games

The case where I rely on a scheduler

Now we are going to look at the scenario where we work with my wife (the scheduler). For the purpose of demonstrating async/await, we would be limiting to one thread. This is what people usually mean when they say JavaScript is single-threaded.

async/await does not need to limit itself to single threads, for instance in Python you could kick off threads inside async functions and Rust’s tokio execution environment can schedule jobs across threads.

import time
import asyncio
from datetime import datetime


def print_with_time(x):
    now = datetime.now()
    print(now, x)


async def send_kid_to_get_coffee():
    print_with_time("kid goes to get coffee")
    await asyncio.sleep(5)
    print_with_time("kid got coffee after 5 seconds")
    return 'coffee'


async def dad_plays_games():
    print_with_time("dad starting to play games")
    await asyncio.sleep(3)
    print_with_time("dad played for 3 seconds")
    await asyncio.sleep(7)
    print_with_time("dad played for 7 seconds")

    print_with_time("dad done playing games")


async def main():
    # kicks off both tasks
    kid_task = asyncio.create_task(send_kid_to_get_coffee())
    dad_task = asyncio.create_task(dad_plays_games())

    # await here yields control so dad task can progress
    coffee = await kid_task

    # here kid_task finished so dad drinks coffee
    print_with_time('dad drinks ' + coffee)

    # wait for dad to finish his gaming session
    await dad_task

asyncio.run(main())

A few important things.

The code snippet above is single threaded
kid_task and dad_task were all added to the event loop at the beginning
Whenever an await was executed the current execution flow can “give up” the control to the scheduler, then the scheduler picks the next place to start.

In python’s asyncio an await does not give up control when the awaited object completes without suspending the current task. For example, directly awaiting a coroutine that returns without reaching a suspension point does not give control back to the event loop. However in JavaScript, await pauses the current async function and schedules its continuation as a microtask.

Let’s try to go through how things happened line by line.

asyncio.run(main()), we created the event loop scheduler and starts running the main function
kid_task = asyncio.create_task(send_kid_to_get_coffee()), creates the task and adds it to the event loop we just created, this task itself is not running yet because the control is still owned by main
dad_task = asyncio.create_task(dad_plays_games()), creates the task and adds it to the event loop we just created, this task itself is not running yet because the control is still owned by main
coffee = await kid_task, control sees the await keyword, tells event loop “I’m pausing here until kid_task is done, take control”
Event loop sees that kid_task should be the next, so it hands kid_task the control
In send_kid_to_get_coffee we print “kid goes to get coffee”, then we hit await asyncio.sleep(5)
Same as above, hitting await asyncio.sleep(5) inside send_kid_to_get_coffee tells the event loop: “I am not doing anything until asyncio.sleep(5) is done, take away my control.
Event loop control and sees dad_task should be next, dad_plays_games gets the control.
dad_plays_games prints “dad starting to play games”
dad_plays_games hits await asyncio.sleep(3), tells event loop “I’m pausing here until asyncio.sleep(3) is done”, takeaway my control.
Event loop takes back control and there isn’t anything going on
After 3 seconds the first asyncio.sleep(3) in dad_plays_games is done, event loop gives dad_task the control, so dad_task prints “dad played for 3 seconds”.
dad_task hits await asyncio.sleep(7), tells event loop “I’m pausing here until asyncio.sleep(7) is done”, take away my control.
Event loop tries to find tasks, await asyncio.sleep(5) in kid_task finished, so event loop gives kid_task the control.
kid_task prints “kid got coffee after 5 seconds” and returns 'coffee'
kid_task was done so await kid_task was done so control was handed back to main, prints “dad drinks coffee”
main says await dad_task, so control was handed back to event loop
After 7 seconds await asyncio.sleep(7) is done so dad_task gets the control, prints “dad played for 7 seconds”
dad_task is done, event loop gives main back control and the whole program is done

shengy@Mac ~/tmp/yo [22:32:21]
> python3 asyncawait.py
2026-08-02 22:32:58.441104 kid goes to get coffee
2026-08-02 22:32:58.441137 dad starting to play games
2026-08-02 22:33:01.442403 dad played for 3 seconds
2026-08-02 22:33:03.442122 kid got coffee after 5 seconds
2026-08-02 22:33:03.442229 dad drinks coffee
2026-08-02 22:33:08.445115 dad played for 7 seconds
2026-08-02 22:33:08.445230 dad done playing games

If there is only one thing to take away, it is that await may suspend the current task and return control to the event loop, allowing other tasks to run.

A bit more about python asyncio

Coroutine and Tasks

Application-level asyncio code mainly works with Coroutines and Tasks, although Futures are the third main awaitable abstraction.

When we call a function that was marked async in python, instead of executing it, it returns a coroutine.

shengy@Mac ~/tmp/yo [21:14:01]
> python3
Python 3.14.6 (main, Jun 10 2026, 10:03:53) [Clang 21.0.0 (clang-2100.0.123.102)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> async def yo():
...     print("yo")
...
>>> yo()
<coroutine object yo at 0x10a45f100>
>>>

A coroutine doesn’t run on its own, calling it just builds the coroutine object, it doesn’t start executing. To actually drive it, you need to either:

await it directly, in which case it runs as part of whoever is currently awaiting it (no new task involved), or
Wrap it in asyncio.create_task(), which registers it with the event loop as its own independent task, so it can run concurrently with other tasks.

import asyncio

async def yo():
    print("yo")

async def main():
    # await a coroutine directly, it runs as part of the main task
    await yo()

asyncio.run(main())

import asyncio

async def yo():
    print("yo")

async def main():
    # creates a new task and runs as an independent task
    task = asyncio.create_task(yo())
    await task

asyncio.run(main())

Note that calling main() also creates a coroutine, and instead of creating a task with it we passed it straight to asyncio.run(), this is how you run the top-level entry point. This is where everything starts.

Garbage Collector Gotcha

In the code snippet above we held a reference to the created task, if we do not do that the code still runs but there is a huge gotcha here.

import asyncio

async def yo():
    print("yo")

async def main():
    # task created here might be garbage collected
    asyncio.create_task(yo())

asyncio.run(main())

Note that we did not hold a reference to the task created (no variable holding the result of asyncio.create_task(yo()))

This means the task might be garbage collected by python’s runtime and give undesired behaviors.

create_task explicitly said that the reference asyncio has to the task is a weak reference, so in order to avoid the created task to be garbage collected, we need to hold references to them.

You might ask why did we create the task and not hold the reference? It’s mainly used for the “fire and forget” case, in the case above the intent was to fire and forget yo() as a task. The right way is to hold them somewhere and once done remove them (to let the garbage collector do its job)

I’ll pull the documentation’s code snippet here

background_tasks = set()

for i in range(10):
    task = asyncio.create_task(some_coro(param=i))

    # Add task to the set. This creates a strong reference.
    background_tasks.add(task)

    # To prevent keeping references to finished tasks forever,
    # make each task remove its own reference from the set after
    # completion:
    task.add_done_callback(background_tasks.discard)

The Bug

OK so I spent a lot of time going through async/await in python’s asyncio. But what was the bug we had?

The bug originated from the following misunderstanding

Event Loop is single threaded, and in single thread programs there are no race conditions.

The above claim is WRONG.

Let’s recap what does await do, await means the current control tells the event loop: “I’m waiting for this operation to finish, so I’ll pause and give the event loop a chance to run other tasks.”

Assume that we want to only say hi or hello if no greeting was said, and tried to come up with the following code


import asyncio

greeting_said = False

async def say_hi_only_if_greeting_not_said():
    global greeting_said
    if not greeting_said:
        await asyncio.sleep(1)
        print("hi")
        greeting_said = True

async def say_hello_only_if_greeting_not_said():
    global greeting_said
    if not greeting_said:
        await asyncio.sleep(1)
        print("hello")
        greeting_said = True

async def main():
    hi_task = asyncio.create_task(say_hi_only_if_greeting_not_said())
    hello_task = asyncio.create_task(say_hello_only_if_greeting_not_said())

    await hi_task
    await hello_task

asyncio.run(main())

The above code would say hi and hello

Let’s try to follow the execution flow line by line again

asyncio.run(main()) kicks off the event loop and main gets control
hi_task = asyncio.create_task(say_hi_only_if_greeting_not_said()) creates hi_task and added it to event loop, main still has control
hello_task = asyncio.create_task(say_hello_only_if_greeting_not_said()) creates hello_task and added it to event loop, main still has control
await hi_task, main tells event loop that it won’t need control until hi_task is done, main gives up control
Event loop gives control to hi_task
hi_task gets control and sees greeting_said == False, executes await asyncio.sleep(1), giving up control
Event loop gives control to hello_task
hello_task gets control and also sees greeting_said == False, executes await asyncio.sleep(1), gives up control to event loop
Event loop gives hi_task control and hi_task prints hi, also sets greeting_said to True and marks hi_task to be done
Event loop picks main because hi_task was done (main was waiting for await hi_task), then main executes await hello_task, gives up control, event loop gives control to hello_task
hello_task prints "hello" even if greeting_said was marked as True in hi_task. This is because hello_task already passed the if check
hello_task done and control goes back to main
main finishes

The problem happened because even if the event loop runs in a single thread, whenever we call await you might give up control and other tasks could modify shared states. In this particular example the tasks are giving up control because asyncio.sleep() was called.

Further things to think about

Rust’s async/await allows you to pick the execution environment, meaning you can choose different implementations.
Rust’s famous tokio schedules jobs across threads and allows work-stealing. This means jobs may be executed in parallel.

Map Flatmap and Options

Tue, 21 Jul 2026 07:09:13 -0400

tldr

map creates a new box, flatMap expects you to create the box and pass it along
Use map when the passed in function does not return a box
Use flatMap when the passed in function returns a box
The real power of Option is to chain map, flatMap so you can change things in the box and pass to the next

Overview

Back in 2020 when I joined a web team and the servers were written in Scala. I spent some time learning Scala and got confused about map and flatMap. I think I understood it at that time but since I didn’t write it down after all these years it became vague in my mind. Recently I’m picking up Flink and flatMap came up again. So I decided to re-learn and write up something.

The Problem of Representing Optional Fields

Say we have a User struct in Go and it looks like the following

type User struct {
	ID      int
	Email   string
	Address *string
	ZipCode *string
}

What I am trying to express with this User struct is:

A User has 4 fields, ID, Email, Address, ZipCode.
Among them ID and Email is required, Address and ZipCode is optional.

However when using the User struct the compiler doesn’t force me to do a nil pointer check, so in my code I always need to keep in mind to do a nil check before I reference the Address.

Let’s see how this is solved as a “builtin” in some other languages

Option

Scala has this type called Option.

The User struct above in Scala could be defined as

case class User(
  id: Int,
  email: String,
  address: Option[String],
  zipCode: Option[String]
)

Note that both optional fields are of type Option[String], this means the type of address is an Option which wraps over a String.

When you try to access address, you get an Option, if it contains the actual address (meaning it does exist), the address is Some(String), if it does not contain the actual address (meaning it does not exist), the address is None.

Some code snippets might be helpful

// A user with all fields populated
val user1 = User(
  id = 1,
  email = "alice@example.com",
  address = Some("123 Main St"),
  zipCode = Some("07901")
)

// A user missing the optional fields
val user2 = User(
  id = 2,
  email = "bob@example.com",
  address = None,
  zipCode = None
)

With Option here the compiler forces me to handle the Option at compile time.

The Other Problem of Representing Optional Fields

Referring to the same User struct in Go

type User struct {
	ID      int
	Email   string
	Address *string
	ZipCode *string
}

There is another problem using this struct. Given a User instance which has a nil Address you can not tell if the user did not set it, or if the user explicitly sets it to nil.

Why does this matter?

Let’s assume that in the database we have a User table where its Address and ZipCode column are both nullable.

So when you get a User instance like the following to write to the database the intent is ambiguous.

user := User{
		ID:    1,
		Email: "a@b.com",
	}

Both Address and ZipCode are nil. Should I set both of them to NULL in the database? Or should I leave the columns in the database alone? There isn’t a way to tell at this point.

The root cause is using a pointer in a struct is not enough to distinguish between

I intentionally set this field to nil
I did not provide this field so leave it alone

I made a big mistake here in the first version of this post.

I thought that Option[String] is going to solve the problem, where I could simply replace *string with Option[String]. But it does not.

Because what I needed is to express 3 states for a given field, using Address as an example. I want to know if the user

Value provided: Update the field to "123 Main St".
Intentionally set to NULL: The user wants to clear/delete their address.
Not provided (leave it alone): The user is only updating their Email, so they didn’t include an Address in the request.

Without creating custom abstractions, using Go to solve this problem requires doing a pointer to pointer

type User struct {
	ID      int
	Email   string
	Address **string
	ZipCode **string
}

Now we have a way to map our 3 states:

Not provided: The field is nil
Intentionally set to NULL: The field is a pointer to nil (*Address == nil)
Value provided: The field is a pointer to a pointer to a string (**Address == "123 Main St")

And because the compiler does not force you to check for nil before deref, it’s harder to maintain code using the struct above.

With Option in say Scala (without creating custom abstractions), we would mark the field as Option[Option[String]]. And because the compiler forces us to handle None cases, even though the code might look a little ugly we solved this problem at compile time.

case class User(
  id: Int,
  email: String,
  address: Option[Option[String]],
  zipCode: Option[Option[String]]
)

With nested Options, the 3 states are cleanly mapped:

Not provided: None
Intentionally set to NULL: Some(None)
Value provided: Some(Some("123 Main St"))

Basic Usages

The Option type has a method to tell if it was defined or not.

// anti-pattern code
val user2 = User(
  id = 2,
  email = "bob@example.com",
  address = None,
  zipCode = None
)

// code to print user2.address if was defined

if (user2.address.isDefined) {
    val addressString = user2.address.get
    println(s"$addressString")
}

The above is not how you use Option, imagine you need to get the user’s address and pass it to a service that takes in the address and returns the zipcode. Your code would be like

def getZipCodeByAddress(addr: String): Option[String] = {
    // A zipcode is not guaranteed by an address
    // Hence returning an Option[String]
}

// anti-pattern code
if (user2.address.isDefined) {
  // .get unwraps the Option, returning the raw String
  val addressString = user2.address.get
  val zipCodeOption = getZipCodeByAddress(addressString)
  if (zipCodeOption.isDefined) {
      val zipCode = zipCodeOption.get
      println(s"$zipCode")
  }
}

The code above works but it’s the typical example of “I’m from a certain language, I’m gonna write everything my way”.

In Scala there are multiple ways to write the above such as for-comprehension, but I want to focus on the less syntax sugary ones.

Pattern Matching (Not the Machine Learning One)

Given an Option, we could do Pattern Matching to match against an Option, this also exists in Rust.

// pattern matching

val someAddress = Some("123 Main St")

val displayAddress = someAddress match {
  case Some(address) => s"address is $address"
  case None          => "No address"
}

The above is fine but imagine the following.

Given an optional address, if the address exists call another function that returns an optional zipcode, and then print the zipcode if it exists.

// nested pattern matching

val someAddress = Some("123 Main St")

// Assuming getZipCodeByAddress returns an Option[String]
val displayMessage = someAddress match {
  case Some(address) =>
    getZipCodeByAddress(address) match {
      case Some(zipCode) => s"Found zip code: $zipCode"
      case None          => "Address exists, but zip code lookup failed"
    }
  case None =>
    "No address"
}

println(displayMessage)

This starts to feel very off, let’s see how map and flatMap could help.

map and flatMap

map

If you imagine Option as a box, in the examples above, we were trying to open (unwrap) the box every time.

What if there is a way that allows you to not open the box but change the thing?

map and flatMap is the tool.

Let’s start with map, what map does is it opens the box, applies the function you provided to the item, then creates a new box putting the new stuff in there.

1
2
3

// map for Option

def map[B](f: A => B): Option[B]

The way to read this is:

map is a function that has a type parameter B, it takes in a function f and returns an Option[B].
f takes in an argument of type A and returns a value of type B.

Say we have an Option[String], like Some("yo"), we could use map modify the box’s content from "yo" to "sup"

val x = Some("yo")

def sup(greet: String): String = {
    if (greet == "yo") "sup" else "hello"
}

val y = x.map(sup) // y's type is Option[String]

Note that using map we opened the Option[String] box, used the sup function to convert "yo" to "sup" and wrapped it in a new Option[String] box

Another example is to open the Option[String] box, get the length of the String and put the Int in a new box.

val x = Some("yo")

def getLength(s: String): Int = {
    s.length
}

val y = x.map(getLength) // y's type is Option[Int]

flatMap

1
2
3

// flatMap for Option

def flatMap[B](f: A => Option[B]): Option[B]

The way to read this is:

flatMap is a function that has a type parameter B, it takes in a function f and returns an Option[B].
f takes in an argument of type A and returns a value of type Option[B].

val x = Some("yo")

def maybeSup(greet: String): Option[String] = {
    if (greet == "yo") Some("sup") else None
}

val sup = x.flatMap(maybeSup) // sup's type is Option[String]

Note that using flatMap we opened the Option[String] box, used the maybeSup function to convert "yo" to Some("sup") and returned it.

map vs flatMap

If you look closer to the above examples you might feel confused, more specifically when you put map and flatMap side by side you feel like finding waldo.

The key difference is that map’s function takes in A and returns B, whereas flatMap’s function takes in A and returns Option[B].

Had we used map with maybeSup, the result would be Option[Option[String]]

val x = Some("yo")

def maybeSup(greet: String): Option[String] = {
    if (greet == "yo") Some("sup") else None
}

val sup = x.map(maybeSup) // sup's type is Option[Option[String]] because we used map

Hence the name “flat”, where it flattens nested Options

Observations

map unwraps Option, applies the function, then creates a new Option
flatMap unwraps Option, applies the function, and returns what the function is returning, since the function is required to return an Option, flatMap does not create an Option

The above could also be summarized as

When the function you want to apply does not return Option, use map.
When the function you want to apply returns Option[T], use flatMap.

But wait isn’t this the same on Lists

So far we have been talking about map and flatMap on Options, but for most people map and flatMap feel more natural on List.

The following applies a double function on each element in xs.

// map
val xs = List[Int](1, 2, 3)

def double(x: Int): Int = {
    x * 2
}

val ys = xs.map(double) // ys = List(2, 4, 6)

The following flattens nested Lists without modifying the elements

// flatMap

val xss: List[List[Int]] = List(List(1, 2), List(3), List(4, 5), List(6, 7))

def f(xs: List[Int]): List[Int] = {
    xs
}

val xs = xss.flatMap(f) // xs = List(1, 2, 3, 4, 5, 6, 7)

To be honest the above example isn’t a good one, I picked it because it demonstrates “flatten” a nested list to a single list.

It is confusing because f’s signature is List[Int] to List[Int] whereas flatMap for Option takes A to Option[B].

The reason this is correct is because the A that f takes is the List[Int] inside List[List[Int]].

The following scratch pad might help

List[List[Int]].flatMap(f: List[Int] => List[Int]): List[Int]

List[A].flatMap(f: A => List[B]): List[B]

A == List[Int]
B = Int

We can pick another example where I have a list of space separated strings and I want to return a List[String]

val sentences = List(
  "Option is a Monad",
  "Scala is strictly typed"
)

def splitWords(sentence: String): List[String] = {
  sentence.split(" ").toList
}

val words = sentences.flatMap(splitWords) // words is of type List[String]

// flatMap(f: String => List[String]): List[String]

The point I am trying to deliver is, map and flatMap work the same way on Option and List. Because Option and List both wrap around the inner type and support some specific operations.

Option in Rust

Rust uses Option a lot, let’s see how Rust does pattern matching for Option

let some_num = Some(123);

// if some_num contains a number, plus one to it
// else return 0 as default
let y = match some_num {
    Some(num) => num + 1,
    None => 0,
};

Rust also supports this if let syntax

let some_num = Some(123);

if let Some(num) = some_num {
    println!("{num}");
}

But in the section above we know that we could use map and flatMap to work with Option. How does it work in Rust?

map and flatMap (and_then) in Rust

The full doc is here.

I’ll point out that in Rust flatMap is called and_then.

It doesn’t feel like they are the same, but the signature of and_then is the same as flatMap

1
2
3

pub fn and_then<U, F>(self, f: F) -> Option<U>
where
    F: FnOnce(T) -> Option<U>,

f takes in T and returns Option<U>

It might feel a bit off to associate and_then with flatMap, but the way Rust handles its functions to work with types like Option focuses on boolean logic, whereas Scala focuses on the shape of the data structure.

It feels more natural when we use and_then on Option.

Let’s go back to the first example where we have address: Option<String> and try to use get_zipcode_by_address(addr: &str) -> Option<&str>

let some_address = Some("123 Main St");

fn get_zipcode_by_address(addr: &str) -> Option<&str> {
    Some("07901")
}

let some_zipcode = some_address.and_then(get_zipcode_by_address);
// some_zipcode's type is `Option<&str>`

The and in and_then checks to see if the Option contains an address; if it does, it passes it to the function and returns Option<&str>. Rust’s methods to work with Option focus on boolean logic. See doc here.

or_else is also interesting, it applies f when Option is None.

For completeness’s sake, I’ll add the map example with Rust here as well

let x = Some("yo");

fn sup(greet: &str) -> String {
    if greet == "yo" {
        return String::from("sup");
    }
    return String::from("hello");
}

let y = x.map(sup); // y's type is Option<String>

If you think about it, doing an and_then on a Vec doesn’t really make sense. Hence Rust actually uses flat_map on Vec. There is no and_then on Vec in Rust, even though they are essentially the same thing.

The example where we have a list of space separated strings and I want to return a Vec<&str> would be

let sentences = vec![
    "Vec is a Monad",
    "Rust is strictly typed",
];

fn split_words(sentence: &str) -> Vec<&str> {
    return sentence.split(" ").collect();
}

// Note: Rust uses into_iter().flat_map for collections instead of and_then
let words: Vec<&str> = sentences.into_iter().flat_map(split_words).collect();

Summary

What have we gone through so far

We started from Go to explain why nil is not enough and Option is needed.
Then went to Scala to learn about Option and Pattern Matching
Then learned about map and flatMap, what is the difference
Then went to Rust to see map and and_then in Rust

Option, List are called Monads

Here is how Haskell defines a Monad, tbh I can’t fully follow it.

My simpler (loose) version is the following.

A Monad is a type that wraps around another type that supports an interface/trait

The wrapper function to take the inner type and return the Monad type. Some("stuff")
Implements flatMap, e.g. Option[String].flatMap[B](f: String => Option[B]): Option[B]
Three rules, see Haskell doc here
- Left Identity
- Right Identity
- Associativity

Further things to think about

Future is also a Monad in both Rust and Scala
Although Promise in JavaScript feels like a Monad, it isn’t.
How other syntax sugars like async/await in Rust, for comprehension in Scala work.
Read Haskell’s Monad definition

MSB and LSB, MSb and LSb

Sun, 12 Jul 2026 09:37:19 -0400

tldr

Distinguish between Most/Least Significant Byte vs Most/Least Significant bit
Most Significant means if you add 1 how big of a deal you change the number
Big Endian and Little Endian affects how the bytes are ordered in the binary, but in your program it’s always assembled correctly

Overview

This week I was working on an inner source project and learned something new. I never knew that there exists the concept of Most Significant bit before.

Hence as usual, vibe with Gemini/Claude, learn again and write it down.

In this post I’ll use MSB and LSB for Most Significant Byte and Least Significant Byte. Use MSb and LSb for Most Significant bit and Least Significant bit

Why is it called Most/Least Significant

I learned this when I was interviewing with Redline Trading Solutions in 2016. Seems like they got merged/acquired by Pico?

Anyways, the dude told me the reason a byte or bit is called Most Significant is because if you add one to it, how much does it change the number.

So if the allowance I give my kid is 20 bucks per paycheck, then 2 in the 20 would be the Most Significant digit (because he then gets 30 bucks), where as 0 would be the Least Significant digit (because he would only get 21 compared to 30).

Note that I used digit instead of Byte or bit. This is because I’m only explaining what is Most Significant or Least Significant.

Bytes

So let’s talk about Most Significant Byte vs Least Significant Byte first.

This is to describe when there are two bytes, which byte is the Most Significant Byte.

The sentence above is a bit vague, but let me explain.

So say you have a uint32_t (unsigned int 32bits), and your code is like

#include <stdint.h>
#include <stdio.h>

uint32_t num = 0x12345678;

int main() {
    printf("num = 0x%X\n", num);
    // this would print "0x12345678"
}

Because this is a C program and num is a global initialized variable, it belongs to the data segment, hence it’s going to be in the output binary.

Then the question is, how are 0x12, 0x34, 0x56, 0x78 ordered in the binary?

Most people would assume they are in order, which means on disk you would see them from left to right 0x12, 0x34, 0x56, 0x78

But the reality is there exist two orders.

Big Endian (`MSB` First)

If 0x12, 0x34, 0x56, 0x78 is in the binary where the Most Significant Byte is first, then this is called Big Endian.

Little Endian (`LSB` First)

If 0x78, 0x56, 0x34, 0x12 is in the binary where the Least Significant Byte is first, then this is called Little Endian.

What does it mean if it’s Big Endian or Little Endian?

Big Endian and Little Endian describes how the bytes of a multibyte object are arranged in address order or in a defined serialization.

So Intel Chips and Apple M Chips are Little Endian, that means for the chip to get 0x12345678, the hex has to be 0x78, 0x56, 0x34, 0x12.

You can verify the program above with the following (since nowadays most chips are little endian)

gcc a.c && xxd -g 1 a.out | grep "78 56 34 12"
00008000: 78 56 34 12 00 00 00 00 00 00 00 00 00 00 00 00  xV4.............

Note that regardless of Big Endian or Little Endian, your code sees 0x12345678. The endianness decides what is in the binary in the case above.

One important one is when we try to send 0x12345678 over the network.

Networking is Big Endian?

I remember in my networking class my professor said networking is Big Endian, but I misinterpreted it, what he meant was the protocol information is always Big Endian when you send it over the wire. Say you are sending some payload to some address’s port 4660 which is hex 0x1234, it would be over the wire in the order of 0x12, 0x34.

The actual payload however, is usually just raw bytes, so on a Little Endian machine, trying to send a uint32_t 0x12345678 over the wire would have it sent as 0x78, 0x56, 0x34, 0x12.

uint32_t payload = 0x12345678;
send(socket, &payload, sizeof payload, 0);

// On a Little Endian machine the payload over the wire would be
// 0x78, 0x56, 0x34, 0x12

There is a function called htonl to convert from host byte order to network byte order.

1
2

uint32_t wire_payload = htonl(payload);
send(socket, &wire_payload, sizeof wire_payload, 0);

One thing to note is that when my professor said “Networking is Big Endian”, he meant the TCP, UDP, IP header information themselves are sent as Big Endian. NOT the payload as we saw above.

The other thing is that there exist other protocols that requires headers to be sent over the wire and they are not universally Big Endian, it’s per protocol. Some of the ones that Gemini brought up are

USB (Universal Serial Bus) is strictly Little Endian
SMB/CIFS (Windows File Sharing) is Little Endian
Bluetooth (HCI layer) is Little Endian

And again when we say Little Endian for all three above we are talking about the protocol headers, NOT the payload.

Bits

Ok we’ve talked a lot about bytes and let’s talk about bits. This is the part that got me confused because I didn’t know there was a difference in bit numbering conventions.

In my previous knowledge, bit 0 is always on the right side (meaning it’s the least significant one because if you add one to it, the number doesn’t change drastically). This is called the LSb 0 convention.

See example below for 0x12, its binary is 0b0001_0010

bit 76543210
    00010010

Apparently there exists another convention called MSb 0, where the Most Significant bit is assigned the index 0. So the above would be

bit 01234567
    00010010

TCP

MSb seems to be prevalent in networking protocols, for instance TCP

  TCP Header Format

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Source Port          |       Destination Port        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Sequence Number                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Acknowledgment Number                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Data |           |U|A|P|R|S|F|                               |
   | Offset| Reserved  |R|C|S|S|Y|I|            Window             |
   |       |           |G|K|H|T|N|N|                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Checksum            |         Urgent Pointer        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Options                    |    Padding    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                             data                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                            TCP Header Format

The top row is the tens digit, the bottom row is the ones digit, so you read them together as one column. This ruler restarts at 0 for every 32-bit row of the header. For example, in the row containing the control bits (the 4th row), the column with 1 on top and 1 on the bottom is bit 11 — that’s the ACK flag, set on every packet after the initial handshake to acknowledge received data. It runs from bit 0 through bit 31 this way, left to right, for each row.

Nevertheless, the Most Significant bit is bit 0.

BlazingMQ

BlazingMQ uses MSb as well. See MessagePropertiesHeader

    //   +---------------+---------------+---------------+---------------+
    //   |0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7|
    //   +---------------+---------------+---------------+---------------+
    //   | R |PHS2X| HS2X|   MPAW Upper  |          MPAW Lower           |
    //   +---------------+---------------+---------------+---------------+
    //   |   Reserved    |    NumProps   |
    //   +---------------+---------------+

Note that bit 0 is the MSb. One difference between BlazingMQ and TCP in terms of bit numbering is that BlazingMQ marks bit 0 to bit 7 for each byte, whereas TCP treats all 32 bits consecutively.

Raspberry Pi Datasheet

If you go to RP2040 Page 434 you would find the datasheet of UARTCR Register.

This shows how to set individual bits in the 32-bit register. Note that it uses LSb, so bit 0 means the least significant bit (almost every modern chip uses this convention)

So if I want to enable SIRLP: SIR low-power IrDA mode., because it’s bit 2 on the datasheet, and I know the convention is LSb. So in my code I’d actually set bit 2 with something like

uint32_t UARTCR_REGISTER = 0;

void SetSIRLP(int value) {
    // assert value is 0 or 1 only
    UARTCR_REGISTER = (UARTCR_REGISTER & ~(1 << 2)) | (value << 2);
}

So when would I need to know about MSBs or LSBs? It’s when something is wrong and you need to inspect the live memory.

Say the code you compiled and flashed to your raspberry isn’t working as you expected, and you want to check if you actually set the thing correctly on the hardware.

You would use a hardware debugger to read the memory at the address of your UARTCR_REGISTER while the chip is running, and look for 4 bytes. And because you know RP2040 is Little Endian, the first byte in memory would be the one that contains your SIRLP bit, read that out and look for bit 2 to confirm.

bit 0 to bit 7 or bit 0 to bit 31

You might have noticed in the examples above when describing a uint32_t, there are two numbering schemes that apply regardless of whether we use MSb or LSb.

Assume that we are talking about a uint32_t with LSb. It could be marked from bit 0 to bit 31 like the following

0x12345678

3322 2222 2222 1111 1111 1100 0000 0000
1098 7654 3210 9876 5432 1098 7654 3210
---- ---- ---- ---- ---- ---- ---- ----
0001 0010 0011 0100 0101 0110 0111 1000

Or it could be marked for each byte from bit 0 to bit 7

0x12       0x34       0x56       0x78

7654 3210  7654 3210  7654 3210  7654 3210
---- ----  ---- ----  ---- ----  ---- ----
0001 0010  0011 0100  0101 0110  0111 1000

Note that Big Endian vs Little Endian wouldn’t matter in the diagram above because we are viewing the hex in our program, so the value is always 0x12345678 in our code.

This numbering choice is independent of MSb/LSb, and we’ve actually already seen both options in the examples above: TCP numbers the whole 32 bits at once (bit 0 to bit 31) using MSb 0, while BlazingMQ numbers each byte separately (bit 0 to bit 7, repeating per byte) also using MSb 0. This section just shows that the same two choices, whole-word vs per-byte apply under LSb 0 too.

Both ways of marking the bits are valid, but I think using bit 0 - bit 31 feels easier to handle in code.

Update

While talking to JG about this post a few things came up and I think it’s worth it to note it down.

When sending a byte over the wire there could be LSb first or MSb first, meaning are we sending the right most bit first over the wire or the left most bit first over wire (since bits on wires are high/low voltages)

Backward and Forward Compatibility

Sun, 05 Jul 2026 14:25:47 -0400

tldr

Distinguish between “breaking change” vs “backward compatible change”
There are writers and readers
backward/forward compatible changes strictly speaking are from reader’s perspective
Even when talking about schema evolutions, compatibility is based on if the reader can read old/new data produced by the writer

Overview

I’ve heard about Avro before but I never used it. Also I was confused about “backward compatibility” and “forward compatibility”. Sometimes I use it in my day to day work but I have the feeling that I’m not using it correct.

So I went to vibe with Gemini and Claude to figure things out. Ironically Gemini did a poor job despite I pay it 20 bucks a month, where Claude explained it nice to me even on the free version.

Again this is me learning things with LLMs and trying to write it down, there might be mistakes but for now it’s the best to my knowledge that I believe what I’m writing is correct.

My confusions

There are 2 confusions before I started to dig a bit deeper.

The usual things I say at work, does it make sense?
What is forward compatibility?

The usual things I say at work

My new backend change is backward compatible, we can move it without coordinating the frontend

After talking with Claude I think the above is understandable but technically incorrect. What I actually mean is:

My new backend change is not a breaking change, we can move it without coordinating the frontend.

If I really want to use the term backward/forward compatible to describe it I would say:

My new backend change is not a breaking change because our clients are forward compatible.

The above is an example of publishing server changes, but similar conversations also happen when people decide if they should bump the major of their semver.

You can’t remove that function because it would break backward compatibility, if you want to do that you have to bump the major.

Where the above should be technically

You can’t remove that function because it would be a breaking change, to release a breaking change you need to bump the major.

What is forward compatibility?

Can an old reader handle new data? If yes then the reader is forward compatible with new data.

This really is it. But it gets a bit confusing when people like me misuse “backward compatible” with “non-breaking”. I’ll try to explain what backward/forward compatibility is in the following sections.

Note that I didn’t explain what is a reader and what is a writer formally. Let’s have a look together.

The definition

Let’s put aside the difference between breaking changes and backward compatible changes for a second and construct our mental model with the following.

There are two parties and a schema

Producer/Writer
Consumer/Reader
A schema

When we talk about backward compatibility, it is always referring to if the reader can handle old (looking backward) data.

When we talk about forward compatibility, it is always referring to if the reader can handle new (looking forward) data.

Schema changes (aka schema evolution) could be categorized as backward compatible or forward compatible by testing if an old reader can handle new data (forward compatibility) and if a new reader can handle old data (backward compatibility).

How can a reader be backward compatible

Assume that I have a v2 reader and it has the following code


type Data struct {
    Name string `json:"name"`
    Age int `json:"age,omitempty"`
}

func handlePayload(payload json.RawMessage) {
    var data Data
    err := json.Unmarshal(payload, &data)
    if err != nil {
        panic(err)
    }

    fmt.Printf("%+v\n", data)
}

This reader can take v1 payloads like

1
2
3

{
    "name": "Bingus"
}

It can also take v2 payloads like

{
    "name": "Bingus",
    "age": 21
}

The fact that the v2 reader could take a payload with only name in it (the v1 payload) means the reader can handle old data, hence it is backward compatible.

How can a reader be forward compatible

Same example but let’s go to the v1 reader.


type Data struct {
    Name string `json:"name"`
}

func handlePayload(payload json.RawMessage) {
    var data Data
    err := json.Unmarshal(payload, &data)
    if err != nil {
        panic(err)
    }

    fmt.Printf("%+v\n", data)
}

This reader can take v2 payloads like

{
    "name": "Bingus",
    "age": 21
}

Because age is simply ignored.

The fact that the v1 reader can take a v2 payload means the reader can handle new data, hence it is forward compatible

Common schema changes

People also say if a schema evolution (schema change) is backward or forward compatible or not.

When they say that they meant with the new schema change from v1 to v2, how does a v2 reader handle v1 data? how does a v1 reader handle v2 data?

Note that this is only a general discussion, different protocols have different tools to handle changes, like Avro has an alias thing to help renaming

Common schema changes are

Adding an optional field

backward compatible?

Because the field is optional, a v2 reader should be able to handle v1 data (where the field was not presented). This means the change is backward compatible.

forward compatible?

A v1 reader should be able to ignore the new field, hence it is also forward compatible.

verdict

Adding an optional field to the schema is full compatible

Adding a required field

backward compatible?

Adding a required field means a v2 reader needs the new field, so v2 reader won’t be able to work with v1 data, meaning the schema change is not backward compatible.

forward compatible?

The v1 reader doesn’t know about the new required field, so when it receives v2 data, it simply ignores the extra field. Hence, the schema change is forward compatible.

verdict

Adding a required field to the schema is forward compatible

Removing a required field

backward compatible?

By removing a required field, the v2 reader would be fine with old data because it no longer reads it. Hence it is backward compatible

forward compatible?

The v1 reader won’t be fine because new data would not send back a field required by v1, hence this is not forward compatible

verdict

Removing a required field from the schema is backward compatible

Renaming a field

This is a bit tricky because people would say: I can create a new field with a new name and keep my old field. That could be in the “add a new optional field” or “add a new required field” based on how you do it.

Let’s assume that we are (1) only renaming, and (2) the field is a required field

backward compatible?

A v2 reader won’t be able to handle old data because it is looking for a required field that does not exist. Hence not backward compatible.

forward compatible?

A v1 reader won’t be able to handle new data because it is looking for a required field that does not exist. Hence not forward compatible.

verdict

Not compatible at all.

The role the schema plays

Note that in the golang example above the behavior of “ignoring new fields” was handled in application logic. Some schemas provide ways to verify if your schema change is backward compatible or forward compatible by simply checking if the schema change added a new optional field or added a required field or removed a required field etc.

This is where a schema registry would come into play. You can configure your schema to only allow full compatible changes. That way if any changes would introduce incompatibility it would be rejected.

Difference between different schemas

As you can see in the golang example above, respecting an optional field is baked in the application code, the JSON schema itself doesn’t really enforce anything on the application side. The application could totally treat an optional field as a required field. This is something that is really fragile (relying on programmers to do the right thing). Hence different solutions came out.

I’m only touching on the surface of these since I don’t have much experience and today’s topic is backward/forward compatibility.

I might be writing a new post when I drill down to Avro.

JSON Schema

This is the one that relies on programmers to treat the fields the right way. Not much to be talking about here tbh.

Avro

Avro provides two schemas, the writer schema and the reader schema. When you add a new optional field you are also required to set up the default value of the field in the reader’s schema. This way when a reader with v2 tries to read data provided by a v1 writer, the new fields would have default values defined by the reader’s schema.

This is what makes the change backward compatible. (v2 reader reads v1 data)

Avro also provides ways to rename by using the alias keyword.

Protobuf and Thrift

Both protobuf and thrift use one schema, but they generate source code and use that opportunity to set default values of optional fields.

Back to my initial confusion about my daily talk

I think the best is to start distinguish between “non-breaking/breaking” change vs “backward/forward” compatible change.

Technically speaking “my server change is fine because clients are forward compatible” is correct, but it does sound weird doesn’t it?

Future things

Dive deeper into Avro, Protobuf and Thrift

Quiz time (LLM Generated)

A new version of a mobile app (v3) can still open files saved by the old version (v1). Is this backward or forward compatible? Which party is being tested — the file, or the app?
Your coworker says: “We can’t remove this API field, it’ll break backward compatibility.” Rewrite this sentence using the more precise term.
True or false: “Forward compatible” means data from the future can be read.
A schema adds a new required field. Is this backward compatible, forward compatible, both, or neither? Justify using the reader/data relationship, not just the label.
In Avro, you add a new optional field to your schema and give it a default value of 0. Which schema — the reader’s or the writer’s — needs to declare that default for backward compatibility to work?
A protobuf message removes field number 7 and, six months later, a teammate adds a brand-new field and accidentally reuses number 7. What goes wrong, and for which compatibility direction (backward/forward) does this cause problems?
Fill in the blank: “My library’s v2.1.0 release is backward compatible” really means, strictly speaking, that ______ is forward compatible with ______.
GraphQL doesn’t really have a “forward compatibility” problem the way JSON or protobuf do. Why not — what about GraphQL’s design makes the unknown-field problem mostly moot?
Scenario: a server adds a new field to its response payload and does not bump its major version. An old client, written before this field existed, keeps working fine. Using strict terminology, write one sentence describing this fact from the client’s perspective, and one sentence describing it from the server’s perspective (loose/producer convention).
Rank these schema changes from “safest” to “most dangerous,” in terms of compatibility breakage, and say which direction (backward/forward/both) each risks: (a) renaming a field with no alias, (b) adding an optional field, (c) removing a required field, (d) adding a required field.

Answers (remove display:none in the following div yourself)

A new version of a mobile app (v3) can still open files saved by the old version (v1). Is this backward or forward compatible? Which party is being tested — the file, or the app?

This is backward compatible, the app is being tested

Your coworker says: “We can’t remove this API field, it’ll break backward compatibility.” Rewrite this sentence using the more precise term.

We can’t remove this API field, it would introduce a breaking change

True or false: “Forward compatible” means data from the future can be read.

False, forward compatible means old reader can handle new data that already exists now. Not about future data that does not exist yet.

A schema adds a new required field. Is this backward compatible, forward compatible, both, or neither? Justify using the reader/data relationship, not just the label.

forward compatible new reader can not read old data because the required field is missing old reader can read new data because the old reader doesn’t know about the new field

In Avro, you add a new optional field to your schema and give it a default value of 0. Which schema — the reader’s or the writer’s — needs to declare that default for backward compatibility to work?

The reader’s schema, default values are always defined in reader’s schema

A protobuf message removes field number 7 and, six months later, a teammate adds a brand-new field and accidentally reuses number 7. What goes wrong, and for which compatibility direction (backward/forward) does this cause problems?

field numbers should not be reused old reader reading new data would be reading the wrong field, hence not forward compatible new reader reading old data would also be reading the wrong field, hence not backward compatible

Fill in the blank: “My library’s v2.1.0 release is backward compatible” really means, strictly speaking, that ______ is forward compatible with ______.

client is forward compatible with my new library

GraphQL doesn’t really have a “forward compatibility” problem the way JSON or protobuf do. Why not — what about GraphQL’s design makes the unknown-field problem mostly moot?

Because in graphql client always requires specific fields. The server never returns fields that the client didn’t ask for, so that means all the fields sent to client are fields client knows how to handle

Scenario: a server adds a new field to its response payload and does not bump its major version. An old client, written before this field existed, keeps working fine. Using strict terminology, write one sentence describing this fact from the client’s perspective, and one sentence describing it from the server’s perspective (loose/producer convention).

The client is forward compatible with new data The server is not introducing breaking changes because its clients are forward compatible (ppl also say I deployed a backward compatible change)

Rank these schema changes from “safest” to “most dangerous,” in terms of compatibility breakage, and say which direction (backward/forward/both) each risks: (a) renaming a field with no alias, (b) adding an optional field, (c) removing a required field, (d) adding a required field.

(a) most dangerous, not compatible at all (b) safest, full compatible (c) medium, new reader can read old data, old reader can not read new data, backward compatible, not forward compatible (d) medium, new reader can not read old data, old reader can read new data, not backward compatible, forward compatible

Rust Learning: Modules

Sat, 27 Jun 2026 10:48:21 -0400

Overview

Rust modules is one of those things where I thought I learned it but it keeps giving me surprises.

I was recently contributing to an inner source rust project and got tripped by modules and realized that I don’t actually understand them.

So here I am trying to learn again and write things down.

I’ll go over the basics which explains the concept and syntax. Then I’ll show some idioms people use.

Might not be exhaustive but this is aligning with my understanding.

tldr

Rust’s module tree is explicit, you have to write it yourself.
mod is the keyword for you to build the module tree.
pub mod abc declares module abc and makes it public.
use is the keyword to “import” other modules, it creates shortcuts.
pub use abc creates shortcuts and shares that with users of your module.
To refer something in a sibling module, you need to use relative path super::sibling_name or absolute paths crate::path_to_sibling.

The Basics

Module Tree

I believe this is a rust concept, it means what module structure(tree) does your program has.

This isn’t a thing in languages like Python because in those languages the module tree maps to the file system.

Say in python I want to create my math package with four operators in it, I would do the following in mymath.py

mymath.py

# mymath.py

def add(a, b):
    return a + b

def minus(a, b):
    return a - b

def multiply(a, b):
    return a * b

def divide(a, b):
    return a / b

And in other places they could simply import mymath and use mymath.add() or from mymath import add then just use add()

In rust having mymath.rs is not enough, you need to explicitly tell the compiler you have a module.

`mod` keyword - to declare or define modules in the module tree

mod keyword is the keyword to declare or define modules in rust. Note that declare and define a module are slightly different.

When using the inline syntax mod defines the module, when using the file+folder way mod declares the module.

To define a module, there are three ways.

(1) Inline

src/main.rs

// src/main.rs
mod my_module {
    pub fn add(a: i32, b: i32) -> i32 {
        a + b
    }
}
fn main() {
    let three = my_module::add(1, 2);
    println!("Hello, world! {}", three);
}

In this way we defined and declared the module at the same time.

(2) File and Directory

The above feels like good tool to do ad-hoc things. Almost feels like in C++ where you open a namespace anywhere and starts adding things in it. A more natural way IMO would be using files and directories.

In rust there exists two ways, one of them being “legacy”, and the other one is the recommended way.

(2.1) [legacy] mod.rs + module_name/

You simply create a folder and name the folder name to be the module name.

Then you would need to have a mod.rs in the folder. You would use mod.rs to declare which modules are available to the outside world.

It could also re-export some internal modules to a different path. We’ll go over re-export in the use section.

folder structure

# folder structure
.
├── Cargo.lock
├── Cargo.toml
└── src
    ├── main.rs
    └── my_module
        ├── mod.rs
        └── operations.rs

3 directories, 5 files

src/my_module/mod.rs

// src/my_module/mod.rs

// this matches the file name operations.rs
pub mod operations;

src/my_module/operations.rs

// src/my_module/operations.rs

// pub here is needed otherwise this would be a private function
pub fn add(a: i32, b: i32) -> i32 {
    a + b
}

src/main.rs

// src/main.rs

// this is to declare the module
// module name is the directory name
mod my_module;

fn main() {
    let three = my_module::operations::add(1, 2);
    println!("{}", three);
}

(2.2) [recommended] module_name.rs + module_name/

The above is nice but there is one catch. In a big project there would be a lot of mod.rs files. Later a new way was introduced, essentially pulling mod.rs out of the folder and name it same as the folder (module name).

folder structure

# folder structure

.
├── Cargo.lock
├── Cargo.toml
└── src
    ├── main.rs
    ├── my_module
    │   └── operations.rs
    └── my_module.rs

3 directories, 5 files

I essentially pulled mod.rs out and renamed it. The rest is the same but for completeness sake I’ll put them below.

src/my_module.rs

// src/my_module.rs

// this matches the file name operations.rs
pub mod operations;

src/my_module/operations.rs

// src/my_module/operations.rs

// pub here is needed otherwise this would be a private function
pub fn add(a: i32, b: i32) -> i32 {
    a + b
}

src/main.rs

// src/main.rs

// this is to declare the module
// module name is the directory name
mod my_module;

fn main() {
    let three = my_module::operations::add(1, 2);
    println!("{}", three);
}

Visibility: public or private

You might have noticed that in the code example above there are some pub keywords here and there.

Everything in rust is defaulted to be private. Adding pub simply makes the declared module public and can be accessed outside.

Every module can have child modules, and in order to use something from the child module, the child module must make it visible for the parent.

Let’s look at a concrete example.

Say we have my_module and inside my_module we have operations, let’s say we need some utility functions that are only available for operations but we don’t want it to be exposed to users of my_module.

We would (1) create a sub module under operations, mark the functions as pub and then (2) declare the module in places we need to use. and (3) do not expose it in operations.rs

.
├── Cargo.lock
├── Cargo.toml
└── src
    ├── main.rs
    ├── my_module
    │   ├── operations
    │   │   └── utils.rs
    │   └── operations.rs
    └── my_module.rs

4 directories, 6 files

I’ll only pick the interesting files here

src/my_module/operations.rs

// src/my_module/operations.rs

// modules_playground::my_module::operations::utils
// this is to declare `utils` in the module tree.
// `utils` is a child of `operations`
mod utils;

pub fn add(a: i32, b: i32) -> i32 {
    utils::utility_func();
    a + b
}

src/my_module/operations/utils.rs

// src/my_module/operations/utils.rs
pub fn utility_func() {
    println!("I'm a utility function in my_module::operations::utils module");
}

The pub on utility_func in utils.rs is required, otherwise the function won’t be accessible in operations.rs.

The mod utils; in operations.rs is important, because it brought utils into the module tree. Mounted utils as a child of operations.

Note that in main.rs there is no access to utils::utility_func because it was not made public in operations.rs.

To make utils::utility_func available in main.rs, we could write pub mod utils instead of mod utils in operations.rs. The line pub mod utils does two things.

It mounts the module to the module tree.
It makes the module public for whoever is mounting the current module. (operations) in this case.

It can be confusing because sometimes when slapping on the pub keyword, a single line does two things.

There are also other fine grained control of visibility like pub(crate), pub(super) but we’ll go over them later.

Using modules from sibling

There is a tool called cargo modules, using that tool you can check the module tree.

Running cargo modules structure at the root of the project shows the following output

crate modules_playground
├── fn main: pub(crate)
└── mod my_module: pub(crate)
    └── mod operations: pub
        ├── fn add: pub
        └── mod utils: pub(self)
            └── fn utility_func: pub

The utils module we created is under operations, what if I want to have a utils module under my_module and share it across my_module?

In order to do this, let’s define the utils module under my_module by creating a utils.rs file inside my_module/ directory. Then in my_module.rs we need to declare it with mod utils;

.
├── Cargo.lock
├── Cargo.toml
└── src
    ├── main.rs
    ├── my_module
    │   ├── operations.rs
    │   └── utils.rs
    └── my_module.rs

3 directories, 6 files

src/my_module/utils.rs

// src/my_module/utils.rs

// this needs to be `pub` because it's used with other modules inside `my_module`
pub fn utility_func() {
    println!("I'm a utility function in my_module::operations::utils module");
}

src/my_module.rs

// src/my_module.rs
pub mod operations;

// we do not need to have `pub` here because this `utils` is private for my_module
mod utils;

And in order to use it from operations.rs, you need to either call it with a relative path, or call it with an absolute path.

To call it with a relative path, operations needs to go up to my_module then go down to utils.

To call it with an absolute path, operations can just specify the path from the root.

Let’s see both in action src/my_module/operations.rs

// src/my_module/operations.rs

pub fn add(a: i32, b: i32) -> i32 {
    // relative path
    super::utils::utility_func();

    // absolute path
    crate::my_module::utils::utility_func();
    a + b
}

Visibility: all pub keywords

So now we know that the mod keyword is to declare/define modules. And the pub keywork is to tweak the visibility. There are some variants of pub, I’ll list them below

pub, public everywhere
pub(crate), public in your current crate
pub(super), public to your parent(no grandparents) and sibling and kids of siblings(nephews/nieces)
pub(in path::to::module), public to the specific module path
pub(self) or ignored, public to yourself, essentially private

pub(super) granting access to nephews/nieces seems strange but it’s because of the following rule.

When you grant visibility to a module, you automatically grant visibility to every single module nested inside of it.

`mod` Summary

So we know that mod keyword declares/defines a module in the module tree. And pub keyword can change the visibility of the module. If you write them in the same line it does two things with one line.

pub mod abc; means declaring abc in the current module and make it public to whoever uses the current module.

`use` Keyword

In the last example above (calling sibling modules), we saw how to call the function with different paths

src/my_module/operations.rs

// src/my_module/operations.rs

pub fn add(a: i32, b: i32) -> i32 {
    // relative path
    super::utils::utility_func();

    // absolute path
    crate::my_module::utils::utility_func();
    a + b
}

However it would be tedious to keep writing the path.

use keyword could be used to shorten the path. It feels similar like import in Python, or use in C++.

It’s easier to think that use creates shortcuts.

We could use till the module and call the function like utils::utility_func(). Similar to import utils in Python.

src/my_module/operations.rs

// src/my_module/operations.rs
use super::utils;

pub fn add(a: i32, b: i32) -> i32 {
    utils::utility_func();
    a + b
}

Or we could use the full path and call utility_func() without utils::. Similar to from utils import utility_func in Python.

src/my_module/operations.rs

// src/my_module/operations.rs
use super::utils::utility_func;

pub fn add(a: i32, b: i32) -> i32 {
    utility_func();
    a + b
}

pub use, aka the Facade Pattern

pub use is a way to share the shortcut use created. It’s known as re-export.

Let’s look at a deeply nested module tree example.

.
├── Cargo.lock
├── Cargo.toml
└── src
    ├── main.rs
    ├── my_module
    │   ├── aaa
    │   │   └── bbb.rs
    │   └── aaa.rs
    └── my_module.rs

4 directories, 6 files

crate modules_playground
├── fn main: pub(crate)
└── mod my_module: pub(crate)
    └── mod aaa: pub
        └── mod bbb: pub
            └── fn bbb_func: pub

Use this as an exercise to see if you can fill out what is needed in each file to create the module tree like this.

src/my_module.rs

1
2

// src/my_module.rs
pub mod aaa;

src/my_module/aaa.rs

1
2

// src/my_module/aaa.rs
pub mod bbb;

src/my_module/aaa/bbb.rs

1
2

// src/my_module/aaa/bbb.rs
pub fn bbb_func() {}

src/main.rs

// src/main.rs
mod my_module;

fn main() {
    my_module::aaa::bbb::bbb_func();
}

In main.rs, it is clear that bbb_func() is deeply nested, we could use pub use inside my_module.rs to create a shortcut and make that shortcut available, so clients (main.rs) don’t really need to know about this aaa::bbb:: module tree structure.

src/my_module.rs

1
2
3

// src/my_module.rs
mod aaa; // still need to declare `aaa`, but no need to make it public
pub use aaa::bbb; // creating the shortcut and sharing it

src/main.rs

// src/main.rs
mod my_module;

fn main() {
    // using the shortcut created in `my_module.rs`
    my_module::bbb::bbb_func();
}

If we wanted we could go one step further to create a shortcut of bbb_func on my_module.

src/my_module.rs

1
2
3

// src/my_module.rs
mod aaa; // still need to declare `aaa`, but no need to make it public
pub use aaa::bbb::bbb_func; // creating the shortcut and sharing it

src/main.rs

// src/main.rs
mod my_module;

fn main() {
    // using the shortcut created in `my_module.rs`
    my_module::bbb_func();
}

The Idioms

The pub use above is one of the commonly used idiom/pattern. Let’s go over a few that I saw. This might not be exhaustive but you gotta start somewhere.

using type vs using functions

In my understanding, the convention for functions is to have the module name, the convention for types is to import the full path.

use super::utils;
use super::utils::MyType;

fn foo() {
    utils::utility_func();

    let x: MyType = 0;
}

Prelude

Some packages have stuff that most people would like to use anyways, so they export them in a prelude module and users would use *. Examples like use rayon::prelude::*;, use bevy::prelude::*;, use std::io::prelude::*;

Facade Pattern

We already covered this when we go over pub use. This is essentially creating shortcuts and share them with the users of your module.

The pattern I saw

mod my_module {
    pub struct Type1{}
    pub struct Type2{}

    pub mod my_module {
        pub use super::Type1;
        pub use super::Type2;
    }
}

pub use self::my_module::*; // without `self` there would be ambiguity, the `my_module` in this line is referring to the outer one

I saw this code when working on the inner source project.

I was super confused then I learned that the reason the modules are structured like this, is to give clients the ability to either refer Type1 or refer it with my_module::Type1.

The gotcha I got was the last line didn’t have self::, so it had ambiguity where rust didn’t know which my_module it is referring to. Because the outer sibling module and the nested module share the same name, omitting self:: causes ambiguity for the compiler over which my_module is being targeted. Adding self:: explicitly tells the compiler to look at the local module in the current scope (the outer one), fixing the ambiguity.

Rust Modules are Completely Closed

Rust modules feels like C++’s namespaces (probably because of the use keyword). C++ namespaces could be “opened” anywhere and have things added to it. It then begs the question, can rust modules be “opened” anywhere and have things added to it?

No, modules are completely closed in rust.

Rust Polars Parquet

Wed, 17 Jun 2026 07:08:39 -0400

Overview

First time I heard about Parquet is from an old friend. I recently moved to a new team that seems to work with parquet files. Hence I thought it might be a good idea to do some drills provided by LLMs. Plus I found that Polars’s original implementation is Rust, just another reason for me to practice Rust. Here is the repo of the LLM guided parquet practice.

I did some other drills like the iterator drill, the country-cli drill. But I got lazy and didn’t do the write up and now my memory isn’t that fresh.

tldr

Do write ups after finishing practicing for best return.
Use LazyFrame whenever possible
group_by destroys the rows, over keeps them
DataFrame transforms rows

What is a Parquet file?

I won’t (can’t) go into the details of what a Parquet file is. In order for me to practice the drills I needed to understand what a Parquet is at a very high level. And that is the level I’m going to explain now.

Parquet, or more specifically Apache Parquet is a file format. Files encoded in this format usually have .parquet as the suffix, the examples I used to do the drill is here.

This file format is designed for efficient data storage and retrieval. There are a few features/traits of this format

it is a binary format
it is column-oriented
it carries each column’s type information
it is compressed well (2024 Jan NY cities taxi rides is like 50mb)

Binary Format

Again, I won’t (can’t) go into the binary format itself, I think it is fun to drill down to the binary format and understand it. But for practical reasons, learning how to use it has a higher priority now. Maybe sometime later :)

What is Column Oriented?

I hear this term a lot and I don’t think I 100% understand it. I’ll try to write down my current understanding.

Before column-oriented became a buzz word every web scale kid talks about, most formats/tools we used are row-oriented.

What that means is each record has to have data of the full row. Say we have a csv file that stores basic student information and their test scores for different subjects

id,name,email,english_score,math_score,art_score
1,aaa,aaa@bb.com,84,90,NULL
2,bbb,bbb@bb.com,NULL,94,NULL
3,ccc,ccc@bb.com,NULL,NULL,98

In the example above, not all students have test scores for all 3 subjects, but we still needed to have NULL in their record. This is because we view the data as rows.

One issue with having NULLs in rows is wasted space.

In the csv above, 5 NULLs were there just as placeholders to indicate that the student didn’t take that test(yeah yeah yeah I know you could leave it empty but that’s missing the point).

You might have realized that this csv could be modeled with a relational database like Postgres and apply some normalization techniques.

Student Table
- id(pk)
- name
- email

English Test Table
- id(pk)
- student_id(fk)
- score
- questions
- answers

Math Test Table
- id(pk)
- student_id(fk)
- score
- questions
- answers

Art Test Table
- id(pk)
- student_id(fk)
- score
- questions
- result_s3_url

Aside of the detour, let’s come back to column oriented.

Had the csv example above be stored in a parquet file, it would be something like

// schema
id: Int32
name: String
email: String
english_score: Int32
math_score: Int32
art_score: Int32

// data
id:[1,2,3]
name:[aaa,bbb,ccc]
email: [aaa@bb.com, bbb@bb.com, ccc@bb.com]
english_score: [84] (Definition Levels [1, 0, 0])
math_score: [90,94]       (Definition Levels [1, 1, 0])
art_score: [98]        (Definition Levels [0, 0, 1])

// Definition Levels is at a high level how parquet knows which field is NULL

Some other benefits that come with column-oriented storage

When performing SELECT name, math_score on a row on a row-oriented format/database, you need to load the entire row in memory then throw away the columns you don’t need, whereas SELECT name, math_score on parquet could only pull out the columns you needed.
Because columns are of the same type, compression algorithms could perform very well, resulting in a very small file size.

Apache Arrow?

When doing research Apache Arrow keeps coming up. This stackoverflow post answer is awesome.

After reading it my understanding is

Apache Arrow is an in-memory format specification for columnar data
Arrow Files is a serialization protocol that allows you to serialize the data structures created by Apache Arrow. (no compression)
The Arrow Protocol is designed for you to just map the file in memory(mmap) without deserialization to use it
Whereas parquet does compression and is more for archiving purposes

Preparing the repo and printing schema

The first drill is to simply load the parquet file and print the schema.

I initially came up with

1
2
3

let f = File::open("./data/yellow_tripdata_2024-01.parquet")?;
let mut pr = ParquetReader::new(f);
println!("{:?}", pr.schema()?);

The above would return

shengy@Mac ~/repos/ruststuff/one-more-time/polars-practice (main|dirty:1) [07:11:07]
> cargo run
   Compiling polars-practice v0.1.0 (/Users/shengy/repos/ruststuff/one-more-time/polars-practice)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.87s
     Running `target/debug/polars-practice`
Schema { fields: {"VendorID": Field { name: "VendorID", dtype: Int32, is_nullable: true, metadata: None }, "tpep_pickup_datetime": Field { name: "tpep_pickup_datetime", dtype: Timestamp(Nanosecond, None), is_nullable: true, metadata: None }, "tpep_dropoff_datetime": Field { name: "tpep_dropoff_datetime", dtype: Timestamp(Nanosecond, None), is_nullable: true, metadata: None }, "passenger_count": Field { name: "passenger_count", dtype: Int64, is_nullable: true, metadata: None }, "trip_distance": Field { name: "trip_distance", dtype: Float64, is_nullable: true, metadata: None }, "RatecodeID": Field { name: "RatecodeID", dtype: Int64, is_nullable: true, metadata: None }, "store_and_fwd_flag": Field { name: "store_and_fwd_flag", dtype: Utf8View, is_nullable: true, metadata: None }, "PULocationID": Field { name: "PULocationID", dtype: Int32, is_nullable: true, metadata: None }, "DOLocationID": Field { name: "DOLocationID", dtype: Int32, is_nullable: true, metadata: None }, "payment_type": Field { name: "payment_type", dtype: Int64, is_nullable: true, metadata: None }, "fare_amount": Field { name: "fare_amount", dtype: Float64, is_nullable: true, metadata: None }, "extra": Field { name: "extra", dtype: Float64, is_nullable: true, metadata: None }, "mta_tax": Field { name: "mta_tax", dtype: Float64, is_nullable: true, metadata: None }, "tip_amount": Field { name: "tip_amount", dtype: Float64, is_nullable: true, metadata: None }, "tolls_amount": Field { name: "tolls_amount", dtype: Float64, is_nullable: true, metadata: None }, "improvement_surcharge": Field { name: "improvement_surcharge", dtype: Float64, is_nullable: true, metadata: None }, "total_amount": Field { name: "total_amount", dtype: Float64, is_nullable: true, metadata: None }, "congestion_surcharge": Field { name: "congestion_surcharge", dtype: Float64, is_nullable: true, metadata: None }, "Airport_fee": Field { name: "Airport_fee", dtype: Float64, is_nullable: true, metadata: None }}, metadata: {} }

Later during some refactoring, instead of getting schema from ParquetReader, I got the DataFrame from ParquetReader and called the schema() method on it.

// in main
let f = File::open("./data/yellow_tripdata_2024-01.parquet")?;
let pr = ParquetReader::new(f);
let df_jan = pr.finish()?;

// basic::drill_1_print_schema
pub fn drill_1_print_schema(df: &DataFrame) {
    println!("{:?}", df.schema());
}

shengy@Mac ~/repos/ruststuff/one-more-time/polars-practice (main|clean) [07:08:21]
> cargo run
   Compiling polars-practice v0.1.0 (/Users/shengy/repos/ruststuff/one-more-time/polars-practice)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.53s
     Running `target/debug/polars-practice`
Schema { fields: {"VendorID": Int32, "tpep_pickup_datetime": Datetime('ns'), "tpep_dropoff_datetime": Datetime('ns'), "passenger_count": Int64, "trip_distance": Float64, "RatecodeID": Int64, "store_and_fwd_flag": String, "PULocationID": Int32, "DOLocationID": Int32, "payment_type": Int64, "fare_amount": Float64, "extra": Float64, "mta_tax": Float64, "tip_amount": Float64, "tolls_amount": Float64, "improvement_surcharge": Float64, "total_amount": Float64, "congestion_surcharge": Float64, "Airport_fee": Float64}, metadata: () }

I just noticed that the schema printed by DataFrame has less information than the schema printed by ParquetReader though.

agy answer for this

The reason is that ParquetReader is reading the physical Parquet/Arrow schema. This physical schema includes a lot of low-level file metadata, like whether a specific column is_nullable, dictionary encodings, and parquet-specific metadata. On the other hand, DataFrame.schema() prints the logical Polars schema. Polars simplifies this representation in memory to just be a mapping of Column Name -> DataType because that’s all the logical query engine needs to know to plan operations.

DataFrame and LazyFrame

Starting from drill 5 I realized that the best practice is to take in LazyFrame if possible. LazyFrame is similar to Rust’s iterators, you can write all the operations/expressions/transformations, but it won’t be actually executed until you call collect(). And when it is executed Polars would help optimize it so it’s as fast as possible.

On the other hand operations on DataFrames are executed immediately, so there is no opportunity for Polars to help optimize it based on a new transformation down the line.

I first learned about lazy vs eager in programming languages. This, this and this are some example links.

rolling_window feature requires strings

In extra drill 1 I needed to use a method called rolling_mean. It was not available and later I found that it is under the crate feature rolling_window.

After installing this feature it was still failing with mysterious messages. Turns out the fix was to have strings also included in the features. I believe rolling_window feature depends on strings feature.

agy says

The underlying reason here isn’t necessarily that rolling_window always depends on strings. It’s because in your specific drill, you are performing a rolling window over a datetime column (tpep_pickup_datetime). Polars heavily relies on the strings feature when formatting, parsing, or interacting with datetime objects under the hood. Polars is incredibly modular, so hitting these weird feature-flag combinations is a very common rite of passage in Rust Polars!

How to read the types, Expr, PlSmallStr

When working through the basic drills I learned to read the type hints from Polars. I got very confused about when should I pass in a string like stuff vs when should I pass in col("stuff")

Examples are

let x = df
    .lazy()
    .group_by([col("tpep_pickup_datetime").dt().hour().alias("hour")])
    .agg([col("total_amount").mean(), col("trip_distance").mean()])
    .sort(
        ["hour"],
        SortMultipleOptions::new().with_order_descending(false),
    );

Why in this case group_by takes col("tpep_pickup_datetime") and why does sort take "hour"?

What I learned is to follow the types.

Expr

If we look at group_by’s type. It is the following

`1`	`pub fn group_by<E: AsRef<[IE]>, IE: Into<Expr> + Clone>(self, by: E) -> LazyGroupBy`

This means group_by takes in a slice of IE as reference, where IE is Into<Expr>. Meaning anything that implemented the Into<Expr> trait could be passed in and the result would be Expr.

So all the babbling above means group_by takes in a slice of Exprs

And in Polars, col("stuff") is an Expr.

Note that since it takes Into<Expr>, the following also works

let x = df
    .lazy()
    .group_by(["payment_type"]) // here
    .agg([col("tip_amount").mean()])
    .sort(
        ["payment_type"],
        SortMultipleOptions::new().with_order_descending(false),
    );

I didn’t pass in [col("payment_type")], instead I passed in ["payment_type"].

Here an Into happened and converted "payment_type" to an Expr

PlSmallStr

The sort method in the example above takes in ["payment_type"], but this is not converting it into an Expr.

See sort’s type

`1`	`pub fn sort(self, by: impl IntoVec<PlSmallStr>, sort_options: SortMultipleOptions) -> Self`

sort’s by parameter takes IntoVec<PlSmallStr>. Honestly I don’t know the exact details of how PlSmallStr works, but the way I read this is: it takes something that could be converted into a vector, and that something is a small string.

So when I see PlSmallStr, I know it doesn’t take Expr, so I wouldn’t pass col("stuff") to it.

DataFrame, LazyFrame

As I work on more drills, specifically extra drill 3 when I was working on over.

I noticed that group_by is a method on the Frame, whereas over is a method on the Expr. We’ll go deeper to group_by and over later.

The DataFrame methods(DataFrame, LazyFrame) often return the DataFrame. Learned from extra drill 2, most DataFrame methods transform the rows, they don’t filter. More of the below as well.

group_by vs over

Although I have been using GROUP BY in sql a lot I think I didn’t fully master it in my mind. Now with the drill I think I got a better mental model.

I think it is very important to visualize things in your mind before you go and write code. This applies to algorithms as well, it is always easier for me to “see” how the graph is being traversed and how the code is handling that.

Back to group by, let me describe how I visualize it in my mind.

Assume that we have 5 rows in the DataFrame

ID	Color	Shape	Weight
1	red	ball	10
2	green	cone	12
3	blue	cone	13
4	red	cone	14
5	green	ball	15

I want to get the average weight by color.

I would group_by color. And in my mind, after group by color, the table is broken into rows And the rows are grouped by color scattering into groups/heaps (搓堆儿 in Beijing Dialect)

It looks something like the following in my mind

Now I want to calculate the average weight of each of these groups. So I apply the calculation I have in agg for each group. In the current case it’s simply calculating the average of weight. So each group would have one number, which is the average weight of that group.

Then the final output would be two columns, the group_by field, in this case the color. And the aggregated result.

Color	Avg Weight
red	17
green	13.5
blue	13

The above is the simple example, there are a few more things to note.

group by destroys the original DataFrame

If you haven’t noticed yet, group_by destroys the original DataFrame and splits it into groups. It might not be apparent in other languages but in rust you can see it clearer because group_by takes in self, it actually consumes it.

group by multiple columns

The other thing is you could group by multiple columns, the main difference is how many groups(堆儿) would you have. In the example above if we group by (color, shape), we would have the following groups

(red, ball)
(green, cone)
(blue, cone)
(red, cone)
(green, ball)

Which doesn’t make much sense since each group has one row, but you get the idea :)

agg takes in a slice

In our example above we only calculated the average of weight of each group. But agg can take in a slice. What does it mean to pass in a slice with two elements?

This means for each group you could calculate multiple things and each of them would be a new column in the final result.

For instance we could not only calculate the average weight but also calculate the number of rows in the group, and end up with a result like

Color	Avg Weight	Count
red	17	2
green	13.5	2
blue	13	1

Now group_by is all g, let’s talk about over

over

I believe over is actually called window functions. In the drill it first came up at extra drill 3.

In some sense group_by and over is very similar, they all perform calculation on a subset of data.

But from this drill I figured out a way to visualize over.

First things first, over does not destroy the DataFrame. So when I visualize over, instead of breaking the DataFrame, I visualize it as adding a new column to the entire table.

Using the color ball example we have, say we want to calculate average weight with over, the way I would visualize it is to

Add a column to the DataFrame
Express the calculations
Specify the calculations over what
Give the column a name

(1) Add a new column

`1`	`lf.with_column(...);`

ID	Color	Shape	Weight	???
1	red	ball	10	???
2	green	cone	12	???
3	blue	cone	13	???
4	red	cone	14	???
5	green	ball	15	???

(2) Specify the calculation

1
2
3

lf.with_column(
        col("Weight").mean() // not finished yet
);

(3) Specify the calculation over what

1
2
3

lf.with_column(
        col("Weight").mean().over([col("Color")])? // question mark is because over returns PolarsResult
);

(4) Give the column a name

1
2
3

lf.with_column(
        col("Weight").mean().over([col("Color")])?.alias("Avg Weight By Color")
);

So after all these the DataFrame should look like

ID	Color	Shape	Weight	Avg Weight By Color
1	red	ball	10	17
2	green	cone	12	13.5
3	blue	cone	13	13
4	red	cone	14	17
5	green	ball	15	13.5

Notice that the original DataFrame is still there, we just added a column.

The transforms rows mindset

While doing extra drill 2 I came across learning the idea that most DataFrame transforms the rows.

The drill wanted me to filter by payment_type == 1 or payment_type == 2.

Following is the filter on LazyFrame, but something interesting is happening.

lf.filter(
    col("payment_type")
    .eq(lit(1))
    .or(col("payment_type").eq(lit(2))));

If we ignore filter, ignore the payment_type == 2 part, the code above is simplified as

`1`	`col("payment_type").eq(lit(1))`

As we were going through the different types above, col returns Expr, eq consumes Expr and returns Expr.

I initially thought that the eq would do the filtering for me, but it turns out that it only transforms the DataFrame.

The example is explained in the comments. But let me try to write it down one more time.

Say our column of payment_type is now [1, 2, 1, 1, 2].

col("payment_type") would give you [1, 2, 1, 1, 2]

Then col("payment_type").eq(lit(1)) would transform [1, 2, 1, 1, 2] to [true, false, true, true, false] by applying eq.

If we want to actually filter here we would slap on a filter like

`1`	`lf.filter(col("payment_type").eq(lit(1)));`

In the drill we wanted to calculate the percentage of people paying by credit cards(payment_type == 1).

Say col("payment_type") is [1, 2, 1, 1, 2].

payment_type == 1 would result in [true, false, true, true, false].

So the percentage is 3 / 5 (3 true out of 5 total).

There is a trick applied here, where after we got [true, false, true, true, false], we did cast(DataType::Float64). This casts true to 1.0, false to 0.0.

So [true, false, true, true, false] transformed into [1.0, 0.0, 1.0, 1.0, 0.0]. And if you run the mean on this you actually get the percentage of trues. Which is the percentage of people paid by credit cards.

I assume I already confused you, the code below probably explains it better.

Let’s look at how this calculates the percentage. Assuming the rows that fall into one specific PULocationID group have the payment types [1, 2, 1, 1, 2], here is how the expression evaluates for that group:

let x = lf
    .filter(
        col("payment_type")
            .eq(lit(1))
            .or(col("payment_type").eq(lit(2))),
    )
    .group_by([col("PULocationID")])
    .agg([
        col("payment_type")          // [1, 2, 1, 1, 2]
            .eq(lit(1))              // [true, false, true, true, false]
            .cast(DataType::Float64) // [1.0, 0.0, 1.0, 1.0, 0.0]
            .mean()                  // 3.0 / 5.0
            .alias("credit_pct"),
        col("payment_type")          // [1, 2, 1, 1, 2]
            .eq(lit(2))              // [false, true, false, false, true]
            .cast(DataType::Float64) // [0.0, 1.0, 0.0, 0.0, 1.0]
            .mean()                  // 2.0 / 5.0
            .alias("cash_pct"),
    ]);

or on lit

If we go back to the filtering part, it asks: filter by payment_type == 1 or payment_type == 2.

In the first try I actually wrote

`1`	`lf.filter(col("payment_type").eq(lit(1).or(lit(2)))); // WRONG`

This actually passed compile, but this is wrong, because or is applied on lit(1), not on the result of eq

`1`	`lf.filter(col("payment_type").eq(lit(1)).or(lit(2))); // ALSO WRONG`

This is wrong as well, because or(lit(2)) would always be true

`1`	`lf.filter(col("payment_type").eq(lit(1)).or(col("payment_type").eq(lit(2)))); // CORRECT`

The above is the correct one. Not gonna lie the need to repeat col("payment_type") is a bit annoying.

If using LazyFrame construct it with scan_parquet

The drills started by loading a DataFrame via ParquetReader. Later, agy told me that doing this is actually bad because the entire DataFrame is read into memory.

In order to only read the columns needed and leave the rest out of memory, agy suggested LazyFrame::scan_parquet.

I tried it out and it actually worked :)

Some other small things

The following are things that I noticed during the practice but I don’t think it’s that much to go over, hence grouping them in a h2 here.

with_column vs select

with_column adds new columns, where select would slice other columns and only keep the ones you selected.

columns could do math

You could do calculations for entire columns. basic drill 8 simply divided two columns.

#[allow(unused)]
pub fn drill_8_new_trip_percentage(lf: LazyFrame) -> Result<(), Box<dyn std::error::Error>> {
    // 8. Add a new column tip_percentage calculated as tip_amount / fare_amount * 100. Print the first 5 rows showing only fare_amount, tip_amount, and tip_percentage.

    let x = lf
        .filter(col("fare_amount").gt(2.5))
        .with_columns([(col("tip_amount") / col("fare_amount") * lit(100.0)).alias("tip_percentage")])
        .select([col("fare_amount"), col("tip_amount"), col("tip_percentage")])
        .sort(
            ["tip_percentage"],
            SortMultipleOptions::new().with_order_descending(true),
        )
        .limit(20);
    println!("{}", x.collect()?);
    Ok(())
}

POLARS_FMT_MAX_ROWS to show more rows

When printing the results, I noticed that some of the rows are omitted. A quick Google search shows that setting the POLARS_FMT_MAX_ROWS environment variable would allow more rows to be displayed.

Rust Learning: Deref

Fri, 29 May 2026 21:50:26 -0400

tldr

deref() returns reference because
- Rust desugars *a to *a.deref()
- Rust needs the size at compile time
- We don’t want to consume by calling deref()
deref coercion can happen
- on function parameters
- on method call expressions
- on assignments with type annotations
as_ref() and as_deref() always return references (or containers of references), which always implement Copy

Overview

I was practicing rust by doing this and this.

While chatting with Claude and trying to learn, I saw that there is this as_deref() call in the iterator chain.

This triggered some memory (trauma) about Deref and I decided to give it another shot with the help of Claude.

The following is (as usual), what I learned from teacher Claude.

The Confusion

When I first saw Deref trait, my instinct/past experience is telling me this is an “interface” that you could implement on your custom type so when you do *my_instance it triggers the specific “dereference” logic and gives you the underlying value.

Then if we look at Deref’s definition here, you can tell the required method is returning a reference instead of a concrete.

pub trait Deref {
    type Target: ?Sized;

    // Required method
    fn deref(&self) -> &Self::Target; // <----- here, returning a reference instead of Self::Target
}

So to summarize my confusion: Why is deref() in Deref trait returning a reference when it’s “dereferencing”?

The Answer

Apparently in Rust, when you call * on a type that implemented the Deref trait it would call deref() then still slap on the * operator

let boxed_value = Box::new(7);

println!("{}", *boxed_value);
// *boxed_value would be converted to *boxed_value.deref()
// where the desugar is *&i32, which gives i32

But Why?

The above seems to make sense but why can’t the deref() function return Target instead of &Target?

Claude explained that it’s because

Target might not be Sized, so its size might not be known at compile time. In Rust, types must be Sized to be passed or returned by value because the compiler needs to know how much stack space to allocate. For types like str or [T], they must live behind a pointer (like &str).
Returning Target would move it out from the containing struct, causing issues.

Deref Coercion

Now since we are reading/learning about Deref, the term Deref Coercion would be shown and we would be confused.

Deref Coercion in function parameters

Let’s look at an example below.

String implements Deref and its Target is str, its deref() returns &str.

yo() takes &str but passed &String, Rust would do a implicit conversion (coercion) from &String to &str.

fn yo(s: &str) {
    println!("{}", s);
}

let name = String::from("hi");
yo(&name); // here &String is coerced to &str

Note that Rust would do this as many times as needed. So if we have a Box that has a String in it, this would work the same.

fn yo(s: &str) {
    println!("{}", s);
}
let boxed_name = Box::new(String::from("hi"));
yo(&boxed_name); // Box<String>.deref() gets &String then String.deref() gets &str

Deref Coercion in method call expressions

Deref Coercion not only happens when passing function arguments, but also happens when we call methods.

let name = String::from("hi");

name.contains("h");
// .contains(&self, pat: P) exists on `str` but not `String`

The method call coercion doesn’t require you to add & yourself because in Method-call expressions the derefs are being considered.

The first step is to build a list of candidate receiver types. Obtain these by repeatedly dereferencing the receiver expression’s type, adding each type encountered to the list, then finally attempting an array unsized coercion at the end, and adding the result type if that is successful.

Deref Coercion in assignments with type annotations

Deref Coercion could also happen when doing an assignment with type annotations. This is kinda self explanatory so I’ll just skip.

1
2

let name = String::from("hi");
let s: &str = &name; // &String coerced to &str on assignment

1
2

let boxed_name = Box::new(String::from("hi"));
let s: &str = &boxed_name; // &Box<String> coerced to &String coerced to &str on assignment

as_deref() on Option

If you still remember, the reason I started this post is I saw this as_deref call in the iterator chain and got me thinking.

But all the discussions above don’t really help, do they?

Although the place where I was confused is on Vec, later practice shows that it might be more beneficial to understand as_deref on Options.

What does as_deref() on Option do?

Let’s look at as_ref() first

To speak about as_deref() on Options let’s have a look at as_ref() first.

Option’s as_ref says it

Converts from &Option to Option<&T>.

So given &Option<String> I can get an Option<&String>.

Yeah yeah sure sure, but what exactly does this mean?

Think about the following, if I have an Option<String>, and I have another function that takes in a &String. Is there a way for me to pass the String in the Option as a borrow to the function without consuming the original Option?

as_ref() is the answer, because it converts &Option<String> to Option<&String>, this gives you the ability to look inside the option without consuming it.

Let’s have a look at an example

fn yo(s: &String) {
    println!("{}", s);
}

let value = Some("stuff".to_owned());
if let Some(inner) = value {
    yo(&inner);
    // value is consumed here, no longer available
    // inner's type is actually String, hence the need to borrow before passing to `yo`
}

let another = Some("other stuff".to_owned());
if let Some(inner) = another.as_ref() { // as_ref() takes &self, so another is borrowed here instead of consumed
    yo(inner);
    // here `inner`'s type is actually &String
}
// `another` is still available

Now deref() should feel natural

What if the function above requires &str instead of &String?

here is where as_deref() comes into play, it not only does what as_ref() does, it also calls deref() as it sees fit to do the conversions.

fn yo(s: &str) {
    println!("{}", s);
}

let another = Some("other stuff".to_owned());
if let Some(inner) = another.as_deref() {
    // inner's type is actually `&str`
    yo(inner);
}
// another is still available here

Granted the as_deref() is not strictly needed in this specific case because &String would be coerced to &str anyways.

Then the natural question is, what is the case where I absolutely needs as_deref() over as_ref()?

The answer is when yo() requires Option<&str>

fn yo(x: Option<&str>) {
    println!("{:?}", x);
}

let another = Some("other stuff".to_owned());
yo(another.as_deref());
// another is still available
// side note: yo does consume the option but as_deref() provided a copy
// as_ref() and as_deref() always return references (or containers of references), which always implement Copy

But How are these implemented?

Now we know why does as_ref() and as_deref() exist, when and why to use them. The next question is, how are they implemented?

How are as_ref() implemented?

Let’s have a look at Option’s as_ref()

// &self is a short hand of self: &Self
pub const fn as_ref(&self) -> Option<&T> {
    match *self {
        Some(ref x) => Some(x),
        None => None,
    }
}

1
2

let value = Some("stuff".to_owned());
let x = value.as_ref(); // x's type is Option<&String>

value.as_ref() is being called
as_ref(&self) takes &self, which is a short hand of as_ref(self: &Self)
Which means Self is Option<String>
Which means self’s type is &Option<String>

Now we are clear about the types of self and Self. Let’s go to the match

We are matching *self (note that this could be written better with matching ergonomics)
As we said, self’s type is &Option<String> so matching *self means matching Option<String>
Now we are looking at Some(ref x) matching on Option<String>
- ref x in a pattern means “instead of moving the matched value, give me a reference to it”.
- So we got a reference of String which is &String
- And we returned it as Some(x) which is of type Option<&String>

// with matching ergonomics as_ref() could be written as
pub const fn as_ref(&self) -> Option<&T> {
    match self {
        Some(x) => Some(x), // because `self` is &Option<String>, `x` is automatically bound as &String
        None => None,
    }
}

How are as_deref() implemented?

Let’s see the code first.

pub const fn as_deref(&self) -> Option<&T::Target>
where
    T: [const] Deref,
{
    self.as_ref().map(Deref::deref)
}

And let’s use our example

1
2

let value = Some("stuff".to_owned());
let x = value.as_deref(); // x's type is Option<&str>

value.as_deref() being called
as_deref(&self) takes &self, which means self: &Self which means self’s type is &Option<String>
self.as_ref() is called, we get Option<&String> back
We call map with Deref::deref, so Option<&String> becomes Option<&str>
- This is because String’s Deref Target is str
- and deref() returns &Target so deref() returns &str

Bonus: DerefMut

DerefMut is the mut version of deref. If we have a &mut String, Rust can coerce it to &mut str if DerefMut is implemented.

Rust Learning: Lifetime

Wed, 20 May 2026 07:36:50 -0400

Disclaimer

I’m in the process of learning and figuring out my confusions, there might be mistakes. What I’m writing here is more of me trying to figure things out and some of my current understandings.

tldr

Can be used in functions, in structs, in traits
lifetime subtyping
'long:'short means 'long outlives 'short, also 'long is a subtype of 'short
Three syntaxes
- Right after & means annotating the lifetime of the reference
- As generic lifetime parameters
- In trait bounds
'static
- Right after & means this reference needs to be available for the processes lifetime
- T: 'static means
  - For reference fields in T, the lifetime has to be valid for the entire program
  - Or T owns the fields
- F: 'static where F is closure means
  - F should own the captured data (move closure)
  - Or F only capture 'static data.
Type outlives lifetime
- T: 'a
  - For reference fields in T, the lifetime has to outlive 'a

Overview

I recently came across learning Rust (again, like the 100th time). I thought maybe if I start writing things down, it helps solidifying my understanding a bit.

This post is about some of my confusions and thoughts that occurred when I was reading the Rust book.

My first confusion

In Chapter 10 Section 3, the example function was trying to explain how to write a function that returns the longer string without taking ownership.

1
2
3

fn longest<'a>(x: &'a str, y: &'a str) -> &'a str {
    if x.len() > y.len() { x } else { y }
}

I’ve seen this many times, but this time I wanted to dig a bit deeper.

The book says:

The function signature now tells Rust that for some lifetime ‘a, the function takes two parameters, both of which are string slices that live at least as long as lifetime ‘a. The function signature also tells Rust that the string slice returned from the function will live at least as long as lifetime ‘a.

In practice, it means that the lifetime of the reference returned by the longest function is the same as the smaller of the lifetimes of the values referred to by the function arguments. These relationships are what we want Rust to use when analyzing this code.

The part that got me thinking was both of which are string slices that live at least as long as lifetime ‘a.

Looking at the function signature, I couldn’t see where it expressed at least as long as. Say if I have a function that takes two ints, I know I need exactly two ints.

Again, long conversations with Gemini, the reasoning seems to be called lifetime subtyping. And it’s similar to the OOP concept that I am familiar with.

In the following code, we are passing in a Cat to a function that takes an Animal, and it seems normal.


class Animal {
public:
    virtual string name() = 0;
    virtual int weight() = 0;
};

class Cat : public Animal {
public:
    string name() { return "Cat"; }
    int weight() { return 20; }
};

void logAnimal(Animal *a) {
    std::cout << a.name() << " " << a.weight() << std::endl;
}

Cat *c = new Cat; // let's ignore the memory leak
logAnimal(c);

This is polymorphism 101: a Cat inherits from Animal. Wherever I need an Animal, I could pass in a Cat because Cat inherits from Animal; in other words, Cat is an Animal.

What I didn’t know was, there is another “in other words”: Cat is a subtype of Animal.

With the detour above out of the way, let’s have a look at the function again:

1
2
3

fn longest<'a>(x: &'a str, y: &'a str) -> &'a str {
    if x.len() > y.len() { x } else { y }
}

There are 4 'as, they are not all the same, the first one longest<'a> is where we are declaring a generic type 'a.

The later 3 are using 'a declared by us

If you look at lifetime 'a as a type, then combine what we said about polymorphism and subtypes, things start to make sense.

The parameter x: &'a str is saying if the passed in argument’s lifetime “satisfies” 'a then it’s good. “satisfying” here in lifetime means the lifetime passed in can not be gone before 'a. In other words, the lifetime passed in has to live at least as long as 'a.

So say the lifetime of the argument is 'b, what we are saying above is: 'b has to be at least as long as 'a.

I happen to know that there is a term in Rust called outlive. If we have two lifetimes 'long and 'short, then the syntax 'long:'short in trait bounds means 'long outlives 'short. We’ll come back to this when we talk about lifetime in trait bounds.

So now it is clear to me why did the book say both of which are string slices that live at least as long as lifetime ‘a. It is because the lifetime parameter we declared in our function signature is like a “Base Class”, you could pass in an argument with a lifetime which has a longer lifetime and it would satisfy the parameter.

The other thing is when you pass in two different lifetimes to longest, rust would find out the overlap and bind it to 'a

Lifetime on structs

With above it is clear to me that lifetimes in Rust are special generic types, and fn longest<'a>(x: &'a str, y: &'a str) -> &'a str is a generic function.

Naturally after a generic function you would think about a generic class.

This is where I got confused yet another time by the syntax. But to be fair I think the confusion comes from the generic syntax not from lifetimes(they are all generics though)

struct User<'a> {
    name: &'a str,
}

fn main() {
    let name = "my name";
    let u = User { name };
    println!("{}", u.name);
}

So far things seems pretty normal, struct User<'a> defines a struct which has a generic type parameter (in this case it’s a lifetime) 'a.

And the lifetime is used to say that the string slice you refer to that represents the name needs a lifetime, and that lifetime needs to outlive 'a

I started to feel confusing when I wanted to create a constructor.

In Rust methods are defined outside of the struct using impl.

The following is what I did at first try.

// WRONG

impl User<'a> {
    fn new(name: &'a str) -> Self {
        Self { name }
    }
}

The compile error is “undeclared lifetime”, and the fix is to add <'a> after impl

error[E0261]: use of undeclared lifetime name `'a`
 --> src/main.rs:5:11
  |
5 | impl User<'a> {
  |           ^^ undeclared lifetime
  |
help: consider introducing lifetime `'a` here
  |
5 | impl<'a> User<'a> {
  |     ++++

impl<'a> User<'a> {
    fn new(name: &'a str) -> Self {
        Self { name }
    }
}

This got me thinking, why do I need to have <'a> after impl, isn’t User<'a> enough to say that my type is taking a generic lifetime parameter?

impl<'a> is where 'a is declared as a new generic parameter. Without it, Rust sees 'a in impl User<'a> as an undeclared name — the same way using a variable before declaring it would be an error.

So now with the syntax issue resolved lifetime on structs are clear. Let’s recap

When defining a struct you can take in lifetimes
lifetimes are special generic lifetime parameters hence they follow the same generic syntax
The generic lifetime parameter is instantiated when we create an instance, it binds to the lifetime it get passed to (So our struct “passes through” the lifetime to it’s own fields)
The lifetime took in could annotate reference fields in the struct (which the struct does not own)
Rust would make sure that the lifetime we took in outlives the instance of our class

See Gemini generated example below (well Gemini just took it from here)

// 1. We declare the generic lifetime `<'a>` right after the struct name.
// 2. We apply it to the `part` field.
struct ImportantExcerpt<'a> {
    part: &'a str,
}

fn main() {
    let novel = String::from("Call me Ishmael. Some years ago...");
    let first_sentence = novel.split('.').next().unwrap();

    // The struct instance `i` now holds a reference to the data inside `novel`.
    let i = ImportantExcerpt {
        part: first_sentence,
    };

    println!("Excerpt: {}", i.part);
}

This struct has the single field part that holds a string slice, which is a reference. As with generic data types, we declare the name of the generic lifetime parameter inside angle brackets after the name of the struct so that we can use the lifetime parameter in the body of the struct definition. This annotation means an instance of ImportantExcerpt can’t outlive the reference it holds in its part field.

When we created an instance of ImportantExcerpt, it also took in a lifetime parameter, the lifetime i took in is the lifetime of novel because ImportantExcerpt took in first_sentence which is a &str which refers to novel. The lifetime tracks where the data lives(novel), not what variable you used to access it(first_sentence).

So as long as novel outlives i Rust would not complain.

Side note: Gemini actually hallucinated pretty hard here, in one version it keeps saying that 'a is the lifetime of i. I’m actually considering subscribing to Claude

Lifetime on Traits

Self Review Note:

After learning lifetimes on structs, I was looking at lifetime on Traits, the following is an example Gemini gave me. It seems like the trait bound T: 'a is redundant and is implied by Rust.

But let’s go through the process together

The Lifetime Bound

use std::fmt::Display;

// We declare a lifetime 'a and a generic type T
struct Logger<'a, T>
{
    borrowed_data: &'a T,
}

impl<'a, T> Logger<'a, T>
where
    T: Display + 'a, // This means T must implement Display and outlive 'a
{
    fn info(&self) {
        println!("INFO: {}", self.borrowed_data);
    }

    fn error(&self) {
        println!("ERROR: {}", self.borrowed_data);
    }
}

fn main() {
    let novel = String::from("Call me Ishmael. Some years ago...");

    let l = Logger {
        borrowed_data: &novel,
    };

    l.info();
    l.error();
}

And here T outlive 'a means if T contains any references, those references need to outlive 'a (those references needs to live longer than 'a)

Side topic: I learned that in rust different impl blocks can have different trait bounds, also the struct definition can have trait bounds. Seems like the idiomatic way is to keep the struct clean for bounds and have the bounds on the impl blocks

When staring at the code snippet above I just feel like there is something still not making sense. Logger takes in a T and references it borrowed_data: &'a T, 'a is the lifetime that is passed in, T is also the type that is passed in, aren’t they the same?

fn main() {
    let novel = String::from("Call me Ishmael. Some years ago...");

    let l = Logger {
        borrowed_data: &novel,
    };

    l.info();
    l.error();
}

In the above T is String, and 'a' is the lifetime of novel, feels like they are the same thing to some extent?

Claude says

1
2
3

struct Logger<'a, T> {
    borrowed_data: &'a T,  // <-- this implies T: 'a, always
}

So two more questions here

Why/how does it imply T: 'a?
In what cases do I actually need to put T: 'a in the trait bound?

For (1), the reason why rust can imply T:'a from &'a T is because

&'a T means a reference to T for a duration of 'a
Which means T needs to live at least as long as 'a
Which means T outlives 'a
Which is T: 'a

For (2) I don’t have a good answer now, but the most common case seems to be traits slapping on the 'static lifetime

The static Lifetime Bound

Here is an example of having 'static in a trait bound.

pub fn spawn<F, T>(f: F) -> JoinHandle<T>
where
    F: FnOnce() -> T + Send + 'static,
    T: Send + 'static,

Having 'static here on T means if T has any reference fields, they have to outlive 'static (which means they have to live as long as the program), or T has to own the fields.

The 'static on F is saying F should own the captured data (move closure) or only capture 'static data

Summary

The deeper I dig the more questions I have. I did find a few links that seems worth checking out

I did have a simliar feeling of leaning C++ again, where you just keep having questions over questions over questions and at some point you don’t remember why you are figuring these things out. And after you think you are crystal clear, how are you going to apply this in your own code? I’m still not sure what is the answer of that. Maybe I need to write a bit more Rust to internalize these concepts?

My Two Cents on Postgres Isolation Levels

Mon, 27 Apr 2026 20:35:03 -0400

tldr

PostgreSQL is awesome
read committed takes a new snapshot for every SQL statement
repeatable read only has one snapshot
repeatable read fails when the row it tries to modify was changed by other transactions
repeatable read hang -> wakeup -> dies when the row of the lock it’s waiting is updated by other transactions
serializable == repeatable read + serializable snapshot isolation
serializable handles read write cases
transactions are nodes in precedence graph
t1 reads row1 and t2 writes row1 means t1 -> t2 in the precedence graph
PostgreSQL serializable is optimistic

Overview

I always had a vague understanding of database isolation levels, recently I discovered that I had a major misunderstanding of it. I talked to Gemini for hours and “I think” I got it. This post is to verify that I actually got it with examples.

This post is specific for PostgreSQL.

When you go to the official PostgreSQL Transaction Isolation documentation it throws a lot of concepts to you and (at least for me) it was very confusing.

For SQL there is a standard, isolation levels are defined by the standard. The standard uses what “phenomena” is allowed to define the isolation level. This is like C++ has standards and each compiler implements it differently.

Honestly I sometimes still get confused by this table, I think the main reason is I don’t understand the “phenomena” by the bottom of my heart. Also it’s very hard to relate this to my day to day problems. I’m going to try to use the way that I hope someone would have explained this to me to write this post.

For now the things to understand is

There is a SQL Standard about isolation levels. It is defined by what phenomena is allowed at each level.
Each database implementation might implement it differently. Some might guarantee more than what the standards asks for. Also the standard doesn’t say if the implementation should use optimistic concurrency control or pessimistic concurrency control.

Optimistic Concurrency Control vs Pessimistic Concurrency Control

Before we go a bit further I think it’s nice to understand what is Optimistic Concurrency Control(OCC) vs Pessimistic Concurrency Control(PCC). In my understanding, OCC is where you are optimistic that there won’t be much conflicts, and if a conflict happens, you simply die and retry it. PCC is where you are pessimistic, and you believe that conflicts are gonna happen a lot, so you use locks to sort things out. When there is a conflict, you just wait for your turn.

The mistake I made was I thought PostgreSQL’s Serializable was pessimistic (it’s optimistic)

Example

Assume there is a bathroom in a building to be shared, an optimistic concurrency control scenario would be like there isn’t many people in the building, so we don’t even have a lock on the bathroom door, anyone could just walk in. Most of the times there won’t be anyone in there so you just walk in and use the bathroom, if you got unlucky and opened the door where someone else was in the bathroom, you come out (abort) and retry later.

Now assume that there are a lot of people in the building, it is highly likely that when you are using the bathroom someone wants to use it as well. We would add a lock to the bathroom door and have the key at the front desk, anyone who wants to use the bathroom has to wait at the front desk for the key then go to the bathroom. This is pessimistic concurrency control.

Now we have an understanding of optimistic concurrency control and pessimistic concurrency control let’s have a look at PostgreSQL’s isolation levels.

Let’s setup the environment

Right now postgres 18.3 is the latest version, let’s just use that.

Run docker pull postgres:18.3

Then let’s spin up the database with

docker run --name pg_isolation_test \
  -e POSTGRES_USER=testuser \
  -e POSTGRES_PASSWORD=testpass \
  -e POSTGRES_DB=isolation_db \
  -p 5432:5432 \
  -d postgres:18.3

You can use docker ps to confirm that postgres is up and running

CONTAINER ID   IMAGE           COMMAND                  CREATED              STATUS              PORTS                                         NAMES
883e64ddfcbc   postgres:18.3   "docker-entrypoint.s…"   About a minute ago   Up About a minute   0.0.0.0:5432->5432/tcp, [::]:5432->5432/tcp   pg_isolation_test

Now open two terminals. In both run docker exec -it pg_isolation_test psql -U testuser -d isolation_db

In any of them let’s use the following schema and test data

CREATE TABLE account (
    id SERIAL PRIMARY KEY,
    name VARCHAR(50),
    balance INT
);

INSERT INTO account (name, balance) VALUES ('Alice', 100);
INSERT INTO account (name, balance) VALUES ('Bob', 200);

Now we should have the test table, test data, and two connections to try out things.

How to reason about transactions on paper

I found the most easy way to reason about transactions on paper is to use a two column table, where each column represents a single transaction and time goes from top to bottom. You could use excel or google sheets to try this out.

Given our test table, assume we are running the following SQL.

Two transactions trying to withdraw 100 from Alice’s account.

BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;

UPDATE account
SET balance = balance-100
WHERE name = 'Alice' and balance >= 100;

COMMIT;

Would the SQL above drop the balance in database to below 0? Before we go and work it out on paper, I’d like to touch on one more thing.

Postgres uses MVCC, and the golden rule of MVCC is

Readers do not block Writers, and Writers do not block Readers. But Writers ALWAYS block Writers

When you run UPDATE, DELETE, SELECT FOR UPDATE they all trigger an exclusive row lock. This is irrelevant to the isolation level. Also the lock is only released when the transaction ends (COMMIT; or ROLLBACK;)

Read Committed Scenario On Paper: Tx1 and Tx2 both update before commit

See the table below and let’s walk through it

At T3, Tx1 UPDATE the balance to 0. And Tx1 locks Alice’s row because it is doing an UPDATE
At T4, Tx2 is blocked by the row lock
At T5, Tx1 commits and Alice’s balance is set to 0. Also Tx1 releases the lock. This would unblock Tx2, where Tx2 would re-evalulate the where clause and since balance is 0 it fails the where clause hence update 0 rows
At T6, Tx2 commits, nothing interesting happened since it updated 0 rows

So after this Alice’s balance is 0, records in database is still correct.

Read Committed Scenario On Paper: Tx1 commits before Tx2’s update

Let’s try to let Tx1 commit before Tx2’s statement runs

At T3, Tx1 sets the balance to 0, Tx1 locks Alice’s row
At T4, Tx1 commits, Tx1 releases Alice’s row lock
At T5, Tx2 sees balance is 0 (since the isolation level is read committed), so where clause fails, note that since the where clause fails, Tx2 does not lock any rows
At T6, Tx2 commits but nothing changed

So after this Alice’s balance is 0, records in database is still correct.

How to try out transactions with a test database

No matter how much you convinced yourself on paper, without trying it out there is still a chance what you believed is wrong. I can’t tell how many times I was dead sure about something and it turns out to be wrong.

So let’s try to verify the above in our testing environment.

The Real Deal: Read Committed, Tx1 and Tx2 both update before commit

The Real Deal: Read Committed, Tx1 commits before Tx2’s update

The Four Isolation Levels in Postgres

Read Uncommitted

This is in the SQL standard but PostgreSQL does not allow this, even if you set the transaction isolation level to Read Uncommitted it acts as Read Committed due to MVCC. Honestly I don’t have enough knowledge of MVCC, but I could probably write something about it later when I learn about it.

For now I’ll put the “golden rule of MVCC” which I have above here again for emphasis.

Readers do not block Writers, and Writers do not block Readers. But Writers ALWAYS block Writers

Read Committed

Say you have a transaction with multiple selects and the isolation level is READ COMMITTED, if anything was committed by another transaction between your SELECTs, it would be showing in your up coming SELECT.

Every statement uses a new snapshot of the database.

[Gemini Review] Nuance: Query vs. Statement While we say “every statement,” technically Postgres takes a new snapshot at the start of every query. For simple commands, they are the same. But if you run a complex query with CTEs (Common Table Expressions) or a query that calls a VOLATILE function multiple times, that entire query shares a single snapshot.

You would be able to see uncommitted results within your own transaction.

Single SQL statements all default to READ COMMITTED. Say you do a UPDATE account SET balance=100 WHERE name='Alice', this by default runs as READ COMMITTED, to change the default to say SERIALIZABLE you would need to SET SESSION CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL SERIALIZABLE;


BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;
-- Alice's balance starts as 100

SELECT balance FROM account WHERE name = 'Alice'; -- 100

-- another transaction updated Alice's balance to 70
SELECT balance FROM account WHERE name = 'Alice'; -- 70

-- another transaction updated Alice's balance to 30
UPDATE account SET balance = 100 where name = 'Alice' AND balance=30; -- this goes through but this also grabs a lock
SELECT balance FROM account WHERE name = 'Alice'; -- 100

UPDATE account SET balance = 20 where name = 'Alice'; -- this goes through because the current transaction has the lock
SELECT balance FROM account WHERE name = 'Alice'; -- 20

COMMIT;

The actual tests also show how the UPDATE in our transaction got the locks

When would using READ COMMITTED isolation level in postgres cause issues?

READ COMMITTED is a noice default, but there are two cases where it falls short.

Application relying on stale data

When the application logic stores information in memory and instead of using SQL where clauses it uses a “stale” in memory value to check if it could update or not.

// pseudo go code, ignoring error checking and rollback
func withdraw100() {
    tx.Begin()

    var currentBalance int
    tx.QueryRow(ctx, "SELECT balance FROM account WHERE name = 'Alice'", accountName).Scan(&currentBalance)

    if currentBalance >= 100 {
        newBalance := currentBalance - 100

        giveClient100Bucks()

        tx.Exec(ctx, "UPDATE account SET balance = $1 WHERE name = 'Alice'", newBalance)
    }

    tx.Commit()
}

Here imagine two withdraw100() were ongoing at the same time without committing, both of them would withdraw the money and set the new balance to 0. Where we allowed withdrew for 200 but in the system it shows that the balance is 0.

To fix the above we could

Do SELECT FOR UPDATE and lock the row, pessimistic concurrency control.
Add a balance check in UPDATE’s WHERE to make sure balance is 100, tx.Exec(ctx, "UPDATE account SET balance = $1 WHERE name = 'Alice' AND balance=$2", newBalance, currentBalance) If updated rows is 0 then we retry. This is optimistic concurrency control.
Use REPEATABLE READ, we will go through the details more but the gist is the transaction would fail if any of the following is true
- Rows our transaction is writing to is different than the snapshot took at the beginning. (Technically in REPEATABLE READ the snapshot is taken at the first non transactional control statement executes, first SELECT/UPDATE/DELETE etc, BEGIN doesn’t take the snapshot)
- Rows our transaction is writing to is Locked by another transaction

When SQL is doing statistics/aggregation calculation

Say in our example we want to calculate the SUM and AVG of balance in account table. Depend on how you write the SQL you might run into issues.

If you run ONE sql statement to get statistics it would always be accurate. Even if there is 1 million rows because in postgres when you run a SELECT statement it gives you the snapshot at that exact time, even if you are running with READ COMMITTED, you won’t read other committed messages because you are running ONE sql statement.

1
2

-- SAFE
SELECT AVG(balance), SUM(balance) from account;

However if you run two selects using READ COMMITTED, you would run into issues.

-- NOT SAFE
SELECT AVG(balance) from account;

-- Any other commits to account would be included in the SUM but not the AVG
SELECT SUM(balance) from account;

Repeatable Read

Repeatable Read in postgres falls into optimistic concurrency control, meaning if you use this isolation level your transaction might fail and you would need to be ready to rerun it.

When transactions run in REPEATABLE READ postgres takes a snapshot at the beginning of the transaction.

A REPEATABLE READ transaction would fail if any of the following is true

Rows our transaction is writing to is different than the snapshot took at the beginning. (Technically in REPEATABLE READ the snapshot is taken at the first non transactional control statement executes, first SELECT/UPDATE/DELETE etc, BEGIN doesn’t take the snapshot)
Our transaction is waiting for a lock on a row, the moment the other transaction commits the change our transaction wakes up and dies. Note that even the other transaction committed the same value as the snapshot we still die.

If multiple transactions collide on the exact same row under REPEATABLE READ, it becomes a “One Winner, All Others Lose” scenario.

It is important to remember that a Repeatable Read transaction fails when it tries to modify a row and that row was changed since the snapshot.

Below I’m listing a few properties of repeatable read.

The snapshot is taken at the first non transactional control statement executes
The snapshot is the entire database, even if you only selected 1 row (note that this is an illusion but you can think of it as the entire database in your mind)
Reads in a repeatable read won’t be affected by commits by other transactions
Postgres repeatable read won’t get newly inserted rows after the snapshot is taken

BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;

-- now there are two rows in account but since there is no non transactional control statement the snapshot is not taken

-- other transaction inserts new rows

-- even only selecting one row creates the snapshot of the entire database
SELECT * from account where name='Alice';

-- other transaction updates Alice's balance

-- this still shows the snapshot taken
SELECT * from account where name='Alice';

-- other transaction updates Bob's balance

-- this still shows the snapshot taken even if Bob wasn't the first selected row
SELECT * from account where name='Bob';

-- other transaction inserts new rows

SELECT * from account; -- this still shows the one and only snapshot

COMMIT;

After the properties let’s check out cases where your Repeatable Read would abort

In the repeatable read transaction if you write to a row but that row was committed by another transaction.

BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;

SELECT * FROM account WHERE name='Pog'; -- creates snapshot for transaction

-- other transaction updates Pog's balance to 100, note that the other transaction finished the commit

-- current transaction attempting to update the same row would fail
UPDATE account SET balance=10 WHERE name='Pog';

COMMIT;

In the repeatable read transaction if you are waiting for a lock of a row but that row was changed by another transaction. As long as the other transaction updated the value(doesn’t even need to be different than snapshot, could be updated to the same value), it would kill the waiting transaction. In short it hang -> wakeup -> dies

BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ;

SELECT * FROM account WHERE name='Pog'; -- creates snapshot for transaction

-- other transaction holds the lock for Bob

-- our transaction tries to get the lock for Bob
-- note that we are only trying to get the lock not even updating it
-- because the other transaction has the lock this would block
-- however the moment the other transaction commits we wake up and die
-- EVEN if the other transaction updated the value to be same as the snapshot
SELECT * FROM account where name='Bob' FOR UPDATE;

COMMIT;

Where Repeatable Read passes but it’s wrong

Let’s have a look where repeatable read passes but the result isn’t correct. The famous example is the oncall doctor example. Note that we are not using database constraints. Given the following table and data

-- 1. Create the table
CREATE TABLE doctors (
    id SERIAL PRIMARY KEY,
    name VARCHAR(50) UNIQUE NOT NULL,
    on_call BOOLEAN NOT NULL
);

-- 2. Insert our initial state (Alice and Bob are both on call)
INSERT INTO doctors (name, on_call) 
VALUES 
    ('Alice', true),
    ('Bob', true);

Right now both Alice and Bob are on call. The business rule is: at any given time there must be at least one doctor on call.

Now let’s see the following scenario

This would pass under repeatable read, but the final result is all doctors are not on call.

The reason this is passing under repeatable read is both transactions wrote to different rows and those rows didn’t change.

Tx1 wrote to Alice’s row which passes the snapshot check
Tx2 wrote to Bob’s row which passes the snapshot check

We’ll see how serializable can help to solve this.

Serializable

I’ve always been confused about the word “Serializable” and why/how is it related to database isolation levels.

The strict definition is

A schedule of concurrent transactions is “Serializable” if its final outcome (the state of the data) is mathematically identical to the outcome that would have been produced if those exact same transactions had been run strictly one at a time, in some sequential order.

In other words this means as long as there exists one sequential order of the transactions in play that could produce the same final result as the actual result then the transactions are serializable.

When multiple transactions are running concurrently it might yield different results, for them to be serializable, as long as the actual final outcome could be achieved by one sequential execution order then the transactions are serializable.

The above is very important, assume that we have two transactions Tx1, Tx2. When they run concurrently there might be two possible final results, resultA and resultB. Among them assume resultA can be achieved by running Tx1 and Tx2 in a specific order, but resultB can not be achieved by any sequential order of Tx1 and Tx2. This would result in transactions passing when the final result is resultA and transactions failing when the final result is resultB. Meaning that the same transcations running in serialization may or may not trigger the serialization error based on how they were executed.

Now let’s (1) look at a real example and (2) think more about how can this confusing definition help us in practice. Like when would I know if I should mark my isolation level serializable.

For (1), let’s check the doctor example again

There are two transactions

Tx1 which tries to take Alice off oncall only if after that there is at least one other doctor on call.
Tx2 which tries to take Bob off oncall only if after that there is at least one other doctor on call.

The final result is

Had we run these transactions one by one can we achieve this result?

The answer is no

If we run Tx1 first and commit, Tx2 won’t let Bob leave on call because he is the last doctor on call.
If we run Tx2 first and commit, Tx1 won’t let Alice leave on call because she is the last doctor on call.

So here we see the actual final result is in a state where no possible sequential execution of Tx1 and Tx2 could achieve, hence these two transactions are not serializable.

When these two transactions are ran under serializable, the first one committed would pass and the other one won’t be able to commit.

[Gemini Review] Nuance: Who actually gets killed? Saying “the first one to commit wins” is a simplification. Under the hood, Postgres’s SSI attempts to abort the transaction that causes the least amount of wasted work. It specifically looks to kill the “pivot” transaction in a conflict. Depending on the timing of locks and reads, Postgres might abort the first transaction that tries to commit, or it might kill a transaction mid-flight before a commit is even attempted.

Let’s see this in action so you know I’m not capping.

The transaction that tries to remove Alice from on call

BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;

SELECT count(*) FROM doctors WHERE on_call = true;

-- App logic evaluates: 2 - 1 = 1 (Safe)

UPDATE doctors SET on_call=false WHERE name='Alice';

COMMIT;

The transaction that tries to remove Bob from on call

BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;

SELECT count(*) FROM doctors WHERE on_call = true;

-- App logic evaluates: 2 - 1 = 1 (Safe)

UPDATE doctors SET on_call=false WHERE name='Bob';

COMMIT;

This feels like “some sort of cycle happened” and postgres caught it. The following is a way for you to be able to look at a few transactions and determine in your mind (or on paper) if the transactions are serializable in postgres. Just bear with me and I promise I’ll explain it later.

If there exists an ongoing transaction, you create a node in the graph
If a transaction Tx1 reads from a row, and another transaction writes to the same row, draw an edge from Tx1 to Tx2
If a cycle forms in the graph, then these transactions are not serializable

Let’s walk through the doctor on call example on paper.

At T2, there exists two serializable transactions in the system. So we draw two nodes in the graph. No edges yet
At T3, Tx1 read from two rows, the Alice row and the Bob row. No edges yet
At T4, Tx2 read from two rows, the Alice row and the Bob row. No edges yet
At T7, Tx1 write to the Alice row, we know that Tx2 read from Alice row, so we draw a directed edge from Tx2 to Tx1. Even if we have edges there are no cycles yet
At T8, Tx2 write to the Bob row, we know that Tx1 read from Bob row, so we draw a directed edge from Tx1 to Tx2. Now we have a cycle. (Tx1 <-> Tx2), hence the transactions are not serializable.

The graph we created above is called Precedence Graph in database theory. The actual technique postgres used to implement serializable is Serializable Snapshot Isolation, this is also known as SSI.

[Gemini Review] Deep Dive: Dangerous Structures and False Positives While thinking about “cycles” is the mathematically correct way to understand serializability, actually traversing a massive graph of concurrent transactions to find cycles is too slow for a database. Instead, Postgres looks for a specific heuristic called a “dangerous structure” (specifically, two consecutive read-write conflicts: T1 -rw-> T2 -rw-> T3). Because this is a heuristic, Postgres will sometimes abort a transaction that matches this pattern even if a full cycle never actually formed. SSI trades perfect precision for performance, meaning it occasionally yields “false positives” and kills perfectly safe transactions.

One other way to think about serializable in postgres is

In postgres Serializable == REPEATABLE READ + Serializable Snapshot Isolation

Hope you haven’t forgot about (2) “when do I actually need serializable?”.

The answer is whenever your need to read and write based on the results of read you should consider using serializable. I used the word consider here because a lot of problems could be solved by Locks and Constraints. You don’t always need to slap on serializable to achieve your goal.

SSI only works on transactions that run under the SERIALIZABLE isolation level

One thing to be careful is that the graph cycle check algorithm we just went through above only applies when transactions are running under serializable isolation level. In the doctor on call example, if one transaction runs in serializable and the other runs in repeatable read, we would still get to the no one on call final state.

The reason is because the read/write tracking and edge drawing from read to write only tracks transactions that are running in serializable isolation level. If a transaction is not running in serializable, it is invisible to SSI. Let’s see this in action.

ww, wr, rw

If you look at the actual error that was thrown when I was trying to commit the serializable transaction you would see

ERROR: could not serialize access due to read/write dependencies among transactions

Why does it say “read/write”? Why isn’t this “write/read”? Are there any implications of this?

Turns out there actually is. We just mentioned that the underlying theory of SSI is Precedence Graph. In Precedence Graph there are three cases where we would draw an edge from one transaction to the other. The three cases are

write/write: Tx1 writes a row first and then Tx2 writes the same row
write/read: Tx1 writes a row first and then Tx2 reads the same row
read/write: Tx1 reads a row first and then Tx2 writes the same row

Now there are three questions

What is the difference between read/write and write/read again?
Why isn’t there a read/read conflict?
Why in the algorithm above you only draw an edge when read/write happened?

For (1), the difference between read/write and write/read is the order. read/write means for the same row Tx1 read it first and Tx2 wrote to it. Where as write/read means for the same row Tx1 wrote first then Tx2 read from it.

The answer of (2) is kinda trivial, read/read does happen but it won’t affect the final result hence the transactions are by definition serializable.

Things are more interesting when we look at (3).

Postgres’s SSI only tracks read/write conflicts. The reason is because write/write and write/read are handled by different mechanisms.

Remember this one liner?

In postgres serializable == repeatable read + SSI

When you run a transaction under serializable, it has all the properties of repeatable read. And it turns out that repeatable read + locks can handle both write/write and write/read. This is the reason why SSI only cares about read/write.

Let’s look at write/write.

In postgres write/write is handled by locks, we’ve seen this in repeatable read hang -> wakeup -> dies.

In the repeatable read transaction if you are waiting for a lock of a row but that row was changed by another transaction. As long as the other transaction updated the value(doesn’t even need to be different than snapshot, could be updated to the same value), it would kill the waiting transaction. In short it hang -> wakeup -> dies

Now lets look at write/read.

A write/read happens when Tx1 first writes and Tx2 reads from the same row, but if you scroll up a bit you would see that in repeatable read postgres takes one snapshot for the entire transaction. When Tx2 reads after Tx1 wrote it actually reads from the snapshot, so whatever Tx1 wrote doesn’t really matter to Tx2.

With this sorted out all is left for SSI to solve is read/write.

Hence why we only draw the directed edge when there is a read/write.

Further things to think about

At this point we can reason about transactions with abstractions like “snapshot” and “precedence graph”, but all these are hard to actually implement. There also exists many other questions to learn about, I’ll probably try to learn them at some point, but for now I’m just listing them as baits for you to be curious.

What exactly is MVCC? How exactly does it work in postgres?
How does postgres actually get the entire snapshot for a single repeatable read transaction?
How does postgres keep track of the precedence graph effectively?
How does postgres do cycle checks of the precedence graph so fast?
How does other databases handle repeatable read and serialization?
For databases that has Read Uncommitted (Microsoft SQL Server), why is it the case?
What does MySQL or Microsoft SQL Server guarantee for repeatable read and serialization?
How does MySQL or Microsoft SQL Server handle repeatable read and serialization? Can you see why/how the implementation relates to the guarantee they provide?

Learn Bubbletea Part 3

Thu, 23 Apr 2026 21:13:10 -0400

tldr

Use v2 instead of v1
Each component is a separate import
Component styles could be set by their own methods
Need to call component’s Update in my own Update

Overview

This practice is to use the existing component library Bubbles. As for now, the latest version of the library is v2, I did feel the documentation was lacking and I had to read code to understand how to use it. But it doesn’t feel that bad.

Code is here.

Demo

Requirements

Use three bubbles/textinput models (Username, Email, Password).
Use Tab to cycle focus between the three inputs.
Mask the password input with asterisks (*).
On Enter (if all fields are valid), clear the screen and print the JSON representation of the user.

Implementation

part2 was a bit messy because I committed all code and when starting to write the blog I needed to go back and remember what changed.

This time I made a commit whenever I felt like I got something. So I could replay the commits and write about them.

Install the component library

diff

The goal here is to setup the library and hook it up. The biggest gotcha is I didn’t realize I need to call the component’s Update in my own Update.

func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    // ...

	var cmd tea.Cmd
	m.textinput, cmd = m.textinput.Update(msg)
	return m, cmd
    // ...
}

Add header and footer

diff

This is just to make things look a little pretty. Nothing interesting.

Use three textinputs

diff

After this the things missing were

tab/shift+tab to switch focus
enter to “submit”
email valid check
styles

Most of the changes here is to change one textinput inside our model to a slice of textinputs

type model struct {
	textinputs       []textinput.Model // <--- here
	currFocusedInput int
	payload          payload
	submitted        bool
}

The other thing is that by default all textinputs are in a row. I had to add a sb.WriteString("\n") to make them in a column.

func (m model) View() tea.View {
	var sb strings.Builder
	for _, ti := range m.textinputs {
		sb.WriteString(ti.View())
		sb.WriteString("\n") // without this the input boxes were in a row
	}

	body := sb.String()
	all := lipgloss.JoinVertical(0, headerView(), body, footerView())
	return tea.NewView(all)
}

Setup focus code

diff

Focus is simple, in the previous change I already added currFocusedInput in the model, though I didn’t set it up.

The setup is really simple, you check for tab and shift+tab in Update, and you change currFocusedInput and call the corresponding Focus(), Blur() method.

The Blur() method call was a bit surprising, but again it does make sense

Enter to submit and show payload

diff

Also very simple, when enter is hit in Update set submitted = true. And in View if submitted is true then json.MarshalIndent and render that result.

Extra: set Focus and Blur style

diff

Here I used textinput.SetStyles() to modify styles of the component. Surprisingly it feels nice to use.

I also decided to Align the names and input boxes, I did it by calculating how many extra spaces I needed to add after the : of the prompt

func NewUserInputs() model {
    //...

	// align text
	// find max prompt, set all to it
	maxPromptLength := 0
	for i := range xs {
		maxPromptLength = max(maxPromptLength, len(xs[i].Prompt))
	}
	for i := range xs {
		suffixLength := maxPromptLength - len(xs[i].Prompt)
		xs[i].Prompt += strings.Repeat(" ", suffixLength)
	}

    // ...
}

Extra: invalid email check and invalid styles

diff

There are two things here.

How to check if an email is invalid
How to apply the invalid style

For (1) I thought I could use https://pkg.go.dev/net/mail#ParseAddress to verify email addresses, but seems like this it not a good fit for the “user input submit”, it’s more for email clients to parse out information. I wouldn’t want “ bungus@localhost” to be considered a valid address.

So good old stackoverflow regex ftw

// Source - https://stackoverflow.com/a/67686133
// Posted by Riyaz Khan, modified by community. See post 'Timeline' for change history
// Retrieved 2026-04-18, License - CC BY-SA 4.0
func isEmailValid(e string) bool {
	emailRegex := regexp.MustCompile(`^[a-z0-9._%+\-]+@[a-z0-9.\-]+\.[a-z]{2,4}$`)
	return emailRegex.MatchString(e)
}

For (2) I’m not sure if the way I did it is correct, I created styleInvalidInput and in Update I check if email is valid or not and swap out the email’s textinput’s style based on if it’s valid or not.

func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    // ...

	if !m.isValid {
		curr := m.textinputs[1].Styles()
		curr.Focused.Text = styleInvalidInput
		curr.Blurred.Text = styleInvalidInput
		m.textinputs[1].SetStyles(curr)
		return m, nil
	}
	curr := m.textinputs[1].Styles()
	curr.Focused.Text = styleCurrentRow
	curr.Blurred.Text = lipgloss.NewStyle().Foreground(lipgloss.Color("7"))
	m.textinputs[1].SetStyles(curr)

    // ...
}

After all it does work :)

Learn Bubbletea Part 2

Sun, 12 Apr 2026 17:03:45 -0400

tldr

Handle styles at the end
Override styles with Inherit
Use tea.WindowSizeMsg to get window dimensions

Overview

This part feels easy. Essentially trying to learn how to use lipgloss to style things. I really like how this set of libraries are designed, feels very good to use.

Demo

Requirements

By Gemini

Create a lipgloss.Style for completed tasks (e.g., green, strikethrough).
Create a lipgloss.Style for the selected row (e.g., blue background, bold text).
Add a bordered box around the entire application.
Add a fixed footer at the bottom using lipgloss.JoinVertical.

By myself on the fly

Make sure that footer is at the bottom
Make sure it adapts to the entire terminal

Implementation

The code is here, I’ll skip the details of each step, instead I’ll just talk about the parts that I found interesting.

Where to apply the styles?

After reading a bit about lipgloss, I thought it was simply define a style and apply it in View().

It is pretty much that but a few things still went wrong.

I tried to apply the styles after every if branch when I go over the tasks. Didn’t feel good because I feel like I’m mixing the content with the style.


// Buggy and BAD
func (m model) View() tea.View {
	log.Println(m)
	var sb strings.Builder

	cursor := " "
	for i, task := range m.tasks {
		style := styleBase
		if i == m.cursor {
			cursor = ">"
			// how to add styles
			style = style.Inherit(styleCurrentRow)
		} else {
			cursor = " "
		}

		marker := "[ ]"
		// use taskID instead of i to help with delete
		if _, ok := m.selected[task.id]; ok {
			marker = "[x]"
			s := fmt.Sprintf("%s%s %s\n", cursor, marker, task.name)
			fmt.Fprintf(&sb, styleCompleted.Render(s))
		} else {
			marker = "[ ]"
			s := fmt.Sprintf("%s%s %s\n", cursor, marker, task.name)
			fmt.Fprintf(&sb, styleBase.Render(s))
		}
	}
	fmt.Fprintf(&sb, "Press q to exit")

	return tea.NewView(sb.String())
}

Apart from style.Render caused an issue rendering \n(See the next section), it feels wrong to do it this way. Instead I created a copy of style at the top of View, and override (Inherit) the style as I go over the model. Then in the end call style render once


func (m model) View() tea.View {
	log.Println(m)
	var sb strings.Builder

	cursor := " "
	for i, task := range m.tasks {
		style := styleBase
		if i == m.cursor {
			cursor = ">"
			// how to add styles
			style = style.Inherit(styleCurrentRow)
		} else {
			cursor = " "
		}

		marker := "[ ]"
		// use taskID instead of i to help with delete
		if _, ok := m.selected[task.id]; ok {
			marker = "[x]"
			style = style.Inherit(styleCompleted)
		} else {
			marker = "[ ]"
		}

		row := fmt.Sprintf("%s%s %s", cursor, marker, task.name)
		row = style.Render(row)
		// put the \n after style.Render otherwise Render adds stuff
		row += "\n"

		fmt.Fprintf(&sb, row)
	}

	body := sb.String()
	body = lipgloss.NewStyle().
		Height(winHeight - footerHeight). // footerHeight is a constant that I set to 2, which is the height of the footer
		Render(body)
	footer := "Press q to exit"

	// body's height was calculated by winHeight so footer is pushed to the bottom
	all := lipgloss.JoinVertical(0.2, body, footer)
	all = styleBorderedBox.Width(winWidth).Height(winHeight).Render(all)
	return tea.NewView(all)
}

Issue calling Render with “\n”

The following is the buggy behavior caused by what I believe is that passing in a trailing “\n” into lipgloss’s Render function. Still not 100% sure why this is an issue, might come back later for this. The fix was to not pass “\n” in the end to Render

I was trying to use JoinVertical to join the body and footer together, it did work but it used the content’s width as the bounding box. So I gave myself an additional challenge to

Make sure that footer is at the bottom
Make sure it adapts to the entire terminal

lipgloss is a pure styling library, it does not know the current terminal’s width and height. To get that I needed to listen to the WindowSizeMsg inside Update and update the current window’s winWidth and winHeight.


func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
	switch msg := msg.(type) {
	case tea.WindowSizeMsg:
		winWidth = msg.Width
		winHeight = msg.Height

    // ...
    }
}

Then I would utilize this information in my View function


func (m model) View() tea.View {
    // ...

	body := sb.String()
	body = lipgloss.NewStyle().
		Height(winHeight - footerHeight).
		Render(body)
	footer := "Press q to exit"

	// body's height was calculated by winHeight so footer is pushed to the bottom
	all := lipgloss.JoinVertical(0.2, body, footer)
	all = styleBorderedBox.Width(winWidth).Height(winHeight).Render(all)
	return tea.NewView(all)
}

Learn Bubbletea Part 1

Sun, 12 Apr 2026 00:16:57 -0400

tldr

IUV
Init, Update, View
Init() Cmd, Update(Msg) (Model, Cmd), View() View

Why Learning Bubbletea?

I wanted to write a GUI/TUI tool and I came across this library which powers lazygit called gocui. It was a bit too bare bones for me and while chatting with Gemini this bubbletea library came into my attention.

I read the tutorials and I really liked it. So here I am asking Gemini to come up with a set of practice projects to hand write to learn the library instead of vibe coding it.

All the code is hand written and if I have issues I try to learn it the old way. I believe this is the best way to learn things instead of having Gemini/Claude generate 1000 lines of code that works but I don’t understand why.

The code is here

Let’s Start

Demo

Requirements

The requirements are very similar to the first tutorial, I was able to do it without looking at the tutorial but only consulting the libraries documentation. The only thing different is the ability to delete items in the list.

A hardcoded list of string tasks.
Use j/k or arrows to move a cursor up and down.
Pressing Enter toggles a [ ] to [x] next to the task.
Pressing d deletes a task.
Constraint: Do not use lipgloss or bubbles. Just standard Go string formatting (fmt.Sprintf).

The Idea

The idea of bubbletea is simple, you have a model and it implements the Model interface

// Model contains the program's state as well as its core functions.
type Model interface {
	// Init is the first function that will be called. It returns an optional
	// initial command. To not perform an initial command return nil.
	Init() Cmd

	// Update is called when a message is received. Use it to inspect messages
	// and, in response, update the model and/or send a command.
	Update(Msg) (Model, Cmd)

	// View renders the program's UI, which can be a string or a [Layer]. The
	// view is rendered after every Update.
	View() View
}

I memorized it as IUV. Similar to AUV which is a term in Beijinglish to express sarcasm.

So for the todo list we need to create a model, which contains all the state the todo list needs.

Init function would populate it but since we are hard coding things so it doesn’t matter for now.
Update would be called by the framework and you check what Msg you get then update the state correspondingly, note that all you are changing in Update is just the state.
View’s job is to read the state from the model and render what is needed (as a string for now)

model

type task struct {
	id   int
	name string
}

type model struct {
	tasks    []task           // list of items
	cursor   int              // where the current cursor points to
	selected map[int]struct{} // taskID checked or not
}

The tutorial did not use []task as tasks in model, I had to change to this because when you are deleting items the indices gets shifted up. Meaning the selected state would be wrong. To fix this I decided to give each task an initial id

Init

For now this could be empty, if we needed to read todo information from disk here Init would return a Cmd which can do that and populate the model state.

1
2
3

func (m model) Init() tea.Cmd {
    return nil
}

Update

The main idea of Update is to switch on the Msg type and update the model state, you return the model after update (similar to react’s update state), and also you can return a Cmd to chain behavior (would have a better understanding of this when I’m done with later phases of the learning)

The following code shows how j/up are handled to move the cursor up, and how enter/space are handled to toggle an item.

Note that Update

Type switches on msg
Checks if it is a tea.KeyPressMsg
Updates model status based on what key was pressed. (Only updates the model status)

func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
	switch msg := msg.(type) {
	case tea.KeyPressMsg:
		if msg.String() == "j" || msg.String() == "down" {
			if m.cursor < len(m.tasks)-1 {
				m.cursor++
			}
			return m, nil
		}
		if msg.String() == "enter" || msg.String() == "space" {
			taskID := m.tasks[m.cursor].id
			if _, ok := m.selected[taskID]; ok {
				delete(m.selected, taskID)
			} else {
				m.selected[taskID] = struct{}{}
			}
		}
	}
	return m, nil
}

View

View’s job is simply render the view given the current model status. For now my understanding is the view is a string, you construct it based on the model state, then return it with tea.NewView(s)

Let’s see the code of how do I mark the current item as “current”, and how do I render an item being checked

The logic is basically

For all tasks in state
1. If index matches current cursor, set cursor text to be * else " "
2. If current task ID is being selected, render [X] instead of [ ]
3. Append current task’s view (the line) to StringBuilder
return tea.NewView(sb.String())

func (m model) View() tea.View {
	var sb strings.Builder

	cursor := " "

	for i, task := range m.tasks {
		if i == m.cursor {
			cursor = ">"
		} else {
			cursor = " "
		}

		marker := "[ ]"
		// use taskID instead of i to help with delete
		if _, ok := m.selected[task.id]; ok {
			marker = "[x]"
		} else {
			marker = "[ ]"
		}

		fmt.Fprintf(&sb, "%s%s %s\n", cursor, marker, task.name)
	}

	sb.WriteString("\n\nPress q to exit\n")
	return tea.NewView(sb.String())
}

Misc

The difference between hand writing code and having Gemini to generate code is that you face issues that you won’t even realize they would be issues on the way. Fixing/Figuring out those issues is also part of the learning.

Following are the issues I faced

tea.Println() returns Cmd

I was trying to print out the model state, and I remembered that fmt.Println should not be used because TUI applications have the entire shell hijacked.

However tea.Println() didn’t really print anything for me, looking up a bit I realized that tea.Println() returns a Cmd, so to have it work it would need to be say returned in Update, I tried it out it did work but it wasn’t what I was expecting. Maybe later I will be able to find out how to use it properly.

If you plugin tea.Println("j pressed") as below, whenever you press j, the message would be printed at the top of the TUI

func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
	switch msg := msg.(type) {
	case tea.KeyPressMsg:
		if msg.String() == "j" || msg.String() == "down" {
			if m.cursor < len(m.tasks)-1 {
				m.cursor++
			}
			return m, tea.Println("j pressed")
		}
	}
	return m, nil
}

tea.LogToFile for debugging

Because tea.Println wasn’t working as I expected, I looked around and found that tea.LogToFile is the thing I needed.

You call tea.LogToFile in main, setup the file you want to log to, and after that standard log.Println would log to the file you setup. This became very handy to check out the state when I was working on the Update method to delete items

func main() {
	f, err := tea.LogToFile("debug.log", "debug")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	p := tea.NewProgram(
		tasktracker.NewTaskTracker([]string{"AAA", "BBB", "CCC", "DDD"}),
	)
	_, err = p.Run()
	if err != nil {
		panic(err)
	}
}

fmt.Fprintf(&sb) to write to StringBuilder

This was not really related to bubbletea. In Update I was trying to use strings.StringBuilder, and I ended up writing things like

1
2

var sb strings.StringBuilder
sb.WriteString(fmt.Sprintf("[ ] %s", task.name))

The Linter gave me an error saying since strings.StringBuilder implements Write, I would just use fmt.Fprintf

1
2
3

// BUGGY CODE
var sb strings.StringBuilder
fmt.Printf(sb, "[ ] %s", task.name) // BUG

However the above was wrong, strings.StringBuilder did not implement Writer, *strings.StringBuilder did. Meaning the code should be

1
2

var sb strings.StringBuilder
fmt.Fprintf(&sb, "[ ] %s", task.name)

delete index caused an issue with selected

This one was one of the fun bugs that maybe Gemini is going to one shot and you never realize there is an issue.

The original model definition was

type model struct {
	tasks    []string
	cursor   int
	selected map[int]struct{}
}

And the Update code to delete was

// BUGGY CODE
func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
	switch msg := msg.(type) {
	case tea.KeyPressMsg:
		if msg.String() == "d" {
			i := m.cursor
			delete(m.selected, i)
			m.tasks = append(m.tasks[:i], m.tasks[i+1:]...) // messes up item IDs
		}
	}
	return m, nil
}

Say I have 4 tasks as the following

0  [ ] AAA
1  [x] BBB
2 >[ ] CCC
3  [x] DDD

selected[int]struct{} = {1:{}, 3:{}}

If I delete CCC at index 2, DDD would shift up, where DDD would then become the item with index 2, meaning the selected map no longer has DDD marked as checked.

selected has index=3 as checked, when CCC was deleted, DDD’s “id” changed from 3 to 2 but the selected status for DDD was not updated.

I thought about going in to the selected map and update all ids from the deleted index, but decided the right way is to have items have their own IDs instead of relying on the index to be their ID.

Another thing was if I am deleting the last item in the list, m.cursor would be out of bounds, I need to check for that and pull it back in range.

// when deleting last line pull cursor back
if m.cursor == len(m.tasks) {
    m.cursor--
}

The End

This is only part 1 of 5. We’ll see if I can keep up with all 5 :)

Fish Practice

Sat, 11 Apr 2026 01:17:01 -0400

tldr

string replace FROM TO input
string split SEP input

What is this about

I’ve been switching to fish for a while. The reason I switched was because I wanted to be able to comfortably write for loops in the interactive shell, utilize various string functions. Unfortunately the only thing I got used to is to do ctrl-f to autocomplete. This is a small drill I came up with to try to improve a little using fish

The adhoc drill

Generate files from a.txt to z.txt
Rename all from *.txt to *.yo.txt
Replace all *.yo.txt to *.sup.txt

Step 1: Generate files from a.txt to z.txt

This step took me a bit longer than I expected. Turns out there is no simple way in fish to generate a to z. the fish native way would be string split "" "abcdefghijklmnopqrstuvwxyz" to tap into bash it would be bash -c "echo {a..z}"

Let’s go with bash

`1`	`set xs (bash -c "echo {a..z}")`

Now I want to for loop $xs and create the .txt files

1
2
3

for x in (string split ' ' $xs)
    touch "$x.txt"
end

Step 2: Rename all from .txt to .yo.txt

This step was nice, string split splits a.txt into a txt, and since fish is 1-based index, I set name to the filename without the extension.

Then it’s the mv command

for x in (ls)
      set name (string split '.' $x)[1]
      mv "$name.txt" "$name.yo.txt"
  end

Step 3: Rename all from .yo.txt to .sup.txt

I could do the same trick, pull out the name and manually write the sup part in mv, but instead I wanted to try out string replace

for x in (ls)
    set new (string replace 'yo' 'sup' $x)
    mv $x $new
end

My Two Cents on Visitor Pattern

Wed, 08 Apr 2026 00:03:00 -0400

tldr

node accepts visitor, visitor visits node

Overview

Design patterns were (and still are) something I found hard to fully understand ever since I started to learn them. Either I found myself memorizing the pattern and “think” I understood it, or I found myself “knowing” some of it but still don’t fully understand them.

Today I’d like to write down something of my current understanding of the Visitor Pattern. This is more of my confusions and ah-ha moments when I was learning this (maybe for the 5th time cause I keep thinking I got it but then actually forget about it)

Why is this pattern needed?

Took me a while to really understand why do we need a Visitor Pattern.

Polymorphism allows you to call different code based on the instance, where visitor pattern allows you to call different code based on the instance and a visitor.

This allows you to decouple data from behavior: the data stays in the class, and the behavior lives in the visitor. When the class tree is big (70 concrete nodes) and you need adhoc behaviors, this pattern helps to keep your code clean. One of the classic/standard places to use Visitor Pattern is when dealing with Abstract Syntax Tree.

In the classic shape example below .area() calls the corresponding method because it’s based on the instance of the object


import math

class Shape:
    def __init__(self):
        pass

    def area(self):
        pass

class Circle(Shape):
    def __init__(self, _radius):
        self.radius = _radius

    def area(self):
        return math.pi * self.radius ** 2


class Square(Shape):
    def __init__(self, _size):
        self.size = _size

    def area(self):
        return self.size ** 2

circle = Circle(2)
square = Square(2)
shapes = [circle, square]
for shape in shapes:
    print(shape.area())

All above seems good enough, if I want to get the xml export string of the shape I would just add a new method to Shape and implement them on all the subclasses.

class Shape:
    def __init__(self):
        pass

    def area(self):
        pass

class Circle(Shape):
    def __init__(self, _radius):
        self.radius = _radius

    def area(self):
        return math.pi * self.radius ** 2

    def xml_export(self):
        return f"<circle radius={self.radius} />"


class Square(Shape):
    def __init__(self, _size):
        self.size = _size

    def area(self):
        return self.size ** 2

    def xml_export(self):
        return f"<square size={self.size} />"

This however has two problems

The classes become massive and bloated. A Circle class shouldn’t need to know about XML, JSON, or database connections. It violates the Single Responsibility Principle.
The code change needs to happen on the concrete types, say you want to separate your library code and client code, you probably don’t want to make changes on your Circle class in library, especially if this is a real library, you won’t be able to accommodate for every client usage. Something Something Open Close Principle

How about we just pull all calculations out to functions

What if we could move the logic to somewhere outside of the concrete classes?

Let’s see what if we just pull the area method out from all shapes and do it with a function

def area(shape):
    res = 0
    if isinstance(shape, Circle):
        res = shape.size ** 2
    elif isinstance(shape, Square):
        res = shape.size ** 2
    return res

circle = Circle(2)
square = Square(2)
shapes = [circle, square]
for shape in shapes:
    print(area(shape))

This is kinda OK but for every new shape we add we need to remember to update area(), which might be a problem if there are a lot of different functions, say we have an xmlexport(shape) and a jsonexport(shape) then we would need to update in 3 different functions. Another issue is type casting might be heavy in some languages vs utilizing the languages builtin polymorphism.

Visitor Pattern

The following code isn’t that useful, but it’s trying to mimic the area() functionality using an AreaVisitor. Note that accept() and visit_xxx() are returning values to be printed in the for loop.


import math

class AreaVisitor:
    def __init__(self):
        pass
    def visit_circle(self, shape):
        return math.pi * shape.radius ** 2
    def visit_square(self, shape):
        return shape.size * shape.size


class Shape:
    def __init__(self):
        pass

    def accept(self, visitor):
        pass


class Circle(Shape):
    def __init__(self, _radius):
        self.radius = _radius

    def accept(self, visitor):
        return visitor.visit_circle(self)


class Square(Shape):
    def __init__(self, _size):
        self.size = _size

    def accept(self, visitor):
        return visitor.visit_square(self)


circle = Circle(2)
square = Square(2)
shapes = [circle, square]
area_visitor = AreaVisitor()
for shape in shapes:
    print(shape.accept(area_visitor))

The concrete classes provides all information needed and no behavior. Each of them has an accept(visitor) method, which takes in a visitor and calls the corresponding visitor function with self.

shape = Circle(2)
area_visitor = AreaVisitor()
shape.accept(area_visitor)

"""
calls Circle::accept with current instance as self, which is a circle (first dispatch)
    visitor.visit_circle(self) # calls area_visitor.visit_circle(shape) where shape is the circle (second dispatch)
"""

Let’s also try to add in the TagExportVisitor

class TagExportVisitor:
    def __init__(self):
        pass
    def visit_circle(self, shape):
        return f"<circle radius={shape.radius} />"
    def visit_square(self, shape):
        return f"<square size={shape.size} />"

circle = Circle(2)
square = Square(2)
shapes = [circle, square]
tag_export_visitor = TagExportVisitor()
for shape in shapes:
    print(shape.accept(tag_export_visitor))

"""
Outputs

<circle radius=2 />
<square size=2 />
"""

The Classic Visitor Pattern

In the classic visitor pattern accept() and visit() do not return anything. Visitors are stateful, they maintain the result.


import math

class TotalAreaVisitor:
    def __init__(self):
        self.total_area = 0
        pass
    def visit_circle(self, shape):
        self.total_area += math.pi * shape.radius ** 2
    def visit_square(self, shape):
        self.total_area += shape.size * shape.size

class Shape:
    def __init__(self):
        pass

    def accept(self, visitor):
        pass


class Circle(Shape):
    def __init__(self, _radius):
        self.radius = _radius

    def accept(self, visitor):
        visitor.visit_circle(self)


class Square(Shape):
    def __init__(self, _size):
        self.size = _size

    def accept(self, visitor):
        visitor.visit_square(self)

circle = Circle(2)
square = Square(2)
shapes = [circle, square]
total_area_visitor = TotalAreaVisitor()
for shape in shapes:
    shape.accept(total_area_visitor)
print(total_area_visitor.total_area)

Gemini:

Why GoF Designed It This Way

In 1994, strongly typed languages like C++ didn’t have generics (). If accept() returned a value, you would have to hardcode the return type (e.g., int). That would mean every visitor would be forced to return an int.

By making the return type void (None) and storing the result as a class member, GoF decoupled the operation from the return type. TotalAreaVisitor can store a float, while TagExportVisitor can store a List[str], and the Shape interface doesn’t have to care at all.

My feeling is we don’t need to strictly abide by the “classic” version, do whatever you need to do. For instance Go’s ast doesn’t even use the visit_xxx() interface, instead it just does visit(node) and type switches inside. You can also see the evaluate example below returning results of sub expressions, if strictly abiding the no return rule, the visitor would need to maintain the stack in its state.

What does this `Accept` even mean

Took me a while to realize that Accept is the instance “accepting” the visitor into the “house”.

Go ast’s Visitor

Golang’s Visitor interface doesn’t really go with the classic pattern. Gemini said this is because

Go prefers smaller interfaces than large interfaces
Had Go’s visitor interface go with the classic version the interface would end up too big and to implement a visitor which only wants to check the comments you need to stub out all other methods

What is this `Walk` thing had to do with `Visitor Pattern`

Visitor Pattern doesn’t really need to be Walking anything, in the examples above, the “Walk” is essentially the for loop. However one of the places visitor pattern works well is when traversing Abstract Syntax Tree, which essentially is a tree composed of different nodes.

So the “Visitor walks through the nodes in the tree”, and each of the nodes “accept” the visitor.

Evaluating an AST

I asked Gemini to generate an example of using the Visitor Pattern to evaluate and print an AST tree that handles Add and Multiply

The recursive part was a bit confusing for me but once I accepted the fact that expr.left.accept(self) would dispatch to evaluator.visit_add when left is Add it was clearer.

    +
   / \
  1   2

The tree above would have the following calls

Add(1, 2).accept(evaluator)
    evaluator.visit_add( Add(1, 2) )
        # Evaluates the left side
        Number(1).accept(evaluator)
            evaluator.visit_number( Number(1) ) -> returns 1

        # Evaluates the right side
        Number(2).accept(evaluator)
            evaluator.visit_number( Number(2) ) -> returns 2

    returns 1 + 2 (which is 3)

from abc import ABC, abstractmethod

# ==========================================
# 1. The AST Node Interface
# ==========================================
class Expr(ABC):
    @abstractmethod
    def accept(self, visitor: 'Visitor'):
        pass

# ==========================================
# 2. Concrete AST Nodes (Data Structures)
# ==========================================
class Number(Expr):
    def __init__(self, value: float):
        self.value = value

    def accept(self, visitor: 'Visitor'):
        return visitor.visit_number(self)

class Add(Expr):
    def __init__(self, left: Expr, right: Expr):
        self.left = left
        self.right = right

    def accept(self, visitor: 'Visitor'):
        return visitor.visit_add(self)

class Multiply(Expr):
    def __init__(self, left: Expr, right: Expr):
        self.left = left
        self.right = right

    def accept(self, visitor: 'Visitor'):
        return visitor.visit_multiply(self)

# ==========================================
# 3. The Visitor Interface
# ==========================================
class Visitor(ABC):
    @abstractmethod
    def visit_number(self, expr: Number):
        pass

    @abstractmethod
    def visit_add(self, expr: Add):
        pass

    @abstractmethod
    def visit_multiply(self, expr: Multiply):
        pass

# ==========================================
# 4. Concrete Visitors (The Logic)
# ==========================================

class EvaluateVisitor(Visitor):
    """Walks the tree and calculates the mathematical result."""

    def visit_number(self, expr: Number):
        # Base case: just return the number
        return expr.value

    def visit_add(self, expr: Add):
        # Recursively evaluate the left and right sides, then add them
        left_val = expr.left.accept(self)
        right_val = expr.right.accept(self)
        return left_val + right_val

    def visit_multiply(self, expr: Multiply):
        # Recursively evaluate the left and right sides, then multiply them
        left_val = expr.left.accept(self)
        right_val = expr.right.accept(self)
        return left_val * right_val


class PrintVisitor(Visitor):
    """Walks the tree and returns a formatted string with parentheses."""

    def visit_number(self, expr: Number):
        return str(expr.value)

    def visit_add(self, expr: Add):
        left_str = expr.left.accept(self)
        right_str = expr.right.accept(self)
        return f"({left_str} + {right_str})"

    def visit_multiply(self, expr: Multiply):
        left_str = expr.left.accept(self)
        right_str = expr.right.accept(self)
        return f"({left_str} * {right_str})"

# ==========================================
# 5. Usage
# ==========================================
if __name__ == "__main__":
    # Let's build an AST for the equation: (3 + 5) * 2
    # Tree structure:
    #       *
    #      / \
    #     +   2
    #    / \
    #   3   5

    ast_root = Multiply(
        Add(Number(3), Number(5)),
        Number(2)
    )

    # 1. Evaluate the math
    evaluator = EvaluateVisitor()
    result = ast_root.accept(evaluator)

    # 2. Print the equation
    printer = PrintVisitor()
    equation_string = ast_root.accept(printer)

    print(f"Equation: {equation_string}")
    print(f"Result:   {result}")

About

Tue, 07 Apr 2026 23:05:32 -0400

A place for me to post random things.

If you can’t write it down you don’t understand it

Opinions are my own.

FAQ

Why luckieboutique.com

This domain used to be my wife’s online shop. Unfortunately it went Chapter 11, so I took it and used it as my blog.

Hello World

Mon, 06 Apr 2026 21:50:33 -0400

Poggers

bingus bungus bowsers burritos

Yun Sheng's Site

New Domain (Finally)

New Domain Purchased at Hawaii

async await in asyncio

tldr

Overview

What is async/await about?

The case where I wait for my coffee

The case where I use a worker thread and callback

The case where I rely on a scheduler

A bit more about python asyncio

Coroutine and Tasks

Garbage Collector Gotcha

The Bug

Further things to think about

Map Flatmap and Options

tldr

Overview

The Problem of Representing Optional Fields

Option

The Other Problem of Representing Optional Fields

Basic Usages

Pattern Matching (Not the Machine Learning One)

map and flatMap

map

flatMap

map vs flatMap

Observations

But wait isn’t this the same on Lists

Option in Rust

map and flatMap (and_then) in Rust

Summary

What have we gone through so far

Option, List are called Monads

Further things to think about

MSB and LSB, MSb and LSb

tldr

Overview

Why is it called Most/Least Significant

Bytes

Big Endian (MSB First)

Little Endian (LSB First)

What does it mean if it’s Big Endian or Little Endian?

Networking is Big Endian?

Bits

TCP

BlazingMQ

Raspberry Pi Datasheet

bit 0 to bit 7 or bit 0 to bit 31

Update

Backward and Forward Compatibility

tldr

Overview

My confusions

The usual things I say at work

What is forward compatibility?

The definition

How can a reader be backward compatible

How can a reader be forward compatible

Common schema changes

Adding an optional field

backward compatible?

forward compatible?

verdict

Adding a required field

backward compatible?

forward compatible?

verdict

Removing a required field

backward compatible?

forward compatible?

verdict

Renaming a field

backward compatible?

forward compatible?

verdict

The role the schema plays

Difference between different schemas

JSON Schema

Avro

Big Endian (`MSB` First)

Little Endian (`LSB` First)

`mod` keyword - to declare or define modules in the module tree

`mod` Summary

`use` Keyword