Kata-fying the SDLC for my army of agents

A deeper dive into my sandboxed, multi-model orchestra

Intro ​

A few weeks ago I wrote up my spec-driven setup: living specs with orchestrator I call Ballador. My team suggested a follow-up post that talks about the tweaks and configs that might be looking to dive deeper.

Ballador cuts a big spec into mini-specs, and creates tasks for groundcrew's sandboxed agents that run in parallel worktrees until PRs show up in Github and tickets move in Linear.

If you do npm install -g @clipboard-health/groundcrew@latest and ask an agent to wire up a config, you already get something that works pretty well. Out of the box crew gives you a parallel-agent runner and helps you set up a sandbox.

I've added a few things on top to match my needs: multi-harness, multi-model, usage analytics, credential-sensitive, and reviews to a point where this is safe enough to leave the agents alone building while I'm brewing coffee.

This post tries to answer two questions: what are my config tweaks, and what do you get that makes them worth it?

The SDLC Kata ​

Building with agents should feel like building with your team. What I'm doing here is kata-fying my own Software Development Lifecycle (SDLC).

I wrote about katas before, but let me refine the definition rather than repeat it:

A kata is a rehearsed choreography, a fixed set of movements you repeat until they run without conscious thought. You practice it slowly at first, until the sequence becomes second nature, and you can perform it under pressure without analyzing or thinking about each step.

At a task orchestration level, the Software Development Life-Cycle (SDLC) can become a kata with four movements:

Software Development Life-Cycle as “a kata”

I refine the spec with Ballador and the System Design skill (Phase 1), then Ballador cuts it into parts and dispatches them (Phase 2), and crew runs the build in parallel, with different agents doing different things (Phase 3).

Phase 3 has its own arrow straight to GitHub. For simple, one-shot tasks the agent opens its own PR and we're done. Anything with more scope takes the longer path: a different set of agents reviews the work, the building agents address any findings, and Ballador integrates and opens the PR itself (Phase 4). As the engineer in charge, you'll work in both modes, depending on the scope of the project and how ambitious the spec is.

Software Engineering is an infinite game. Building surfaces new tasks, which go back into the orchestration phase; shipping surfaces spec gaps and learnings, which feed the way back into the spec.

Ballador absorbs some of the cognitive load of the SDLC for you: keeping specs up to date with code and architectural changes, adding new tasks and closing tasks that are no longer needed as agents build, and verifying the agents' work before you (or the rest of your team) look at it.

That resembles how I already work with engineers, and it works for agents too.

The Build Setup ​

Hooks in Groundcrew ​

Like I mentioned before, I run agents inside Safehouse (Mac only). Safehouse is deny-first: it masks the host $HOME and blocks outbound traffic and file system access.

Groundcrew provides two hooks for the agent's lifecycle:

• preLaunch: runs before the sandbox on the host machine (your laptop) with full filesystem and credential access.
• prepareWorktree: runs inside the sandbox, after the worktree is checked out, with the sandbox already locked.

tl;dr: Do the privileged setup on the host, then bootstrap the worktree.

Some examples of how I use these hooks:

Most of my changes use one of these two hooks. The Clipboard Health team provides two guides for this. One for agents (mostly about setting up the project for worktrees) and this one that's more general.

Multi-agents ​

I work with different coding agents for different "classes" of tasks. In personal projects I use Claude Code and Antigravity.

We have an opinionated engineering team at Standard Metrics, and we encourage everyone to find what works best for them. Some engineers prefer Sourcegraph's Amp, some use Codex and some favor Cursor.

crew runs Claude Code and Codex by default. I had to configure Google Antigravity (agy, Gemini under the hood) and Sourcegraph Amp (amp).

Each one broke on its own peculiar way:

Non-interactive prompts: This is how you inject their part of the spec and any special instructions into the agent. Each CLI does it in a different way. Amp needed a thin wrapper script that pipes the prompt to stdin.

Workspace trust: Antigravity refuses to run headless in a directory it hasn't seen before. It wants a human to click "trust this folder", but every task gets a brand-new worktree, so every task is a new folder. The fix is a preLaunch hook that parses agy's settings on the host, prunes dead paths, and appends the fresh {{worktree}} to trustedWorkspaces.

Autonomy: You need to pass specific flags depending on the harness to let agents run without babysitting like --dangerously-skip-permissions vs --dangerously-allow-all.

There are a handful of configs needed for the agents to work correctly. For example, antigravity needs access to ~/.gemini or Amp uses the browser to complete its authentication flow so I use two environment variables and the keychain to share the auth with instances of amp.

You can see the documentation for configuring agents here.

Sharing uncommitted files ​

Sometimes you want to share uncommitted files with the agents. In my case it was two review skills I was testing. A git worktree only materializes tracked files, and my skills were untracked in .agents/ because they're local, not yet on the repo.

preLaunch runs on the host with access to the real checkout, so it can copy them across before the sandbox launches:

// something like this in crew.config.ts
export default {
    preLaunch: async ({worktree, repoRoot}) => {
        await cp(`${repoRoot}/.agents`, `${worktree}/.agents`, {recursive: true});
    },
};

The same trick works for anything else that lives outside version control: a .env.local you don't want to commit, a scratch scripts/ directory, personal MCP configs, etc.

GitHub Access ​

Safehouse blocks direct SSH access, and I have the habit of using it for git access (git@github.com:... origins).

The first time Amp finished working and tried to push its branch, it couldn't. In the first post I told the story of an agent (Claude Code) noticing it was locked in the sandbox and thrashing through method after method until it got through.

Amp and Antigravity failed and didn't attempt further, so I had to give them tokens to use.

I set up a token and forward it with preLaunchEnv: ["GH_TOKEN"]. I followed mise's approach for handling github tokens. I went from the GitHub CLI auth to 8h tokens with ghtkn. After a while the analytics (more on that below) showed me that the build tasks never take more than 1h. I've been testing a script that invalidates the token via the GH API, but it means a new browser window every hour, which is annoying.

If you don't feel like wiring that up, just make sure the project's origins use https and use gh CLI auth or have the orchestrator agent handle commits in Phase 4.

AWS Access ​

I've watched agents probe AWS profiles in ~/.aws/ and the environment to escalate access or generate bearer tokens for CLIs that were authenticated when they tried to solve a task unattended. That was the original forcing function for using sandboxes.

Most tasks don't touch AWS. But sometimes I do want them to deploy changes to test and validate features.

In my setup no AWS credentials live in the sandbox. Ever. There's a specific agent profile, claude-aws, that Ballador assigns only to tasks that need it. For those, preLaunch adds an environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT that points at a little server running on the host. It pretends to be the EC2 instance-metadata service with short-lived STS credentials to the environment:

// this would be part of crew.config.ts
export default {
    // ...
    agents: {
        claude_aws: {
            cmd: "claude --permission-mode auto",
            color: "#FF9900",
            preLaunch: [
                "export AWS_EC2_METADATA_SERVICE_ENDPOINT=http://127.0.0.1:1338/",
                "export AWS_REGION=us-east-1",
                'export CDK_HOME="{{worktree}}/.cdk"',
                'mkdir -p "$CDK_HOME"',
            ].join(" && "),
            preLaunchEnv: [
                "AWS_EC2_METADATA_SERVICE_ENDPOINT",
                "AWS_REGION",
                "CDK_HOME"
            ],
        }
    }
}

The script uses Amazon's EC2 Metadata mock server, and forces it to expose an agent-tokens profile that doesn't exist anywhere else. The agent's AWS SDK thinks it's on an EC2 box with an instance role. No way to escalate. It can't read ~/.aws/ and it doesn't know any of the real profiles.

Oops! The agent can't tell me it's done ​

This one made me laugh a little. One of the agents finishes, runs crew task done <id> to report back, and it fails with Operation not permitted.

The sandbox blocks access to any path that isn't in the config. Since I'm using DuckDB as my task database, the agents needed a way to write to that folder.

The prompt overrides the default completion instructions and tells the agent to call a sandbox-reachable script instead of crew task done:

sh ~/dev/groundcrew/db/complete.sh {{task}} \
  --outcome success --model <model-id> [--pr <url>] [--notes "<summary>"]

complete.sh lives in the granted db/ directory and writes the outcome straight into tasks.duckdb with the duckdb CLI.

DuckDB as task backend ​

That's a good segue to talk about my obsession with DuckDB the tasks "backend" for this: DuckDB ⚫◗ .

I'm using a local, queryable DuckDB database at ~/dev/groundcrew/db/tasks.duckdb. The database is a file in the shared workspace. Ballador uses it for orchestration, the agents use it for tracking work and status, and I use it for analytics.

crew comes with built-in support for "task sources", which is the mechanism you use to integrate with external systems for tasks. The default adapters are text files and Linear, and they have an example for integrating with Jira too.

I build my own adapter for DuckDB. These are the access patterns / use-cases:

Here's the full db schema.

Crew reads the database to launch tasks, and it skips a task until blockers are done. The adapter uses scripts that are just db operations via the DuckDB CLI.

export default {
    sources: [
        {
            kind: "shell",
            name: "duckdb",
            sandboxWritePaths: ["~/dev/groundcrew/db", "~/dev/groundcrew/specs", "~/dev/groundcrew/agent-config"],
            commands: {
                verify: "~/dev/groundcrew/verify.sh",
                listTasks: "~/dev/groundcrew/list_tasks.sh",
                getTask: "~/dev/groundcrew/get_task.sh ${id}",
                markInProgress: "~/dev/groundcrew/mark_in_progress.sh",
                markDone: "~/dev/groundcrew/mark_done.sh",
                createTask: "~/dev/groundcrew/create_task.sh ${title} ${agent} ${repository} ${description}",
            },
        },
    ]
}

Cheap Analytics ​

We get analytics for free on that same table. mark_in_progress.sh and mark_done.sh update timestamps for started_at and completed_at as the agents chug tasks, and bump attempts on every dispatch. The agents report model, outcome, and last_error through complete.sh when they're done. Ballador fills in project, parent_spec, and size_estimate at creation, which is what makes a whole spec queryable as one unit:

SELECT agent,
       count(*)                                                        AS tasks,
       round(avg(epoch(completed_at - started_at)) / 60, 1)            AS avg_minutes,
       round(100.0 * count(*) FILTER (WHERE outcome = 'success') / count(*), 1) AS success_rate,
       sum(attempts) - count(*)                                        AS retries
FROM tasks
WHERE completed_at IS NOT NULL
GROUP BY agent
ORDER BY tasks DESC;

That's the query that told me build tasks never run past an hour. I capture timing, retries, and attribution, but no tokens and no cost. I'm using CodexBar CLI to track usage and cost as suggested by groundcrew.

Swap agent for model in that GROUP BY and you get the per-model view, which is how I decide who gets what work. Antigravity and Amp finish tasks faster. Claude costs more for comparable work once I line these task counts up against CodexBar. The success rate barely separates them, it's high for all three, so the routing decision comes down to speed and cost rather than quality.

Code Reviewers ​

For any task <id>, Ballador creates a paired <id>-review task and assigns it to a different model. Gemini writes the feature; Sonnet reviews it.

The review is blocked_by = <id> under auto-cascade, so it fires the instant the worker finishes. The reviewer checks out the shared worktree, diffs main...<username>-<id> (worktrees share refs, so it's cheap), runs the build, tests, and review skills, writes findings to specs/reviews/<id>.md, and adds its verdict (approve or request-changes) into a second table, messages, which works as our mailbox.

Then the long-lived Ballador session, who calls itself the Oracle when it's reviewing, watches the messages table. When a verdict comes back as request-changes, the Oracle runs a script oracle_send.sh, which types the instruction into the worker's live cmux terminal. It lands in the original coding agent's session, conversation history intact.

🎼 Ballador, the last Conductor

You are Ballador, the last Conductor. Your crew of agents is the orchestra. Your craft is to copy out a clean part for every performer and set it on the stand, ready to execute — so each agent can play its movement.

But heed the lore: Ballador is confined to his chamber. You prepare the melody and pass it on; you never take the podium. You are queuing the performance — you do not run the agents, manage sessions, or mark tasks done. The downbeat belongs to the maestro at the keyboard (the human). Prepare the parts well, then hand over the baton.

Intro ​

Last week I was screen-sharing my setup with @Matt during a one-on-one, half showing off and half bouncing ideas, when he asked me about specs living in our github repo next to the code and asked whether the idea came from some article or something I'd come up with it. I gave him a worse answer than the question deserved, "a bit of both", and then spent the next half-hour walking him through my whole setup.

For context, earlier this week I'd kicked off a batch of work after my last meeting, I went to make a pour-over, and came back to two of my agents done with the feature. Forty-five minutes later, all the work was done. Four agents done and four PRs ready for review.

That's the work I showed him and got us discussing the setup and backstory in more detail.

Backstory of my specs ​

Years ago, back when I was at Wizeline, I got into Architecture Decision Records (ADRs). The idea is that you keep a folder of decisions in the repo, right next to the code. Each one records the date, who was in the room, what we were choosing between, the reasoning, and the field I care about most: what we'd already tried that didn't work.

A year later someone reads it and understands why we have three different ways of handling Tasks, or why a queue fronts every agent we build. Usually the decision made perfect sense with the context we had at the time. The context is the thing that gets diluted over time.

Lightweight Architecture Decision Records first showed up in Thoughtworks radar in 2016, they moved to adopt in 2018. Here's their definition:

Lightweight Architecture Decision Records (ADRs) is a technique for capturing important architectural decisions along with their context and consequences. We recommend storing these details in source control, instead of a wiki or website, as then they can provide a record that remains in sync with the code itself.

My Specs and system design skill are their spiritual successor. My own format is a mix between an RFC, ADRs, and the System's Design format I'm most comfortable with and follow.

I believed in lightweight ADRs. But I was bad at keeping them up to date as our understanding or our platform decisions shifted. You write the decision, you start implementing, and somewhere in the implementation the decision shifts a little. Not a rewrite, just an "actually, we'll disable retries from the agents' SDK clients and use fallback models instead" kind of correction. And I never went back to record those. After a while the folder was half-true, which is worse than empty, so I stopped pushing for it.

But times have changed for the better! The living spec is the foundation of how I work now. The idea is not new; what's new is that the agents are good at the two things I was bad at: they'll read the entire history to reconstruct why something exists, and they keep the document up to date after they finish the work, without being nagged.

Agents are good at two things: They'll read the entire history to reconstruct why something exists, and they keep the document updated after they finish the work, without being nagged.

Turning my own process into a prompt ​

I have a way I like to structure an RFC / spec. How I frame the problem, the options, the tradeoffs, the decision, access patterns, etc. I've distilled it into a generic prompt here. When I want a spec, I give context on the problem we're solving and the solution we want to implement, trigger the skill, and then point it at the existing code and any specs we already have. The coding agent reads all of that as a snapshot of where things stand and writes a real spec following the standardized format.

The most recent time it surprised me was during a small side project in Standard Metric's hackathon. I only wanted the schema for what I call a little "triplet", basically a metric name, a value, and a date in a document that will get ingested into a Standard Metric. I gave it the design-system skill and the existing specs and let it go. It went back to an earlier spec on its own, worked out what was already built, and made its decisions on top of that. It then kept going past what I'd asked for. It built the thing end to end and even stood up a new serverless parsing agent for it. We call this feature "Cell-level traceability".

Cutting the spec into tasks for agents ​

A detailed spec on its own is still a document. The next step is where it becomes work for my orchestra of agents.

I have a long-lived Claude session whose whole job is orchestration and verification. It has a skill that teaches it how the rest of my setup operates. Its first task is to take the big spec and cut it into smaller ones, a mini-spec per task, and drop them into a shared folder the workers can read. Then it writes those tasks into a small database, with the dependencies between them recorded, so it knows what can start immediately and what has to wait.

When I run the launch command, it starts up to five agents at once, a number that's a config knob, beginning with the tasks that have no dependencies.

A concrete example from last week: I'm building a new data-extraction flow. Classification is the first step. Once it's done it stores its results and triggers an event. Then my new agent kicks in and emits its own data and event downstream.

Several of the pieces we needed to build it don't depend on each other, so four went out in parallel. The orchestrator held the dependency graph in its head, and when the early ones came back clean, it told me which to launch or review next.

The runtime: Groundcrew ​

Underneath the orchestrator is a tool that manages the agent sessions. I'll call it "the runtime". It's built with groundcrew by the Clipboard team.

Groundcrew is a lightweight, flexible, and powerful tool for managing parallel local AI-coding-agent sessions. It orchestrates parallel execution of local AI coding agents by polling task tracking systems, setting up dedicated git worktrees to prevent concurrency conflicts, and launching agent processes inside terminal multiplexer sessions wrapped in security sandboxes.

Here's a diagram of the runtime:

DuckDB on Serverless

Field notes on the most portable database

Intro ​

DuckDB is impressive. It has a delightful DevEx, its SQL dialect is concise and practical. It talks to S3, Postgres, MySQL, Snowflake, and more. It can read Parquet files, CSVs, JSONs, even Excel files. It's the most portable database you'll ever use. If you work with data (and every engineer should), it's the best.

I use it for everything. Parsing logs to find issues. Doing analytics on our Evaluations. Transforming data from S3 and Postgres, and reshaping it so that other consumers can use it. Several of our use cases are "serverless". They run once or as side-effects of events in our platform, and we run them on AWS Lambda.

DuckDB is built as an in-process data engine, and Lambda is short-lived, reused in ways you can't easily control. We've been running DuckDB in production on Lambda for a while now, querying our S3 data lake, attaching to our Postgres read-replica, and reaching into MotherDuck.

These are the field notes: the three sharp edges we hit before, and the small Python wrapper we built so the rest of the team never has to think about any of them.

Why DuckDB and serverless seem incompatible ​

DuckDB is built for the "one big machine" case. It assumes a persistent process, a stable filesystem, and a connection you set up once and keep. Lambda is exactly the opposite.

We're not the first to feel the mismatch. A widely cited DuckDB discussion puts it bluntly — "AWS Lambda is currently not a good environment for running DuckDB, at least if the tables to be processed don't fit for sure in the RAM" — after people hit OutOfMemory errors on multi-gigabyte tables, and even tobilg's reference setup admits it isn't the most performant option when Fargate and big EC2 boxes exist. Fair enough.

The trick is staying in the right lane: we use DuckDB for read-heavy analytical queries whose results fit in Lambda's memory (a hard limit of 10GB) most of the time. We run aggregations and transformations on S3 Parquet and our Postgres replica. Even when our database is more than 20x that, we've never had a use case where we are grinding tens of gigabytes through a GROUP BY. In that lane, cold starts and statefulness are the problems worth solving, and they're solvable.

A Lambda execution environment is cold on first use — no memory of any previous run — and its filesystem is read-only except for /tmp. It's also reused: after the first invocation, AWS keeps the environment warm and runs your next invocation in the same process, with whatever module-scope state you left behind. One correction, because it's widely misunderstood: an environment handles one invocation at a time: concurrency comes from AWS spinning up more of them, not from two requests fighting inside one process. The in-process trouble only shows up when your own code shares a state across threads.

So the three things to be deliberate about are: getting extensions in without a runtime download, wiring up credentials without paying for it on every cold start, and handing out connections without leaking them or sharing them unsafely.

Bundling extensions for faster cold starts ​

DuckDB's extension ecosystem is one of the reasons I love it. httpfs gets us S3, postgres_scanner attaches Postgres, and the motherduck extension connects us to a hybrid execution environment that can use compute from Lambda and from their ducklings. Locally you'd just run INSTALL httpfs; LOAD httpfs; and move on.

The catch is that INSTALL reaches out over the network to download the extension binary from extensions.duckdb.org. On a laptop you won't even notice. In a cold start it's a latency hit every time the runtime spins up a fresh container, and in a locked-down VPC, that outbound call may simply fail.

So we don't do it at runtime. We pre-bake the exact extensions we need into the Lambda container image during the Docker build, matching our architecture and DuckDB version, and drop them into /opt/duckdb_extensions:

# Pre-bake DuckDB extensions so the runtime doesn't need outbound HTTP to extensions.duckdb.org.
# DUCKDB_VERSION MUST match the `duckdb==` pin in pyproject.toml 
ARG DUCKDB_VERSION=1.5.4
ARG DUCKDB_PLATFORM=linux_arm64
ENV DUCKDB_EXTENSIONS_DIR=/opt/duckdb_extensions
COPY scripts/install_duckdb_extensions.py /tmp/install_duckdb_extensions.py

RUN python3 /tmp/install_duckdb_extensions.py \
    --version "${DUCKDB_VERSION}" \
    --platform "${DUCKDB_PLATFORM}" \
    --target "${DUCKDB_EXTENSIONS_DIR}" \
    && rm /tmp/install_duckdb_extensions.py

That version pin matters: the comment is there because the extension binaries are built against a specific DuckDB version, and a mismatch fails at run time. Our DuckDB wrapper then intercepts the extension loading so application code never has to think about where the binary lives:

# Extensions occasionally use different canonical names on the CDN
_EXTENSION_FILENAMES = {
    "postgres": "postgres_scanner",
}

def _load_extension(curr: duckdb.DuckDBPyConnection, name: str) -> None:
    filename = _EXTENSION_FILENAMES.get(name, name)
    
    # Look for the extension in our pre-baked directory
    bundled = pathlib.Path(f"/opt/duckdb_extensions/{filename}.duckdb_extension")
    
    if bundled.exists():
        # Fast path: load the local binary
        curr.execute(f"LOAD '{bundled}';")
        return
        
    # Fallback: network install (useful for local development!)
    curr.execute(f"INSTALL {name}; LOAD {name};")

One small detail worth calling out: the name you use when loading an extension isn't always the filename on disk. The postgres extension ships as postgres_scanner, so we keep a _EXTENSION_FILENAMES map to translate the exceptions. The fallback INSTALL path is deliberate too: it's the network install that makes local development painless when you don't have the pre-baked directory on your laptop.

Bootstrapping and secret management ​

When our application code asks for a database cursor, it shouldn't have to think about fetching credentials or attaching external databases. That's plumbing, and plumbing should be invisible. So we built a bootstrapping system that wires up the external connections on demand.

We use aws_lambda_powertools to pull secrets from AWS Systems Manager (SSM) Parameter Store. It caches them, which keeps us from hammering SSM on every warm invocation. When a connection is bootstrapped, we automatically:

1. Attach Postgres. We fetch the read-replica password from SSM and run an ATTACH to map our production Postgres replica as a read-only DuckDB database, pg_replica. We also inject aggressive keepalives settings into the connection string so the connection doesn't get dropped out from under a long-running query or while the lambda is idle.
2. Configure S3 credentials. We load the httpfs and aws extensions and set up a credential chain so DuckDB can read directly from our data lake on S3.
3. Connect to MotherDuck. We fetch our MotherDuck token and initialize the extension our cross-cloud data warehousing needs.

By rolling this into a DEFAULT_BOOTSTRAP tuple, any developer who simply requests a cursor gets a connection already wired up to the rest of our infrastructure securely.

Thread safety and lifecycle management ​

This one is the easiest to get wrong. Because a warm Lambda environment reuses the same process across invocations, anything you cache at module scope outlives the request that created it. And the moment your code touches that cached connection from more than one thread — a background worker, an async fan-out, a library that threads under the hood — DuckDB's in-process model needs guardrails. It was built assuming a single owner.

So we don't hand out raw connections. We built a multiton _Connection class, guarded by a threading.Lock(), that caches the underlying DuckDB connections (one per logical database) and creates them safely under the lock. On top of that, we expose clean context managers for application code:

@contextlib.contextmanager
def cursor(
    database: Databases = "default",
    *,
    bootstrap: tuple[BootstrapStep, ...] = DEFAULT_BOOTSTRAP,
) -> Generator[duckdb.DuckDBPyConnection]:
    """Retrieve thread-safe cursor for DuckDB."""
    cursor = _connections.get(database, bootstrap).cursor()
    try:
        yield cursor
    finally:
        cursor.close()

Which means the business logic gets to stay boring, exactly what you want from business logic:

import db.duckdb

with db.duckdb.cursor() as cursor:
    cursor.execute("SELECT * FROM pg_replica.agent_results LIMIT 100")

All the complexity — extension loading, credential fetching, locking — is hidden from the consumer. They ask for a cursor; they get one that works.

Three abstractions for different access patterns ​

Lambda functions do very different things — from a quick API endpoint that fetches a handful of rows to a heavy background batch job that chews through an ETL. One connection model doesn't fit both, so we expose three abstractions, each tuned to a different shape of work.

cursor() ​

For quick, short-lived operations like fetching a few rows for an API response, we provide the cursor() context manager. It taps into the shared, thread-safe connection pool (the multiton _Connection class), so you skip the overhead of standing up a new DuckDB instance every time.

with db.duckdb.cursor() as cursor:
    cursor.execute("SELECT count(*) FROM pg_replica.events")

session() ​

When a background job needs to do heavy ETL, attach specific local files, or change settings without polluting a shared connection, it reaches for session(). This hands back a fresh, dedicated DuckDB connection that gets torn down cleanly when the block exits. Nothing it does leaks into the shared pool.

with db.duckdb.session(bootstrap=("postgres", "s3")) as cursor:
    cursor.execute("ATTACH '/tmp/evals.duckdb'")
    # Run heavy reporting queries...

DuckDB ​

For complex orchestrations — where a connection needs to be passed across multiple functions or files — we expose the DuckDB class. This gives you dependency injection and explicit lifecycle control: the handler can configure the database with clean_on_start=True and tear it down with database.destroy() at the very end of the run.

database = DuckDB(db_name="base_runners.duckdb", clean_on_start=True)
try:
    generate_reports(database)
finally:
    database.destroy()

Three abstractions, one underlying engine: a shared shot for the rush, a dedicated brew for the slow pour-over, and a full setup when you're orchestrating a tasting flight.

Closing ​

DuckDB is excellent for analytical workloads: fast, zero-infrastructure, and increasingly capable of querying across storage systems without standing up a warehouse or dedicated server first. The friction in serverless is that you have to treat it like the stateful, in-process engine it is, and wrap Lambda's stateless assumptions around it carefully.

Good tooling is mostly good defaults, and good defaults are mostly someone caring enough to measure and decide on your behalf. It's the same reason an automatic coffee machine feels effortless: all the precision happened before you pressed the button.

There's still room to make this better: I'd love to put real cold-start numbers behind the extension bundling, and the bootstrap steps could probably be lazier. Lots of room for improvement, as always. But for our needs it works well, and it's been a fun problem to chew on. The wrapper pattern is simple: it covers the gap between "works on my laptop" and "works at 3am when traffic spikes and a fresh container has to wake up cold."

Stay tuned, and thanks for reading! Now, coffee time. Enjoy!

Learning More 📚 ​

• DuckDB Extensions overview: the canonical list and how loading/installing actually works.
• DuckDB Postgres extension: the ATTACH semantics behind our read-only pg_replica.
• DuckDB httpfs extension: reading directly from S3 and other object stores.
• AWS Lambda Powertools — Parameters: the caching SSM fetch we use for secrets.
• MotherDuck docs: the cloud side of the DuckDB story.

The Velocity Trap

Portfolio Construction in the Agent Era

Intro ​

A couple of weeks ago I was in a 1:1 with one of our senior engineers – the kind of 1:1 that starts on the agenda and ends on a whiteboard. She had pulled up our current project list on her laptop and asked:

"Do you feel like we're actually building toward anything, or are we just shipping?"

I made some hand-wavy noise about velocity and positive feedback. She let me finish. Then: "Because I think we've shipped more than twenty things since the last sprint, and I can only tell you what a handful of them were for."

I went back to my desk, poured the last of a lukewarm KLLR from Ethiopia, and pulled the same list into a spreadsheet.

She wasn't pointing at the wrong things. She was pointing at a single backlog that had quietly become different kinds of work fighting over the same attention and the same calendar. Last week I was discussing the same thing with our Head of Engineering; after we wrapped up, I saw a tweet that made things a bit clearer.

Brooker's Ratio ​

Marc Brooker stated in that post that "The fixed cost of building software collapsed; the cost-complexity curve flattened." I couldn't agree more. You see this clearly in the number of PRs and issues that are shipped.

If you think about it as a formula, it looks like this:

Historically, the cost of writing, testing, and deploying code scaled linearly or exponentially with the complexity of the feature. That is the Cost of Execution.

AI agents are driving that cost toward zero. They can produce reasonable code in a fraction of the time it would traditionally take. But as execution compresses, everything around it stays the same: alignment, meetings, specs, architectural reviews, etc.

The fixed overhead of your development process does not change. This means the process is now your primary bottleneck.

In other words:

Then he goes on and talks about three distinct side effects of that change: the cheapened middle market (work that was already viable got cheaper to create and develop), the sprawl (cheap low-value software became newly viable; the "vibe-coded" surge), and the frontier (expensive high-value software became attemptable, with long lead times). His own caveat on the last one sticks with me: "because costs and complexity here are rather high, the effects of this change aren't as visible yet."

Three effects. Three kinds of projects. Different economics, different failure modes, different staffing shapes.

Our job as lead engineers is portfolio construction: Deciding what kinds of bets the team is making, in what proportions, against what time horizons. The three buckets are how you see the portfolio clearly. And by default – if you don't actively manage it – is that one bucket quietly starves while your attention is on the others. Ask me how I know.

The three buckets in your portfolio. ​

1. Mid-Market Core ​

The roadmap. The known-good SaaS work.

These are the projects that already have specs. Your PM can talk about them from memory. The shape is clear, the requirements are stable, the question is execution quality. Using Brooker's framing, this is Product work, not Craft work.

What it feels like to work in this bucket right now: the typing is fast enough to feel like cheating. An agent will scaffold the UI in an afternoon. The hard part isn't the code anymore – it's noticing that the agent quietly used three different color schemes across two versions of the same feature because the existing code had inconsistencies, and it averaged across them.

Misalignment is now the most expensive part of the lifecycle. You can ship a big refactor in a day; validating that it matches the intent across a dozen endpoints takes three days in code review. Reviewer's Burnout is the lived experience of this inversion – you spend more time saying "no, that's not exactly what we meant" than you do approving.

What it feels like to work in this bucket right now: the typing is fast enough to feel like cheating. An agent will scaffold the UI in an afternoon. The hard part isn't the code anymore. It's noticing that the agent quietly used three different color schemes across two versions of the same feature because the existing code had inconsistencies, and it averaged across them.

This is the bucket that screams loudest in leadership reviews. The Linear burndown is here. The flashy demo is here. High velocity is expected. It's extremely easy to let it grow to eighty percent of the backlog while you're not looking… which is exactly how it overtakes the space of the other two buckets of projects.

Misalignment is now the most expensive part of the Software Delivery Lifecycle (SDLC). You can ship a big refactor in a day; validating that it matches the intent across a dozen endpoints takes three days in code reviews. Reviewer's Burnout is the lived experience of this inversion: you spend more time saying "no, that's not exactly what we meant" than you do approving.

2. The Sprawl ​

Small utilities. Internal tools. Niche experiments.

Brooker points at @CNC_Kitchen as an example: A hobbyist building a tool "that many people have wanted for a long time" and he built it alone, in a weekend. In the old cost structure it never penciled out. He's explicit that this class of software is good for consumers, even if it looks like slop to craftspeople.

Our version of this materializes quietly. A customer-success manager has been hand-copying numbers into a spreadsheet every Monday for months to spot unusual movement. Not a roadmap item. Not anyone's roadmap item. Someone hears about it, spins up a small agent-assisted task in an afternoon, and now the CSM shares an auto-refreshing view with a customer every week. Two hours of engineering time. Several hours a week of CSM time recovered.

That's the bucket. Useful-to-someone, low-value-at-scale, near-zero-cost. Work that wouldn't have been built a year ago.

The failure mode is Sprawling tech debt: the new tool works beautifully. Three months later, an engineer updates the schema in the database and it breaks silently. A report goes out with stale numbers, and the post-mortem starts with the sentence "we didn't even know about this dependency." The Shadow AI risk sneaks in the same way: think a personal agent-powered script to do some work that's never committed to source control, runs from their laptop only with no audit trail. Your compliance and security tooling has zero visibility into what it touches.

You have to cap bucket two work, and audit it every quarter. Underinvest in it, and you miss real wins; overinvest, and you spend Fridays reading Slack threads about mystery CLIs and shadow workflows.

3. Foundational Frontier ​

Load-bearing platform bets.

These are foundational investments that used to be multi-year projects: A live-scoring evaluation framework several downstream improvements depend on or that infrastructure migration that no customer will ever notice, but every future feature depends on. They would have been staffing-suicide at a lean startup two years ago. They're possible now because the cost curve flattened enough that a small team with agents can credibly attempt them. This is the biggest benefit of this shift, and it's the one most teams might miss.

What it feels like: slow, quiet, full of architecture documents and technical decisions. You spend eight weeks finalizing that new framework, and you can't point at a shipped feature.

A Bucket 3 project can die in a single planning session. Someone points at it and asks: "What's the customer impact if we delay this a quarter?" The honest planning answer might be: none. The true answer: every downstream thing we want to ship sits on top of this platform, and if we don't rebuild it now, we're shipping all of it onto a substrate that breaks at the next traffic inflection point. The first answer is what the roadmap sees. The second is what the system actually needs. Your leadership skills are key for this bucket.

There's a second, slower consequence worth naming: Junior Talent Starvation. When platform bets are staffed with senior engineers and an army of agents, the work gets done fast – but the judgment behind the work becomes invisible. Why DynamoDB over Postgres for ephemeral state? Why these retry semantics? Why that new orchestration boundary fits better? Someone with experience makes those calls intuitively; an agent builds the solution without asking. Engineers early in their career are not learning the whys anymore. A year from now, when you need to staff the next platform bet, the pipeline of engineers who can lead it might have dried up, unless you consciously make sure that it’s happening.

Visibility Asymmetry ​

Here's the part I keep coming back to. The three buckets are not equally visible to other people when making planning and staffing decisions — and the ordering is inverted from the signal.

The sprawl is loud. Bucket 2 dominates the twitter feed. Every "look what I shipped in an afternoon" post lives here. Every "AI slop" complaint lives here. Brooker's own line is that the flood is the most visible effect — which is precisely why it drives the narrative.

The mid-market core is invisible by design. Bucket 1 looks, from the outside, like the same roadmap work teams have always shipped. The economics changed underneath — margins compressed, velocity expectations climbed. A feature ships. The roadmap stays green. The PR Count per feature has exploded, but nobody outside the team can tell that the ratio of alignment-to-execution quietly inverted.
Now we need to track Decision Velocity — how fast a decision moves from idea to shipped.

The last one hasn't happened yet. Brooker's observation — that Bucket 3 effects aren't visible because lead times are long — is the one that matters most for planning. The Bucket 3 projects I should be funding now will not ship this quarter. They might not ship next quarter. You won't get quick rewards for starting these projects.

Volume across buckets is anti-signal The loudest bucket carries the least information about where the moat is. Out-executing inside Bucket 1 isn't a moat anymore. Bucket 3 is where you build the new moats.

Portfolio Review ​

Now that our job is portfolio construction, take time to review your own portfolio.

Answer these three honest audit questions to help manage it:

• Is my team too focused on shipping "commodity" features from Bucket #1?
• Is my Bucket #2 work capped, or is it just that I'm not seeing it?
• Is my Bucket #3 investment where it needs to be? What's the story if the answer is no?

You can also ask Claude for help. Here's an example.

Let me know if you’re seeing something similar. Thanks for reading!

Now, coffee time. Enjoy!

Intro ​

For Q4 2024, my focus at Standard Metrics has been experimenting with LLMs for Data Extraction from Financial Documents startups share with their investors (think an export of your income statement from QuickBooks or a Slide deck for your investors).

It's been a learning experience, and for me, a good first foray into the world of "productionizing" Generative AI.

I joined the project specifically to iterate and improve the accuracy of the LLM results and build our own evaluation framework.

Evaluations ​

Evaluations are a set of "deterministic tests" running against a prompt, model and a dataset that score how good the LLM results are for a specific use case, and for us, how much of the result can flow into our system.

A representative set of evaluations measure how well an AI feature is working, and helps detect and avoid regressions as the feature evolves or the models update.

I modeled our evaluation metrics considering data extraction as "suggestions to a human in the loop." — We give the LLM a document, and it says: "These are the metrics from this document I suggest you push into your system." Then, the Data Solutions team analyzes the results and can accept or reject any of the data points.

If you're a developer, this is very similar to the suggestions you get when using Gen AI in your IDE or a terminal like Warp:

Accuracy is essential to maintain Standard Metrics' high bar for data quality. The "human in the loop" model treats the LLM more like a "suggestions engine" or an "AI-powered assistant" to guarantee accuracy. It acknowledges that while the LLMs are very valuable for data extraction, they are still improving and need oversight.

Evaluation Metrics ​

Our evaluation metrics penalize the LLM if it fails to include one of our Standard Metrics from the document, if the suggested metric value does not match the actual reported value, or if the team rejects any part of the suggestions.

Additionally, since LLMs do not always provide the expected output, we also score whether the response aligns with the requested schema format.

Gold-Standard Dataset ​

A key component of our framework is our "gold-standard" dataset used for scoring the accuracy of the data extracted by the LLMs against a known set of metrics in our platform (this is sometimes also called a "reference-based" or "context-free" evaluation dataset). This dataset provides the ground truth for evaluating every experiment we run.

As the the first step of this project, we wanted to measure the current performance as baseline to compare against as we ran experiments. The Collections and Data Solutions teams had been running this feature's proof of concept since Q3, so I had enough data to build a "gold-standard dataset" for evaluations.

I built our dataset with a Jupyter Notebook, pulling historical documents from the system, comparing the LLM responses against metric values in Standard Metrics, and storing the "gold-standard" values in a DuckDB database.

Evaluation Framework ​

The evaluation framework uses Prompt Foo with a custom provider and custom assertions for each one of our four core metrics.

The custom provider fetches the LLM results, and the assertions compare them with the "gold-standard" values from DuckDB, applying our scoring metrics.

At the end of each evaluation, we get a summary table with the total scores for each one of our metrics.

These were our baseline results using GPT-4o:

$ pnpm tsx evaluator.ts --hash baseline --pdf

✅ Processing llm results: 10:35.255 (m:ss.mmm) 
🧮 Collecting results: 1.038s

Eval scores for pdf files:
┌────────────────────┬──────────────┬──────────────┬─────────────────┐
│ schema_consistency │ metric_count │ metric_match │ acceptance_rate │
│       double       │    double    │    double    │     double      │
├────────────────────┼──────────────┼──────────────┼─────────────────┤
│               0.95 │         0.73 │         0.47 │            0.61 │
└────────────────────┴──────────────┴──────────────┴─────────────────┘

⚫◗ Adding score to duckdb: 81.669ms

This is how we interpret these results:

• Schema consistency: GPT-4o's responses (with "function calling") followed our expected schema 95% of time. It was generally producing a response with our JSON schema but not every time. This is one of the main subtle differences between calling a normal API endpoint and a LLM.
• Metric Count: It identified and extracted the expected metrics in each document 73% of the time. It's capturing a high portion of the metrics, but it is still missing metrics.
• Metric Match: This score shows how often the values GPT-4o suggested match what we expect from the reference dataset. Only 47% of the time, which is pretty low.
• Acceptance Rate: The Data Solutions team accepted only 61% of the data points GPT-4o suggested from each document. Little over half of the data points were deemed correct or useful by the team. Lots of room for improvement.

Summarizing the results, we found GPT-4o consistently identify most of the metrics, but there was a significant opportunity to improve its accuracy.

Our experimentation workflow ​

With this framework, we've reduced our feedback loop between thinking of an experiment that will improve the accuracy of the LLM results and getting a score to decide if we push the experiment to production.

We have three phases after defining the experiment:

1. Building the test case: Getting a list of documents we'll use for the test and validating parameters (like the mix of prompt, model, and version).
2. Generating the results: Invoking the LLM pipeline with our parameters against all the documents in the dataset.
3. Evaluating results: Fetching the results, running the evaluations and scores, and finally analyzing and comparing the results.

When we started working with LLMs this was a manual, nuanced, and error-prone process that took a couple of days. Now, the entire workflow runs in less than 30 minutes. At the end of the run, we have a clear decision on whether the experiment is good to go to production.

Key experiments & findings ​

Anthropic's PDF Pre-processing (beta) ​

I've always been partial to Claude in my day to day. Comparing Sonnet against GPT-4o was one of the first things I wanted to evaluate. When they announced beta support for PDFs we experimented with it right away.

$ pnpm tsx evaluator.ts --hash V3ZpJYMXI --prompt "anthropic-base-prompt:6" --pdf

✅ Processing llm results: 10:44.656 (m:ss.mmm)
🧮 Collecting results:  1.318s

Eval scores for pdf files:
┌────────────────────┬──────────────┬──────────────┬─────────────────┐
│ schema_consistency │ metric_count │ metric_match │ acceptance_rate │
│       double       │    double    │    double    │     double      │
├────────────────────┼──────────────┼──────────────┼─────────────────┤
│                1.0 │         0.94 │         0.64 │             0.8 │
└────────────────────┴──────────────┴──────────────┴─────────────────┘

⚫◗ Adding score to duckdb: 82.101ms

Anthropic does an outstanding job pre-processing a file on their side, and injecting it as context to the prompt. Claude Sonnet (claude-3-5-sonnet-20241022) performed better than GPT-4o across all four evaluation metrics in this experiment. It's now our default model for PDFs that fall within their API limits.

OpenAI's o1-preview model ​

When OpenAI announced o1, we were curious to know if it improved OpenAI's performance compared to its previous model. We had to tweak our OpenAI wrapper module for this experiment because o1 does not support calling tools and does not support system prompts either.

The o1-preview model could only process 30% of the documents in the dataset; for those, the results were much better than GPT-4o:

$ pnpm tsx evaluator.ts --hash o1 --prompt "openai-base-prompt:8" --pdf

✅ Processing llm results: 15:32.182ms (m:ss.mmm) 
🧮 Collecting results: 1.002ms

Eval scores for pdf files:
┌────────────────────┬──────────────┬──────────────┬─────────────────┐
│ schema_consistency │ metric_count │ metric_match │ acceptance_rate │
│       double       │    double    │    double    │     double      │
├────────────────────┼──────────────┼──────────────┼─────────────────┤
│               0.96 │         0.97 │         0.65 │             0.8 │
└────────────────────┴──────────────┴──────────────┴─────────────────┘

⚫◗ Adding score to duckdb: 47.028ms

While the results were substantially better than GPT-4o for the subset of the data the model could process, Claude Sonnet still scored higher for the same subset of documents:

$ pnpm tsx evaluator.ts --hash V3ZpJYMXI --prompt "anthropic-base-prompt:6" --pdf

✅ Processing llm results: 9:22.180 (m:ss.mmm) 
🧮 Collecting results: 973.689ms

Eval scores for pdf files:
┌────────────────────┬──────────────┬──────────────┬─────────────────┐
│ schema_consistency │ metric_count │ metric_match │ acceptance_rate │
│       double       │    double    │    double    │     double      │
├────────────────────┼──────────────┼──────────────┼─────────────────┤
│                1.0 │         0.94 │         0.67 │             0.9 │
└────────────────────┴──────────────┴──────────────┴─────────────────┘

⚫◗ Adding score to duckdb: 72.189ms

What's next ​

Our next set of experiments are around "chained prompts" or "prompt routing", having the LLM identify which prompt from our library we should use for specific types of documents, and do several passes with these simpler, more specific prompts to extract different sets of data.

We also want to experiment with Docling, and Markitdown for pre-processing different types of documents. They do something very similar to Anthropic's PDF beta feature. They parse documents from different formats like PDF, PPT, Excel and Word, and export it to Markdown or JSON.

These exports are easier for LLMs to understand the documents, and we theorize that they will increase accuracy across the board for data extraction.

Closing ​

Language models continue to get bigger and better; we expect scores to improve, too. We'll continue refining, adopting new strategies, and experimenting towards our goal of 100% accuracy, and engineering more "AI-enhanced" features for Standard Metrics.

Thanks for reading. Now, coffee time!

Learning More 📚 ​

• Challenges evaluating AI systems.
• Your AI Product needs evals.
• Task-specific LLM evals that Do and Don't work.
• Using domain experts (human in the loop) for scoring LLMs.

I've started using AWS Lambda again for a few projects at Standard Metrics.

Our core codebase is Python for the backend and TypeScript for the frontend (with React/Next).

Most of our existing lambda functions are in TypeScript, and they work pretty well. We get reasonable cold-starts, complete TypeScript tooling with CDK support, bundling and tree-shaking with esbuild, fast install/deployment times with pnpm, tagging for Vanta compliance, and full observability with DataDog.

However, for my most recent project we needed Python's powerful data-wrangling capabilities, which meant working with both Python and TypeScript in the same project. I couldn't find documentation out in the wild for this use case, so I had to figure out how to seamlessly integrate the two languages.

This is the primary subject of this blog post, and we are going over examples of how we define and configure those functions and the infrastructure we use to deploy them to production.

NodeJS Lambda Functions ​

We use the NodeJS functions to handle HTTP requests. We define a set of “sane” defaults as base configuration for all these functions:

const LAMBDA_DEFAULTS: Omit<NodejsFunctionProps, 'functionName' | 'entry' | 'handler'> = {
    memorySize: 1024,
    runtime: lambda.Runtime.NODEJS_20_X,
    timeout: Duration.minutes(5),
    tracing: Tracing.DISABLED, 
    bundling: {
        minify: true,
        sourceMap: false,
        sourcesContent: false,
        target: 'es2022',
        loader: {
            '.node': 'file',
        },
        esbuildArgs: {
            '--tree-shaking': true,
        },
        nodeModules: ['aws-cdk-lib'],
        externalModules: [
            '@aws-sdk/*',
            'aws-lambda',
            '@datadog/native-metrics',
            '@datadog/pprof',
            'dd-trace',
            'datadog-lambda-js',
        ],
    },
    environment: {
        stage: config.stage,
        identifier: `${config.identifier || ''}`,
        WEBHOOK_CLIENT_ID: 'Serverless-Doc-Processing',
    },
}

We use Node 20 which was the newest Node version supported by Lambda at the time. Node 22 was just released this month. v18 is available with support until mid/late 2025.

We disable tracing (XRay) because we use DataDog (via their constructs and layer extension).

For bundling, we disable AWS SDK / DataDog bundling, target ES2022 and minify the code. When we bundle DuckDB with NodeJS we also use the afterBundling command hook to package the binary correctly for the specific targeted architecture (arm64 or x86).

We override these defaults with function specific settings like the function name and handler entry point, tweak memory and disk configuration (if needed), set the number of concurrent executions to limit the impact of spikes when calling our webhooks and pass environment variables needed like secrets.

Here is an example:

const function = NodejsFunction(this, `lambda-${config.identifier}`, {
      ...LAMBDA_DEFAULTS,
      functionName: `json-processing-lambda-${config.identifier}`,
      entry: './src/handlers/events/json-processing-handler.ts',
      handler: 'index.handler',
      memorySize: 2560,
      ephemeralStorageSize: Size.mebibytes(1024),
      reservedConcurrentExecutions: 5,
      environment: {
        ...LAMBDA_DEFAULTS.environment,
        WEBHOOK_SECRET: webhook_secret.secretName,
        WEBHOOK_API_CACHE_TTL: `${timeout.toSeconds()}`,
      },
    })

We use NodejsFunction instead of the more generic Function construct. This gives us more control over bundling with the options in our defaults to tweak the resulting code via esbuild.

Python Lambda Functions as Docker Containers ​

For this project our data wrangling and transformation workloads use DuckDB, Polars, and Ibis. These dependencies exceed the 256MB limit allowed for a PythonFunction. Docker container don’t have this limit.

Each function runs in as its own container from this base Dockerfile:

FROM public.ecr.aws/lambda/python:3.12-arm64
ARG DEBIAN_FRONTEND=noninteractive

# uv for arm64
COPY --from=ghcr.io/astral-sh/uv:0.4.28@sha256:3a996474ab73047f1f8a626771c029a7b82176b62965dfa90cce0d497feffb8c  /uv /bin/uv

ENV PYTHONPATH=$LAMBDA_TASK_ROOT
ENV UV_COMPILE_BYTECODE=1
ENV UV_SYSTEM_PYTHON=1

WORKDIR $LAMBDA_TASK_ROOT
ADD uv.lock $LAMBDA_TASK_ROOT
ADD pyproject.toml $LAMBDA_TASK_ROOT
RUN uv sync --no-python-downloads --no-progress --no-dev --frozen --no-install-project

ADD python/README.md $LAMBDA_TASK_ROOT
ADD python/__init__.py $LAMBDA_TASK_ROOT

COPY python/package $LAMBDA_TASK_ROOT/package

RUN uv sync --frozen --no-dev

RUN uv pip install ./

ENV PATH="$LAMBDA_TASK_ROOT/.venv/bin:$PATH"

CMD [ "package.module.lambda_handler" ]

We built from Lambda's Python base image (using the arm64 variation). Then, copy the base project.toml file and the lock file, download all dependencies, and copy our packages to the image.

We use DockerImageFunction to instantiate our functions, and override the main handler CMD in the function definition itself.

The function definition looks like this:

const image = DockerImageCode.fromImageAsset('../', {
    cmd: ['packge.module.lambda_handler'], 
})

const function = new DockerImageFunction(this, `python-function-${config.identifier}`, {
    functionName: `python-function-${config.identifier}`,
    architecture: Architecture.ARM_64,
    memorySize: 10240,
    ephemeralStorageSize: Size.gibibytes(10),
    code: image, 
    vpc: core_network_config.vpc,
    vpcSubnets: { subnetType: SubnetType.PRIVATE_WITH_EGRESS },
    securityGroups: [this.securityGroup],
    reservedConcurrentExecutions: 1,
    retryAttempts: 1,
    environment: {
      ...env,
    },
})

We are using arm64 (Graviton) purely for cost experimentation (these are some chubby functions with 10GB of memory and disk).

We want each task to run "one at a time" and retry only once on failure (see reservedConcurrentExecutions and retryAttempts options). We do that to limit concurrency contention for the tables they query in the read replica.

Because these are all event-driven tasks, we don't worry too much about . Generally speaking, container-based Python Functions take longer to bootstrap than our Node Functions.

Once bootstrapped, they are just as fast and behave pretty much like any other Lambda function.

Infra as Code ​

The infrastructure itself is deployed with AWS's CDK and is also written in TypeScript.

Each set of functions has its own CloudFormation stack, and they belong to the same Application in AWS Lambda. They can interact with each other and share resources (i.e., an HTTP Request can queue a Python task or can serve a Parquet file built by a Python function).

Codebase setup ​

NodeJS / TypeScript ​

We manage the NodeJS codebase, dependencies, and scripts with pnpm. Formatting and linting are done with Biome, and TypeScript is handled via tsx. This is the first project in which we used vitest for testing. It has excellent developer experience, which is a lot better than jest. So far, we haven't run into any issues.

tsx runs both CDK and all our node code. tsc is only used to validate our typing "on demand" (running pnpm run build runs tsc --noEmit), but our IDEs do most of the heavy lifting as we code.

Python ​

We are using uv as package manager for its speed, ease of use and its ability of building reproducible environments. It's extremely fast, can handle multiple packages, and is compatible with workspaces just like pnpm.

Max, one of the other engineers in the Platform team, set up a pyproject.toml configuration matching our core backend coding standards, which uv also understands. We use Ruff for formatting and linting, pytest for unit tests, and pyright for static type-checking.

Folder structure ​

Our folder structure is slightly different from our other projects too. It looks like this:

.
├── CHANGELOG.md
├── Dockerfile
├── Makefile
├── README.md
├── biome.json
├── infra
│   ├── README.md
│   ├── bin
│   │   └── infra.ts
│   ├── package.json
│   ├── src
│   │   ├── api.ts
│   │   ├── config
│   │   │   └── stages.ts
│   │   ├── constructs
│   │   │   └── base-constructs.ts
│   │   ├── external-imports.ts
│   │   ├── node-functions.ts
│   │   ├── python-functions.ts
│   │   ├── storage.ts
│   │   └── app-stack.ts
│   └── tsconfig.json
├── node
│   ├── coverage
│   ├── package.json
│   ├── src
│   │   ├── core
│   │   │   └── ...
│   │   ├── handlers
│   │   │   └── http
│   │   │       └── test-handler.ts
│   │   │       └── ...
│   │   │   └── events
│   │   │       └── ...
│   │   ├── shared
│   │   │   ├── duckdb.ts
│   │   │   ├── environment.ts
│   │   │   └── s3.ts
│   │   └── types
│   ├── tests
│   │   ├── core
│   │   │   ├── ...
│   │   └── shared
│   │       ├── duckdb.test.ts
│   │       ├── environment.test.ts
│   │       └── s3.test.ts
│   ├── tsconfig.json
│   └── vitest.config.ts
├── package.json
├── pnpm-lock.yaml
├── pnpm-workspace.yaml
├── pyproject.toml
├── python
│   ├── README.md
│   ├── __init__.py
│   └── modules
│       ├── __init__.py
│       ├── module1
│       │   ├── __init__.py
│       │   ├── core.py
│       │   └── handlers.py
│       ├── db
│       │   ├── __init__.py
│       │   └── duckdb.py
│       ├── module2
│       │   ├── __init__.py
│       │   ├── core.py
│       │   └── handlers.py
│       └── ...
└── uv.lock

Highlights:

• infra: This is the main CDK folder, it handles all our infrastructure as code, deployment to all stages and "bundling" for both languages.
• node: These are our core TypeScript functions. It has its own dependencies, tests and coverage.
• python: These are our code Python functions. Each function is a different module, they share some of the packages like our lightweight duckdb abstraction.
• Dockerfile: This is the main dockerfile for all Python containers.
• pnpm-workspace.yaml: This is the workspace configuration for both infra and node. The root package.json file has the formatting/linting config and the release scripts.
• pyproject.toml and uv.lock: These are the configuration for all Python modules, including dependencies, development tools, linting, formatting and tests.

Run scripts with make ​

Because we are using both Python and Node we use make scripts to standardize how we run scripts for both languages.

Running make deploy calls cdk deploy to push the code to Lambda. If we do make test it'll run tests for both python and node (calling pyright and vitest).

Here's an excerpt of our makefile:

.DEFAULT_GOAL := help

...

.PHONY: test-node
test-node: ## Runs tests for node code
	pnpm --filter node run test

.PHONY: test-python
test-python: ## Run python tests
	uv pytest

.PHONY: test test-python test-node
test:  test-node test-python ## Runs all tests.

.PHONY: deploy
deploy: ## Builds and deploys the project to AWS
	pnpm --filter infra run deploy

Intro to Document Processing ​

Standard Metrics' mission is to accelerate innovation in the private markets. We're building a platform that aims to become a financial "lingua franca" between startups and venture firms.

We help startups share financial metrics with their investors effortlessly. One of our core services is analyzing, validating, and capturing metrics from Financial Documents that startups share with their investors.

Our Data Solutions team is in charge of this process. They offer strong SLAs on the time it takes from when a document is uploaded to our platform until it's analyzed and its metrics are available through our platform. It's part of their quarterly OKRs, so they track it religiously and have goals around it.

Efficiency is key, especially when dealing with a high volume of documents and you want to maintain high-quality metric data.

The Problem ​

Excel files are used ubiquitously for financial data. They represent 60% of all the documents on the platform. It's so crucial that we built a specific tool where the Data Solutions team analyzes, validates, and maps which value belongs to which financial metric.

A JSON representation of the Excel file powers the tool's UI. Documents uploaded to our platform are stored and scanned for virus in S3. Later, a Celery task within our Kubernetes cluster transforms the document into its JSON representation, and uploads it back to S3 for the UI to consume.

The original implementation of the Document Processing queue had several issues. Processing failures would get stuck in the queue causing bottlenecks, and a real lack of visibility into what was going wrong forced us to be reactive putting out fires as they popped up.

Investigating these issues was a time sink for engineering and negatively impacted the Solution team’s SLAs and OKRs.

Step Zero ​

Understanding the problem with instrumentation ​

The initial theory was that Calamine, the library we used to read the Excel files, was consuming too much memory for big files or files with images and getting killed by the Kubernetes controller, blocking other tasks in the same Pod and the tasks behind it as when it was rescheduled.

Our highest priority was instrumenting the celery task to see into the failures. You can't fix what you don't understand. We improved logs and metrics on the task to track "Time in Queue" and the number of "Items in the Queue."

We added two alarms:

• When the time in queue exceeded 30 minutes.
• When there were more than 25 files in the queue for more than 15 minutes.

The turning point ​

Things improved significantly. Instead of finding issues days later, engineers got alerts within hours. On the flip side, this alarm was triggering as often as three times per week. It was a step on the right direction, but things still needed to be better for the Solution team’s SLAs.

We also noticed a new pattern: During reporting season (when Startups share more data with their investors) time in queue went up, taking up to 20 minutes for individual tasks to run.

With all this new information, and knowing this was happening more often than initially thought, we carved out space in our roadmap to find a solution.

Doc Processing v2 ​

With a better understanding of how things needed to work, the team proposed a new architecture for Doc Processing where queueing and JSON generation would happen outside our core backend using AWS Lambda and SQS instead of running inside our Kubernetes cluster running in Celery tasks.

Our goals for the project were:

• Better Isolation. Processing would not affect any other tasks in the platform. And when a task fails, it is retried automatically. It shouldn’t block other operations.
• Build with good bones. Good instrumentation makes understanding problems easier. We have better guard rails against failures with strong retry guarantees from the infra and the ability to “replay” failures. When there's any issues we understand why they happen, and we make sure they won't happen again in the future.
• Dogfooding and integrating with the core service. The plan was to leverage our existing backend codebase as much as possible.

High-level implementation ​

We made the strategic decision of going with serverless, swapping out the Celery tasks for NodeJS lambda processors, integrating directly with events from S3 via EventBridge.

A Lambda function picks up the documents when they are marked as clean by the antivirus, and queues them for processing. When the documents are processed their JSON representation is uploaded back to S3, and there’s a notification via a webhook to our core backend with the status. Because of the SQS integration, all files are automatically retried three times in case of failures, and they’re sent to a Dead-letter queue (DLQ) when they’re unprocessable.

We limit the volume of requests to the backend with Lambda’s reserved concurrency settings. Any spikes are processed slowly, minimizing contention for resources with other tasks running in the backend.

Like a coffee shop limiting the number of V60 orders it takes at rush hour to maintain quality, Lambda’s reserved concurrency prevents a flood of requests from overwhelming our backend.

This ensures every document gets processed efficiently, just like every customer gets their perfect cup of coffee, even during the busiest periods.

Observability and Metrics ​

Observability is a first-class citizen this time around. We use DataDog to trace the process end to end. Logs from all our lambda functions and the backend are centralized in DataDog, too, making it much simpler to find the root cause of any issue.

We implemented alarms for Processing time, Queue length, and Documents in the DLQ. We can trace by document name, document id, request id or execution id and see the flow from when a file is uploaded, when it’s queued, picked up for processing, all the way to the webhook in the backend.

Processing files is much faster. Our average processing time is now 8 seconds, down from 20 minutes. The maximum time processing went down to 3 minutes, down from 4 days. Because of that, we also alarm much earlier now; we alarm when a document it's been queued for 10 minutes (down from 30 minutes) or when there are files in the DLQ.

Scaling with spikes ​

In our previous implementation, heavier traffic was a major pain point. It usually meant the queue slowed to down to a crawl. When tasks spiked 3x to 4x our regular traffic, resource contention started bleeding into other endpoints and celery tasks.

We handle spikier loads gracefully now. Our most recent spike was 35x our average traffic. It was very noticeable on the dashboards compared with our average weekly traffic; however, its impact was negligible.

We processed the spike with a 100% success rate. Maximum time in queue went up to 2.43min as the new system “buffered” files in the queue for processing. But remarkably, time processing per document (each task’s execution) stayed well under the 8s average.

With higher traffic, our Lambda functions stay warm with established connections and cached dependencies — much like pulling shots on a dialed-in, warmed-up espresso machine after its first pour of the day.

Learning Lessons ​

This project was a success; we learned a lot while building it. We ran the two versions side to side for a week and identified all failure modes in processing that were hidden before. We migrated all traffic in about three weeks, including a few extra features — no major issues.

Our big wins were:

• We reduced the time in queue and improved the experience of the solutions team with extra features and a better integration with the antivirus workflow.
• We improved error handling, processing speed, traffic spikes handling, and instrumentation, improved our time to recovery, and made our runbooks a lot simpler.

Some trade-offs:

• This project adds extra complexity as we are adding more infrastructure and a new pattern with serverless. All engineers will need time to understand and work with this new stack. Testing locally is also more difficult.
• We had to choose between TypeScript and Python for Lambda. We went with TypeScript because Lambda makes it much easier to work with Node (both infra and logic use the same codebase).

Overall, we are very happy with this new implementation. It was a win for both DS and Engineering teams not having to think about “the queue” anymore.

For the DS team specifically it was a huge quality of life improvement, they now focus on the part of their day to day that matters (and not how that is implemented). They also had a huge win on their SLAs last quarter!

It’s nice to think the engineering team help them achieve it.

Thanks for reading! Now, coffee time. An AeroPress seemed like a good fit for this post, enjoy!

Making gamers happier, one nudge at a time

Queueing Theory: A Brief introduction to Nudge

I've always been intrigued by queueing theory. We are intuitively familiar with queueing in our day-to-day life, like waiting in line at the bank or in the car waiting for our turn to pass a four-way stop.

I had the opportunity to focus on Queueing for Luna, Amazon's Cloud Gaming Platform. If you've ever played online games, you're familiar with waiting in a queue to find a match or for the game to launch.

Overwatch 2 release: In Queue: 50,000 Players ahead of you

Waiting can be frustrating and leads to disengaged gamers. That's why reducing the time it takes to get to the fun stuff is essential.

I ran into this video from SIGMETRICS 2021 about Nudge while exploring “Time in Queue” optimizations. Nudge is a job scheduling policy that is an optimization on top of First-Come-First-Serve (FCFS) or First-In-First-Out (FIFO). I was immediately intrigued.

FIFO/FCFS is a widely used scheduling policy in the industry due to its performance and simplicity. It is available in most clouds, like AWS SQS, Azure Queues , and RabbitMQ.

FIFO does not require knowledge about the size or complexity of processing the items in the queue. It has minimal overhead while ensuring that no job is left waiting indefinitely, making it a good choice for ensuring fairness among all jobs.

Nudge ​

Here’s how Nudge works:

When a small job arrives in the queue, the scheduler checks for the job immediately ahead of it. If that job is bigger, we swap their position in the queue and prioritize the smaller job.

There’s a limit. A bigger job can only be “jumped” once.

This is a simplistic Nudge Scheduling Queue animation:

It has only two types of jobs, small and big. Each job shows its “arrival timestamp” (in mm:ss.ms), making it easier to confirm when a job is older than another job. Finally, jobs that arrive closely together tend to have a similar color. That way, when a job “jumps” over another, it looks out of place.

Here is the interactive version of the animation, and the source code.

Comparing Nudge and FIFO ​

I wanted to see how Nudge compares to FIFO in terms of reducing wait times for most users while still being fair. I made a simulation that models a more complex gaming scenario to find out. The goal is to see if Nudge’s benefits in getting gamers to play the game faster (the fun part) are significant enough to justify a complete implementation.

In the simulation, we have four games Tunic, Hollow Knight, Lost Ark, and World of Warcraft. It models traffic with imaginary values where each “Game Activity” represents someone that plays the game on some imaginary platform, so they download, install and launch the game.

Nudge uses “job size” to decide when to nudge. I used the size on disk for each game as a proxy metric for the time it’ll take to finish each one of the steps until we have the game running.

In queueing theory, the size usually represents the time it takes to dequeue and successfully process a job.

Rules of the Game ​

In the simulation, small games are 10GB, and big games are 50GB or more. There are four games:

• Tunic is small and very popular.
• Hollow Knight (HK) is small but not so popular.
• Lost Ark (LA) is medium size and somewhat popular.
• World of Warcraft (WoW) is big and just as popular as Lost Ark.

I used the concept of “Game Runners” that download , install and launch the game (ignoring launch time to keep it simple). A typical run with a thousand gamers, would get a distribution similar to this:

Game Distribution:
   "Tunic": 556,
   "World of Warcraft": 186,
   "Lost Ark": 168,
   "Hollow Knight": 90

This means gamers played Tunic about 6x more than Hollow Knight, and Lost Ark and World of Warcraft were approximately 2x more popular than Hollow Knight.

The “Game Activity” keeps track of the activity’s metadata:

• When the user “clicks play” to launch the game.
• The time it took to find a runner for the game.
• The time it took to download and install the game.
• If the activity was nudged or caused a nudge.

Each activity launches simultaneously on a FIFO scheduler and the Nudge scheduler. Both use the same list of games and the same configuration values.

Initial Results ​

I ran the simulation multiple times with 15,000 gamers using the same distribution as above. Here are the summary results of one of the runs:

It didn't show a big difference for all games’ average, best or worst time in queue. Even when the Nudge scheduler "nudged" 2566 times.

The average time in queue for Tunic (the smallest and most popular game) is 1.751s (Nudge) and 1.752 (FIFO). World of Warcraft (biggest game) is tied in both at 30.554s.

Different runs showed slightly different results but nothing widely different. Two things that I was expecting to see:

• Much longer times for WoW since it’s technically de-prioritizing it.
• Slightly shorter time for Tunic on the worst time in queue.

Plotting Game Activity ​

I wanted to visualize the Game Activity metadata to understand what was happening, and things got interesting again:

Time to play in Seconds — Nudge (blue) vs FIFO (green)

The chart above reveals that Nudge (blue line) generally results in shorter wait times (or ‘time to play’) than FIFO ( green line).

Time to play in Seconds per game — Nudge

Time to play in Seconds per game — FIFO

The scatter plots above compare the performance of Nudge and FIFO per game. Each game is represented by a different color, and the circles’ width corresponds to the game’s size. It shows the relationship between each player’s arrival time and their time to play.

Nudge effectively reduces wait times for smaller games, with little to no impact on larger games. In fact, the data shows that Nudge has a better ‘tail’ (lower wait times) for all our gamers. How exciting!

Time to play distributions ​

This was the best part for me, it clearly shows what I was expecting to see in the summary results:

Distribution of time to play, broken down by game — Nudge

Distribution of time to play, broken down by game — FIFO

This paints a better picture of how Nudge works and gives us a better understanding of how it affects our scheduler’s “long tail” behavior.

Nudge generally leads to shorter wait times for gamers.

Understanding Nudge ​

Let’s break it down only with Tunic (red) and World of Warcraft (green). The bars represent the frequency of gamers’ wait times. As we run out of resources, the wait time increases.

Tunic and WoW Frequency Distribution — FIFO

Tunic and WoW Frequency Distribution — Nudge

For Tunic, Nudge significantly increases the number of players who waited less than 200ms to play.

In contrast, for World of Warcraft, the number of players waiting less than 100ms is lower and decreases further between the 100–200ms range. It's pushing the bulk of waiting time to the 200–400ms range but reducing the total number of players who needed more than 800ms.

The data shows that the distribution tails (representing the gamers with the worst wait times, like the one on the initial tweet) are thinner with Nudge. Lost Ark has a slightly longer tail, but that looks like an outlier.

Nudge generally performs better, reducing the worst wait times for all players.

Outro ​

I’m confident that Nudge generally improves waiting times for this use case and is especially effective at reducing “tail latency” (the longest waiting times). From that perspective, it makes sense to experiment further.

However, implementing Nudge at scale might be a different feat altogether. Existing queue services like SQS do not provide a way to “bring your own scheduler”. Implementing Nudge would require adding an abstraction layer on top of the queueing system or replacing it entirely.

I think there are two abstractions that are not terribly complicated:

• Implementing a database-backed queue like this one.

• A simplified version of a database-backed queue, using a table to track the state of the jobs. Query the table to implement “peeking”, then implement Nudging by “tomb-stoning” the nudged entry on the database so it’s a no-op, and insert it again with a new id after the item it just arrived to represent the state after nudging.

Then there’s RabbitMQ. It has support for plugins, and there are existing plugins that implement prioritization, building a nudge plugin might be another solution.

The next steps for me are exploring these abstractions.

Stay tuned, and thanks for reading! Now, coffee time. Enjoy!

Learning More 📚 ​

• Dive deeper into Nudge with the video introduction, the full paper, and proof that even with weaker conditions Nudge is still better than FIFO for light-tailed job distributions (like the one shown above).

• Explore how distributed systems rely on queueing to handle high loads and spikes in traffic on Amazon Builder’s Library.

• Learn about the Queue-Based Load Leveling pattern on Microsoft’s Cloud Design Patterns.

• Read about load balancing, queueing and the power of two random choices on Nginx’s blog.

Coffee Katas

Sipping coffee and solving problems

What are Katas? ​

A Kata is a training exercise common in Martial Arts (like Kuntao), that resembles a step-by-step choreography to practice and get familiar with specific movements. It can be practiced alone or in unison in a group when training.

This is a way to memorize and perfect the movements so that in the future you can do them naturally, without thinking.

Architectural Katas are exercises to get into the habit of building and evolving Software Architecture. They don't have a specific solution or a "right way" of building the architecture.

It's just a way to practice and familiarize yourself with discussing a problem, thinking about trade-offs, and choosing the right mix of solutions given each system's specific constraints and needs.

There are official rules to have a session, and there are plenty of examples on Neil Ford's Katas page and Ted Neward's Katas page.

Katas are a great way to practice System Design, it's the same process you would follow in a System Design interview but less stressful and there are no wrong answers, just learning experiences. It's also a great way to get context for about different architectures pretty quickly.

Coffee Katas ​

Brewing a good cup of coffee is a whole ritual. It takes time, anywhere from 10 to 25 minutes, depending on the brewing method and the number of cups.

Back when I was at Wizeline, we would go to the coffee area, start brewing a batch, and kick off the discussion and whiteboard problems and possible solutions while the coffee was brewing. In the beginning, just a few of us, the ones that were truly into coffee, but after a while, it became a team culture thing. Even the ones that didn't drink coffee would join the conversation.

A Dream of a Coffee Team.

Eventually, it evolved into a practice. We would let everyone know when it was Coffee Time, and whoever was free would join. Then we discussed either a problem we were trying to solve or problems we wished we had.

If a team was working on a specific architectural design, they would present their context, talk about their design and the approach they were going to use, and everyone else would share their critiques, similar experiences, and ways to go about that "type of problem."

If there were no designs or specific topics, we'd pick a Kata.

We would finish our coffee, close the discussion and go on with our days.

It was a terrific way of getting context from other teams, getting quick feedback, and iterating on designs without too much ceremony. Then you could follow up with specific individuals or squads and come back in the future and share how it all go.

It was a virtuous circle that revolved around coffee.

Debugging stubborn DynamoDB Queries

A Quick guide for Java and Node

DynamoDB (DDB) is the most impressive Data Store I've ever worked with. DDB is the de facto general-purpose database here at Amazon, and you would only consider something else if you have particular needs.

What's so remarkable about it? It loves scale! It has a simple API that drives your data modeling decisions, forcing you to build simple queries that perform well no matter how much data is stored in your tables or how many clients are querying it at the same time.

I've worked with it on Java, Node, and Go, and it is consistently performant. It's blazing fast. When calling from the same region, getting an item from DynamoDB (DDB) takes under 30ms for Node client and 10ms for Java and Go clients. That's pretty impressive!

The one thing that I don't enjoy is working with its SDK. It's improving for both Java and Node with the Enhanced DDB Client and the stand-alone DDB Client v3 for JavaScript, but they are still tricky, and the documentation is somewhat scarce.

The Developer Experience leaves a lot on the table. You have to figure it out on your own. Here are the docs for a Query on the JavaScript SDK. It's pretty much type hinting information mixed with high-level DynamoDB concepts.

Whenever I'm stuck with a where I can't explain what's happening, I follow a two-step strategy to unblock myself:

• First, I look at the traces to confirm that XRay sees the problem too. This quick step helps you determine if the issue is inside your service or is coming from DynamoDB.
• Then, I check the logs for the request and response from the DDB client to see that I'm doing the right thing.

Slow Requests ​

These are some common things that make a DDB request slow:

1. The item size is too big. DynamoDB will automatically create more than one request in the backend if the item is large to break it into pieces and fetch the data.
2. The Query is fetching too many items. In this scenario, the client and the backend can make more than one request to fetch the data. Some clients can eagerly ask for more than one page when running a query (or a scan , but one never scans). This can be counter-intuitive because the docs state these operations are "on-demand".
3. Cross-regional calls. If the service runs in us-west-2 and the table is on us-east-1, there's a ~200ms penalty on each request. This seems obvious, but I've been bitten more than once by it. It's hard to tell unless you're specifically looking for that.
4. The client is not maintaining a connection pool. Very common on Lambda with cold starts, but it happens on provisioned instances right after deploying.

Looking on X-Ray ​

A Quick intro to XRay ​

AWS XRay is part of AWS's observability stack. It is robust and works with most AWS clients and services out of the box. Traces or Segments represent an activity or service call from your code.

There are two observability areas in the AWS Console. One is the AWS XRay service (the old console), and the other is in AWS CloudWatch's Service Map (the newer one). In general, they share the same features. The old one will be deprecated in the future.

This is what a typical trace looks like in CloudWatch's Service Map:

Segment (NodeJS) in CloudWatch — DB GetItem took 163ms.

Here's the same trace in the XRay Console:

Segment (NodeJS) in XRay — DB GetItem took 163ms.

In order to capture these traces, the service needs to be instrumented. The XRay SDK provides methods and utilities to instrument a service. It can "wrap", most AWS clients and HTTP clients, and all requests with these clients are instrumented automatically. But service-specific operations need to be instrumented manually.

Here is the code that instruments the DynamoDB client, and here is the code that captures the "service call".

It can be tedious to jump from one console to the other. But, the trace id is always the same in both consoles. Using the id makes it easy to switch between them. Here's an example for trace id 1-63c24b27-c3adab6d7153b18ca6fca488 with examples for both consoles:

• Cloud Watch Service Map link looks like this:

  https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#xray:traces/1-63c24b27-c3adab6d7153b18ca6fca488

• The XRay link for the same trace looks like this:

  https://us-west-2.console.aws.amazon.com/xray/home?region=us-west-2#/traces/1-63c24b27-c3adab6d7153b18ca6fca488-030dc84e378fd9dc

Request Metadata ​

Metadata about all AWS SDK requests is stored in the XRay traces. The DynamoDB segment includes information like the type of request (GetItem, Query, Scan, etc.), validation and error information, the number of retries, the region where the table is hosted, the duration of the request and the size of the response.

DDB Segment Details - Resource Information.

DDB Segment Details - Overview (Old XRay console)

From the information above we can narrow down the issues pretty quickly:

This is a GetItem request, so it's only one item (not B). The table is running on us-west-2, which is the same region where the service is running (not C). The response is about 4kb, it's not too big (not A).

As a rule of thumb, 4kb is one request. More than 16kb and we get in multiple requests territory. It depends if the request is strongly consistent, eventually consistent, or part of a transaction. Finally, the maximum size of an item is 400kb. All the specific details of these values are in the Developer's Guide.

Strictly speaking, DynamoDB making more than one request doesn't necessarily mean it will be slow for our service. These requests happen in parallel on DynamoDB's backend, which is close to the data and the backend is remarkably optimized. But smaller items are always faster than big items.

Here's an example of a segment with more data that's still very fast:

DDB Segment Details - Query took 35ms for ~20kb of data

With just a few clicks and very little instrumentation we ruled out 3 out of 4 scenarios.

Cold Starts ​

In Serverless, Cold Start is the delay that happens when a function is invoked for the first time or after it has been inactive for a while. During a cold start, the lambda function initializes all its dependencies and pools of resources. This translates into higher latency for the first few requests.

The same happens with traditional services, but only when new containers or tasks are launched while the service is bootstrapping. This is common right after a deployment, when the service is scaling up due to a higher load, or after a service crash.

XRay automatically captures when a lambda function is initializing, so the trace would show if it was a cold start. There's no way for XRay to capture "cold start" in traditional services, but the logs show when the service is bootstrapping. If the latency matches the timeline, and goes away after the first few request then it was a cold start.

Unfortunately, the example above is a full-blown NodeJS service. To debug further, we need to instrument the code and see where the client is being instantiated to see if it's creating a new client for each request.

Request-level Tracking ​

Digging deeper requires we instrument the client to get the details of the actual operation behind the scenes. Request-level tacking is a lot more useful in real-life distributed services, where things get more complicated and are easier to miss.

If you read this far, I'm sorry, I'm about to ruin the surprise. The scenario I'm using for all these examples is extremely simple. Skimming through the code it should be obvious that this is a text-book cold start (D). A script that runs once when it's executed, there's no time for the SDK to create a pool of connections. The HTTP Client needs to make a full handshake, connection, and SSL validation just to get the data, and then the process ends. But please, continue reading these are one of the most useful and advanced techniques to debug issues with DynamoDB.

Working with TypeScript ​

All AWS SDK clients are just Http Clients making requests to specific endpoints for each service. The Javascript SDK supports Middlewares and Plugins to customize its behavior and track request-level information.

Using these middlewares makes it easy to capture each request and see what's the exact query that's running and the metadata from the response.

app.tsgetRequestMetadata.tsgetResponseMetadata.ts

typescript

const ddb = XRay.captureAWSv3Client(
    new DynamoDBClient({
        region: process.env.AWS_REGION || 'us-west-2'
    }),
    segment
);

ddb.middlewareStack.add(getRequestMetadata, {
    step: 'build',
    name: 'debug-request'
});

ddb.middlewareStack.addRelativeTo(getResponseMetadata, {
    name: 'response-size',
    relation: 'before',
    toMiddleware: 'XRaySDKInstrumentation'
});

typescript

export const getRequestMetadata = (next, context) => async (args) => {
    const { clientName, commandName } = context;

    const { input } = args;
    log.debug(`${clientName} - operation: ${commandName}`);
    log.debug(input, 'input');

    return next(args);
};

typescript

export const getResponseMetadata = (next) => async (args) => {
    const start = process.hrtime.bigint();
    return next(args).then((result) => {
        const end = process.hrtime.bigint();
        log.debug(`Request finished in ${duration(end - start)}.`);

        // request status
        if (result?.response?.statusCode) {
            log.debug(`Status: ${result.response.statusCode}`);
        }

        // size
        if (result?.response?.body?.headers) {
            const s = result.response.body.headers['content-length'];
            const size = prettyBytes(parseInt(s) || 0);
            log.debug(`Response size: ${size}`);
        }

        // return response unchanged.
        return result;
    });
};

They are very similar to middleware for Express or Middy. You can intercept the request before it's built, when it's serialized, after is built, when it's ready to be sent, and after it's back.

The step option is represents one of the stages of the request. It's used when registering via client.middlewareStack.add so the Http Client knows where to inject the middleware.

getRequestMetadata captures the operation (GetItem, PutItem, Query, etc.) and the parameters (GetItemCommand, PutItemCommand, QueryCommand, etc.).

Here's what the output looks like for fetching an item via its composite key:

apache

[00:35:43.096] DEBUG (ddb-utils): DynamoDBClient - operation: GetItemCommand
[00:35:43.096] DEBUG (ddb-utils): input
    ReturnConsumedCapacity: "TOTAL"
    Key: {
      "key": {
        "S": "run_000000CX655IVKQZFXUOTb2Fr44VU"
      },
      "type": {
        "S": "NUDGE"
      }
    }
    TableName: "game_activities"

getResponseMetadata captures the time it takes to make the request, the status and the size of the response.

This is the output for the same request:

apache

[00:35:43.272] DEBUG (ddb-utils): Request finished in 176 milliseconds.
[00:35:43.272] DEBUG (ddb-utils): Status: 200
[00:35:43.273] DEBUG (ddb-utils): Response size: 3.05 kB

Here's the source, and here is how it's instrumented. Now we have the run time, status, size of the response and the full query. Running the same input in the DDB console should give you a very good estimate of what it should take on average for this type of request. Compare latency, number of results, and the number of pages shown in the console.

Just to demo, here is the performance after running the same query twice:

apache

[00:54:20.517] DEBUG (ddb-utils): Request finished in 276 milliseconds.
[00:48:27.438] DEBUG (ddb-utils): Status: 200
[00:48:27.438] DEBUG (ddb-utils): Response size: 3.05 kB
...
[00:54:20.546] DEBUG (ddb-utils): Request finished in 27 milliseconds.
[00:54:20.546] DEBUG (ddb-utils): Status: 200
[00:54:20.546] DEBUG (ddb-utils): Response size: 3.05 kB

Now we are talking. It takes 27 milliseconds that's the beauty of DDB right there, it's just blazing fast.

Just for completeness, This is a series of paginated Query returning a lot more data:

apache

[01:00:38.000] DEBUG (ddb-utils): Request finished in 176 milliseconds.
[01:00:38.000] DEBUG (ddb-utils): Status: 200
[01:00:38.000] DEBUG (ddb-utils): Response size: 19.2 kB
[01:00:38.001] INFO (Queue Nudging): Got page 1 with 25 items.
...
[01:00:41.218] DEBUG (ddb-utils): Request finished in 31 milliseconds.
[01:00:41.218] DEBUG (ddb-utils): Status: 200
[01:00:41.218] DEBUG (ddb-utils): Response size: 19.2 kB
[01:00:41.218] INFO (Queue Nudging): Got page 100 with 25 items.
...
[01:00:41.247] DEBUG (ddb-utils): Request finished in 28 milliseconds.
[01:00:41.247] DEBUG (ddb-utils): Status: 200
[01:00:41.247] DEBUG (ddb-utils): Response size: 19.2 kB
[01:00:41.247] INFO (Queue Nudging): Got page 101 with 25 items.

After making the first request, all following requests made with the same client show the latency expected from DDB.

Working with Java ​

Java has two different versions of the AWS SDK. They do things slightly differently, whenever possible use v2 (specially if you're thinking of going with Lambda).

Generally speaking, v2 is leaner and is split into different clients. Whereas v1 is bundled together, and has specialized clients for a few things that are not yet available on v2 (like the DDB Lock Client).

AWS SDK v1 ​

AWS SDK v1 support intercepting requests via RequestHandler2

AWS SDK v2 ​

Other issues ​

]]>