But first: I’ll be at KubeCon North America next week in Atlanta. Can I buy you a coffee/beer/etc there in exchange for insights on your DevSecOps challenges (and maybe some feedback on Archodex)?

Let's chat at KubeCon!

Now, here’s what we’ve been up to since my last post:

• My friend Apurva Jantrania has joined the fun! We’re now two ex-AWS Principal Engineers working to solve secrets management and other DevSecOps challenges. Don’t worry, this is still a labor of love with no outside investment, which means we get to focus solely on building the best software we can.
  

• We have a fun new Playground experience. We know you don’t have time or energy set up new services just to figure out what they can do. This short-circuits trying Archodex to a 3-minute interactive demo.

  

• You can now sign up and start using Archodex! Consider this a soft-public-launch where more of our friends can try it out and help us by providing feedback. We’re not going to be broadcasting about Archodex yet, though feel free to send people our way if we can help them.
  

• We’ve released the source code for all our components as Fair Source. This gives you the right to improve or repair Archodex however you need and allows us to build a sustainable project for the long term. I’m really excited about developing Archodex in the open!
  

• You can now self-host the Archodex service wherever you like. Some folks need their data to remain in their control and/or on their own premises. You can even run the agent in logging-only mode, which works even when completely firewalled off from all networks.
  

• If you’re curious how our agent is able to performantly introspect your API interactions to discover secrets usage, you can take a look at the technical details documentation of our agent (or even check out our source code!). We had to really innovate to solve a bunch of tricky eBPF challenges!
  

So that’s what we’ve been working on lately. If you’ll be at KubeCon, please reach out!

Let's chat at KubeCon!

A common theme emerged from these efforts: It is really hard for engineering organizations to keep track of their IT resources, how they interact, and who accesses them. This post details the problem and provides a teaser of the “Platform Observability” solution I’m building to solve it.

New Relic’s El Dorado

I started as employee number ~80 at New Relic. My goal was to architect the first Browser Observability product. Six weeks before the public beta launch we realized we needed to sync with the Ops side of the org about our ingestion requirements. There was already a service, simply named “Beacon”, doing a tiny amount of browser telemetry ingestion (e.g. page load latency) for customers. We were extending it to ingest much more observability data (JS errors, XHR timing data, etc.). We needed to confirm that the service could handle this additional load.

But who owned “Beacon”? Thankfully, New Relic engineering fit on one floor of Portland’s Big Pink tower, so this only took a couple days to nail down. Beacon had become a long forgotten service because it just “worked” and hadn’t been touched for a while. The original code had been written by a few of the first five employees at the company, who now had titles like CTO and CPO. Getting answers to questions that normally might take a quick conversation between engineers became days of back and forth as the organization tried to remember long-forgotten details of the service. Thankfully we nailed down these details and launched on time and without issue.1

After I left New Relic in 2014 the company continued on its rapid startup-to-public-company ascent. But the problem of tracking IT resources and who owned them only grew. I’ve been told of offsites where key engineering leaders attempted to remember and diagram all the microservices that formed the New Relic platform. Inevitably these attempts ended in failure. Ward Cunningham, inventor of Wikis and a Principal Engineer at New Relic at the time, led an effort to solve this challenge. That effort, through a few fits and starts, became El Dorado, New Relic’s attempt to produce a map of all IT resources.

El Dorado was a massive undertaking. Unfortunately, only the frontend became open source. The backend was much larger, ingesting information from dozens of sources including git repos, project management tools, and HR systems. Whether or not the project uncovered untold riches is debatable, but it is clear that the problem of understanding New Relic’s IT footprint was painful enough to spend significant resources on.

Stackery’s Architecture Diagrams

In 2017 I co-founded Stackery as a DX platform for serverless cloud applications. A key feature was its visual drag-and-drop interface for editing your Infrastructure-as-Code (IaC) templates. While the editing functionality was useful, it was only near the end of our time as an independent company that we realized one of the most impactful benefits of our solution was simply being able to visualize the architecture embedded within IaC templates. One of our customers said to me: “We pay $20k/yr for Lucidchart just to be able to spend our Architects’ valuable time to maintain all our architecture diagrams by hand.” Stackery focused on individual serverless applications, but this customer was asking for more: Automatic documention of their entire IT footprint.

A (Marauder's) Map of your IT

The Marauder's Map was a magical document in the Harry Potter series. After waving one’s wand and saying, “I Solemnly Swear I am up to No Good”, the map would “unlock” to show everyone’s location, in realtime, at Hogwarts School. What if we could do the same for organizations’ IT footprints?

This is what I’m working on. Archodex is the first Platform Observability solution. It efficiently observes traffic between your services, including encrypted HTTPS API requests and responses, picking out the important details that enable automated documentation and deep introspection of your IT footprint.

Let’s imagine an engineer comes to you and says: “We need to rotate the staging sprockets DB credentials.” But now you need to know which services use those credentials to ensure you can rotate them safely. This map shows you that the Kubernetes sprockets service running in the default namespace accesses the stg/sprockets/db_creds secret in the vault.acme.com Hashicorp Vault service. Now you have the information you need to make sure your credential rotation can be done safely.2

Let’s imagine another scenario: A security engineer wants to understand more about who and what can access a secret. Here we see that the GitHub user @txase (me!) has read the prod DB secret in the vault. Let’s click the link to find out more details.

The image on the left above is show first, and the image on the right appears when we click “Show Event Chains”. The Event Chain details tell us: I, GitHub user @txase, Assumed myself (a GitHub Actions detail) to Invoke the workflow .github/workflows/deploy.yml in the txase/archodex-demo-sprockets repo, which Read the prod/sprockets/db_creds secret. Archodex provides a rich level of detail to help engineering organizations understand their IT footprint, including how resources and people interact.

More to Come

This is just a tease of Archodex. You can find out more about planned functionality at its website, and you can request early access if you need to solve painful problems today. Current plans are to make Archodex free for individual and team-sized usage, available for folks to download and run for free themselves, and offer it as a managed service. I’m polishing it up to share publicly, stay tuned!

In between things I built a POC side-project to scratch an itch and learn more about AWS Aurora DSQL. I’m a big believer in the value of a horizontally scalable relational database, and I wanted to find out how well it worked. (If you came here specifically for DSQL takes, you can jump straight to them here.)

Motivation

My password manager has been bugging me. It’s become a bit onerous to manage sharing items with family, and it felt like it got really bloated/slow recently. It turns out there was a good reason it got slower1, but speed wasn’t my only issue, and it got me wondering what I could do to solve my daily annoyances with it. I wanted my own password manager solution that worked how I wanted it to. And it turns out I can have my cake and eat it to!

Password managers aren’t new. Bruce Schneier created the first password manager in 1997, and although the tech has changed considerably since then, password managers should be considered a commodity rather than a luxury good at this point. It turns out there is such a “commodity” password manager solution via the combination of the official Bitwarden clients (e.g. the their browser extensions, iOS and Android apps, and web app) + the community open source Vaultwarden backend service for cross-device syncing.

You might be thinking: Hang on, I’m going to trust a community open source app with all my passwords? This concern is mooted by the fact that all encryption and decryption occurs inside the official Bitwarden clients. The API service, which is what Vaultwarden provides, merely stores and retrieves already encrypted items. In theory, you could even make your vault database and files public because they cannot be decrypted without your master password.

But also, I wanted to play with the new Amazon Aurora DSQL service, which promises a truly serverless2 PostgreSQL-comptible(-ish) database :).

A Serverless Password Manager Backend

Having my own password manager with cross-device syncing requires operating a cloud service. But I don’t want to deal with the typical operational headaches of managing an always-on service. I previously built a company to help people run cloud apps with less operational headaches using serverless services. I wondered if I could find a password manager with a serverless backend.

Enter Vaultwarden. Vaultwarden is a reimplementation of the official Bitwarden backend API service. What makes it ideal here is that it is written in Rust. Rust software both initializes and executes fast. This is important because serverless APIs (especially low-throughput APIs like a personal Vaultwarden service) often execute with cold starts, where response latency includes process initialization.

Vaultwarden Serverless Architecture

The current Vaultwarden implementation is architected as a stateful single-node service. It saves vaults and other files directly to the filesystem, serves the official Bitwarden web app assets (rebranded as “Vaultwarden” assets) from a directory on disk, and connects to SMTP servers or uses a local Sendmail installation to send emails. And it uses a relational database (Postgres, MySQL, or SQLite) for account records and to store encrypted items. These would all be a pain to operate and maintain long-term as a personal project, and there are AWS Serverless services for each component that greatly reduce this operational toil. Inserting these services results in the following Vaultwarden Serverless architecture:

CloudFront CDN
├─ API Lambda Function
│  ├─ Data S3 Bucket (for encrypted file attachments, etc.)
│  ├─ Aurora DSQL Database
│  └─ Amazon Simple Email Service (SES)
└─ Web App Assets S3 Bucket

You can find the code for this Vaultwarden Serverless POC here.

AWS Lambda for the API Service

AWS Lambda is used for the API service. Personal password manager workloads don’t hit the API frequently (vaults are cached client-side), so we can give our Rust function a ton of memory and CPU to reduce execution time while still staying within the monthly free-tier usage limit. We use the Lambda Function URL feature with the Lambda Web Adapter so we don’t have to modify any code to make the API just work inside a Lambda Function.

Amazon S3 for Data and Web Assets

Amazon S3 is the OG serverless service3. Although folks will tell you it is not a filesystem, it can be used to replace a filesystem as a general store of files. For example, instead of saving vault item attachments to /app/data/attachments/<encrypted name> on the local disk, we can save it to s3://<data bucket>/attachments/<encrypted name> in an AWS S3 Bucket. Similarly, instead of serving web assets from /app/web-vault/<asset filename>, we can serve them from s3://<web asset bucket>/<asset filename>. There is also support for streaming files from S3 via presigned URLs, meaning our API Function is left mostly to process API requests rather than act as a proxy for file downloads.

Amazon Simple Email Service (SES) for Sending Emails

Amazon SES is used for sending email. Not because sending email is hard, but because sending email that won’t go straight to spam folders is hard. After SES is configured for sending email from our domain with DKIM, SPF, and DMARC, we can then use it to send email on our behalf.

Amazon CloudFront to Combine Routes and Serve from a Custom Domain

Amazon CloudFront can be fiddly to work with, but solves a couple challenges for us. It routes web app asset requests straight to our web assets bucket and routes API requests to our Lambda Function URL. CloudFront also makes it easy to serve the backend from a custom domain.

Amazon Aurora DSQL for the Database

We finally come to Aurora DSQL, AWS’ first truly-Serverless relational database offering. Vaultwarden can speak PostgreSQL. But can it work with DSQL, and how much effort will it take?

How Difficult is it to Refactor a Workload for DSQL?

Aurora DSQL is a new service in Preview, meaning most functionality is present and working, but there are both known and unknown issues, some of which will be resolved before GA. The feature scope for GA can also change, meaning we can’t fully know for sure which features will or will not be supported by then. In rare cases, Preview functionality has been removed before GA due to security or operational concerns. Altogether, this means insights noted here are based on a point-in-time snapshot of service functionality that may or may not be representative of the service at GA. With that caveat, here are my key learnings from refactoring Vaultwarden to use DSQL:

Connecting / Authenticating

DSQL provides a PostgreSQL-protocol-compatible endpoint, meaning any existing PG client should be able to talk to it. However, DSQL only supports TLS-encrypted connections authenticated via temporary credentials that are valid for only 15 minutes. AWS is absolutely making the right decision here in enforcing authentication and encryption best-practices, but it may not be easy to refactor existing applications and libraries to use this approach. Vaultwarden uses the synchronous Diesel crate for database queries, and adding temporary credentials for authentication required a decent amount of new and complex code. Connection encryption and authentication alone will require significant application developer involvement in order to adopt DSQL for existing workloads. Even if a workload uses Amazon Aurora PostgreSQL with IAM authentication today, it will need to be modified to use the new DSQL SDK to generate signed credentials. You cannot simply swap in an Amazon Aurora DSQL endpoint for an existing Amazon Aurora RDS endpoint.

Migrations

DSQL only supports one DDL statement per transaction. It is very common for existing migrations to contain multiple DDL statements, however. For example, a migration may move data from columns in one table to a new table by creating the new table, copying the data to the new table, then deleting columns from the original table. The creation of the new table and the deletion of columns from the old table involve (at least) two separate DDL statements. You also want this entire migration to occur within one transaction to ensure either all the data is copied into the new table, or the database is left in its original state. We don’t want to end up in some half-way state with a new table and some partially copied and/or partially deleted data spread across two tables.

This is a significant cause for concern, but one that is not necessarily a deal-breaker. It means that migrations have to be performed with even more attention, and that migrations should be written such that an operator can manually complete or revert individual migrations if they fail (for any potential reason) part-way through. The tradeoff is that while DSQL should make day-to-day operations easier (e.g. because it can horizontally scale, it should have less potential for failing at larger data set sizes), it may make migrations more challenging. At least migrations happen at known points in time instead of other kinds of failures that can happen at random times.

Foreign Keys

DSQL does not support Foreign Keys (FKs). This is a more commonly known DSQL limitation, and was the first concern I audited the Vaultwarden codebase for. Many database schemas define FKs to ensure data consistency. For example, a schema could use FKs to ensure that when a user account record is deleted all the other records associated with the user are also deleted. FKs are also often used for upsert (i.e. insert-if-not-exists-otherwise-update) functionality, though there are other ways this can be accomplished.

Whether FKs are a concern for your workload depends on its schema and existing queries. In Vaultwarden, FKs are defined for all database backends (PostgreSQL, MySQL, and SQLite), and are used in MySQL and SQLite queries for upsertion, but are not used in PostgreSQL queries. The Vaultwarden 2FA database model save() function demonstrates an example of this implementation difference.

To make Vaultwarden work on DSQL I needed to remove all FK definitions from the schema. I accomplished this by initializing a local PostgreSQL database for Vaultwarden then running pg_dump to get all the table definitions. I then searched and removed all FK definitions. Lastly, I saved this schema as one large initial migration.

Adding Columns with Default Values or Constraints

Later in development I rebased my changes on top of the latest changes from upstream. The upstream changes included a migration that added a column to a table. This is one of the most trivial kinds of migrations that can be performed, as no existing data was touched. However, even this simple migration caused a headache.

It turns out that DSQL cannot add columns with default values or constraints. This isn’t clearly and explicitly stated anywhere in the documentation. The only mention of it is within a page that ostensibly describes the “supported” subsets of PostgreSQL commands. The Supported PostgreSQL features in Aurora DSQL page currently lists the following as a supported operation:

Did you catch the issue when your eyes scanned this row of information, which appears at the end of a long table of “supported” statements in the AWS documentation? Let me give you a hint by providing the syntax of the ADD COLUMN clause from the official PostgreSQL docs:

ADD [ COLUMN ] [ IF NOT EXISTS ] column_name data_type [ COLLATE collation ] [ column_constraint [ ... ] ]

...

and column_constraint is:

[ CONSTRAINT constraint_name ]
{ NOT NULL |
  NULL |
  CHECK ( expression ) [ NO INHERIT ] |
  DEFAULT default_expr |
  GENERATED ALWAYS AS ( generation_expr ) STORED |
  GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( sequence_options ) ] |
  UNIQUE [ NULLS [ NOT ] DISTINCT ] index_parameters |
  PRIMARY KEY index_parameters |
  REFERENCES reftable [ ( refcolumn ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ]
    [ ON DELETE referential_action ] [ ON UPDATE referential_action ] }
[ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]

The problem is that the DSQL ADD COLUMN statement cannot have constraints like a DEFAULT value or a requirement that the value be NOT NULL let alone all the other potential constraints one might need.

I want to be clear that my main concern is not that you cannot add a column with constraints after a table is created. That is a real and significant concern, and I sincerely hope they address it by GA as it is a very common element of database schemas. Instead, my main concern is that this needs to be documented much better. AWS will quickly burn a lot of good will if they confuse customers into thinking their workloads can operate on DSQL, and only find out later on that they can’t. The DSQL docs have two pages where people are likely to look for issues using the service with existing workloads: Unsupported PostgreSQL features in Aurora DSQL and Known issues in Amazon Aurora DSQL. But this significant ADD COLUMN limitation cannot be found on either page, nor is it clearly stated in the “supported features” page even though the statement is listed there.

Both the underlying ADD COLUMN limitation and this documentation issue give me significant concerns in using DSQL for critical workloads.

Conclusion

There is a lot to love about this Vaultwarden Serverless concept. There are also significant concerns with DSQL, which is a critical piece of the solution. How do we reconcile the concerns with the benefits?

I’ll start first with workload-agnostic takeaways. The core Lambda + S3 + CloudFront + SES components are rock-solid. They can also be used with any off-the-shelf database backend, including other AWS solutions like Aurora Serverless. But based on the learnings from this POC it would be hard to recommend DSQL for most workloads due to:

• The inability to perform multiple schema changes in a single transaction

• The lack of support for basic functions like adding a column with constraints

• Poor information architecture within the DSQL docs that make it hard to assess all the limitations relative to upstream PostgreSQL up front

However, my takeaway for Vaultwarden is more nuanced. The database requirements are well defined (e.g. no current usage of Foreign Keys beyond their definitions in the schema) and the data can be easily backed up via pg_dump. Further, the Bitwarden clients cache database-stored items locally and can re-export them all without needing access to the API service. If my API service went down, I could still migrate all my items to the official Bitwarden service (or other third-party service) from an export from any one of my clients. I also feel comfortable operating on top of DSQL for my own personal use, knowing that I can dig into the details and address issues if they arise. But I also wouldn’t recommend others deploy Vaultwarden using DSQL unless they can do the same.

That said, I love the promise of DSQL, and I hope that over time many of these limitations will disappear. It would be great to have a truly serverless relational database that worked for the majority of existing and new workloads.