The architecture of modern forum software

Storyden, that's the modern forum software I'm referring to. Even though it's more than just a forum! But anyway, let's get into the innards!

Starting in the middle

At the core, Storyden's behaviour is defined as an OpenAPI (opens in a new tab) specification. This specification is hand-written, optimised for readability because it's intended to be read (opens in a new tab).

I opted to write the specification and generate the code because I really value static, declarative documents from which the boring bits can be mass-produced. I don't really enjoy writing func(w http.ResponseWriter, r *http.Request) functions by hand, dealing with decoding the JSON and encoding the responses and errors. OpenAPI allows me to work with functions that look like this:

AccountUpdate(ctx context.Context, request openapi.AccountUpdateRequestObject) (openapi.AccountUpdateResponseObject, error) {
  // access `request.Name`
  // respond with an `Account` struct
  // handle errors by returning (nil, err)
}

It also allows me to generate client code for both Golang (for end-to-end tests, my favourite flavour!) and TypeScript. The frontend for calling the above example looks like this:

const updatedAccount = await accountUpdate({ name: "Southclaws" });
// updatedAccount: { name: "Southclaws", ... }

Which eliminates a metric ton of work for me and other contributors!

But it's more than just a time saver, it's documentation and, most importantly, a contract! A contractual interface that's agreed upon by developers before diving into implementation details behind the interface.

Now I don't take the spec part that seriously, it's a useful layer to write some details about things that may not be obvious just from the operation name and parameters. But there's no formal MAY, SHOULD, MUST lingo in there, it's just a useful source of truth from which everything else is built on.

Speaking of APIs...

Also, this makes Storyden API-driven. You can bin the stock frontend and build your own if you want! You can also build other services in whatever language you want that call these APIs to automate certain tasks where WebAssembly plugins and integrations aren't quite enough.

Content-type driven handlers

Another neat thing you can do with OpenAPI is define HTML form friendly handlers. Most JSON APIs accept application/json from a fetch request. But if we want to support JS-less frontends that want to use HTML forms in all of their natural beauty, it's as simple as:

AccountUpdate:
  content:
    application/json:
      schema: { $ref: "#/components/schemas/AccountMutableProps" }
    application/x-www-form-urlencoded:
      schema: { $ref: "#/components/schemas/AccountMutableProps" }

This requestBody (opens in a new tab) schema permits two content types that use the exact same underlying schema. This re-use means certain endpoints ✳︎ can trivially be set up to support non-JS clients as long as their HTML form field IDs match the fields documented in the schema.

HTML forms are important because some folks disable JavaScript for good reasons: privacy concerns, bandwidth constraints and device processing power.

The Backend

I've already teased some Go code so if you've not checked out the repository (opens in a new tab), by now you can probably guess the language of choice.

I chose Go because I like Go, and the things I like about Go fit quite nicely into Storyden's goals particularly how you can compile it almost anywhere for almost anywhere else to a single binary. No deep trees of dynamic source files to package up, no version managers to get confused about, plus it's got a decent type system!

I won't go into more detail about the codebase itself, it's a pretty standard idiomatic Go codebase with a few opinionated bits like initialisation-time dependency injection, that's a topic for another post.

The data model

This section won't go into every single table but a brief overview of the most important parts.

Posts

The most interesting and important part of the model is the posts table. It's organised as a directed acyclic graph where each post has two parent relationships:

• root post:
  • if the post is a reply within a thread then this is the first post in that thread
  • if the post is the start of a thread, this is empty
    • you can find all threads by simply querying for posts with no root
• reply-to:
  • you can also reply to specific posts within a thread, independent of the root post
  • the reply-tree is similar in principle to that of Reddit, Hacker News or Lobste.rs

There are a few benefits to this approach, a lot of older forums would model "Threads" and "Replies" as two separate tables, but this often leads to some duplication of common fields as well as making certain operations a bit more awkward such as merging two threads, moving posts between threads or promoting posts to top-level threads.

There are some downsides though, Threads and Posts are not identical so there are a few fields that are only used for one but not the other (such as slug and title.)

Authentication

If you look at the "Account" schema, you may notice the lack of two fields that are usually a standard in any database schema with a "user" model:

• email
• password

This omission is intentional. While ideating Storyden, one of the values I chose was that Storyden is a platform the the next era of internet culture (or something like that...) and the two things I'm not entirely certain will be guaranteed in 20 years time are emails and passwords.

Okay, the emails one is a stretch, but passwords I strongly believe should be optional.

And so, this fact is true right down to the data model. Instead of encoding these concepts as fundamentals on the account table, they exist as "Authentication methods" which use a separate table on a one-to-many basis against accounts.

This makes it trivial to facilitate a choice of authentication methods for each account and allows individual communities to customise exactly how they want to allow users to register and log in.

Database tools: SQL and Ent

I have a complicated (opens in a new tab) relationship (ha!) with relational databases, but it's a very necessary evil for such a project. While I tend to avoid pasting raw SQL into string-literals in favour of code-generated type-safe interfaces, there's a healthy balance of both.

Ent (opens in a new tab) does most of the CRUD legwork, raw SQL does anything that requires a recursive CTE or an optimised join. That's really all there is to it. The schema has no migration strategy at the time of writing, but this will likely become a necessity as the product matures. I'll likely choose Atlas (opens in a new tab) for that task but suggestions are always welcome!

The main reason I chose Ent was the code generation part, the vast majority of boring queries are CRUD and don't really require too much complication or custom code. Ent also generates the structs too and provides a fairly neat way to traverse the graph of relations.

API: OpenAPI generated code

The OpenAPI specification mentioned above is turned into Go code using a library called oapi-codegen (opens in a new tab) which does a decent job of generating all the schemas and interface. All developers need to do is satisfy the interface.

The Frontend

My frontend tool of choice hasn't really changed since I started doing frontend work professionally. Storyden uses Next.js because I like React but also like server-side rendering and shipping HTML.

Next.js has had an admittedly rocky 2023 since the "App directory" chaos and there are lots of new frameworks on the block trying to dethrone it, but I've never really been one to hop between frameworks (terrible frontend dev, aren't I?)

User interface

My weapon of choice for styling is Panda (opens in a new tab), which by no surprise is a code-generation tool. Panda allows you to specify a design system as a (fairly) declarative document and generate all the code and CSS statically. This means the frontend doesn't need to run JavaScript to style things.

Which may sound odd but... look, the frontend world has had a rough decade okay!

So we're shipping static HTML and static CSS like the Good Old Days (opens in a new tab). Great! But how do you make it pop when everything is static?

Well, I lied, it's not all static, it's ✨ Progressively Enhanced 💫 (opens in a new tab) (very few things make me proud to be British, but the government design system is just amazing) which means static HTML and CSS gets sent to the browser to render everything fast, then bits of JavaScript join the party a little later to jazz it up a bit.

What this means in reality is we can have all the bells and whistles of what you'd expect from a modern web application while still retaining the qualities of what makes a great web site.

Ark UI

For the actual components, I've chosen Ark UI (opens in a new tab) which is a neat little headless component library providing all the standard widgets you might expect on a user interface. It pairs quite nicely with Panda CSS and together these two tools power the entire layout and interface elements of Storyden.

The road from Chakra to Panda

A short side note, Storyden (and most of my products) started life with Chakra UI (opens in a new tab), which is an amazing library by the very talented Segun Adebayo. For various reasons I chose to move away from Chakra UI after Segun published this post (opens in a new tab) and I discovered that Panda is a better fit for the project.

To learn more about why Storyden moved from, I wrote a short thread about that (opens in a new tab). And if you're interested in the technical details of how to migrate from Chakra UI to Panda CSS, I also wrote a guide (opens in a new tab)!

SWR

The underlying request state for the React code is managed by SWR (opens in a new tab), a neat little library from the Vercel team which I fondly remember the release of. It does a few handy things that facilitate instantaneous reactivity to interactions that result in mutations and data access.

I won't go into the details but it's a fantastic tool for building web applications that feel like local apps.

How OpenAPI is used

For the client code generation, I chose a tool called Orval (opens in a new tab) which generates code which utilises SWR as well as all the TypeScript types that match the OpenAPI schemas and the Go structs on the other end.

Data retrieval

Getting data (via GET requests) is done via hooks that look like this:

export const useAccountGet = <
  TError = UnauthorisedResponse | NotFoundResponse | InternalServerErrorResponse
>(options?: {
  swr?: SWRConfiguration<Awaited<ReturnType<typeof accountGet>>, TError> & {
    swrKey?: Key;
    enabled?: boolean;
  };
}) => {
  const { swr: swrOptions } = options ?? {};
 
  const isEnabled = swrOptions?.enabled !== false;
  const swrKey =
    swrOptions?.swrKey ?? (() => (isEnabled ? getAccountGetKey() : null));
  const swrFn = () => accountGet();
 
  const query = useSwr<Awaited<ReturnType<typeof swrFn>>, TError>(
    swrKey,
    swrFn,
    swrOptions
  );
 
  return {
    swrKey,
    ...query,
  };
};

Which roughly just wrap a useSwr hook call and sprinkle in some type annotations.

Note that there's no actual schema validation happening here with a tool such as Zod (opens in a new tab) because the assumption is that the backend is conforming to the OpenAPI specification too. Given that Storyden is in control of both sides of this in the monorepo, it's a compromise I'm willing to make.

Data mutation

Mutations to data such as create, update and delete (POST, PUT, PATCH and DELETE) are done via functions that look like this:

/**
 * Update the information for the currently authenticated account.
 */
export const accountUpdate = (accountUpdateBody: AccountUpdateBody) => {
  return fetcher<AccountUpdateOKResponse>({
    url: `/accounts`,
    method: "patch",
    headers: { "Content-Type": "application/json" },
    data: accountUpdateBody,
  });
};

Which can be easily called in event handlers such as button clicks or form submissions, etc. The fetcher is a client written by hand which handles a few extra details such as CORS, cookies and errors.

Server Side Rendering

SSR and RSC are a hot topic right now, but I won't go into why. I'm bullish on it and I find the mental model productive (though the reality is a little rough around the edges.)

For a full rundown, I highly recommend this post by Josh Comeau (opens in a new tab)!

Storyden's view of this is that any content consumption screen must be server side rendered. That is any feed of posts and the posts themselves, as well as other stuff like the knowledgebase and people's profiles.

How this works in the code is all pages start life as async function components:

export async function FeedScreen(props: Props) {
  const data = await server<ThreadListOKResponse>({
    url: `/threads`,
    params: {
      categories: [props.category],
    } as ThreadListParams,
  });
 
  return <Client category={props.category} threads={data.threads} />;
}

This performs the initial API call with any query parameters passed in from the Next.js page load. It then passes the result to a component called Client which is in another file.

Client which is defined in a sibling module looks like something like this:

"use client";
 
export function Client(props: { category: string; threads: ThreadList }) {
  const { data, error } = useThreadList(
    {
      categories: [props.category],
    },
    {
      swr: {
        fallbackData: props.threads && { threads: props.threads },
      },
    }
  );
 
  if (!data) return <Unready {...error} />;
 
  return <MixedPostList posts={data?.threads} />;
}

As outlined earlier, these generated hooks such as useThreadList are thin wrappers around useSwr so there are a few important things happening here:

• the first argument contains the query parameters for the actual API endpoint, these are often the same as the parameters in the browser's address bar.
• the swr option in the second argument means the hook will immediately return the provided data while revalidating in the background. This is called Pre-fill data (opens in a new tab) and it allows this client component to be rendered server-side using the data fetched in the server-only component above but continue to provide the benefits of SWR when it renders on the client.
• because we're using fallbackData, the data part of the return value is always present but TypeScript forces us to check due to the discriminated union return type.
• MixedPostList renders immediately with the data we have on the server
  • once the browser renders this, it'll render again after useSwr has re-fetched

Most screens in Storyden follow this pattern, with some extra bits that make certain things easier such as mutations and pagination. But it's pretty much the same idea throughout.

Conclusion

Ultimately, my goal is to make Storyden secure, modern and easy to contribute to. There's not much more to say on this, but feedback is always welcome so if you have opinions or thoughts on the direction any of this should move in, open an issue (opens in a new tab)!