The Macro Filesystem: Redesigning Our Permissions and Sharing Model

Access management is one of the hardest systems we've built at Macro, and the one we've rewritten the most. The design we finally landed on does the most work under the hood and is the easiest for users to understand.

This is one part of a multi-part piece on how we design the Macro filesystem (MFS). Our focus in this article is the access management (permissions and sharing) system.

Some background on the Macro Filesystem

The Macro Filesystem ("MFS") is a unique problem because, unlike single-purpose software (Linear for tasks, Slack for chat, Notion for notes), Macro needs a flexible schema that works across all of these content types. Every entity in Macro, whether it's a document, a project, a chat, or an email, lives in one workspace under one access model.

MFS is also different from, and more complex than, a POSIX filesystem that just needs to store files on one machine:

• MFS is real-time collaborative for certain entity types, like markdown docs, and version-controlled. It needs to support different types of collaboration for different entity types. A POSIX filesystem can assume a single machine, one owner editing a file at a time, and read/write/execute permissions.
• MFS needs to be globally available in under 100ms for every user, no matter where they are.
• MFS is multi-modal, not just files: channels, video calls, emails, and more over time.

Beyond these requirements, MFS should be simple for users and developers to use. That is our design goal with Macro generally: it is why the app is built in Rust and SolidJS, and why we put so much effort into product design.

Designing permissions and sharing for MFS

When we talk about permissions, we mean which users and agents should have access to an entity. Sharing is closely related: it is the act of granting permissions and notifying users or agents of the grant via email, push notification, or Macro message.

In designing sharing and permissions for MFS, we had the following goals:

1. MFS's sharing model should extend nicely to agents, not just humans.
2. For humans, sharing should work the same way for every entity type as much as possible. Sharing a transcript of a call should not work differently than sharing a document.
3. Sharing should be simple. We are not concerned with obtuse enterprise use cases, at least not yet. We are concerned with avoiding gotchas, protecting privacy, and making it very easy to use.
4. Bad abstractions that create headaches for each block team (for example, the email team or the docs team) should be discarded. Where flexibility is needed, it should be allowed.

The first sharing system

MFS has had two main share models. The initial model was fairly standard and mirrored how Google Drive and other cloud drives let you share items.

• Items can be made publicly accessible.
• Items can be shared with individual users or your entire organization.
• If a project (folder) is shared with you, then all items within that project are also shared with you.

To support simple sharing we set up a UserItemAccess table that recorded every shared entity and who it was shared with. We made it purely user based: one row per user per item they could reach. Sharing a project with your organization meant writing a row for every member, so adding or removing someone from the org meant fanning that change out across every shared item. That was not ideal, but shared item counts were small and membership rarely changed, so it held up fine early on. (We have since dropped organizations entirely in favor of a new team system.)

Finding all the items a user could access meant looking up your user id in UserItemAccess and, for every project, recursively drilling down into it to grab all sub-items. By all accounts this seemed fine at first and gave us everything we needed.

The eureka moment

As time went on and channels (group messaging) were released, we frequently found ourselves sending docs to one another through a channel only to be hit with "I don't have access to this".

This is nothing unique to Macro. It happens half the time someone pastes a Google Docs link in Slack, or a Figma in a Notion. So we didn't initially recognize this was a problem we could solve.

One day, though, we were sitting in the office and thought...

"WAIT, we can fix this sharing foot-gun problem!"

Introducing: channel-based sharing

Because Macro has everything in one system, we thought: why not just "cascade" the permissions from the channel to the document? (Cascade was the first word we used to describe it.)

Everybody in the channel should obviously have access to the document, whether it's a DM to Teo or a doc dropped into #engineers. You would never send something to someone and not want them to be able to open it.

At first the idea was a webhook that would auto-share things with the members of a channel using the share system we already had. Sharing an item in a channel would grab every user in that channel and insert UserItemAccess rows for them. That bought us the behavior we wanted, but it piled onto the same problem as before: now channel membership changes also had to fan out across items.

This all worked okay... but...

The problem: it was slow

Copying Google Drive seemed fine. The webhook idea seemed fine. But it was slow.

As the number of items a user had grew with each new entity type, the recursive read query got slow, especially on a cache miss. For some power users it could take over 10 seconds to grab their accessible items, depending on database load and other factors. The cause was the shape of their data: large nested projects alongside many root-level items, which the query planner couldn't walk efficiently.

On top of the read cost, the write side was a liability of its own. Every channel or org membership change meant remembering to grant or revoke the right items, which was easy to get wrong. This was unacceptable to us, and I began designing a new system to fix both halves.

Why not rework the whole sharing system around channels?

Instead of a webhook side effect, what if the sharing system itself was channel-based, instead of working like Google Drive?

We explored the edge cases, found nothing fatal, and decided to rebuild sharing around channels.

Making channel-based sharing fast

The new design flattens everything into a single table and requires no recursion at read time, at the cost of becoming more write heavy. The recursion doesn't disappear; it moves to write time, which happens far less often and tolerates latency much better. That trade works for us because the vast majority of items aren't deeply nested, and it leaves the door open to push writes onto an event-based pipeline later without touching the read path.

CREATE TABLE entity_access
(
  entity_id UUID NOT NULL,        -- document_id, project_id, chat_id, email_thread_id
  entity_type EntityType NOT NULL, -- document, project, chat, email
  source_id TEXT NOT NULL,         -- channel_id, team_id or macro_user_id if creator
  source_type TEXT NOT NULL,       -- channel, team, user if creator
  access_level AccessLevel NOT NULL, -- the access level granted to the source
  granted_from_project_id UUID     -- set when access came from a shared project
);

Like the rest of Macro, the implementation is open source at github.com/macro-inc/macro.

How does the entity_access system fix the previous iterations' shortcomings?

Instead of creating records per user, we create them per source for channels and teams. Removing a user from a channel or a team is as simple as deleting them as a member of that channel or team. No more side effects.

The source of an entity_access row can be a channel, a team, or a user (for item creators).

granted_from_project_id tells us which shared project produced the row. If that project is later deleted or moved, it becomes much easier to update all the affected records.

Adding an item to a project

In this example we have the following project structure:

A/ -- owner 1
-- B/ -- owner 2
---- C/
------ add item here

Say we are adding an item to project C. Programmatically, we need to walk up the tree to get all parent project ids, including project_c itself. That gives us [project_a, project_b, project_c]. Next, we get all channel and team source entries and access levels for those projects.

In this example:

• project_a was shared with channel_1, so channel_1 should have view access
• project_b was shared with team_1, so it should get comment access
• project_c was shared with team_2 with edit access

With that information we insert the following records:

Alongside the channel and team sources, we also fetch each parent project's owner (owner1, owner2) and insert an owner row for them, so they keep access to items added anywhere beneath their projects. The <user_id> row with no granted_from_project_id is the creator of the new item itself.

Granted, there is more work upfront to insert correct access for a new item in a project. But our workload is very read heavy, and we should prioritize reads over insertion speed.

Moving a sub-project

In this example we have the following projects:

A/
-- B/
---- C/
------ document_a
------ document_b

X/
-- Y/
---- Z/

We are going to move project C from project B into project Z.

We walk up the project tree to get all parent project ids (project_a, project_b). Notably this excludes project_c, since none of its own permissions change. project_c can still be shared with the same teams and channels it was before, with no changes required.

Next we get all items in project_c (document_a, document_b), and then:

DELETE FROM entity_access
WHERE entity_id = ANY([document_a, document_b, project_c])
  AND granted_from_project_id = ANY([project_a, project_b]);

This removes the implicit permissions that parent projects granted while project_c lived inside them. Finally, we run the same steps as adding an item to a project for everything in project C, including project C itself, under its new parents.

Deleting a project

When a project is deleted, all of its items are deleted as well. The process is the same as it was with UserItemAccess, with a slightly different query: delete from entity_access where granted_from_project_id = x OR entity_id = x.

How we access items

Now, instead of the recursive tree walk over projects to find accessible items, we simply get the user's channels and teams (easily cacheable queries) and then run one query on entity_access with source_id = ANY([channels, teams, user_id]).

Because a user can reach the same entity through multiple sources (for example, a channel that grants view and a team that grants edit), this query can return more than one row for a single entity. When that happens, the highest access level wins, so in that example the user ends up with edit.

Conclusion

Sharing is now as easy as sending someone a message, and we can retrieve everything a user has access to quickly. Flattening entity access into a single table reshaped the system around the access pattern that actually mattered to us:

• Reads got fast. The recursive tree walk became a single indexed lookup, taking our worst-case power user from over 10 seconds on a cache miss down to about 600ms.
• Membership changes lost their side effects. Grants are stored per source rather than per user, so adding or removing someone from a channel or team is just adding or removing a member.

There are still downsides, of course. The recursion wasn't eliminated; it moved to write time, where granting access still has to walk the project tree. Our workload is overwhelmingly read heavy, which made that trade easy to stomach. And the nature of the writes lets us set up an event-based pipeline if we ever have enough volume to need to process them asynchronously.