Releasing Org-roam v2

6 months and over 200 commits later, I’m finally releasing Org-roam v2!

Org-roam v2 is an overhaul of Org-roam. It is slimmer and more performant, and has first-class support for linking to headlines. It also features a revamped side-buffer, a rich completion interface and a clean public interface for community extensions to work with.

Org-roam in 2021?

Over the course of the year, new plain-text note-taking apps have been developed. The most notable ones (imo) are Obsidian (which now has a mobile app) and Logseq. These products both look fantastic, and are under active development. It also appears possible to use Logseq as a web interface to Org-roam, but I’ve never tried it.

I think Org-roam still best serves the niche community of users who take notes in Emacs using Org-mode. There is simply no good alternative to both Org-mode and Emacs.

Why the rewrite?

I started contemplating a rewrite some time in December 2020. Github issues were piling up, and I could not keep up. Org-roam began as a simple (70 LOC) file-based linking system with a backlinks buffer, but it quickly accrued new features, many of which are brittle and interact poorly with each other. Org-roam v1 itself had gone through many redesigns, and it accumulated poor architectural decisions. For example, headline linking support was patched in, and many of the functions then had two separate code paths to support both files and headlines. I wanted to see if I could distill what I’ve learned building Org-roam v1 into a simpler, faster v2.

Also, I’d like to take this opportunity to clarify what Org-roam means to me, and hopefully seeing things from my point of view will allow you to be more empathetic of the current situation. I’m less crazed about note-taking paradigms than most people think. I still use and maintain Org-roam because it removes the friction in taking, accessing and synthesizing notes, and to achieve this I only need to use a tiny subset of its features. This makes it difficult for me to maintain or fix user-contributed features. Being the sole maintainer is really really hard: beyond my full-time job I have little mental capacity left to comprehend and resolve these issues that I don’t have first-hand experience with. With Emacs Lisp being a tinkerer’s language, I’d expected more open-source contributions, but it is what it is. It’s incredibly fun to work with Emacs Lisp once you learn the ropes! There’s no other programming environment that lets you step through your code the way Emacs does. I guess this was also in part due to the codebase for Org-roam being incredibly huge, and a good part of it consisted of hacky extensions of Org: there were parts that I myself could no longer comprehend.

Part of this rewrite involves narrowing Org-roam’s scope, making it more manageable for me. The new Org-roam is designed to fulfill a singular purpose: it acts as dual representation to your plain-text Org-mode notes. It mirrors key information about your notes in a SQLite database, in a format that’s convenient for programatic queries. The main features (primarily creating and accessing notes) will continue to have first-class support. In addition, a public interface for accessing the database is provided, and some simple guidelines are laid for those who wish to build extensions on top of Org-roam.

With that, I can stay focused on keeping the core high quality, while those in the community who are more enthusiastic and exploratory about note-taking workflows can experiment with the public interface. This also allows me to be accountable to those who have shown support for my work. I hope that the restructuring will also invite more contributors.

What took me so long?

Because Org-roam v2 is a near complete rewrite, it provides a unique opportunity to make breaking changes. I wanted to get it right so users don’t have to jump through hoops to migrate their notes over again.

2021 hasn’t exactly been a great year for me either. I’ve been chugging along whenever I had the time and willpower, which turned out to be a rare combination. I am indebted to many of the users in the community for continuously demonstrating passion for the project, providing early feedback, and answering any queries. I’m not as active in Slack or Discourse as I’d like to be, but I do read them from time to time. I’m particularly grateful to Noboru (@nobiot), who has been relentless in helping users and writing documentation, a practice he’s been keeping up for a while now. Boris (@d12frosted) has also been influential in some of the early architectural decisions of v2. I also appreciate everyone who provided support, monetary or otherwise. I hope you like what v2 has to offer.

What’s different?

The key difference between v1 and v2 is that v2 makes no distinction between a file note and a headline note. In v2, Org-roam views notes in terms of nodes, where a node is any headline or top-level file with an ID. This means that IDs are now compulsory.

This one requirement reduces a lot of overhead. We no longer need special hooks maintaining the validity of file: links, and no longer have to treat headline and file notes differently. Org-roam v2 removes 7000 lines of code while only re-adding about 4000, and even ships with a few more features than Org-roam v1.

Other notable new features include:

• A large number of additional properties are now cached for later usage. This includes the node’s TODO state, priority, and the scheduled/deadline time.
• Keeping the database in sync is much quicker than it was in v1, despite more things being cached
• Org-roam v2 ships with a new buffer based on magit-section for displaying backlinks and other widgets

Wtf bro you just broke my config

Some people seem to think that I don’t care about backwards compatibility, and I think it’s okay releasing breaking changes. I can assure you it brings me more pain than pleasure doing so. I’m expecting a tirade of “XXX is broken” from unknowing package upgrades. And guess who has to deal with that? That’s right. Me.

The crux of the issue is that there are now two versions of the same package, which operate very differently (file links vs id links) but are in fact very similar. The main difference is that one of them (v1) will no longer be actively maintained by me. And when I say no longer actively maintained by me, it really means no longer actively maintained, because… you know why. I created a PR to MELPA to provide the Org-roam v1 so both packages can be easily installed, but the MELPA maintainers make a fair point that MELPA shouldn’t really be distributing orphaned packages.

Therefore, only Org-roam v2 will be distributed on MELPA, under the same name. That means that if you unknowingly upgrade your package from MELPA and obtain v2, then your config will break. You’ll however see an obnoxious warning that you’re now on v2. Those who wish to use v1 can obtain it from the repo, but I encourage you to try give v2 a try.

As a side note, always pin your packages! Even the poorest package managers for language ecosystems lets you specify the version of the package you want to install. Package maintainers can publish the breaking changes under a new major release and call it a day, without thinking about whether it will break production deployments, beacuse the responsibilty is on user to use the right version of the package. This infrastructure, while not the default for Emacs, is available, and you should use it! I like straight, but there’s many others like quelpa.

One reason I’m less concerned about pushing out this breaking change is because migration is actually simple. To upgrade your notes from v1 to v2: simply run M-x org-roam-migrate-wizard (more details here). After which you just need to familiarize yourself with the new commands, which behave very similarly to those in v1.

Why merge now?

People who rely heavily on org-roam-server will be displeased to find that it is not yet compatible with Org-roam v2. Shouldn’t I wait for full feature-parity before releasing v2?

This is once again an issue with the packaging infrastructure for Emacs: these extensions should be able to pin the Org-roam dependency and upgrade in their own time, but alas. org-roam-server and org-roam-bibtex, while under the Org-roam organization, are community extensions. The original author of org-roam-server has expressed that he is now busy and would be unable to help out with the migration, and I’m not sure how long that would take either. And I think it’s odd to delay the release of a main package based on its dependencies. Kudos to Mykhailo (@myshevchuk) for keeping org-roam-bibtex compatible with v2 though!

v2 as a standalone package is ready for public consumption, and that’s why I’m merging it.

What’s next?

Org-roam v2 provides the necessary caching infrastructure and a relatively stable query interface to build extensions upon. I hope this encourages more active experimentation. Some ideas that sound good in my head but are unexplored include:

1. Integrating mail: there seems to be decent Org integration for notmuch, mu4e
2. Integrating news: working with elfeed, gnus or whatever people use this days
3. Building a polished publishing infrastructure via Org export

While org-roam-server, is not yet compatible with v2, I’m pleased to see that there are people stepping up to build it. It shouldn’t take too long to reach feature-parity. There’s also a lot more that can be done with the Org-roam buffer that I have yet to explore.

Following which, I plan to build one or two more core functionalities that I believe would be useful. The one that’s next up on my roadmap at the moment is org-roam-agenda. Using the Org-roam database, it should be possible to build a version of org-agenda that is incredibly fast and scales across thousands of files.

That’s all I have for now. Happy note-taking!