In the seven months since we said "Hello, World!" Banyan has been hard at work bringing our vision of a secure, affordable, distributed data storage service to life. We're proud to announce the latest step in this process, a new distributed filesystem and data wallet: BanyanFS.
BanyanFS will allow us to continue to offer secure and reliable storage solutions, even as the data we store scales up to petabytes. At the same time, we’re concretizing the idea of a data wallet with a viable implementation. But we’re not just building it for our own use case. We want to collaborate with the community to build BanyanFS into an open standard so you can use it to bring your own vision to life.
Wait, a data wallet?
Yep, a data wallet! This term has been thrown around for a while and we’re finally making it a reality. A data wallet is like a crypto wallet for other data, so you only need to manage a few pieces of metadata like signing keys to have access to all your data, safe and sound, wherever you are.
Why a new filesystem?
Banyan's goals are ambitious. We want to provide an excellent experience for users storing everything from a handful of files up to petabytes of data without sacrificing on security or decentralization. While there are existing filesystems that get close to achieving what we want to do, none go all the way. After experimenting with other solutions, we decided the only way to fulfill our commitment to security, reliability, and usability is by creating BanyanFS.
What's a filesystem again?
A filesystem is simply the code that manages storing data and how you see it. Traditional filesystems only exist on one device and frequently require external solutions for things like encryption. If you want to store data in a distributed way using a traditional filesystem, you have to manually copy the files separately, without any automatic mechanism for synchronizing different copies.
BanyanFS operates like a traditional filesystem — it manages how data gets stored and adds an interface to it — while also coming equipped with end-to-end (E2E) encryption. It enables sharing access to specific parts of the filesystem with others without altering security guarantees, allows users to store data on multiple devices across different locations, uses a versioned journal so users can return to a previous data state at any time, and ensures all copies of data are automatically synchronized.
How does BanyanFS do it?
BanyanFS learns from earlier distributed filesystem projects like UnixFS and WNFS while adapting those lessons to Banyan's specific use case. We’re proud to say that we’ve solved the problems inherent to building an E2E encrypted, distributed filesystem that can still be used for secure collaboration while also providing superpowers like time traveling to previous states and automatic synchronization across devices.
We believe that E2E encryption is the best solution for security in distributed contexts. But using encryption in distributed systems makes collaboration technically challenging. BanyanFS uses an asynchronous Diffie-Hellman key exchange algorithm with hierarchically-organized keys so that authorized users can share granular access to the filesystem with anyone at any time. This means that BanyanFS lets you share limited or full access to individual files and folders without needing to give access to the whole filesystem. As a bonus, this system – combined with a versioned CRDT journal – means every change to the filesystem is associated with a specific key. As long as the key is safe and unique, you will always know who does what and when.
As we briefly mentioned, BanyanFS uses a versioned journal built on operation-based conflict-free replicated data types, or CRDTs, for time traveling and auto synchronization. Operation-based CRDTs mean your data is always in the same state, no matter where it’s stored, while using a versioned journal means you can rollback or re-apply any changes at will.
BanyanFS takes advantage of the properties of its journal as well as the nature of its encryption to associate every change with a specific user key. That means you always know who made what change and when. This is great for single users and businesses that need to ensure data attribution and provenance.
Using a versioned journal would normally require keeping around old data, even when you don’t need it anymore – otherwise, there would be no way to return to a previous version. Because we use operation-based CRDTs, we don’t actually need to do that. That’s why BanyanFS provides facilities for users to mark their data as no longer needed so that it can be deleted locally as well as remotely.
All of this great functionality would mean nothing if it wasn’t usable. That’s why we’ve carefully designed the metadata format for BanyanFS so that users can stream a view of the filesystem as they need it. Plus, BanyanFS is written in Rust and compiles to WebAssembly so it can run in the browser or almost anywhere else our SDK (coming soon) can be embedded. That means you can responsively browse all your files right in the browser – even if you have petabytes of data.
BanyanFS is interoperable with other data backends. Right now we support local disk, IPFS through UnixFS compatibility, and Filecoin; and are working on an S3-compatible API, to name just a few. If we don’t support your preferred data backend yet, join our Discord server and help us build support for it!
Is this only available to Banyan customers?
No! Our goal is to make secure, distributed storage accessible to everyone — and so BanyanFS is open source. You can see it right now on GitHub. Our engineers will be working on documentation, helpful diagnostic utilities, and general repository polish in the coming weeks. We’re excited to see what the community does with it!
That sounds pretty cool! Is it ready to use?
Yes, V1 of BanyanFS is now live running directly in the browser as a WebAssembly module, so go ahead and try out the world’s first data wallet. We’d love to get your feedback and collaborate with you as we build BanyanFS into a community-backed open standard!
