Hey everyone,
after almost a year of work I'm very excited to announce Irdest release
v0.6.0!
This is probably the biggest update to Irdest (and in particular the
router ratmand) since the beginning of this project. In this e-mail I
want to highlight some of the changes and outline the future of
development from here on out.
Architectural changes
---------------------
The biggest change in our architecture is the change from "discrete"
messages to streamed messages. Previously, when trying to send a 2GB
payload ratmand would load all of it into memory, blockify it, encrypt
it, slice it, then send it. This had the massive limitations that the
router had to hold the whole message state in memory, and any error that
occured during transmission lead to data loss. Not to mention that the
router could easily run out of memory and crash.
This has now been changed. Every operation in ratmand is a stream, no
more than a few dozen megabytes are stored in memory at a time, and
things are streamed to and from a database journal on disk. This _can_
currently become quite large (several GB), which is something we will be
improving in future. It also means that the router crashing (or being
killed by external forces) half-way through a send procedure now is a
recoverable scenario.
User/ developer facing changes
------------------------------
The client facing API has been completely overhauled, ditching protocol
buffers in favor of a very simple hand-rolled header-payload frame
format (called "microframe"). The Rust client library implementation
(libratman) handles encoding and decoding of all of these payloads, but
the format is simple enough that alternate implementations won't be
difficult to write for third-party developers. There's some
documentation on this format in the developer manual.
Additionally we have now split the CLI from a single tool into two:
ratcat, responsible for sending and accepting message streams, and
ratctl, responsible for managing addresses, subscriptions and streams,
accessing peer and system diagnostics, and in future managing contact
aliases and other subscriptions as well.
In summary
----------
Overall this is a very exciting release of Irdest. Not all components
have been updated to the new APIs and the main focus of development has
been on making ratmand, ratcat, and ratctl as stable and easy to use as
possible. If you're interested in a full breakdown of the changes you
can check out the changelog [1] for this release. And as always, you
can download the new version installer from our website [2]. It's
recommended to delete any existing state from ~/.config/ratman,
~/.config/ratcat, and ~/.local/ratman before installing this version.
While we can't promise stability guarantees for the on-disk state just
yet, this is a huge step forward to a stable release of Irdest and I'm
very excited to see how future development unfolds. I'm also excited to
see what people can and will do with this new version.
Cheerio,
~ Kookie
[1]: https://git.irde.st/we/irdest/-/blob/0.6.0/CHANGELOG.md
[2]: https://irde.st/downloads/
I am pleased to announce that with the merge of the request 129 [1] we now are publishing package for OpenWrt for the Armv7 architecture which will support quite a large number of old routers.
OpenWrt [2] is an open source router operating system which has a large support for old and new router architectures. This enables ratmand to run on routers and thus become sort-of access points in the peer-to-peer communication.
OpenWrt packages are mostly tar/gzipped files with a certain structure with your binary. The current package is build using our CI/CD and is published on Irdest’s Git host (an example job run that published this is at [3]).
The demo video of installation can be seen on our PeerTube instance [4].
This is first of many steps and improvements in this area and hoping to do more for better support of all major architectures. This is critical because if routers can be deployed in a crowded area, they can help with communication loads and user devices do not have to do as much work for relaying.
[1] https://git.irde.st/we/irdest/-/merge_requests/129
[2] https://openwrt.org/
[3] https://git.irde.st/we/irdest/-/jobs/53255/artifacts/browse
[4] https://diode.zone/w/qexDpmaP6TZU866TYyCsmB
Hey everyone,
it's been a while since the last update. I'm sending this e-mail to
both the announcements and community list to a) break the ice on the
community list and b) make it easier/ more approachable for people to
reply to the announcement on the community list.
So far the community list hasn't really gotten used, since we mostly
communicate on Matrix. In the future I suspect the community list
would be the easiest place to report issues, and contribute code
changes, since our GitLab requires account verification, which can be
annoying.
In the meantime, let's get into todays announcements:
Irdest has received two new NLnet grants this year! One for myself,
and one for Aman, who has been part of the Irdest community for many
years, and I'm very excited to get to work with him finally :)
My NLnet grant
--------------
The Irdest project has been in a strange state for this past year (and
a bit). Since we forked off from qaul.net, there was a kind of
pressure to change things on a technical and social level, to work
towards a new vision for the project. And I think, mostly, we have
succeeded at that.
At the same time, the result of this "exploration" has been technical
debt at many levels, inconsistencies that have slowly accumulated, and
more and more bugs that are the result of them.
This is going to be the focus of my grant work: to bring Irdest into a
more unified state, to integrate features that were written as part of
the last NLnet grant but haven't been able to properly land in a
release. And to make the codebase easier to understand, work on, and
maintain in the future. The core of the Irdest project (Ratman) used
to be just one component of a larger tower, but now it _is_, at least
in many ways, the tower. This needs to be reflected in the way the
project and repository is structured.
New release
-----------
Today I am tagging release/0.5.0!
It brings a lot of simplification to the codebase. All of the ratman
utility crates (-netmod, -client, -types) have been consolidated into
"libratman", which also replaces the "ratman" crate name, since there
is little point to having a crate on crates.io that purely represents
an executable target (the daemon). The daemon is now called
"ratmand". The ratman tools are now no longer special parts of the
ratman tree, and have instead been moved to the clients directory,
with the rest of the applications that interact with ratmand over its
client API.
Additionally I changed the configuration format from JSON to KDL [1].
I've been deliberating about this change for a long time now, and
while the code for parsing the configuration is now a lot more
complex, it also allows us to upgrade configurations in-place in the
future, can handle comments more easily, in-place edits, nested
structures without getting unwieldy to edit, etc. The syntax will
most likely be new to people, but I feel it's straight forward enough
(especially in the way that we use it) that it should be easy enough
to get used to.
There are some other smaller things that you can read about in the
release announcement! [2]
Aman's grant
------------
And finally, I want to briefly talk about Aman's NLnet project: part
of the shift in "vision" for Irdest has been to move away from the
idea of building another meshing "app", and instead focus on the idea
that technical infrastructure requires work and maintenance, and to
democratise this work and maintenance.
Making it easy to take existing router hardware, and augment it with
easy to use and configure meshing capabilities, in our opinion, is
fundamental to getting a project like Irdest not only to succeed, but
to stick around any one hype wave. I have installed and forgotten
about too many "apps" that were too single minded in their approach to
communication. So, with this in mind: Aman's grant focuses on
building ratmand and any other necessary tools for OpenWRT and
building images/ packages that people can install via the existing
OpenWRT package infrastructure. Additionally, he will be working on a
netmod driver for embedded bluetooth boards. Potentially this could
re-use some of the serial driver infrastructure we already have in
place for the LoRa modems, but we'll see.
In conclusion
-------------
Anyway, I am very excited about the future of this project. There are
still many many things to do, and I hope to be able to provide updates
on how things are going more frequently. In the meantime I will start
using the announcement mailing list for actual release announcements,
plus probably I will start posting RFCs (requests for comment) on the
community list regarding our upcoming specification and similar!
Thank you for reading my somewhat rambly thoughts on the future of
development. If you have any questions, feel free to reply to this
e-mail, or start a new thread (on the community list).
Read you all soon,
- Kookie
---
[1]: https://kdl.dev
[2]: https://git.irde.st/we/irdest/-/tags/release%2F0.5.0
Irdest April 2022 announcement
══════════════════════════════
Dear readers,
it's been a few months since our last announcement and a lot of things
have happened since then!
Grant funding updates
─────────────────────
In the last e-mail I hinted that we were going to open/ rework our
NLnet [1] grant funding agreement. This has now happened and I'm very
excited to welcome new people to the core Irdest development team.
Together we will work on everything from driver integration,
user-facing clients, better packet encoding and encryption, and
testing frameworks. A full breakdown of milestones is available in
our wiki [2].
I'm very excited for this summer of development and the progress we
will make towards an Irdest beta. In the meantime, we also have
another small release ready for you!
Ratman v0.4
───────────
This version of Ratman fixes some significant bugs that existed in 0.3
that prevented various peering mechanisms to work properly. A lot of
testing has gone into this version to make sure it is easier to use
and less prone to crashing. /Note that this is a breaking update/.
Following is a short overview of significant changes.
• Ratman stores runtime settings in a configuration file and has
better default configuration values, which allow you to start the
daemon without any arguments
• We now provide a simple web dashboard on `localhost:8090' where you
can see existing addresses on the network. This dashboard will be
expanded in the future
• The Ratman daemon can now also be launched via the –daemonize and
–pidfile flags to make integration into operating systems easier.
It now also supports logging into syslog
• Network floods are now namespaced via an address, instead of going
to every participant. Filtering still has to be done on the
client-side, which will likely change in the next release
• Various bug fixes, performance, and documentation improvements :)
You can read more details in our new CHANGELOG [4]. Overall I am very
happy with this release and would like to thank the following people
for making this version of Ratman possible 🖤
• Bluju <bluju(a)Bluju.local>
• Lux <lux(a)lux.name>
• Manos Pitsidianakis <el13635(a)mail.ntua.gr>
• Yureka <yuka(a)yuka.dev>
• embr <git(a)liclac.eu>
• oooh chew <gx_339-4(a)pm.me>
You can download Ratman v0.4.0 from your friendly neighbourhood
package repository, or from <https://irde.st/downloads>
If you have questions or comments, feel free to drop by our community
mailing list [4] or our matrix channel [5].
~ Kookie
Citations
─────────
[1]: <https://nlnet.nl/project/Irdest/>
[2]: <https://hedgedoc.irde.st/M-v5_NZLTTid27-hndogZg>
[3]: <https://git.irde.st/we/irdest/-/blob/develop/CHANGELOG.md>
[4]: <https://lists.irde.st/archives/list/community@lists.irde.st/>
[5]: <https://matrix.to/#/#chat:irde.st>
Dear readers,
I'm very excited to make this announcement today, approaching the one
year anniversary of the Irdest fork from the previous project's
leadership. In the past year a lot of things have happened. We have
gotten funding via the NLnet [1] European research grant program, as
well as participated in the Google Summer of Code for the first time
(technically anyway).
But while these are amazing milestones in the life of this project,
today's announcement is about something more technical.
[1]: https://nlnet.nl/project/Irdest/
The announcement
----------------
I'm extremely happy to announce the release of Ratman v0.3.0 [2], a
_stand-alone userspace packet router_ for Linux, NetBSD, MacOS, and
Windows (the latter three with only very minimal testing however). On
the surface Ratman provides a TCP socket API, allowing
applications to interact with a decentralised Irdest network. In the
backend Ratman connects to a set of networking channels via specific
drivers (called "net modules").
This architecture enables peer-to-peer connections between Ratman
instances on a wide variety of platforms with implicit network roaming
support. Currently the only net module included is a _TCP overlay_
but we are working on many others!
[2]: https://irde.st/downloads
The reasoning
-------------
I have been working on this project in some way or another since 2016
and in all that time we have gone through a lot of phases in the
design of what Irdest (and it's predecessor) should, and could be. In
many ways its identity in the last few years was still deeply rooted
in the initial application written back in 2012: a phone app that
allowed you to connect to people around you.
However as time went on, the more convinced I became that this
approach to building mesh networks did not scale. Not only has mobile
technology moved in a vastly different direction, the use-cases for
these purely ad-hoc networks are so few and far between that focusing
on it exclusively felt like the wrong design decision to make.
There are a lot of exciting prospects to mesh routing, with research
papers of the last decade demonstrating many mechanisms by which mesh
routing can be made more efficient and scalable. Irdest (meaning the
fork) was born out of a desire to build a platform to implement and
test these mechanisms in a real-world setting, while also providing
easy access to these tools to anyone who was interested in them.
With this release we are officially departing from the goal of
creating "an app" and instead focusing on the infrastructure that
other applications can use to build their networking on top of.
Of course Ratman can still run on phones, and we do have a phone app
on the roadmap which will act as a testbed to wireless peer-to-peer
connection mechanisms (among other things), while also integrating
with the Android and iOS VPN interfaces. The shift in focus here
however is the building of infrastructure that other applications
should use, not to create the next messenger or social network that
people will have to install and ignore on their phones.
Our hope is that the technology we create can be re-used by other
projects to improve the state of decentralised mesh routing projects
overall.
What else?
----------
As I have mentioned we have funding from an NLnet grant to do work on
Irdest. While this grant was initially meant for me to work on this
project full time I have since found a day job and would like to
extend the grant to anyone else who wants to contribute to the
project. The current set of milestones will have to be revised (with
this shift in project focus) but I'm hoping to release a set of
milestones in the next month or so and work with NLnet to extend the
scope to anyone else wanting to work on any of them. More details on
this will follow!
Summary
-------
I'm very excited about this step towards a new project identity and I
hope that this announcement has made you at least a bit curious to try
out Ratman in some way or another. The current daemon will very
likely be most interesting to those already running and administrating
networks and there is still a lot of work ahead of us to make this
project more approachable and understandable to users without network
administration backgrounds.
Nonetheless I think this is a big step in the right direction and I'm
excited to see what the future may hold.
If you have questions or comments about this announcement, feel free
to reply to this list, or join our Matrix chat room at #chat:irde.st.
Cheerio!
- Kookie
