rns.recipes

Thanks @p1ld7a! First of all, I just pushed a new set of changes to Aleph. If you already updated, can you please try running the very latest commit? The "Over-optimization is the root of all evil" one?

There's unfortunately not much that stands out immediately in that log file. For the next run, can you try increasing the loglevel to debug instead? That should give a much better picture.

One thing I do notice, though, is all those TCPInterfaces. Is there a specific reason you're using those, and not BackboneInterface? Can you try changing your TCP interfaces to Backbone in your config? That would help rule out whether it is anything going wonky in the TCPInterface code, as opposed to BackboneInterface (they're mutually compatible, but BackboneInterface is much faster and much more efficient in all regards).

Also @K8, there was a TX stall bug in one of the previous commits, that I think you might be running now. Race in EPOLLOUT due to my silly modify guard experiment. Sorry for that one ;) It should be fixed now in the latest commit.

LOL, "4 cores and they're all sad" @K8. Don't know why, but I find that so funny :)

Actually, I have a pet theory about why performance is often so much worse on some systems, especially VPSes, and it has to do with memory bandwidth, so I ran some probing. In a synthetically engineered, completely worst-case scenario, I'm seeing some pretty insane memory bandwidth consumption. Now, this is admittedly completely synthetic, and designed to expose the absolute worst-case scenarios, which are unlikely to happen in real usage, but still, it serves as an insight into what to target.

The short version of it is that I think I can reduce 331 gigabytes (for that test scenario) of memcopies down to zero, which is bound to have some effect.

I got completely sidetracked by this today, and decided to dive into it, so I'll hopefully have something ready later. Or much later, depending on how it goes. I'll be back.

I have updated the config of the server: https://github.com/drupol/infra/commit/079f68b244b95d400bdd01603f1078e791217d9f

I will now try to fetch the sources from reticulum and keep you posted.

@Mark: would it be OK for you to sync the GH mirror ? Fetching with reticulum is currently not possible on my end.

If over-optimization is the root of all evil, why is it so much fun? 😈

That’s interesting about memory bandwidth, I’m sure there lots of room for improvement there. I was just starting to do some memory profiling to understand why my instance’s memory usage seems to keep growing over time.

I have made the changes, but 10 minutes later, it was down: https://gist.github.com/drupol/65cd41db93bfedf38954a6225793ef72

loglevel is at 7, I am surprised I don't have more logs to show.

You can find the complete rnsd config at: https://github.com/drupol/infra/blob/079f68b244b95d400bdd01603f1078e791217d9f/modules/reticulum/reticulum-server.nix#L199

p1ld7a wrote:

I have made the changes, but 10 minutes later, it was down: https://gist.github.com/drupol/65cd41db93bfedf38954a6225793ef72

loglevel is at 7, I am surprised I don't have more logs to show.

You can find the complete rnsd config at: https://github.com/drupol/infra/blob/079f68b244b95d400bdd01603f1078e791217d9f/modules/reticulum/reticulum-server.nix#L199

You have panic_on_interface_error enabled, so based on the logs it seems to be behaving as it should. Just disable that!

If you ever want more logs (which most of the time you probably don’t) there’s also loglevel 8 now.

Updated my config, will keep you posted ! https://github.com/drupol/infra/commit/5a0e9b56dc75f292913aa8e14b0fbae8c36f0828

Loglevel 8 is a dreaded and dangerous land. A stream of information so unhinged and profound that few mortals dare consume it. Heed my warning; if you go, you may never return.

Back in the day before ingress limiting, I hear that skilled node operators could spot a spammer in the log stream from 30 paces.

Days of yore, and log-stream gore.

Okay, it wasn't that long, but a bit more involved than I had expected. The latest commit on aleph now uses a zero-copy, per-interface coalescing transmit buffer on BackboneInterface. I've also optimized memcopies almost entirely away on HDLC deframing. And added full, memory-bounding dataplane egress control. Parameters will most probably need tweaking; it's the wild west at HEAD right now, yall. Does it all work? I think so. Will there be bugs? If not, I deserve a beer or something.

The intention of all this memcpy schything is to significantly reduce memory bandwidth requirements for all the core ingress/egress and buffer operations. There's still low-hanging fruit left, but this should take care of most of it.

Whether it provides a decent performance improvement on slower hardware remains to be seen; I haven't tested it yet, caus I need to sleep. But on my laptop at least, it pushes resource transfers from ~670 Mbps to just shy of 1 Gbps, so something it did.

Also, the dataplane egress control serves, among other things, to set hard memory bounds on TX buffer mem consumption (needed the coalescing buffer done first before I could add that), which could be the cause of your RSS growth @K8. I want to yabber on about it all, but I am falling asleep in my chair.... gnight.

Bugs or no bugs, you definitely deserve a beer! Thanks for all the performance improvements packed into this humble patch release (1.5.1 is out). And thanks to K8 for the live profiling, you deserve a beer too :)

Seriously, we're lucky to get all of this for free. I'm so grateful for all the hard work you've put into improving Reticulum. For those who haven't donated to Mark yet, you know what you should do!

Thanks so much Joakim :)

Definitely big thanks for all the work that went into this release. Even better than thanks, though, here's more data! 🎁 https://pastebin.com/p0pdwz65

There were 235 clients when I pulled those numbers. Things are looking better across the board, though there are still some spikes. The CPU usage on the node is MUCH improved, though, and it's been running more smoothly and more quickly than it has in a long time. I'm also dropping a lot fewer packets in the queues. I'll keep an eye on things and see how it looks after a week.

The protocol violation stats are interesting. I only have two interfaces with any (both are just running plain rnsd as far as I know), plus some more from other clients in the logs. It looks like they are all coming from malformed packets with too-high hop counts. Most of them are just 128 hops, and a handful have much much higher counts.