rns.recipes

Also, want to say that Mitch is right that permanent, full ASN range blocks are not an ideal or permanent solution to anything, but for node operators wanting their nodes to actually be available to normal users and home connections, it might be a meaningful stop-gap right now, caus things are getting absolutely hammered.

I'm working on solutions. Please do post any information or clues you find here. I'd really, really like to know what the hell is actually going on here.

Also wondering if someone/something "upgraded" some kind of widely deployed service or system that now integrates some kind of Reticulum functionality, and that this is actively using discovered interfaces to connect in large numbers, with some sort of weirdly broken transport mode enabled.

Also, I forgot to say @Nickie, when I checked your connecting IPs earlier, there was a 72% overlap between Bern and RNS Moscow for the rental ranges.

Mitch wrote:

hi Nickie, look that in your list of ips, you are including some legitimate clients and maybe backbones, i'm on ionos for instance, I shared the list to some real users and they saw their ip in there, the capture filters used are not good, if everyone uses this list with the new rns ipblock in the repo you are cutting out legitimate users and the spammers are still going on!! and have you also considered cgnat connections? a generic block like this should never be used in a real network!! meanwhile my backbone has been up for over 2 hours without problems, the filter above 20 hops and my patches made yesterday was enough, for now it holds and seems to be solid!! it seems to me that the real problem is not in the single clients, it's in the weird rns porting that amplificates announces, and rns that has to verify the ed signature for each of them is heavily fatigued!!!

I've removed the links to the files, they were for quick reference only. This is stop-gap measure at best, I know there are legitimate addresses being blocked. Sorry about that. I've removed all the legitimate discovery peers I could find/parse from my firewall rules.

hi guys! yes, i know everyone is tring to to their best here, i didnt'want to seem rude, sorry if i was so!! and sorry, im writing without translatos again! from my last analysis on the captured milions of annunces i saw that there are some strange aspects on the net, there is one in particular that has his name in the aspect instead of an app or protocol name, and this alone was sending 2% of the whole traffic! with the latest tweaks and hop filters set to 18 my backbone is up and stable, i added filters for the unofficial apps and protocol that i'll share at the end of this message. there is also an app that is really aggressive, all the strange things are included in the ignore filters, maybe you can investigate more on these things, for me the filter works great, now 80%/90% of traffic seems to be legit, the rest are weird announces without any aspect, how is it possible? goodnight guys! [filter] hop_limiter = 18 ignored_types = loquax.home, genie.node, agenda.sync, bench.echo, xprs.chat, xprs.files, xprs.relay, retifs.node, node.mesh, wardrive.ping, wardrive.probe, qortal.core, rfed.node, rfed.delivery, meshage.profile, netptt.hub, rfed.channel, retifs.library, stiqrcloud.node, rfed.link, wisco.network, rnfs.name, bloom.v1, wardrive.beacon, openbridge.files, bridgeos.node, bastion.site, genie.storage

another thing i noticed, many many nodes that 'duplicates' announce don't have the version announced, and the number of nodes like this is huge. they coulb be outdated rns versions or weird llm implementations...this is the base of xprs that is already in my filter lists above, advertised also to share torrents over rns or similar, and is based on rns 1.3.5 so no version announced here! made in dart. consider it just as a hint, i'm really exausted and going to sleep to look at the code RN and what this thing does on the net. updated yesterday but still on old rns!! link: https://github.com/xprs-dev/reticulum-dart/blob/main/doc/reticulum.md and: https://github.com/xprs-dev/server

Nice finds Mitch! Thanks for sharing that list. Adding it does cut down quite a bit on the announce traffic. Once you've had some sleep, and when you have the time, how did you compile it? Did you trawl for weird github repos, or did you set up something automated to check announce frequencies?

Also very interested to hear what you're seeing with those announces without aspects. That shouldn't really be possible, so if you can send me some more information exactly on what you mean, that would be very interesting.

Goodnight and sleep well! :)

I added filtered traffic stats to rnstatus output:

 BackboneInterface[Bern IPv4/0.0.0.0:7822]
    Status    : Up
    Clients   : 446
    Blocked   : 22 IPs
    Mode      : Gateway
    Rate      : 250.00 Mbps, MTU 65536
    Announces : 66.7 Hz ↑   0.1 Hz/c (↓99% / ↑81% of flow)
                312.9 Hz↓  (t:6h/p:0s/g:1) burst on 211 for 3m and 52.25s
    Traffic   : ↑70.94 MB   2.57 Mbps
                ↓32.12 MB   1.78 Mbps
    Filtered  : ↑0 B  0 bps
                ↓37.03 MB  1.18 Mbps

 Totals       : ↑71.68 MB  2.61 Mbps, 3% data (88.95 Kbps), 1.98 Kpps
                ↓32.24 MB  1.78 Mbps, 0% data (0 bps), 1.11 Kpps

 Filtered     : ↑0 B       0 bps
                ↓37.76 MB  1.21 Mbps

 Announces    : ↑58.81 MB  2.12 Mbps, 81% of flow, 149.06 Hz
                ↓38.90 MB  1.78 Mbps, 100% of flow, 447.23 Hz

 Qu. Pressure : 0.0% total, 0 pkts, 801 dropped
                0.0% data, 0 pkts
                0.0% announce, 0 pkts, 748 dropped
                0.0% path request, 0 pkts
                0.0% ingress limiter, 0 pkts, 53 dropped

 Transport Instance <521c87a83afb8f29e4455e77930b973b> running
 Uptime is 3m and 52.14s, 333 entries in link table (332 active)

Lots of other small tweaks and improvements to help with situations like this, including:

• Added ability to load a blocked_ips file from the current config dir
• Added client_limit option to BackboneInterface
• Filter destinations at path table load time
• Drop announces early when held announces is full
• Made max stamp workers and worker niceness configurable
• Use RLock in logging, and reset DPEC on reconnect (thanks K8 and Mitch!)
• Improved path pool and tunnel culling

It's in the source repo if anyone wants to try it out.

Any of you ladies and gentlemen who are still getting mass-connect-spammed? Most of them seem to have evaporated from Bern for now, but I hope that's not just because they moved to someone else.

The wardrive.* aspects are probably the LoRa coverage mapping app here

circfruit wrote:

The wardrive.* aspects are probably the LoRa coverage mapping app here

Yes true, there are also the same wardrive responders in Sofia. They announce from time to time, but rarely.

Ah, that makes sense. If they're used for local wardriving / coverage mapping, it should be fine to filter them on wider interconnects, or am I missing part of the functionality? As in, are they only useful for communicating with locally, when doing RF mapping, for example, or do you still need them reachable from completely distinct regions over internet backbone nodes?

Mark wrote:

Ah, that makes sense. If they're used for local wardriving / coverage mapping, it should be fine to filter them on wider interconnects, or am I missing part of the functionality? As in, are they only useful for communicating with locally, when doing RF mapping, for example, or do you still need them reachable from completely distinct regions over internet backbone nodes?

Those were announces for the wardrive application to find the bot. I'm actually thinking about retiring the "bot" altogether and I've made wardriving fully local with a built-in responder on the RNode and a local signal metrics storage, so it shouldn't generate any wider RNS traffic at all (the downside being requiring my fork of RNode_Firmware) I believe the default announce interval was also pretty tame.

It's fine to filter them out for the wide network.

Hi Mark, good morning and thanks! You are right, "without aspect" was my wrong wording, sorry. Every announce has a name_hash. What I meant: name_hashes we cannot turn back into a name. The wire carries only sha256("app.aspect")[:10], so we get the names by hashing candidates (core names from the RNS/LXMF/NomadNet code, a word list combined as app.aspect, words found in app_data) and checking if they match. About 17% of the announces we received have a name_hash we could not match. Most of them look normal (low hops). Only a few look like junk, for example e820b4049768af64c795: 3.5% of traffic, median 55 hops, one new identity for every destination, so a blackhole per identity is useless. The list was made by a script: a passive tap on Transport.inbound (logs every announce before any filter), then grouping by name_hash with share, identities, sources and hops, and name recovery by hashing. Then I searched GitHub for the names found: the apps exist, some are also on F-Droid. I filtered everything that is not an aspect of the official rns apps. On my backbone I use only the official protocols, in Python, and I don't want to relay third-party apps until i test them personally, so no offense for anyone if i cut out your apps!! My biggest suspects from the traffic were xprs (about 2% of all announces, 22-31% at 20+ hops) and Cardputer Zero, and rnfs is one to watch. Cardputer Zero uses the normal lxmf.delivery and lxst.telephony aspects, so only hop_limiter stops it, not the aspect filter. ...

... Two ideas from the data: 1. ignored_types needs the name, so an unknown name_hash cannot be blocked from config. Could it also take a raw hex name_hash? 2. After hop_limiter there are still 1300-3300 copies per destination in 3 hours, so more than 95% are copies. packet_filter lets SINGLE announces through even when the packet_hash is already known (I understand why, for alternative paths), but then validate_announce runs again on every copy. In my py-spy dumps this was the top CPU cost during the stalls. A small cache of announces already verified could skip the signature check on copies, and routing stays the same. The full report and the two scripts (tap and report) are at the link below, open link and download in the file section. The report script prints source IPs, so I did not post its raw output. To be clear: the scripts, the report and this message were made with an LLM, because I needed to put everything together quickly. The data is real, from my captures. Report, anntap.py, anntap_report.py: https://cloudpaste.io/#/dbzang01

For me the chaos already went away yesterday with hop_limiter 20. Then I went down to 18 and added the ignore list, plus the RLock/DPEC patches. Only one disconnection during the night, and now the backbone has been stable for 10 hours in a row. One more check from my PC (rnpath -t, hops per interface): the links through rmap.world:4242 and :4343, two of the backbones I connect to, look clean (no paths beyond 20 hops). Another public backbone I connect to (aspark.uber.space:44860) brings 252 paths, 52% of them beyond 30 hops and up to 127. I will remove that one from my interfaces. About today being calmer: for me it calmed down after the filters, but I did not see a big drop in the traffic itself. From my PC: 166 new destinations per minute today vs 192 yesterday, and 102 on a normal day (Nov 8). So it is a bit calmer, not back to normal yet. Maybe some of the LLM coded transport nodes and unhealthy app devs read this thread and realized what they was doing :D

Mitch wrote:

[...]

iirc Microsoft was also down yesterday

Oh, not only yesterday. Today all day, Azure, Outlook, OneDrive, Teams were all fucked up. Just around 15:30 CET just Outlook came back, Teams and OD were still dysfunctional.