rns.recipes

GM! New problem found: rnsd self-deadlock in RNS.log on 1.6.0. This morning my backbone node froze for about 2 hours: rnsd still running and rnstatus answering, but 0 bps on all backbone interfaces, nothing written to the logfile and no new connections accepted. py-spy showed the backbone __job thread stuck inside log() at RNS/init.py line 130 (with logging_lock), called from log() at line 147 (the "Exception occurred while writing log message to log file" call in the except branch), called from BackboneInterface.py line 670 ("Accepting socket failed for incoming connection"). logging_lock is a plain threading.Lock, so when a log write fails the except branch calls log() again while holding the lock, and the thread waits on itself forever. Every other thread that logs then queues behind it, and since it is the epoll thread the whole node goes dead, while RPC keeps working because it does not log. The trigger was file descriptors: with the current flood the node reached about 1000 backbone clients, and rnsd under systemd had the default soft limit of 1024 open files, so accept() failed and then open(logfile) failed too. Suggested fixes: use threading.RLock for logging_lock, or do not call log() inside the except and write to stderr instead; back off in the accept path on EMFILE instead of logging every loop; and mention in the docs that backbone nodes need a higher LimitNOFILE (I now run 65536). Without that, about 1000 connections are enough to freeze any public node.

Mitch wrote:

GM! New problem found: rnsd self-deadlock in RNS.log on 1.6.0. This morning my backbone node froze for about 2 hours: rnsd still running and rnstatus answering, but 0 bps on all backbone interfaces, nothing written to the logfile and no new connections accepted. py-spy showed the backbone __job thread stuck inside log() at RNS/init.py line 130 (with logging_lock), called from log() at line 147 (the "Exception occurred while writing log message to log file" call in the except branch), called from BackboneInterface.py line 670 ("Accepting socket failed for incoming connection"). logging_lock is a plain threading.Lock, so when a log write fails the except branch calls log() again while holding the lock, and the thread waits on itself forever. Every other thread that logs then queues behind it, and since it is the epoll thread the whole node goes dead, while RPC keeps working because it does not log. The trigger was file descriptors: with the current flood the node reached about 1000 backbone clients, and rnsd under systemd had the default soft limit of 1024 open files, so accept() failed and then open(logfile) failed too. Suggested fixes: use threading.RLock for logging_lock, or do not call log() inside the except and write to stderr instead; back off in the accept path on EMFILE instead of logging every loop; and mention in the docs that backbone nodes need a higher LimitNOFILE (I now run 65536). Without that, about 1000 connections are enough to freeze any public node.

Oh yeah, I used to have a patch that changed that to an RLock, I'm not sure what ever happened to it. I guess I moved logging to a dedicated thread instead, where the RLock wasn't needed anymore, and then sent that patch along.

Massive attack going on right now. I have over 800 connections on Bern right now, ~550 of them doing nothing else than dumping crap at full tilt.

Can confirm the same.

Oh, doh! And that patch was apparently forgotten and buried @K8. Well, this is a kind reminder. Now patched.

Mitch, pardon my bluntness, but you do speak quite a lot like an LLM; are you one? Or is it just some kind of work-related inury, or just your style?

And speaking of, there's something seriously bizarre going on here. There's thousands of these connections popping up, practically all coming from rented infrastructure ranges, with a surprisingly high percentage originating from the Azure ASN.

Some connections are dumping whatever they can as if tomorrow depended on it. Some are doing nothing at all, not even sending tunnel synthesis packets. So at least some of these, whatever the fuck they are, looks to be not running RNS, but "something else".

I'm gonna don the festive tin-foil hat and make a weird face while saying this, but it sure does look a bit like some bot swarm took an interest in Reticulum, and got things wrong in more than one way.

How many connections are you ladies and gentlemen seeing on your nodes? Bern is swinging between 800 and 1600. This is, uh... not normal. And it all came on basically within a few hours.

Thanks for chiming in @Nickie! Is it also only happening for you towards nodes that have a discoverable backbone interface?

A few more patches pushed to the repos:

• If a ~/.reticulum/blocked_ips file exists, it will be loaded at startup and add every line as a blocked IP for BackboneInterface instances.
• BackboneInterface now has a client_limit option, defaults to 500.

Mark wrote:

Thanks for chiming in @Nickie! Is it also only happening for you towards nodes that have a discoverable backbone interface?

Seems so regarding discoverable, a couple of local backbones also reported similar activity, currently 1000+ clients and they are pumping full tilt with total bandwidth exceeding 5+ Mbps, continuing to monitor...

Very interesting. If at some point you can grab a rnstatus -a "Client on" | grep "Client on" and send the IP list to me, I'd like to correlate it with the IPs connecting to my nodes. And run in through the analysis scripts I have now. So far it's looking like almost all of it is coming from commercial infrastructure providers, and it's several thousand IPs I have logged now. Something really funky is going on.

Mark wrote:

Very interesting. If at some point you can grab a rnstatus -a "Client on" | grep "Client on" and send the IP list to me, I'd like to correlate it with the IPs connecting to my nodes. And run in through the analysis scripts I have now. So far it's looking like almost all of it is coming from commercial infrastructure providers, and it's several thousand IPs I have logged now. Something really funky is going on.

Given the instability of RNS atm, I've dropped them to:
---SNIPPED---

(legitimate IPs/local backbones/regulars are pruned from the list, but some may still be legitimate clients)

Hi Mark, I'm human, first of all I don't speak English and second thing I'm not super expert in RNS, I'm trying to write you reports on what I find hoping to help solve problems, I manage networks for work and a server that goes down is not my style, I'm famous for 100% uptime wherever I put my hand, so I'm trying patches and various solutions, on my machine, I know Bardolini Reference and I don't let the LLMs read the RNS code but only the logs both on my local machine rnsd and on the backbone server on a VPS for comparison and to understand where the problems are and i use py-spy and some llm made scripts to check where the rnsd locks. This message is handwritten in my language and translated with online translator, the previous ones I had LLM write to make a bug report done well, if it is not welcome I stop, but a server that falls is not good for the whole community, I will continue to look for the solution on my machine to offer a stable service, or I disable it completely, 100% uptime or nothing for me. Also I'm writing from Nomadnet and the text is not formatted, some of my old posts have not even been read, but the problem of 'heavy load' I have already reported previously in v1.5.7. That said, at the moment I receive about 900 / 1100 clients, my usual load was about 600 max. I ran various scripts atm: a script to count the announces and in a few minutes I got 1,463,622!! another script saves me all the announces in a json so I check what arrives. I notice that many announces come from 20 to 127 hops, and is more than 50% of the traffic ATM, some days ago these high hop was about 2%!! many look like test clients made with AI, 12,700 announcements from 'Cardputer zero', many are anonymous peers. many seem to be clients made with AI that generate new identities at each start so you can't even use the blackhole on them. i put a filter that blocks all announces beyond 20 hops and stops them from reaching rns firm verification to buffer the situation. also i see many rnsd-go and other modified rnsd that could have a routing problem and then resend announces with wrong hops or something like that, I'm not sure, it's just an impression from what I can capture and understand from the logs. thanks for your time and i hope we, or you, find a rapid solution, the whole rns network is in crisis from the last 2/3 days!!

Sorry for being a bit rude Mitch! I totally get it, of course using translation is very welcome, and I appreciate all the input you've provided! There's just been so much absolutely crazy and weird stuff going on around here lately I had to ask. Hope you can forgive my bluntness.

But for bug and problem reports, I will say I very much prefer the human-written variant, to me that looks much more "done well", and I will instantly take it a lot more serious. I don't care about typos and the formatting, what I care about is that I can see that a human evaluated and thought about this, so please just write whatever you find in your own words. It makes it much easier for me to filter and evaluate, because I receive around a dozen "bug reports" every week written by LLMs that are not bug reports, but complete fantasy and confabulation.

And by all means, if it helps you debug, let the LLM read the code; just don't use it to generate a new implementation from the code and then claim it is now a "clean-room implementation under AGPL" ;)

i put a filter that blocks all announces beyond 20 hops

Yes, you can also use hop_limiter in the [filter] config section (in case you hadn't seen it, and patched in your own).

many seem to be clients made with AI that generate new identities at each start

Yes, this is very likely to be a significant part of the problem. I am also seeing all kinds of weird announces with the same generic names, under different new identities. And additionally, it seems that a lot of LLM-generated crapware is pretending to be "transport nodes", but "forgot" to implement all the crucial filtering, de-duplication, rate controls, et cetera, et cetera, that actually makes the networks usable and stable.

"Let's make a super fast RNS clone - by just slicing out all the hard stuff, and leaving actual RNS nodes to clean up the mess we're spewing on the networks" - The sloporator's motto

no problem Mark!! my purpose is to have a healty network and i know what you mean, peoples report bugs when theyr mouse dont work and tell that the problem is in your software, i know, and with llms advent, especially the allucinated 0.1B local , the problem and annoiance is real and worse. now im writing also without translator, lets see what happens! :D there is also a strange phenomenon on nomadnet: as you launch a new node, you get immediately 100s of opened links without requesting pages. probably crawlers. someone has a good blackhole list? im using the rmap one but is very small ATM. i analized the milion of announces captured before and this is the report, sorry but i need again the translator here, for the llm report on the capture: Capture analysis: Volume: 1,463,622 announces in 12 minutes, or about 120,000 per minute, with a peak of 217,000 per minute at 10:38 a.m. - Almost all are copies: there are only 13,586 different destinations, with 10,316 identities. On average, each destination arrives 12 times, the 10% more repeated over 329 times, the maximum is 3,914 times from 627 different sources. The reason: out of 1,517 interfaces (984 IPs), 609 send us more than 500 different destinations each. So they don't announce their own stuff: they are clients with active transport that send us back everything they receive. The load is this amplification, not many new nodes. Hops: 30% arrive with 20 hops or more, and the hop_limiter discard them. Who weighs more: "Anonymous Peer" and "Cardputer Zero" (same name_hash 6ec60bc3) alone make 519,000 ads, more than a third of the total. - Stall: at 10:41 only 12,000 announces arrive and at 10:42 zero. It's a rnsd stall there too.

running another capture now, that starts after 30 min from rnsd reboot, in the first minutes there is also the reconnect flood from all the clients reconnecting, but the problem i think is like you said, Mark: some shitty rnsd porting in afrodisiac llm modern languages that don't have all the original rns implementations! i'm seeing exactly a sort of amplified announce ddos, and any announce requires the sign verification that stalls the cpu for long time on small server machines, here the dead ec 12s or my monkey patched 60sec intervention closes everyone out. i noticed a max cpu stall of 4 minutes at 100%!

┌ Mark wrote:
│ How many connections are you ladies and gentlemen seeing on your nodes? Bern is swinging
│ between 800 and 1600. This is, uh... not normal. And it all came on basically within a
│ few hours.

For the last couple of days, I'm monitoring my transport node in more detail. And I have to say, I do not see abnormal behaviour. I have two public backbones (IPv4 and IPv6) with each between 5 and 25 clients connected to it at any one time. Used bandwidth is between 50-200 kbps with peaks up to 500 kbps. It's hard to say for sure, but the used bandwidth appears to have doubled since 9 November or there about. I'm using vanilla rns 1.6.0 and a default configuration. Maybe I'm lucky?

Mees electronics wrote:

┌ Mark wrote:
│ How many connections are you ladies and gentlemen seeing on your nodes? Bern is swinging
│ between 800 and 1600. This is, uh... not normal. And it all came on basically within a
│ few hours.

For the last couple of days, I'm monitoring my transport node in more detail. And I have to say, I do not see abnormal behaviour. I have two public backbones (IPv4 and IPv6) with each between 5 and 25 clients connected to it at any one time. Used bandwidth is between 50-200 kbps with peaks up to 500 kbps. It's hard to say for sure, but the used bandwidth appears to have doubled since 9 November or there about. I'm using vanilla rns 1.6.0 and a default configuration. Maybe I'm lucky?

You most definately are, because other local nodes have been hit a day or two before mine got hit.

In the meantime, I've firewalled the following ASNs on the TCP backbone until this idiocy fizzles out...

• AS8075: Microsoft (Azure)
• AS14618: Amazon (AMAZON-AES)
• AS16509: Amazon (AMAZON-02)
• AS14061: DigitalOcean
• AS51167: Contabo GmbH
• AS40021: Contabo Inc.
• AS24940: Hetzner
• AS16276: OVH
• AS31898: Oracle Cloud
• AS8560: IONOS

Might be a good idea for RNS users to investigate alternative interfaces like I2P and Ygg if they haven't already.

hi Nickie, look that in your list of ips, you are including some legitimate clients and maybe backbones, i'm on ionos for instance, I shared the list to some real users and they saw their ip in there, the capture filters used are not good, if everyone uses this list with the new rns ipblock in the repo you are cutting out legitimate users and the spammers are still going on!! and have you also considered cgnat connections? a generic block like this should never be used in a real network!! meanwhile my backbone has been up for over 2 hours without problems, the filter above 20 hops and my patches made yesterday was enough, for now it holds and seems to be solid!! it seems to me that the real problem is not in the single clients, it's in the weird rns porting that amplificates announces, and rns that has to verify the ed signature for each of them is heavily fatigued!!!

Last night I was seeing around 600 clients connecting to the instance within a few minutes of a restart. Conntrack was showing around 10000 entries of mostly SYN_SENT connections. So it seems like there’s a mix of strategies going on, combining TCP shenanigans like you had mentioned to me with all the other bullshit. I tightened up the kernel’s reverse path filtering in case the IPs were being spoofed but that’s hardly foolproof, and it didn’t help much. I wish I could afford the amount of compute they’re paying for…

Nickie wrote:

In the meantime, I've firewalled the following ASNs on the TCP backbone until this idiocy fizzles out...

• AS8075: Microsoft (Azure)
• AS14618: Amazon (AMAZON-AES)
• AS16509: Amazon (AMAZON-02)
• AS14061: DigitalOcean
• AS51167: Contabo GmbH
• AS40021: Contabo Inc.
• AS24940: Hetzner
• AS16276: OVH
• AS31898: Oracle Cloud
• AS8560: IONOS

Might be a good idea for RNS users to investigate alternative interfaces like I2P and Ygg if they haven't already.

I did the same with OVH and AWS, to start with.

For what it's worth, I noticed the "Cardputer Zero" spam yesterday and tracked it down to this:

https://github.com/vicliu624/trail-mate/issues/112

Thanks for noting that Torlando and telling him (or ChatGPT or whoever is actually running the show) about it!

K8 and Nickie, I'm tracking 2000+ distinct IPs across those ASNs as well, most of it originating from Azure and AWS, but also Contabo and DO. Also seeing quite a bit of TCP weirdness and SYN flooding.

Yeah, this is starting to look umm, kinda weird. At this point I'd say it's more or less out of the "one grumpy guy trying to be annoying" zone, and into something that's actually costing quite a bit of money to maintain.

Reminds me a bit of the DN42 situation about half a year back.

This is seriously strange and intriguing. I mean, there's is a non-zero chance that two thousand people individually decided to set up transport nodes on hosted VPSes/containers, misconfigured them in more or less the same way, all within a few hours of each other. But err... yeah...