rns.recipes

hi all,

maker of STRUDEL and operator of rns.reticulum.at here. i just went through the whole thread.

the „public pages are public” line on my about page was a bit smug in hindsight. i still believe there should be a way to discover pages without clicking through every announce manually (as that was my motivation to build the search in the first place) but i can see how the framing comes across. as for scraping policies: as already mentioned earlier i set up very relaxed scrape policies with around once per day, cooloff and only selective drilldown. i don‘t mean to burden the network or annoy anyone with my little tool.
as for the opt-out: nomadnet pages are essentially clearnet so an opt-out will always be a goodwill option and the next guy might again ignore it. that said, i get the argument about bandwith. what do you guys think of a bandwith-gray-out option? scraper only comes back weekly or monthly?
if that's still a no i will implement an opt-out but i hope you can see that that's a futile approach to the overall problem.

rudi: you’re right about the german on reticulum.at, i got the dns and wanted to put up a quick page not thinking it would draw traffic so quickly. on a side note: i set it up primarily to give local enthusiasts a lead in how to meet other interested people and that goal was fully achieved. we are a small but growing little group that start experimenting with reticulum.

and regarding the „one man show”, that’s accurate regarding the two sites. but we are already a small group in vienna, lower austria and especially some super cracked folks in innsbruck.
i experiment and play around with a technology i find fascinating and so are all the others. i don’t claim to speak for anyone else building in austria but as a network is only valuable with nodes, that is what i am trying to generate more of.

happy for feedback and comments and a big thank you to mark and everybody else who has been building this incredible technology and community. i hope we find ways to make reticulum a success without gate-keeping.

lxmf welcome if you’d rather write directly.

all the best,
axhoff

disclaimer: STRUDEL was called „🔵🔴🟡🟢 ROOGLE Search“ for about two weeks until the owner of roogle.us contacted me and stated he‘s been active since april so i renamed. oversight on my part. i have never been associated with roogle.us or any of its aggressive scraping behavior.

Anonymous wrote:

hey mark how about you just get better at designing to be resilient to bad actors, or admit that one guy can’t do everything and get better at working with others/running this thing like a real grown-up software project instead of a megalomaniacal fever dream

makes a network for everyone
gets mad when everyone starts using it

This is not a resiliency issue, dumbass. It's about the unfair usage of people's resources they have put into the network. You have limits on LoRa usage, electricity and ISP costs, hardware maintenance, and so on.

Rudi Mentaire wrote:

What I want to say is: now that reticulum gets more known, and a lot
of new people are attracted (which is not a bad thing) - it could have
a good impact to guide them more.
…
An example: having a guide like "this is reticulum/nomadnet, your
first steps, what you can expect here and what not" would be very
helpful and could also prevent undesired developments. At least, this
is something I missed, sometimes I felt a bit lost. People must know
how they can do better, else they will fall back to what they used to
do in "the Internet".
…
So I think this is my main point (apart from avoiding negative bias
because dumb people are causing so much harm): a welcome page that
gets people started, that is updated regularly, and guides people that
don't want to cause harm but just lack knowledge at the moment.

Does that make sense? I could offer my help - although I'm a newcomer
(and I also don't have too much energy left all the time). Mark
should probably not get an additional task...

Thanks for the idea, Rudi, I completely agree! I've started on a draft for a wiki page:
https://reticulum.miraheze.org/wiki/Draft:What_is_Reticulum
72a2d9c12215802605779f4366d48335:/page/wiki.mu`path=Draft:What_is_Reticulum

If you or anyone else* have any thoughts or additions/edits, feel free to comment on the Talk page. I'd be more than happy to collaborate with others in writing this up, the more perspectives the better. I guess it should go on to describe NomadNet and what to expect there.

*) Especially Mark, if you see something that's not right.

Well most of you probably won't care. But I started talking to roogle on ways to be way less intrusive and malicious. His scrap times were supposedly every four hours... And he recently said he has stopped scraping to work on limiting how much it scrapped. Take it with a grain of salt if you want. That's just what he told me recently.

Anon wrote:

Well most of you probably won't care. But I started talking to roogle on ways to be way less intrusive and malicious. His scrap times were supposedly every four hours... And he recently said he has stopped scraping to work on limiting how much it scrapped. Take it with a grain of salt if you want. That's just what he told me recently.

its worth taking a look at Beacon's code it is a search engine that already implements everything Mark mentions. The code is also FOSS.

SevenFourTwo wrote:

bergie wrote:

Mark wrote:

• Create an easy way to permanently signal to your scraper that a node never wants to be contacted again. How much of a no-brainer is this?

Hmm, "standardized" path for serving a robots.txt over a Reticulum request?

I think the 20 page limit is way too low. Once we bring our boat's log over to NomadNet, it'll be hundreds of pages that would be totally fine to index. But at the same time indexing every rngit commit page makes no sense.

With robots.txt (or something similar) the party running the serving node could decide what's ok and what's not.

We'd likely need an automated enforcement mechanism of some kind although it would be hard to make one that would work well without complicating the nomad page protocol or reducing anonymity. Stamps could be used but they could waste power.

If (big if!) I understand blackhole correctly, I could see a policy to automatically blackhole a requestor on a target node; because the originator isn't visible in the request packet(s), this would still incur some overhead, but only the existing blackhole processing. A possible implementation outside the reticulum code proper might be a Python script tailing a connection log and identifying a destination that exceeds a request rate limit (too many, too quickly or too many in a given period), and invokes rnpath to blacklist that requestor for some period (e.g., 24h).

Actually, as I was reviewing the above and checking the manual, I realized that Mark appears to have created a mechanism for this already: Blackhole lists (https://reticulum.network/manual/using.html#publishing-blackhole-lists). Maybe a trusted blackhole list source could be made part of the recommended initial configs for a Transport node?

hey mark how about you just get better at designing to be resilient
to bad actors, or admit that one guy [...]

Is this meant as a mere provocation, or as an argument? It works very
well as a provocation, but it is a weak argument. I don't see
Reticulum as purely one-person. I never heard Mark saying "please
don't support or help me". It is also expected and normal that new
kinds of problems occur when software gets more mature. And they are
addressed. More or less right in the same moment they occurred.

A tit-for-tat response that I only mention because it would be
impolite to use in direct speech is: while Mark works very hard and
was getting mad, you did nothing and are getting mad as well.

I know this well from free software development: most of the time
there are only very few people doing all the work, sometimes even
alone. And then there is always a huge group of people who think it
would be helpful to criticize, question decisions, question processes,
etc. But that's not helpful. Doing a part of the work, or offer
financial support, is help. More or less, only that. Developers
never have a lack of ideas or things to fix.

Finally, you may have some point if Mark's chosen path would not have
been successful. By any measure, the opposite is true. Mark probably
knows best how he can achieve good results most efficiently - a larger
organizational construct comes with other problems and new risks
(like, the thing getting out of control). Some people just prefer to
work alone. Etc. etc. etc... (this was already too much text for an
reply to this, I'm sorry ... since I don't think that this remark was
even meant as something thoughtful and helpful, it would be better to
ignore remarks like that. Yep, I failed.).

@axhoff Thank you for coming here and explaining. Well, maybe you might
consider to support Mark with a donation if you can and if you not
already did. Helps a lot.

Anonymous wrote:

Anon wrote:

Well most of you probably won't care. But I started talking to roogle on ways to be way less intrusive and malicious. His scrap times were supposedly every four hours... And he recently said he has stopped scraping to work on limiting how much it scrapped. Take it with a grain of salt if you want. That's just what he told me recently.

its worth taking a look at Beacon's code it is a search engine that already implements everything Mark mentions. The code is also FOSS.

link?

On the topic of sloperations, I see that we've now gotten 2 posts in off-topic asking about home renovation/moving in a format typically seen in undisclosed advertisements.

Third Force wrote:

Anonymous wrote:

Anon wrote:

Well most of you probably won't care. But I started talking to roogle on ways to be way less intrusive and malicious. His scrap times were supposedly every four hours... And he recently said he has stopped scraping to work on limiting how much it scrapped. Take it with a grain of salt if you want. That's just what he told me recently.

its worth taking a look at Beacon's code it is a search engine that already implements everything Mark mentions. The code is also FOSS.

link?

https://github.com/wdunn001/beacon

I am wary of posting such links. One recent addition is the use of a Voronoi Diagram and the reuse of my portable extra from rns-resolver to beacon. crawlers now share data if using beacon (rate limited) and only crawl their own seed decided by locating neighbors and their hop count. This creates a really cool effect the more replications across the mesh the less traffic you see not more and saturated points in the mesh act as just replicated backups instead. I am working on the other direction redirecting clients to their "local expert" that ranks results by hop count. It has some real open questions because of interfaces. the system as designed doesn't know your setup and what interfaces you have so I can use the local experts data on the matter but interfaces complicate that. I am still braining that part out. any way the first half is still plenty useful.

As a counter-measure,maybe a sliding window counter would be feasible for link count. Would prevent lots of slop it seems. You could store two hashmaps with identity as the key and increment for each link. After X minutes alternate between hashmaps(discarding previous data from the hashmap you're switching to). Would not use much resources but possibly give the option to configure per identity link limits.

I don't imagine there's a robots.txt equivalent for NomadNet? Not that it'd do much good as I'm sure these slop scrapers would just ignore it.

I wonder if the reason this bozo isn't re-using links is to try to keep themselves from being blocked.

jlamothe wrote:

I don't imagine there's a robots.txt equivalent for NomadNet? Not that it'd do much good as I'm sure these slop scrapers would just ignore it.

I imagine the intention was to not repeat the mistakes that the internet made, but this requires developers to have forethought and care, something LLMs are incapable of

@A. Non

Yeah, that's more or less what I figured.

I really need to do some more digging into the nuts and bolts of how the protocol works, but as I understand it the whole initiator anonymity thing makes defending against this sort of thing difficult.

It's definitely a problem that needs solving. I just don't know at this point if there is a good way to do it without a fundamental change to the protocol.

There is a good way of solving it, and I have been working on it for a long time. But details matter, so I'm taking as long as is needed thinking it over and prototyping different implementation variants. It's been in the works since stamps were introduced in LXMF (which was the first "testbed phase" for the in-RNS functionality for handling stuff like this).

Mark wrote:

There is a good way of solving it, and I have been working on it for a long time. But details matter, so I'm taking as long as is needed thinking it over and prototyping different implementation variants. It's been in the works since stamps were introduced in LXMF (which was the first "testbed phase" for the in-RNS functionality for handling stuff like this).

Just curious, (this is only a question) after reading this at
https://github.com/wdunn001/beacon

'Crawl policy
Crawling a mesh is not crawling the web. Pages can sit behind LoRa links where the handshake costs more than the page it carries, and the person paying for that airtime did not ask to be indexed. Every limit below is enforced in code rather than by convention, and the environment variable is given so you can check what any given deployment is actually running.

Limit Default Variable
Pages held per node 20 BEACON_MAX_PAGES_PER_NODE
Link depth from a node's index page 2 BEACON_MAX_DEPTH
Delay between page fetches 5s BEACON_FETCH_DELAY
Re-crawl interval, per url 24h BEACON_RECRAWL_HOURS
Pages fetched over one Link up to 8 BEACON_NODE_BATCH
Worker threads 1 BEACON_CRAWL_WORKERS
'

for sites serving pages:
Could they render a page address hash that indicates when changed
that the page content has changed,
so that if page content has not changed
the content should not be scraped ?
Multiple (5, or several ?) queries from a Node to re-read alraady
stable pages could trigger a 24hour (or more!) block ban
and a banning announcement to alert everyone else
that there is a source begging to be banned.

( This is a question, not a suggestion.
The fundamental problem presents itself as a shield-vs-sword struggle.
Perhaps I misunderstood what can and cannot be done.
Any legitimate search engine would operate with a single identifier.
Any "search engine" trying to hide behind mutating identifiers
has obviously malevolent intentions and could be denounced as such ?)

Cheers to all who are trying to help here.

@joakim Thank you - I will look at your draft ASAP. I have been quite busy over the last two days and wanted to use a calmer moment for this. Not that you think I ignored or missed your post...

Correct me if I'm wrong, but from what I understand, couldn't the standard RNS routing table basically be used for search, locally? If not, I think it would be a good idea if nodes could publicize their public endpoints in their announces. Think of it like inverting the search paradigm from pulling (via scrapes), to pushing (via announces). Then it would be a small amount of additional load within the existing announces, instead of tons of scraping. Additionally, every node can choose which endpoints to publish, which seems like a win. This way, the routing table can additionally store public endpoints, and that can be locally searched with no additional network traffic. To assist with content lookup, perhaps announces could also be supplemented with content blurbs, such that a local search process could match queries to blurbs.

holobrine wrote:

Correct me if I'm wrong, but from what I understand, couldn't the standard RNS routing table basically be used for search, locally? If not, I think it would be a good idea if nodes could publicize their public endpoints in their announces. Think of it like inverting the search paradigm from pulling (via scrapes), to pushing (via announces). Then it would be a small amount of additional load within the existing announces, instead of tons of scraping. Additionally, every node can choose which endpoints to publish, which seems like a win. This way, the routing table can additionally store public endpoints, and that can be locally searched with no additional network traffic. Granted, this would only allow destination lookups rather than content lookups, but hey, it's a start of a solution at least.

See "announces" tab for Nomad Network in MeshChatX, or other similar clients.