rns.recipes

What if the real solution is breaking up the world nomadnet into disjoint, smaller, flatter, more "gated" networks instead of everything is 1 big WWW-connected "test-net" with everyone on it. also a solution to the "welcome to my test page" spam which makes "search" "useful" in the first place. maybe it's the same solution.

Eric-the-Read wrote:

Mark wrote:

There is a good way of solving it, and I have been working on it for a long time. But details matter, so I'm taking as long as is needed thinking it over and prototyping different implementation variants. It's been in the works since stamps were introduced in LXMF (which was the first "testbed phase" for the in-RNS functionality for handling stuff like this).

Just curious, (this is only a question) after reading this at
https://github.com/wdunn001/beacon

'Crawl policy
Crawling a mesh is not crawling the web. Pages can sit behind LoRa links where the handshake costs more than the page it carries, and the person paying for that airtime did not ask to be indexed. Every limit below is enforced in code rather than by convention, and the environment variable is given so you can check what any given deployment is actually running.

Limit Default Variable
Pages held per node 20 BEACON_MAX_PAGES_PER_NODE
Link depth from a node's index page 2 BEACON_MAX_DEPTH
Delay between page fetches 5s BEACON_FETCH_DELAY
Re-crawl interval, per url 24h BEACON_RECRAWL_HOURS
Pages fetched over one Link up to 8 BEACON_NODE_BATCH
Worker threads 1 BEACON_CRAWL_WORKERS
'

for sites serving pages:
Could they render a page address hash that indicates when changed
that the page content has changed,
so that if page content has not changed
the content should not be scraped ?
Multiple (5, or several ?) queries from a Node to re-read alraady
stable pages could trigger a 24hour (or more!) block ban
and a banning announcement to alert everyone else
that there is a source begging to be banned.

( This is a question, not a suggestion.
The fundamental problem presents itself as a shield-vs-sword struggle.
Perhaps I misunderstood what can and cannot be done.
Any legitimate search engine would operate with a single identifier.
Any "search engine" trying to hide behind mutating identifiers
has obviously malevolent intentions and could be denounced as such ?)

Cheers to all who are trying to help here.

this is a great question I have considered this beacon however at the moment trusts the node to describe itself. On the open web we would allow the page to provide a sitemap. This is currently not a feature I have fully developed yet.

Anonymous wrote:

I realize now I didn't mention how it describes itself. and a hash is actually an excellent key for exactly that as it can character the compressed map as well the same way a jwt carries data why also being the check for change token you describe. This would be part of https://github.com/wdunn001/meshdata

What I hesitated to write was that the hash
could be as simple as a unix-timestamp affixed to the end of the page-name.
That unix-timestamp could be encoded as BASE-62,
eliminating the overhead of hashing the entire file.

Whenever the page is edited/updated,
just encode the time in BASE62 and replaced the old tag with the new one.
done
(if the content remains the same
then there is no need to update the tag)

correct me if i'm wrong, but if roogle or someone else is using the i2p interface, then can't they just still anonymously crawl and index the network how ever they want to?

if someone is using the i2p interface and they are anonymous through i2p, then how can someone stop them from crawling, indexing and congesting the network?

i am QandA, just made an account here

another question, how would you stop a person if they are hosting their reticulum search engine website on i2p as a hidden service? what i am saying is, if someone wants to crawl the entire network. and they don't make themselves known online, then the only think you can do is block their identity. i can't think of anything else.

I2P would not make difference. Well without I2P I suppose it might be possible for the transport node to just manually block whatever IP address the crawler is connecting from, so they can't access the network at all. But even with I2P I think it should still be possible to automatically detect any spamming node and block it. Like I2P might not have a consistent form of identification that is the same everytime the node connects, but if you just block any sub-interface spamming link requests with a hopcount of 0 then the node will still be blocked upon reconnecting as it would continue spamming and be detected again. I mean you don't really have to limit it to nodes where all the spam has a hopcount of 0, even if it's above 0 (meaning the node is either a transport node transporting spam from another node, or a malicious node trying to pretend to not be the source) you could still block it and argue that it also has a responsibility to block spamming nodes, and that if it doesn't do that then you have to.

And this is all even without mentioning the fact that whether or not you use an IP or I2P interface every packet will include your transport identity. I suppose you can generate a new transport identity every time you reconnect, but like I said you can automatically detect a spamming node and block it. These are just way’s for the transport node to block a crawler though, a better solution for a situation like this would be the rngit node using either some kind of PoW stamp solution where every request has to include a stamp that needs some processing power to calculate, possibly slowing down the crawler and making it more expensive to send so many requests as it would need more electricity, or just blocking the crawlers identity. You could perhaps have a system combing stamps with identities where you only require stamps if multiple links identify with the same identity, and the more simultaneous links with the same identity, the harder the stamps would be to calculate, meaning the more you spam the more processing power you would need for each stamp. That’s specifically for a crawler that doesn’t reuse links though, if it does you could just require stamps if they reach a certain number of requests per second.

There is already a solution and it's the same mechanism used in LXMF for stamps, but applied on the Destination and Identity level.

Necom wrote:

And this is all even without mentioning the fact that whether or not you use an IP or I2P interface every packet will include your transport identity. I suppose you can generate a new transport identity every time you reconnect

Transport identities are ephemeral, they no longer persist. Unless you explicitly set static_transport_identity = yes