Did this article travel forward in time from the year 1999?
In the early days of the internet, there was definitely a good number of techies who were in control of the infrastructure and believed that as long as you don't mess with other people's toys, you should be allowed to roam freely online. But even then, this wasn't the universal consensus. You would still get shown the door for certain behaviors on the Usenet or on web forums. And many ISPs would still drop you for hard porn, gore, or piracy.
But today, the consensus is that tech companies are the guardians of morality. You can get deplatformed quite easily from all the major platforms just for saying things that others disagree with. Your private files in the cloud (and sometimes on the device) get scanned for contraband. Search engines and LLMs are carefully engineered to never say or encourage the wrong things, and to flag certain things for human review. You'd be hard-pressed to find an online platform or a Western ISP that doesn't bow to social pressures.
Blocking bots would solve 98% of the problem. We need something that does just that and only that. Once traffic becomes natural again, we can rethink the abuse problem. Charging per click or even per MB sent is an excellent idea that nobody will ever support. I wonder if that is even technically possible.
Mobile carriers certainly manage to bill per MB. But I don't think people would like their rates.
People forget that a lot of the information pre-web was somewhat pricey, and especially anything routed through a telco. The web drove prices to zero, which has had some bad effects and many very good ones.
Also newsletters, magazines, journals, etc. related to any interest you might have would require a paid subscription. Or a visit to the library, if you could convince them to subscribe.
At least there would be an incentive to reduce size. I could easily got the same amount of information with the same quality when I had 3 MB/month 20+ years ago.
I was thinking about charging per TCP handshake, or even closing their connection if the user is exceeding the typical throughput a human would need to use the site's service. Dystopian, but effective.
I don't know, I run a social media platform for learning and bots are almost never a problem apart from occasional bandwidth spike. Most abuse comes from people, and we've definitely applied the principles from the article, because other way we would just get overmhelmed.
This is still predicated on correctly identifying "bad behaviour". Given distributed attacks and botnets (often utilising residental / "dial-up" equipment, whether desktops, routers, or IoT (the "S" stands for "security") kit), identifying specific network spaces as hostile still posits a great deal of collateral damage / false positive error.
Mind, I'm strongly in favour of what you're advocating, in theory. And I'm well aware that failing to accomplish this will make the Web far less useful for everyone. But the fundamental challenge remains difficult.
Ad-hoc blocking of bad actors is bound to be an endless futile game of wack a mole. The way I see things going, the internet is continuing to move away from an open web and into walled gardens. Those with resources will create large walled gardens like the gardens of Meta, OpenAI and Alphabet, each with their own issues and serving the interests of their owners. Smaller walled gardens will exist, but any time they grow anywhere near the scale of the global web of old, they'll face increasing challenges from bad actors anywhere from spam to scams to ai to propaganda and only those with resources will be able to maintain those walled gardens, and they'll only spend their resources on that if it suits their interests.
All IPs are allocated to CIDR blocks and Autonomous Systems, the latter identified by their Autonomous System Number (ASN). It's reasonably straightforward and tractable to track good/bad behaviour by either, and (thanks to the Law of Large Numbers and Power Laws), there's virtually always a very small number of absolutely horribly-misbehaved blocks from which a large fraction of abuse originates. Moreover, at sufficiently fine detail, it's possible to identify both friendly and hostile address spaces, permitting carve-outs for the former and scaled response against the latter.
The second part of this approach is that defences need not be all-or-nothing, universal, and/or unscaled. A netblock with a few bad actors might be subject to a slight performance penalty. A netblock with no non-hostile traffic could be blocked entirely (or tarpitted or otherwise subject to negative performance impacts). And of course, reputation data can be shared, as a broader view (one which, say, a large CDN or monitoring service might have) is going to provide both earlier warning and greater detail of where hostile activity originates. And individual instances of good behaviour could be excepted from broader blocks.
Ultimately, connectivity providers, whether of data centres or residential / organisational / mobile Internet services, should be encouraged to police their own outbound traffic and take actions themselves in the event of identified abusive behaviour. (That's been a long-standing dream of mine, it's ... stubbornly refused realisation.)
My one-sentence summary is: "Stop accepting rude behavior from crawlers, AI agents, closed-source-browsers, and greedy mega-platforms or else they metastasize and ruin everything."
The problem with these blocks is that they always end up blocking normal users, making the browsing experience worse... the opposite of what the intention was.
I use a forum. The operator decided to block almost every IP range associated with a data center. The problem is that more people are using VPNs due to the spread of geo restrictions, local laws like age verification, etc. And so now I need to disable my VPN - assuming I'm using one - just to access the site.
If there is indeed a lot of people using VPNs then way not to form darknet already? Ask interested site to peer with you. Peer with others, from overlay network, where you and interested parties will be in control. Its the only way imo. We need to build new net from the scratch, using current Internet as transport. VPNs is so easy to use these days, that even no-tech people can use it. All what is need to be done is to provide service by more technical people.
Me too. I was browsing on my old Windows 8 computer that I refuse to upgrade and it did not like my OS. I don't like it either, but I'm not going to install a newer version, out of principle.
There's zero javascript on the page and it reads perfectly in lynx. I'm not sure how your browser could possibly be a variable here, unless TFA's platform is actively blocking certain user agents (which I suppose isn't quite ironic, but would not exactly send the best message to go with the arguments)
Answering this and other similar arguments/observations, and for the benefit of those unable to read TFA:
To answer one potential criticism, it's true that in some sense, blocking and so on for social reasons is not good and is in some theoretical sense arguably harmful for the overall web ecology. On the other hand, the current unchecked situation itself is also deeply harmful for the overall web ecology and it's only going to get worse if we do nothing, with more and more things effectively driven off the open web. We only get to pick the poison here.
If the USA and Europe decide to got this way, they will be (as in many other ways today) followers, rather than the leaders. China already does large-scale net censorship.
Exactly, what this article is arguing for is essentially just the Chinese model of the Internet. The outside is largely inaccessible and the inside is tightly controlled by having political oversight over the large platforms.
No doubt this is effective at achieving the political goal it aims to achieve.
Sure, but the problem is that you can’t easily block the worst actors. And automated blocking by IP or user agent is nearly pointless or counterproductive these days. It’s not entirely clear what actors this author is thinking of, but if you think you can effectively block AI harvesters, you are either kidding yourself, or committing to doing more work for this ethical ideal than is remotely worth the effort.
Huh? Blocking senders as you surf the web based on what you want to see is a completely different problem from blocking requests to your server based on what the intent of the requester is. I can think of no way these problems are similar except in the very narrow technical sense of maintaining a blocklist and attaching it to a request cycle, which is really not the hard part of either of these problems.
Quite obvious is if I don't like you, I can block you. What exactly it is that I don't like is arbitrary and changes with my mood. You're visiting me from a very liberal country that makes fun of me, so I'm blocking your entire country. Or you posted a very mean thing about me on some site somewhere, so now I'm blocking you.
There's a fine example of some one that blocks any link with HN as the referrer.
In the early days of the internet, there was definitely a good number of techies who were in control of the infrastructure and believed that as long as you don't mess with other people's toys, you should be allowed to roam freely online. But even then, this wasn't the universal consensus. You would still get shown the door for certain behaviors on the Usenet or on web forums. And many ISPs would still drop you for hard porn, gore, or piracy.
But today, the consensus is that tech companies are the guardians of morality. You can get deplatformed quite easily from all the major platforms just for saying things that others disagree with. Your private files in the cloud (and sometimes on the device) get scanned for contraband. Search engines and LLMs are carefully engineered to never say or encourage the wrong things, and to flag certain things for human review. You'd be hard-pressed to find an online platform or a Western ISP that doesn't bow to social pressures.
Let's keep them on a steady diet of 403 pages and 99 cyclic captchas a day. And see how that changes their tune.
People forget that a lot of the information pre-web was somewhat pricey, and especially anything routed through a telco. The web drove prices to zero, which has had some bad effects and many very good ones.
It listed like 2 dozen spam control schemes that had been proposed that failed, mostly for social reasons.
If I had the link, I would have simply posted it as the reply.
<https://craphound.com/spamsolutions.txt>
Your post advocates a ( ) technical ( ) legislative ( ) market-based ( ) vigilante approach to fighting spam...
Mind, I'm strongly in favour of what you're advocating, in theory. And I'm well aware that failing to accomplish this will make the Web far less useful for everyone. But the fundamental challenge remains difficult.
All IPs are allocated to CIDR blocks and Autonomous Systems, the latter identified by their Autonomous System Number (ASN). It's reasonably straightforward and tractable to track good/bad behaviour by either, and (thanks to the Law of Large Numbers and Power Laws), there's virtually always a very small number of absolutely horribly-misbehaved blocks from which a large fraction of abuse originates. Moreover, at sufficiently fine detail, it's possible to identify both friendly and hostile address spaces, permitting carve-outs for the former and scaled response against the latter.
The second part of this approach is that defences need not be all-or-nothing, universal, and/or unscaled. A netblock with a few bad actors might be subject to a slight performance penalty. A netblock with no non-hostile traffic could be blocked entirely (or tarpitted or otherwise subject to negative performance impacts). And of course, reputation data can be shared, as a broader view (one which, say, a large CDN or monitoring service might have) is going to provide both earlier warning and greater detail of where hostile activity originates. And individual instances of good behaviour could be excepted from broader blocks.
Ultimately, connectivity providers, whether of data centres or residential / organisational / mobile Internet services, should be encouraged to police their own outbound traffic and take actions themselves in the event of identified abusive behaviour. (That's been a long-standing dream of mine, it's ... stubbornly refused realisation.)
My one-sentence summary is: "Stop accepting rude behavior from crawlers, AI agents, closed-source-browsers, and greedy mega-platforms or else they metastasize and ruin everything."
I use a forum. The operator decided to block almost every IP range associated with a data center. The problem is that more people are using VPNs due to the spread of geo restrictions, local laws like age verification, etc. And so now I need to disable my VPN - assuming I'm using one - just to access the site.
edit: added version
To answer one potential criticism, it's true that in some sense, blocking and so on for social reasons is not good and is in some theoretical sense arguably harmful for the overall web ecology. On the other hand, the current unchecked situation itself is also deeply harmful for the overall web ecology and it's only going to get worse if we do nothing, with more and more things effectively driven off the open web. We only get to pick the poison here.
(From TFA.)
No doubt this is effective at achieving the political goal it aims to achieve.
There's a fine example of some one that blocks any link with HN as the referrer.