“you could block logged out users but that would impact many lurkers”
“regardless you might not be logged in at all, you should still be allowed to browse content”
Fundamentally, what I’m suggesting is a fork in the road. Either an instance admin can set up to eliminate scrapers by making the instance private to only registered users,
or they can maintain their instance as public and deal with more arcane methods to attempt to eliminate scraping.
The issue is that if the infrastructure isn’t in place for the instance operator to decide to make their service private, then everyone is opted in to the Scrapers vs Countermeasures war with no alternative.
Privacy and encryption just work, it seems like not building the infrastructure to enable the network to function with them in place is a mistake.
To me, and to many users, what we want is fast load times, quick federation, and reliable service, all things that benefit from reducing traffic load to only registered users.
Fundamentally, what I’m suggesting is a fork in the road. Either an instance admin can set up to eliminate scrapers by making the instance private to only registered users,
Yeah, it would require perhaps more changes (since instances newly subscribed to a community need the ability to ad hoc fetch content), but even just not showing the website when someone isn’t logged in would probably make a big difference. That might be pretty easy, just redirect requests to load the web app (except the login page) to the login page, and exclude the API. Apps would still get logged out access but I doubt that’s much of a problem compared to the website, since the bots seem to just be indiscriminately scraping web pages.
“you could block logged out users but that would impact many lurkers”
“regardless you might not be logged in at all, you should still be allowed to browse content”
Fundamentally, what I’m suggesting is a fork in the road. Either an instance admin can set up to eliminate scrapers by making the instance private to only registered users,
or they can maintain their instance as public and deal with more arcane methods to attempt to eliminate scraping.
The issue is that if the infrastructure isn’t in place for the instance operator to decide to make their service private, then everyone is opted in to the Scrapers vs Countermeasures war with no alternative.
Privacy and encryption just work, it seems like not building the infrastructure to enable the network to function with them in place is a mistake.
To me, and to many users, what we want is fast load times, quick federation, and reliable service, all things that benefit from reducing traffic load to only registered users.
Yeah, it would require perhaps more changes (since instances newly subscribed to a community need the ability to ad hoc fetch content), but even just not showing the website when someone isn’t logged in would probably make a big difference. That might be pretty easy, just redirect requests to load the web app (except the login page) to the login page, and exclude the API. Apps would still get logged out access but I doubt that’s much of a problem compared to the website, since the bots seem to just be indiscriminately scraping web pages.
Definitely true.