Crawling without permission? That’s a lawsuit
  • kateAEnglish
    111·
    4 天前

    100,000 requests in 11 months? That’s about 12.5 requests an hour

    • grysbok@lemmy.sdf.orgEnglish
      19·
      4 天前

      That’s hardly anything. Facebook has a bot accessing my server’s robots.txt multiple times a second. (My robots.txt used to say “Facebook bot go away” but now I just respond 404 to any requests from the Facebook bot. Pretend I said that all technical and stuff, it’s 2 am and I ought to go to sleep.)

      • kateAEnglish
        7·
        4 天前

        That’s what I’m sayin! I make more than 12.5 requests an hour to lemmy and I’m a human I’m 90.00000000000001% sure

        • grysbok@lemmy.sdf.orgEnglish
          1·
          3 天前

          I wonder if I can get Facebook to give me some of that sweet, sweet cash for the inconvenience of telling them to bugger off…