I’m getting really tired of “verify you’re not a bot” checks everywhere.

I understand these checks help prevent bots, but they seem to appear on so many websites now. Does anyone else find them frustrating?

  • GenosseFlosse@feddit.org
    link
    fedilink
    arrow-up
    17
    ·
    edit-2
    7 days ago

    No, it’s not. Had request from a bot network, always changing ip’s, user agent and request url. There is no pattern there I can block.

    • mesa@piefed.social
      link
      fedilink
      English
      arrow-up
      16
      ·
      7 days ago

      I created a bot black hole. Specific recursive link that if the click it three times it blocks the IP. And the link is “click if ai” URL at the bottom. Works like a charm.

      • GenosseFlosse@feddit.org
        link
        fedilink
        arrow-up
        12
        ·
        7 days ago

        No; the traffic i get seems like it had stored my sitemap from ~10 years ago because the requested urls there are long gone, and now tries to guess other url’s with a similar pattern. It does not seem to follow links from the frontpage.

        • America@sh.itjust.works
          link
          fedilink
          arrow-up
          9
          arrow-down
          1
          ·
          7 days ago

          That’s fascinating. It almost sounds like they’re training a scraper. Which, really, is far and away the worst possible bot to have.

          • GenosseFlosse@feddit.org
            link
            fedilink
            arrow-up
            7
            ·
            7 days ago

            I already looked into something similar, IP banning does not work. One day i got ~120k unique IPs, but the most active one only made ~100 requests. I would gain nothing if i run any script to lookup the IP with each requests (and using up more resources in the process), if it fails 99% of the time because the next hit is a different IP.

            • mesa@piefed.social
              link
              fedilink
              English
              arrow-up
              4
              ·
              7 days ago

              Ok. Sounds like your in a unique situation. Good luck. You seem fucked.

              All the above worked for me.

              • CorrectAlias@piefed.blahaj.zone
                link
                fedilink
                English
                arrow-up
                1
                ·
                edit-2
                7 days ago

                I also have the same problem as who you’re replying to on a few of my websites. The only way to block them is Anubis or Cloudflare WAF.

                I think you might be in the unique situation, to be honest. It may only be a matter of time until your solution also stops working.

                • grysbok@lemmy.sdf.org
                  link
                  fedilink
                  English
                  arrow-up
                  1
                  ·
                  7 days ago

                  Agreed. The current situation seems to be a gazillion bots each making a single request. I’ve had some luck blocking whole ASNs (Alibaba, I’m glaring at you), and Anubis. I’ve had pretty good uptime the last few months.