• Dragonstaff@leminal.space
    link
    fedilink
    English
    arrow-up
    41
    arrow-down
    2
    ·
    5 days ago

    the search model is helpful to is all

    You answered your own question. The search engine indexes your page to send traffic to you. The AI bot indexes your page to plagiarize your content.

    Anecdotally, AI also routinely ignores sites’ robots.txt and spoof their agents to try to hide what they’re doing. A lot of site owners are complaining about the costs of delivering content to web scrapers. Where search indexes might hit a site every day, some AI bots are running every hour and just wasting their bandwidth.