Artificial intelligence/training - Revision history

Rudxain: linkify: Chrome, Windows

2026-02-26T05:46:00Z

linkify: Chrome, Windows

← Older revision		Revision as of 05:46, 26 February 2026
Line 24:		Line 24:
	Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow <code>robots.txt</code> - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.		Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow <code>robots.txt</code> - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.

	Some AI services opt to use separate User-Agent strings, potentially also ignoring <code>robots.txt</code>, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />		Some AI services opt to use separate User-Agent strings, potentially also ignoring <code>robots.txt</code>, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard [[Google_Chrome\|Chrome]] web browser running on [[Microsoft_Windows\|Windows]]. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />

	Less legitimate bots use a wide distribution of IP addresses, further reducing options for the website to protect itself. This is in a clear attempt to bypass IP-based request throttling and rate limiting the website may implement. They are also known to ignore HTTP response status codes that indicate a server error ([[wikipedia:HTTP status code#5xx server errors\|5xx]]), or warnings that the client needs to slow down ([[wikipedia:HTTP status code#429\|429 Too Many Requests]]) or has been entirely blocked ([[wikipedia:HTTP status code#403\|403 Forbidden]]).		Less legitimate bots use a wide distribution of IP addresses, further reducing options for the website to protect itself. This is in a clear attempt to bypass IP-based request throttling and rate limiting the website may implement. They are also known to ignore HTTP response status codes that indicate a server error ([[wikipedia:HTTP status code#5xx server errors\|5xx]]), or warnings that the client needs to slow down ([[wikipedia:HTTP status code#429\|429 Too Many Requests]]) or has been entirely blocked ([[wikipedia:HTTP status code#403\|403 Forbidden]]).

Rudxain: style `robots.txt` as code

2026-02-26T05:38:25Z

style `robots.txt` as code

← Older revision		Revision as of 05:38, 26 February 2026
Line 11:		Line 11:
	While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them (such as [[wikipedia:Alibaba_Group\|Alibaba]]<ref>{{Cite news \|last=Venerandi \|first=Niccolò \|title=FOSS infrastructure is under attack by AI companies \|url=https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies \|access-date=2026-02-23 \|work=LibreNews \|archive-url=http://web.archive.org/web/20260217195639/https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ \|archive-date=17 Feb 2026}}</ref>), causing [[wikipedia:Denial-of-service attack\|distributed denial of service attacks]] which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.		While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them (such as [[wikipedia:Alibaba_Group\|Alibaba]]<ref>{{Cite news \|last=Venerandi \|first=Niccolò \|title=FOSS infrastructure is under attack by AI companies \|url=https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies \|access-date=2026-02-23 \|work=LibreNews \|archive-url=http://web.archive.org/web/20260217195639/https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ \|archive-date=17 Feb 2026}}</ref>), causing [[wikipedia:Denial-of-service attack\|distributed denial of service attacks]] which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.

	Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow [[wikipedia:robots.txt\|robots.txt]], a text file found at the root of a domain that indicates:		Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow <code>[[wikipedia:robots.txt\|robots.txt]]</code>, a text file found at the root of a domain that indicates:

	*Paths bots are allowed to index		*Paths bots are allowed to index
Line 20:		Line 20:
	These rules are typically configured for all bots, with minor adjustments made to individual bots as needed. Additionally, specific web pages may use the [[wikipedia:noindex\|robots meta tag]] to control use of their output.		These rules are typically configured for all bots, with minor adjustments made to individual bots as needed. Additionally, specific web pages may use the [[wikipedia:noindex\|robots meta tag]] to control use of their output.

	While it is good practice for a bot to respect robots.txt, there is no requirement for it, and there is no punishment for not following a website's wishes. It is additionally standard practice, but in no way enforced, that bots use a [[wikipedia:User-Agent header\|User-Agent header]] to uniquely identify itself. This allows a website operator to observe a bot's traffic patterns, potentially blocking the bot outright if its scraping is not desirable. The header also typically contains a URL or email address that can be used to contact the operator in case of anomalies observed in its traffic.		While it is good practice for a bot to respect <code>robots.txt</code>, there is no requirement for it, and there is no punishment for not following a website's wishes. It is additionally standard practice, but in no way enforced, that bots use a [[wikipedia:User-Agent header\|User-Agent header]] to uniquely identify itself. This allows a website operator to observe a bot's traffic patterns, potentially blocking the bot outright if its scraping is not desirable. The header also typically contains a URL or email address that can be used to contact the operator in case of anomalies observed in its traffic.

	Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow robots.txt - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.		Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow <code>robots.txt</code> - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.

	Some AI services opt to use separate User-Agent strings, potentially also ignoring robots.txt, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />		Some AI services opt to use separate User-Agent strings, potentially also ignoring <code>robots.txt</code>, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />

	Less legitimate bots use a wide distribution of IP addresses, further reducing options for the website to protect itself. This is in a clear attempt to bypass IP-based request throttling and rate limiting the website may implement. They are also known to ignore HTTP response status codes that indicate a server error ([[wikipedia:HTTP status code#5xx server errors\|5xx]]), or warnings that the client needs to slow down ([[wikipedia:HTTP status code#429\|429 Too Many Requests]]) or has been entirely blocked ([[wikipedia:HTTP status code#403\|403 Forbidden]]).		Less legitimate bots use a wide distribution of IP addresses, further reducing options for the website to protect itself. This is in a clear attempt to bypass IP-based request throttling and rate limiting the website may implement. They are also known to ignore HTTP response status codes that indicate a server error ([[wikipedia:HTTP status code#5xx server errors\|5xx]]), or warnings that the client needs to slow down ([[wikipedia:HTTP status code#429\|429 Too Many Requests]]) or has been entirely blocked ([[wikipedia:HTTP status code#403\|403 Forbidden]]).

Rudxain: link AI

2026-02-26T05:33:36Z

link AI

← Older revision		Revision as of 05:33, 26 February 2026
Line 22:		Line 22:
	While it is good practice for a bot to respect robots.txt, there is no requirement for it, and there is no punishment for not following a website's wishes. It is additionally standard practice, but in no way enforced, that bots use a [[wikipedia:User-Agent header\|User-Agent header]] to uniquely identify itself. This allows a website operator to observe a bot's traffic patterns, potentially blocking the bot outright if its scraping is not desirable. The header also typically contains a URL or email address that can be used to contact the operator in case of anomalies observed in its traffic.		While it is good practice for a bot to respect robots.txt, there is no requirement for it, and there is no punishment for not following a website's wishes. It is additionally standard practice, but in no way enforced, that bots use a [[wikipedia:User-Agent header\|User-Agent header]] to uniquely identify itself. This allows a website operator to observe a bot's traffic patterns, potentially blocking the bot outright if its scraping is not desirable. The header also typically contains a URL or email address that can be used to contact the operator in case of anomalies observed in its traffic.

	Unethical AI scraper bots do not follow robots.txt - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.		Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow robots.txt - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.

	Some AI services opt to use separate User-Agent strings, potentially also ignoring robots.txt, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />		Some AI services opt to use separate User-Agent strings, potentially also ignoring robots.txt, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />
Line 91:		Line 91:

	On 17 March 2025, the Git source code host SourceHut announced that the service was being disrupted by large language model crawlers. Mitigations deployed to reduce disruption involved requiring login for some areas of the service, and blocking IP ranges of cloud providers, affecting legitimate use of the website by its users.<ref>{{Cite web \|date=17 Mar 2025 \|title=LLM crawlers continue to DDoS SourceHut \|url=https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|website=sr.ht status \|url-status=live \|archive-url=http://web.archive.org/web/20251220125852/https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|archive-date=20 Dec 2025}}</ref> In response to the event, SourceHut founder Drew DeVault wrote a blog post entitled "[https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html Please stop externalizing your costs directly into my face]", discussing his frustrations with having ongoing and ever-adapting attacks that must be addressed in a timely fashion to reduce disruption to legitimate SourceHut users. DeVault estimates that between "20-100%" of his time is now spent addressing such attacks.		On 17 March 2025, the Git source code host SourceHut announced that the service was being disrupted by large language model crawlers. Mitigations deployed to reduce disruption involved requiring login for some areas of the service, and blocking IP ranges of cloud providers, affecting legitimate use of the website by its users.<ref>{{Cite web \|date=17 Mar 2025 \|title=LLM crawlers continue to DDoS SourceHut \|url=https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|website=sr.ht status \|url-status=live \|archive-url=http://web.archive.org/web/20251220125852/https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|archive-date=20 Dec 2025}}</ref> In response to the event, SourceHut founder Drew DeVault wrote a blog post entitled "[https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html Please stop externalizing your costs directly into my face]", discussing his frustrations with having ongoing and ever-adapting attacks that must be addressed in a timely fashion to reduce disruption to legitimate SourceHut users. DeVault estimates that between "20-100%" of his time is now spent addressing such attacks.


	==References==		==References==

Rudxain: wanted: JS

2026-02-23T18:35:51Z

wanted: JS

← Older revision		Revision as of 18:35, 23 February 2026
Line 33:		Line 33:
	*'''Bot check walls''': The user may be required to pass a security check "wall". While usually automatic for the user, this can affect legitimate bots. When a website protection service such as [[Cloudflare]] is not confident as to whether the visitor is legitimate, it may present a [[CAPTCHA]] to be manually filled out. An example is "Google Sorry", a CAPTCHA wall frequently seen when using Google Search via a VPN. An example that's popular in the FOSS community is [https://github.com/TecharoHQ/anubis Anubis].		*'''Bot check walls''': The user may be required to pass a security check "wall". While usually automatic for the user, this can affect legitimate bots. When a website protection service such as [[Cloudflare]] is not confident as to whether the visitor is legitimate, it may present a [[CAPTCHA]] to be manually filled out. An example is "Google Sorry", a CAPTCHA wall frequently seen when using Google Search via a VPN. An example that's popular in the FOSS community is [https://github.com/TecharoHQ/anubis Anubis].
	*'''Login walls''': Should bots be found to pass CAPTCHA walls, the website may advance to requiring logging in to view content. A major recent example of this is [[YouTube]]'s "Sign in to confirm you're not a bot" messages.		*'''Login walls''': Should bots be found to pass CAPTCHA walls, the website may advance to requiring logging in to view content. A major recent example of this is [[YouTube]]'s "Sign in to confirm you're not a bot" messages.
	*'''JavaScript requirement''': Most websites do not need JavaScript to deliver their content. However, as many scrapers expect content to be found directly in the HTML, it is often an easy workaround to use JavaScript to "insert" the content after the page has loaded. This may reduce the responsiveness of the website, increasing points of failure, and preventing security-conscious users who disable JavaScript from viewing the website.		*'''[[JavaScript]] requirement''': Most websites do not need JavaScript to deliver their content. However, as many scrapers expect content to be found directly in the HTML, it is often an easy workaround to use JavaScript to "insert" the content after the page has loaded. This may reduce the responsiveness of the website, increasing points of failure, and preventing security-conscious users who disable JavaScript from viewing the website.
	*'''IP address blocking''': Blocking IP addresses, especially by blocking entire providers via their [[wikipedia:Autonomous system (Internet)\|autonomous system number]], always comes with some risk of blocking legitimate users. Particularly, this may restrict access to users making use of a VPN.		*'''IP address blocking''': Blocking IP addresses, especially by blocking entire providers via their [[wikipedia:Autonomous system (Internet)\|autonomous system number]], always comes with some risk of blocking legitimate users. Particularly, this may restrict access to users making use of a VPN.
	*'''Heuristic blocking''': Patterns in request headers may give away that the request is being made by an unethical bot, despite attempts to act as a legitimate visitor. Heuristics are imperfect and may block legitimate users, especially those that may use less common browsers.		*'''Heuristic blocking''': Patterns in request headers may give away that the request is being made by an unethical bot, despite attempts to act as a legitimate visitor. Heuristics are imperfect and may block legitimate users, especially those that may use less common browsers.

Bananabot: Added archive URLs for 1 citation(s) using CRWCitationBot

2026-02-23T12:16:36Z

Added archive URLs for 1 citation(s) using CRWCitationBot

← Older revision		Revision as of 12:16, 23 February 2026
Line 9:		Line 9:

	==Examples==		==Examples==
	While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them (such as [[wikipedia:Alibaba_Group\|Alibaba]]<ref>{{Cite news \|last=Venerandi \|first=Niccolò \|title=FOSS infrastructure is under attack by AI companies \|url=https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies \|access-date=2026-02-23 \|work=LibreNews}}</ref>), causing [[wikipedia:Denial-of-service attack\|distributed denial of service attacks]] which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.		While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them (such as [[wikipedia:Alibaba_Group\|Alibaba]]<ref>{{Cite news \|last=Venerandi \|first=Niccolò \|title=FOSS infrastructure is under attack by AI companies \|url=https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies \|access-date=2026-02-23 \|work=LibreNews \|archive-url=http://web.archive.org/web/20260217195639/https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ \|archive-date=17 Feb 2026}}</ref>), causing [[wikipedia:Denial-of-service attack\|distributed denial of service attacks]] which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.

	Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow [[wikipedia:robots.txt\|robots.txt]], a text file found at the root of a domain that indicates:		Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow [[wikipedia:robots.txt\|robots.txt]], a text file found at the root of a domain that indicates:

Rudxain: link CAPTCHA

2026-02-23T05:31:38Z

link CAPTCHA

← Older revision		Revision as of 05:31, 23 February 2026
Line 31:		Line 31:
	To protect against unethical crawlers, due to concerns of both intellectual property and service disruption, websites adopt practices that affect the experience of real users:		To protect against unethical crawlers, due to concerns of both intellectual property and service disruption, websites adopt practices that affect the experience of real users:

	*'''Bot check walls''': The user may be required to pass a security check "wall". While usually automatic for the user, this can affect legitimate bots. When a website protection service such as [[Cloudflare]] is not confident as to whether the visitor is legitimate, it may present a CAPTCHA to be manually filled out. An example is "Google Sorry", a CAPTCHA wall frequently seen when using Google Search via a VPN. An example that's popular in the FOSS community is [https://github.com/TecharoHQ/anubis Anubis].		*'''Bot check walls''': The user may be required to pass a security check "wall". While usually automatic for the user, this can affect legitimate bots. When a website protection service such as [[Cloudflare]] is not confident as to whether the visitor is legitimate, it may present a [[CAPTCHA]] to be manually filled out. An example is "Google Sorry", a CAPTCHA wall frequently seen when using Google Search via a VPN. An example that's popular in the FOSS community is [https://github.com/TecharoHQ/anubis Anubis].
	*'''Login walls''': Should bots be found to pass CAPTCHA walls, the website may advance to requiring logging in to view content. A major recent example of this is [[YouTube]]'s "Sign in to confirm you're not a bot" messages.		*'''Login walls''': Should bots be found to pass CAPTCHA walls, the website may advance to requiring logging in to view content. A major recent example of this is [[YouTube]]'s "Sign in to confirm you're not a bot" messages.
	*'''JavaScript requirement''': Most websites do not need JavaScript to deliver their content. However, as many scrapers expect content to be found directly in the HTML, it is often an easy workaround to use JavaScript to "insert" the content after the page has loaded. This may reduce the responsiveness of the website, increasing points of failure, and preventing security-conscious users who disable JavaScript from viewing the website.		*'''JavaScript requirement''': Most websites do not need JavaScript to deliver their content. However, as many scrapers expect content to be found directly in the HTML, it is often an easy workaround to use JavaScript to "insert" the content after the page has loaded. This may reduce the responsiveness of the website, increasing points of failure, and preventing security-conscious users who disable JavaScript from viewing the website.

Rudxain: mention Alibaba as an offender

2026-02-23T05:28:35Z

mention Alibaba as an offender

← Older revision		Revision as of 05:28, 23 February 2026
Line 9:		Line 9:

	==Examples==		==Examples==
	While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them, causing [[wikipedia:Denial-of-service attack\|distributed denial of service attacks]] which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.		While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them (such as [[wikipedia:Alibaba_Group\|Alibaba]]<ref>{{Cite news \|last=Venerandi \|first=Niccolò \|title=FOSS infrastructure is under attack by AI companies \|url=https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies \|access-date=2026-02-23 \|work=LibreNews}}</ref>), causing [[wikipedia:Denial-of-service attack\|distributed denial of service attacks]] which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.

	Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow [[wikipedia:robots.txt\|robots.txt]], a text file found at the root of a domain that indicates:		Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow [[wikipedia:robots.txt\|robots.txt]], a text file found at the root of a domain that indicates:
Line 91:		Line 91:

	On 17 March 2025, the Git source code host SourceHut announced that the service was being disrupted by large language model crawlers. Mitigations deployed to reduce disruption involved requiring login for some areas of the service, and blocking IP ranges of cloud providers, affecting legitimate use of the website by its users.<ref>{{Cite web \|date=17 Mar 2025 \|title=LLM crawlers continue to DDoS SourceHut \|url=https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|website=sr.ht status \|url-status=live \|archive-url=http://web.archive.org/web/20251220125852/https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|archive-date=20 Dec 2025}}</ref> In response to the event, SourceHut founder Drew DeVault wrote a blog post entitled "[https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html Please stop externalizing your costs directly into my face]", discussing his frustrations with having ongoing and ever-adapting attacks that must be addressed in a timely fashion to reduce disruption to legitimate SourceHut users. DeVault estimates that between "20-100%" of his time is now spent addressing such attacks.		On 17 March 2025, the Git source code host SourceHut announced that the service was being disrupted by large language model crawlers. Mitigations deployed to reduce disruption involved requiring login for some areas of the service, and blocking IP ranges of cloud providers, affecting legitimate use of the website by its users.<ref>{{Cite web \|date=17 Mar 2025 \|title=LLM crawlers continue to DDoS SourceHut \|url=https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|website=sr.ht status \|url-status=live \|archive-url=http://web.archive.org/web/20251220125852/https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|archive-date=20 Dec 2025}}</ref> In response to the event, SourceHut founder Drew DeVault wrote a blog post entitled "[https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html Please stop externalizing your costs directly into my face]", discussing his frustrations with having ongoing and ever-adapting attacks that must be addressed in a timely fashion to reduce disruption to legitimate SourceHut users. DeVault estimates that between "20-100%" of his time is now spent addressing such attacks.


	==References==		==References==

Tempo123: Add archival link for citation [1]

2026-02-19T20:05:08Z

Add archival link for citation [1]

← Older revision		Revision as of 20:05, 19 February 2026
Line 78:		Line 78:
	On 15 June 2024, an investigation by Apple blog MacStories found that Perplexity does not follow its own documented policies when accessing content the user requests from the web. In their testing, the scraper pretended to be Chrome 111 running on Windows 10, connecting from an IP address not found in Perplexity's publicly-listed IP address ranges.<ref>{{Cite web \|last=Knight \|first=Robb \|date=15 Jun 2024 \|title=Perplexity AI Is Lying about Their User Agent \|url=https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/ \|url-status=live \|archive-url=http://web.archive.org/web/20260123072701/https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/ \|archive-date=23 Jan 2026}}</ref> MacStories' findings were confirmed by a WIRED investigation.<ref>{{Cite web \|last=Mehrotra \|first=Dhruv \|last2=Marchman \|first2=Tim \|date=19 Jun 2024 \|title=Perplexity Is a Bullshit Machine \|url=https://www.wired.com/story/perplexity-is-a-bullshit-machine/ \|website=WIRED \|url-status=live \|archive-url=http://web.archive.org/web/20260201173126/https://www.wired.com/story/perplexity-is-a-bullshit-machine/ \|archive-date=1 Feb 2026}}</ref> Perplexity responded by removing its list of IP addresses.		On 15 June 2024, an investigation by Apple blog MacStories found that Perplexity does not follow its own documented policies when accessing content the user requests from the web. In their testing, the scraper pretended to be Chrome 111 running on Windows 10, connecting from an IP address not found in Perplexity's publicly-listed IP address ranges.<ref>{{Cite web \|last=Knight \|first=Robb \|date=15 Jun 2024 \|title=Perplexity AI Is Lying about Their User Agent \|url=https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/ \|url-status=live \|archive-url=http://web.archive.org/web/20260123072701/https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/ \|archive-date=23 Jan 2026}}</ref> MacStories' findings were confirmed by a WIRED investigation.<ref>{{Cite web \|last=Mehrotra \|first=Dhruv \|last2=Marchman \|first2=Tim \|date=19 Jun 2024 \|title=Perplexity Is a Bullshit Machine \|url=https://www.wired.com/story/perplexity-is-a-bullshit-machine/ \|website=WIRED \|url-status=live \|archive-url=http://web.archive.org/web/20260201173126/https://www.wired.com/story/perplexity-is-a-bullshit-machine/ \|archive-date=1 Feb 2026}}</ref> Perplexity responded by removing its list of IP addresses.

	On 27 June 2024, [[Amazon]] announced an investigation into Perplexity AI, suggesting the behavior may be considered abusive under Amazon Web Services terms of service:<ref name="perplexity-aws">{{Cite web \|last=Mehrotra \|first=Dhruv \|date=27 Jun 2024 \|title=Amazon Is Investigating Perplexity Over Claims of Scraping Abuse \|url=https://www.wired.com/story/aws-perplexity-bot-scraping-investigation/ \|website=WIRED}}</ref>		On 27 June 2024, [[Amazon]] announced an investigation into Perplexity AI, suggesting the behavior may be considered abusive under Amazon Web Services terms of service:<ref name="perplexity-aws">{{Cite web \|last=Mehrotra \|first=Dhruv \|date=27 Jun 2024 \|title=Amazon Is Investigating Perplexity Over Claims of Scraping Abuse \|url=https://www.wired.com/story/aws-perplexity-bot-scraping-investigation/ \|website=WIRED \|url-status=live \|archive-url=https://web.archive.org/web/20260202160655/https://www.wired.com/story/aws-perplexity-bot-scraping-investigation/ \|archive-date=2 Feb 2026}}</ref>

	<blockquote>		<blockquote>

Banana: Added archive URLs for 12 citation(s) using CRWCitationBot

2026-02-15T23:54:57Z

Added archive URLs for 12 citation(s) using CRWCitationBot

← Older revision		Revision as of 23:54, 15 February 2026
Line 43:		Line 43:
	===Case studies===		===Case studies===
	====Diaspora====		====Diaspora====
	On 27 December 2024, the open-source social network project Diaspora noted that 70% of traffic across its infrastructure was in service of AI scrapers.<ref name="geraspora">{{Cite web \|last=Schubert \|first=Dennis \|date=27 Dec 2024 \|title=In the last 60 days, the diaspora* web assets received 11.3 million requests ... \|url=https://pod.geraspora.de/posts/17342163 \|via=Diaspora*}}</ref> Particularly, the project noted that bots had followed links to crawl every individual edit in their [[#MediaWiki\|MediaWiki]] instance, causing an exponential increase in the number of unique requests being made.		On 27 December 2024, the open-source social network project Diaspora noted that 70% of traffic across its infrastructure was in service of AI scrapers.<ref name="geraspora">{{Cite web \|last=Schubert \|first=Dennis \|date=27 Dec 2024 \|title=In the last 60 days, the diaspora* web assets received 11.3 million requests ... \|url=https://pod.geraspora.de/posts/17342163 \|via=Diaspora* \|url-status=live \|archive-url=http://web.archive.org/web/20251203054837/https://pod.geraspora.de/posts/17342163 \|archive-date=3 Dec 2025}}</ref> Particularly, the project noted that bots had followed links to crawl every individual edit in their [[#MediaWiki\|MediaWiki]] instance, causing an exponential increase in the number of unique requests being made.

	====LVFS====		====LVFS====
	The [https://fwupd.org/ Linux Vendor Firmware Service] (LVFS) provides a free central store of firmware updates, such as for UEFI motherboards and SSD controllers. This feature is integrated with many Linux distributions through the <code>fwupd</code> daemon. For situations where internet access is not permitted, the service allows users to make a local mirror of the entire 100+ GB store.		The [https://fwupd.org/ Linux Vendor Firmware Service] (LVFS) provides a free central store of firmware updates, such as for UEFI motherboards and SSD controllers. This feature is integrated with many Linux distributions through the <code>fwupd</code> daemon. For situations where internet access is not permitted, the service allows users to make a local mirror of the entire 100+ GB store.

	On 9 January 2025, the project announced that it would introduce a login wall around its mirror feature, citing unnecessary use of its bandwidth.<ref>{{Cite web \|last=Hughes \|first=Richard \|date=9 Jan 2025 \|title=Authentication soon required to mirror the entire LVFS \|url=https://lore.kernel.org/lvfs-announce/zDlhotSvKqnMDfkCKaE_u4-8uvWsgkuj18ifLBwrLN9vWWrIJjrYQ-QfhpY3xuwIXuZgzOVajW99ymoWmijTdngeFRVjM0BxhPZquUzbDfM=@hughsie.com/T/ \|website=Linux Vendor Firmware Service (LVFS) mailing list}}</ref> Up to 1,000 files may be downloaded per day without logging in. The author later mentioned on Mastodon that the problem appears to be caused by AI scraping.<ref>{{Cite web \|last=Hughes \|first=Richard \|date=22 Jan 2025 \|title=Commentary citing 'Authentication soon required to mirror the entire LVFS' \|url=https://mastodon.social/@hughsie/113871373001227969 \|via=Mastodon}}</ref>		On 9 January 2025, the project announced that it would introduce a login wall around its mirror feature, citing unnecessary use of its bandwidth.<ref>{{Cite web \|last=Hughes \|first=Richard \|date=9 Jan 2025 \|title=Authentication soon required to mirror the entire LVFS \|url=https://lore.kernel.org/lvfs-announce/zDlhotSvKqnMDfkCKaE_u4-8uvWsgkuj18ifLBwrLN9vWWrIJjrYQ-QfhpY3xuwIXuZgzOVajW99ymoWmijTdngeFRVjM0BxhPZquUzbDfM=@hughsie.com/T/ \|website=Linux Vendor Firmware Service (LVFS) mailing list \|url-status=live \|archive-url=http://web.archive.org/web/20250901012238/https://lore.kernel.org/lvfs-announce/zDlhotSvKqnMDfkCKaE_u4-8uvWsgkuj18ifLBwrLN9vWWrIJjrYQ-QfhpY3xuwIXuZgzOVajW99ymoWmijTdngeFRVjM0BxhPZquUzbDfM=@hughsie.com/T/ \|archive-date=1 Sep 2025}}</ref> Up to 1,000 files may be downloaded per day without logging in. The author later mentioned on Mastodon that the problem appears to be caused by AI scraping.<ref>{{Cite web \|last=Hughes \|first=Richard \|date=22 Jan 2025 \|title=Commentary citing 'Authentication soon required to mirror the entire LVFS' \|url=https://mastodon.social/@hughsie/113871373001227969 \|via=Mastodon \|url-status=live \|archive-url=http://web.archive.org/web/20251019021656/https://mastodon.social/@hughsie/113871373001227969 \|archive-date=19 Oct 2025}}</ref>

	====LWN.net====		====LWN.net====
Line 59:		Line 59:
	</blockquote>		</blockquote>

	He later commented:<ref>{{Cite web \|last=Knop \|first=Dirk \|date=22 Jan 2025 \|title=AI bots paralyze Linux news site and others \|url=https://www.heise.de/en/news/AI-bots-paralyze-Linux-news-site-and-others-10252162.html \|website=Heise \|language=en, de}}</ref>		He later commented:<ref>{{Cite web \|last=Knop \|first=Dirk \|date=22 Jan 2025 \|title=AI bots paralyze Linux news site and others \|url=https://www.heise.de/en/news/AI-bots-paralyze-Linux-news-site-and-others-10252162.html \|website=Heise \|language=en, de \|url-status=live \|archive-url=http://web.archive.org/web/20251018112153/https://www.heise.de/en/news/AI-bots-paralyze-Linux-news-site-and-others-10252162.html \|archive-date=18 Oct 2025}}</ref>

	<blockquote>		<blockquote>
Line 69:		Line 69:

	<!-- COI alert: I, [[User:kirb]], am an admin for The Apple Wiki. Hopefully this is neutral enough?		<!-- COI alert: I, [[User:kirb]], am an admin for The Apple Wiki. Hopefully this is neutral enough?
	-->The Apple Wiki, which documents internal details of Apple's hardware and software, holds more than 50,000 articles. On 2 August 2024, with a repeat occurrence on 5 January 2025, the service was disrupted by scraping efforts.<ref>{{Cite web \|title=Bot traffic abuse \|url=https://theapplewiki.com/wiki/The_Apple_Wiki:Community_portal#Bot_traffic_abuse \|website=The Apple Wiki}}</ref> The wiki contains a considerable amount of information that is scraped by legitimate security research tools, making it difficult for the website to block non-legitimate requests. Efforts to block unethical scraping and protect the wiki have disrupted these legitimate tools. The large article count, combined with more than 280,000 total edits over the wiki's lifetime, create an untenable situation where it is simply not possible to scrape the website without causing significant service disruption.		-->The Apple Wiki, which documents internal details of Apple's hardware and software, holds more than 50,000 articles. On 2 August 2024, with a repeat occurrence on 5 January 2025, the service was disrupted by scraping efforts.<ref>{{Cite web \|title=Bot traffic abuse \|url=https://theapplewiki.com/wiki/The_Apple_Wiki:Community_portal#Bot_traffic_abuse \|website=The Apple Wiki \|url-status=live \|archive-url=http://web.archive.org/web/20260130154004/https://theapplewiki.com/wiki/The_Apple_Wiki:Community_portal \|archive-date=30 Jan 2026}}</ref> The wiki contains a considerable amount of information that is scraped by legitimate security research tools, making it difficult for the website to block non-legitimate requests. Efforts to block unethical scraping and protect the wiki have disrupted these legitimate tools. The large article count, combined with more than 280,000 total edits over the wiki's lifetime, create an untenable situation where it is simply not possible to scrape the website without causing significant service disruption.

	On 1 April 2025, the Wikimedia Foundation indicated that its infrastructure has been under increasing pressure from content scraping bots since January 2024, with the particularly critical metric that "65% of our most expensive traffic comes from bots", despite estimating 35% of all traffic as coming from bots. The bots create traffic patterns that are significantly unlike human traffic patterns, effectively bypassing Wikimedia's caching infrastructure and placing significant load on the core servers. A blog post provides an example where bot traffic caused the [[wikipedia:Wikimedia Commons\|Wikimedia Commons]] service to become unstable during a human traffic spike. The Foundation is considering introduction of a Responsible Use of Infrastructure policy to ensure the continued stability of their services.<ref>{{Cite web \|last=Mueller \|first=Birgit \|last2=Danis \|first2=Chris \|last3=Lavagetto \|first3=Giuseppe \|date=1 Apr 2025 \|title=How crawlers impact the operations of the Wikimedia projects \|url=https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/ \|website=Wikimedia Foundation}}</ref>		On 1 April 2025, the Wikimedia Foundation indicated that its infrastructure has been under increasing pressure from content scraping bots since January 2024, with the particularly critical metric that "65% of our most expensive traffic comes from bots", despite estimating 35% of all traffic as coming from bots. The bots create traffic patterns that are significantly unlike human traffic patterns, effectively bypassing Wikimedia's caching infrastructure and placing significant load on the core servers. A blog post provides an example where bot traffic caused the [[wikipedia:Wikimedia Commons\|Wikimedia Commons]] service to become unstable during a human traffic spike. The Foundation is considering introduction of a Responsible Use of Infrastructure policy to ensure the continued stability of their services.<ref>{{Cite web \|last=Mueller \|first=Birgit \|last2=Danis \|first2=Chris \|last3=Lavagetto \|first3=Giuseppe \|date=1 Apr 2025 \|title=How crawlers impact the operations of the Wikimedia projects \|url=https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/ \|website=Wikimedia Foundation \|url-status=live \|archive-url=http://web.archive.org/web/20260207164851/https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/ \|archive-date=7 Feb 2026}}</ref>

	====Perplexity AI and news outlets====		====Perplexity AI and news outlets====
	[[Perplexity AI]], founded in August 2022, is a large language model that aims to be viewed as a general search engine. It encourages users to consume news through its summaries of stories.		[[Perplexity AI]], founded in August 2022, is a large language model that aims to be viewed as a general search engine. It encourages users to consume news through its summaries of stories.

	On 15 June 2024, an investigation by Apple blog MacStories found that Perplexity does not follow its own documented policies when accessing content the user requests from the web. In their testing, the scraper pretended to be Chrome 111 running on Windows 10, connecting from an IP address not found in Perplexity's publicly-listed IP address ranges.<ref>{{Cite web \|last=Knight \|first=Robb \|date=15 Jun 2024 \|title=Perplexity AI Is Lying about Their User Agent \|url=https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/}}</ref> MacStories' findings were confirmed by a WIRED investigation.<ref>{{Cite web \|last=Mehrotra \|first=Dhruv \|last2=Marchman \|first2=Tim \|date=19 Jun 2024 \|title=Perplexity Is a Bullshit Machine \|url=https://www.wired.com/story/perplexity-is-a-bullshit-machine/ \|website=WIRED}}</ref> Perplexity responded by removing its list of IP addresses.		On 15 June 2024, an investigation by Apple blog MacStories found that Perplexity does not follow its own documented policies when accessing content the user requests from the web. In their testing, the scraper pretended to be Chrome 111 running on Windows 10, connecting from an IP address not found in Perplexity's publicly-listed IP address ranges.<ref>{{Cite web \|last=Knight \|first=Robb \|date=15 Jun 2024 \|title=Perplexity AI Is Lying about Their User Agent \|url=https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/ \|url-status=live \|archive-url=http://web.archive.org/web/20260123072701/https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/ \|archive-date=23 Jan 2026}}</ref> MacStories' findings were confirmed by a WIRED investigation.<ref>{{Cite web \|last=Mehrotra \|first=Dhruv \|last2=Marchman \|first2=Tim \|date=19 Jun 2024 \|title=Perplexity Is a Bullshit Machine \|url=https://www.wired.com/story/perplexity-is-a-bullshit-machine/ \|website=WIRED \|url-status=live \|archive-url=http://web.archive.org/web/20260201173126/https://www.wired.com/story/perplexity-is-a-bullshit-machine/ \|archive-date=1 Feb 2026}}</ref> Perplexity responded by removing its list of IP addresses.

	On 27 June 2024, [[Amazon]] announced an investigation into Perplexity AI, suggesting the behavior may be considered abusive under Amazon Web Services terms of service:<ref name="perplexity-aws">{{Cite web \|last=Mehrotra \|first=Dhruv \|date=27 Jun 2024 \|title=Amazon Is Investigating Perplexity Over Claims of Scraping Abuse \|url=https://www.wired.com/story/aws-perplexity-bot-scraping-investigation/ \|website=WIRED}}</ref>		On 27 June 2024, [[Amazon]] announced an investigation into Perplexity AI, suggesting the behavior may be considered abusive under Amazon Web Services terms of service:<ref name="perplexity-aws">{{Cite web \|last=Mehrotra \|first=Dhruv \|date=27 Jun 2024 \|title=Amazon Is Investigating Perplexity Over Claims of Scraping Abuse \|url=https://www.wired.com/story/aws-perplexity-bot-scraping-investigation/ \|website=WIRED}}</ref>
Line 85:		Line 85:

	====Read the Docs====		====Read the Docs====
	In an early example, on 25 July 2024, open source documentation website Read the Docs detailed cases of abusive bots downloading large amounts of content from the service. Particularly, the significant range of IP addresses used in an aggressive manner rendered existing rate limiting ineffective. Taking action to block traffic identified by Cloudflare as "AI crawlers" reduced bandwidth requirements by 75%, at a cost saving of $1,500 USD/month.<ref>{{Cite web \|last=Holscher \|first=Eric \|date=25 Jul 2024 \|title=AI crawlers need to be more respectful \|url=https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/ \|website=Read the Docs}}</ref>		In an early example, on 25 July 2024, open source documentation website Read the Docs detailed cases of abusive bots downloading large amounts of content from the service. Particularly, the significant range of IP addresses used in an aggressive manner rendered existing rate limiting ineffective. Taking action to block traffic identified by Cloudflare as "AI crawlers" reduced bandwidth requirements by 75%, at a cost saving of $1,500 USD/month.<ref>{{Cite web \|last=Holscher \|first=Eric \|date=25 Jul 2024 \|title=AI crawlers need to be more respectful \|url=https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/ \|website=Read the Docs \|url-status=live \|archive-url=http://web.archive.org/web/20251028100640/https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/ \|archive-date=28 Oct 2025}}</ref>

	====SourceHut and Fedora Linux====		====SourceHut and Fedora Linux====
	On 15 March 2025, an infrastructure manager for the [[wikipedia:Fedora Linux\|Fedora Linux]] open source project discussed an assumed large language model crawling attack against the Prague.io Git source code hosting service. The project made the decision to block the entire country of Brazil for some time, while also blocking access to certain repositories whose traffic was creating significant CPU usage.<ref>{{Cite web \|last=Fenzi \|first=Kevin \|date=15 Mar 2025 \|title=Mid March infra bits 2025 \|url=https://www.scrye.com/blogs/nirik/posts/2025/03/15/mid-march-infra-bits-2025/}}</ref><ref>{{Cite web \|last=Fenzi \|first=Kevin \|date=29 Mar 2025 \|title=Late March infra bits 2025 \|url=https://www.scrye.com/blogs/nirik/posts/2025/03/29/late-march-infra-bits-2025/}}</ref>		On 15 March 2025, an infrastructure manager for the [[wikipedia:Fedora Linux\|Fedora Linux]] open source project discussed an assumed large language model crawling attack against the Prague.io Git source code hosting service. The project made the decision to block the entire country of Brazil for some time, while also blocking access to certain repositories whose traffic was creating significant CPU usage.<ref>{{Cite web \|last=Fenzi \|first=Kevin \|date=15 Mar 2025 \|title=Mid March infra bits 2025 \|url=https://www.scrye.com/blogs/nirik/posts/2025/03/15/mid-march-infra-bits-2025/ \|url-status=live \|archive-url=https://web.archive.org/web/20260215235430/https://www.scrye.com/blogs/nirik/posts/2025/03/15/mid-march-infra-bits-2025/ \|archive-date=15 Feb 2026}}</ref><ref>{{Cite web \|last=Fenzi \|first=Kevin \|date=29 Mar 2025 \|title=Late March infra bits 2025 \|url=https://www.scrye.com/blogs/nirik/posts/2025/03/29/late-march-infra-bits-2025/ \|url-status=live \|archive-url=http://web.archive.org/web/20251018081550/https://www.scrye.com/blogs/nirik/posts/2025/03/29/late-march-infra-bits-2025/ \|archive-date=18 Oct 2025}}</ref>

	On 17 March 2025, the Git source code host SourceHut announced that the service was being disrupted by large language model crawlers. Mitigations deployed to reduce disruption involved requiring login for some areas of the service, and blocking IP ranges of cloud providers, affecting legitimate use of the website by its users.<ref>{{Cite web \|date=17 Mar 2025 \|title=LLM crawlers continue to DDoS SourceHut \|url=https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|website=sr.ht status}}</ref> In response to the event, SourceHut founder Drew DeVault wrote a blog post entitled "[https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html Please stop externalizing your costs directly into my face]", discussing his frustrations with having ongoing and ever-adapting attacks that must be addressed in a timely fashion to reduce disruption to legitimate SourceHut users. DeVault estimates that between "20-100%" of his time is now spent addressing such attacks.		On 17 March 2025, the Git source code host SourceHut announced that the service was being disrupted by large language model crawlers. Mitigations deployed to reduce disruption involved requiring login for some areas of the service, and blocking IP ranges of cloud providers, affecting legitimate use of the website by its users.<ref>{{Cite web \|date=17 Mar 2025 \|title=LLM crawlers continue to DDoS SourceHut \|url=https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|website=sr.ht status \|url-status=live \|archive-url=http://web.archive.org/web/20251220125852/https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|archive-date=20 Dec 2025}}</ref> In response to the event, SourceHut founder Drew DeVault wrote a blog post entitled "[https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html Please stop externalizing your costs directly into my face]", discussing his frustrations with having ongoing and ever-adapting attacks that must be addressed in a timely fashion to reduce disruption to legitimate SourceHut users. DeVault estimates that between "20-100%" of his time is now spent addressing such attacks.

	==References==		==References==

Rudxain: obligatory Anubis mention

2026-01-17T17:58:30Z

obligatory Anubis mention

← Older revision		Revision as of 17:58, 17 January 2026
Line 31:		Line 31:
	To protect against unethical crawlers, due to concerns of both intellectual property and service disruption, websites adopt practices that affect the experience of real users:		To protect against unethical crawlers, due to concerns of both intellectual property and service disruption, websites adopt practices that affect the experience of real users:

	*'''Bot check walls''': The user may be required to pass a security check "wall". While usually automatic for the user, this can affect legitimate bots. When a website protection service such as [[Cloudflare]] is not confident as to whether the visitor is legitimate, it may present a CAPTCHA to be manually filled out. An example is "Google Sorry", a CAPTCHA wall frequently seen when using Google Search via a VPN.		*'''Bot check walls''': The user may be required to pass a security check "wall". While usually automatic for the user, this can affect legitimate bots. When a website protection service such as [[Cloudflare]] is not confident as to whether the visitor is legitimate, it may present a CAPTCHA to be manually filled out. An example is "Google Sorry", a CAPTCHA wall frequently seen when using Google Search via a VPN. An example that's popular in the FOSS community is [https://github.com/TecharoHQ/anubis Anubis].
	*'''Login walls''': Should bots be found to pass CAPTCHA walls, the website may advance to requiring logging in to view content. A major recent example of this is [[YouTube]]'s "Sign in to confirm you're not a bot" messages.		*'''Login walls''': Should bots be found to pass CAPTCHA walls, the website may advance to requiring logging in to view content. A major recent example of this is [[YouTube]]'s "Sign in to confirm you're not a bot" messages.
	*'''JavaScript requirement''': Most websites do not need JavaScript to deliver their content. However, as many scrapers expect content to be found directly in the HTML, it is often an easy workaround to use JavaScript to "insert" the content after the page has loaded. This may reduce the responsiveness of the website, increasing points of failure, and preventing security-conscious users who disable JavaScript from viewing the website.		*'''JavaScript requirement''': Most websites do not need JavaScript to deliver their content. However, as many scrapers expect content to be found directly in the HTML, it is often an easy workaround to use JavaScript to "insert" the content after the page has loaded. This may reduce the responsiveness of the website, increasing points of failure, and preventing security-conscious users who disable JavaScript from viewing the website.

← Older revision		Revision as of 05:38, 26 February 2026
Line 11:		Line 11:
	While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them (such as [[wikipedia:Alibaba_Group\|Alibaba]]<ref>{{Cite news \|last=Venerandi \|first=Niccolò \|title=FOSS infrastructure is under attack by AI companies \|url=https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies \|access-date=2026-02-23 \|work=LibreNews \|archive-url=http://web.archive.org/web/20260217195639/https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ \|archive-date=17 Feb 2026}}</ref>), causing [[wikipedia:Denial-of-service attack\|distributed denial of service attacks]] which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.		While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them (such as [[wikipedia:Alibaba_Group\|Alibaba]]<ref>{{Cite news \|last=Venerandi \|first=Niccolò \|title=FOSS infrastructure is under attack by AI companies \|url=https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies \|access-date=2026-02-23 \|work=LibreNews \|archive-url=http://web.archive.org/web/20260217195639/https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ \|archive-date=17 Feb 2026}}</ref>), causing [[wikipedia:Denial-of-service attack\|distributed denial of service attacks]] which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.

	Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow [[wikipedia:robots.txt\|robots.txt]], a text file found at the root of a domain that indicates:		Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow <code>[[wikipedia:robots.txt\|robots.txt]]</code>, a text file found at the root of a domain that indicates:

	*Paths bots are allowed to index		*Paths bots are allowed to index
Line 20:		Line 20:
	These rules are typically configured for all bots, with minor adjustments made to individual bots as needed. Additionally, specific web pages may use the [[wikipedia:noindex\|robots meta tag]] to control use of their output.		These rules are typically configured for all bots, with minor adjustments made to individual bots as needed. Additionally, specific web pages may use the [[wikipedia:noindex\|robots meta tag]] to control use of their output.

	While it is good practice for a bot to respect robots.txt, there is no requirement for it, and there is no punishment for not following a website's wishes. It is additionally standard practice, but in no way enforced, that bots use a [[wikipedia:User-Agent header\|User-Agent header]] to uniquely identify itself. This allows a website operator to observe a bot's traffic patterns, potentially blocking the bot outright if its scraping is not desirable. The header also typically contains a URL or email address that can be used to contact the operator in case of anomalies observed in its traffic.		While it is good practice for a bot to respect <code>robots.txt</code>, there is no requirement for it, and there is no punishment for not following a website's wishes. It is additionally standard practice, but in no way enforced, that bots use a [[wikipedia:User-Agent header\|User-Agent header]] to uniquely identify itself. This allows a website operator to observe a bot's traffic patterns, potentially blocking the bot outright if its scraping is not desirable. The header also typically contains a URL or email address that can be used to contact the operator in case of anomalies observed in its traffic.

	Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow robots.txt - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.		Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow <code>robots.txt</code> - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.

	Some AI services opt to use separate User-Agent strings, potentially also ignoring robots.txt, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />		Some AI services opt to use separate User-Agent strings, potentially also ignoring <code>robots.txt</code>, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />

	Less legitimate bots use a wide distribution of IP addresses, further reducing options for the website to protect itself. This is in a clear attempt to bypass IP-based request throttling and rate limiting the website may implement. They are also known to ignore HTTP response status codes that indicate a server error ([[wikipedia:HTTP status code#5xx server errors\|5xx]]), or warnings that the client needs to slow down ([[wikipedia:HTTP status code#429\|429 Too Many Requests]]) or has been entirely blocked ([[wikipedia:HTTP status code#403\|403 Forbidden]]).		Less legitimate bots use a wide distribution of IP addresses, further reducing options for the website to protect itself. This is in a clear attempt to bypass IP-based request throttling and rate limiting the website may implement. They are also known to ignore HTTP response status codes that indicate a server error ([[wikipedia:HTTP status code#5xx server errors\|5xx]]), or warnings that the client needs to slow down ([[wikipedia:HTTP status code#429\|429 Too Many Requests]]) or has been entirely blocked ([[wikipedia:HTTP status code#403\|403 Forbidden]]).