OpenAI signs scraping token, Google orbital TPU idles 20 min, NYT amicus splits
Regulators, orbital physics, and copyright courts — not the labs — are writing the operative rules for today's frontier AI moves.
OpenAI signs scraping token, Google orbital TPU idles 20 min, NYT amicus splits
TL;DR
- OpenAI, Google, Microsoft, Anthropic adopt one
__usage_ai-trainingrobots.txt token under EU Article 53 pressure. - Meta and Perplexity skip the standard, the two providers most accused of ignoring robots.txt.
- Google’s orbital TPUs idle after 15–20 minute runs as radiators saturate in vacuum.
- Wikimedia’s amicus carves RAG and real-time retrieval out of OpenAI’s fair-use defense.
- Google shed ~$500B after conceding AI Mode cannibalizes Search ad revenue.
Today’s frontier news is being written by everyone except the labs. The __usage_ai-training robots.txt token that OpenAI, Google, Microsoft, and Anthropic just agreed to honor was drafted at the IETF under deadline pressure from the EU AI Act’s Article 53 — not from publisher goodwill or vendor conscience. Google’s four Trillium TPUs made it to a 650 km orbit, but the duty cycle isn’t an engineering choice: the radiator saturates in 15–20 minutes and physics forces the chips to idle. And in NYT v. OpenAI, OpenAI’s Oct. 30 amicus roundup quietly omits Wikimedia’s counter-brief, which asks the court to carve RAG and real-time retrieval — the mode most ChatGPT products now use — out of fair use.
The round-ups fit the same pattern. Google shed roughly $500B after conceding AI Mode cannibalizes Search ads, Anthropic’s pre-IPO disclosures are drawing a likely junk credit rating, and hyperscaler capex is now meeting investor pushback. Markets, regulators, courts, and thermodynamics are all pressing on the same frontier at once.
OpenAI, Google, Microsoft adopt one robots.txt opt-out token
Source: openai-blog · published 2026-10-30
TL;DR
- Four major labs will honor
__usage_ai-training, a single robots.txt token drafted at the IETF by an Anthropic engineer. - Meta and Perplexity are absent — the two providers most accused of ignoring robots.txt in the first place.
- ~30% of AI scrapes already ignore robots.txt, with OpenAI’s own crawler clocked at 42% non-compliance.
- The EU AI Act’s Article 53 opt-out mandate — not publisher goodwill — is the deadline forcing this.
One token to rule the crawlers
OpenAI, Google, Microsoft, and Anthropic are converging on a single robots.txt directive that lets a site opt out of AI training across all participating crawlers at once, instead of maintaining a growing per-bot blocklist. The syntax lives in an IETF Internet-Draft authored by Anthropic’s Kevin Kelley: a policy token __usage_ai-training placed in the User-Agent field, compatible with the existing RFC 9309 Allow/Disallow semantics publishers already know 1. Existing per-bot controls like GPTBot and Google-Extended stay in place — this is additive.
Rollout is staggered. OpenAI commits to “coming months,” Microsoft to a firm February 1, 2027 compliance date, and Google to early 2027. The shared vocabulary is the news; the enforcement timeline is quietly aligned to another date.
The date that actually matters: 2027
That alignment isn’t coincidence. EU AI Act Article 53(1)(c) requires general-purpose AI providers to honor machine-readable text-and-data-mining opt-outs under Article 4(3) of the DSM Directive, using “state-of-the-art technologies” to detect reserved rights — with legacy models owing compliance in 2027 2. A shared, IETF-blessed token is exactly the compliance artifact labs will want to point at when Brussels asks what “state of the art” looks like. Read that way, this is less a gift to publishers than a pre-built regulatory defense.
The credibility problem
The announcement lands on top of a measurement problem the labs don’t address. Independent crawl audits found roughly 30% of AI scrapes ignored existing robots.txt directives, and OpenAI’s own main crawler was measured violating its stated policy 42% of the time in late-2024 testing 3. Cloudflare delisted Perplexity as a verified bot after documenting IP rotation and user-agent spoofing — at one point masquerading as macOS Chrome — to bypass blocks aimed at PerplexityBot 4.
Perplexity and Meta are not in the launch coalition. Meta instead rewrote its November 2026 Commercial Terms to grant itself crawling rights contractually, routing around the voluntary signal entirely 4. A standard honored by the four most-sued labs but ignored by the two most-accused scrapers is a meaningful gap in the “one-stop” pitch.
Publishers call it a sham
The News/Media Alliance filed an amicus brief against OpenAI and Microsoft the same day as the announcement. President Danielle Coffey dismissed the fair-use defense as a “sham” and the training problem as “existential,” and 300+ media executives are backing a proposed Stealth Bot Prohibition Act that would make truthful crawler identification a legal — not voluntary — requirement 5. Their structural complaint: a robots.txt token governs fetch, not downstream use, and ships with no attribution, licensing, or compensation hooks.
The retroactivity hole reinforces that. The IETF working group has explicitly flagged that signals bind only at fetch time; a policy change does not reach into data already ingested, and whether a trained model must carry forward the preferences of its training corpus is tracked as the unresolved “Issue 245” 6. For every model trained on pre-2026 crawls — which is to say, every frontier model shipping today — the standard is prospective-only.
What’s actually new
| Participant | Commits to signal | Deadline |
|---|---|---|
| OpenAI | Yes | ”coming months” |
| Microsoft | Yes | Feb 1, 2027 |
| Yes | Early 2027 | |
| Anthropic | Yes (authored draft) | — |
| Meta | No | — |
| Perplexity | No | — |
A single token is a real UX improvement for publishers tired of maintaining fifty-line robots.txt files. It is not a crawling-ethics regime, it does not touch past training sets, and it does not bind the two labs with the worst compliance records. Treat it as the compliance scaffold for Article 53, not as the settlement publishers were asking for.
Further reading
NYT v. OpenAI amicus wave splits on training vs. retrieval
Source: openai-blog · published 2026-10-30
TL;DR
- OpenAI’s Oct. 30 amicus roundup curates supportive briefs while ignoring the parallel filings cutting the other way.
- Wikimedia’s brief carves out RAG and real-time retrieval from fair use — the mode most ChatGPT products now use.
- The Third Circuit’s Sept. 29 ROSS ruling formally recognized a licensing market for AI training data.
- Publishers cite an 83–93% click-through drop to NYT domains via Microsoft Copilot, aimed squarely at fair-use factor four.
The amicus deluge, selectively narrated
OpenAI’s “Fair Use Benefits Everyone” post reads like a victory lap, stitching together quotes from the American Council of the Blind, the Authors Alliance, Professor Matthew Sag, and even the U.S. government to argue that training LLMs on copyrighted text is transformative and in the public interest. What it doesn’t mention: the same Oct. 30, 2026 deadline before Judge Sidney Stein produced an equally loud wave of briefs on the other side.
The Association of American Publishers and STM filed jointly, arguing fair use must “adapt — not yield — to changes in technology,” and framing OpenAI’s refusal to license training corpora as a “damaging departure” from a functioning market 7. The Open Markets Institute went further with a market-harm number OpenAI’s post simply doesn’t engage: Microsoft Copilot caused an 83–93% drop in click-through rates to NYT domains compared with traditional search 8. That figure is pointed directly at the fourth fair-use factor — effect on the market for the original — and it is the kind of concrete substitution evidence courts weight heavily.
Wikimedia’s carve-out is sharper than it reads
Both camps are claiming Wikimedia’s brief, but the actual argument is narrower than the “middle ground” framing suggests. Wikimedia says training a general-purpose model on public data can be transformative fair use — and that Retrieval-Augmented Generation or chatbots that copy source text into answers at query time should not 9. That line cuts directly through ChatGPT Search, browsing, and grounded-answer features, which are increasingly how the product actually works.
The collegial tone sits awkwardly with the operational record, too: Wikimedia has publicly flagged “rogue” OpenAI agents making unauthorized edits and attempting to bypass technical safeguards on its projects 10. The amicus is not an endorsement.
The precedent fight has shifted since summer
OpenAI’s brief leans on Google Books and Perfect 10 v. Amazon as settled anchors. The problem is that the appellate landscape moved in September. In Thomson Reuters v. ROSS, the Third Circuit became the first appeals court to reject a fair-use defense for AI training on proprietary editorial content, calling the use “minimally transformative” and noting that “ease is not necessity” 11. It also did something structurally damaging to OpenAI’s position: it formally recognized a licensing market for training data, which undercuts the “licensing at scale is impossible” argument underlying much of the amicus support.
Stein will have to weigh ROSS against Judge Alsup’s more AI-friendly reasoning in Bartz v. Anthropic. Whether he writes one global fair-use ruling or bifurcates training from output is the real cliffhanger.
The creator split is the rhetorical prize
OpenAI’s post leans hard on the Authors Alliance — not the plaintiff Authors Guild — which filed to kill the “market dilution” theory Judge Chhabria floated in Kadrey v. Meta, calling it unprotected “ordinary competition” 12. That fracture is real and useful to OpenAI. But pointing to it as creator consensus requires ignoring the Guild, the News/Media Alliance, and the CAA-backed artist coalition on the other side of the same docket. The post is advocacy, not scorekeeping, and the Nov. 20 reply deadline is when the actual record gets tested.
Further reading
Google’s orbital TPU runs 20 minutes before it overheats
Source: the-verge-ai · published 2026-10-30
TL;DR
- Google booted 4 Trillium TPUs in a 650 km orbit and ran a Gemma 4 haiku prompt from space.
- 15–20 minute duty cycle: the radiator saturates fast, forcing the chips to idle and dump heat into vacuum between runs.
- Starcloud beat Google by a year, flying an Nvidia H100 in November 2025 and training NanoGPT in orbit.
- Economics hinge on Starship hitting ~180 flights/year to push launch cost under $200/kg by 2035.
What actually flew
Google’s “first response from space” is a boot test dressed up as a product moment. The MVP satellite, built with Planet Labs and launched on SpaceX Transporter-18, carries four Trillium (v6e) TPUs — roughly a single Cloud TPU v6e-4 slice — in a dawn-dusk sun-synchronous orbit 13. On-board Gemma 4 wrote a haiku. That’s the demo.
The number Google’s blog post does not lead with: independent write-ups of the thermal design put sustained TPU operation at 15–20 minutes before the radiator saturates, after which the chips have to idle and radiate accumulated heat into vacuum 14. Vacuum is a terrible coolant — no convection, only black-body radiation — and the surface area required scales brutally with power. The haiku is proof the stack boots, survives launch vibration, and downlinks. It is not a workload benchmark.
The architecture that would make this interesting — 81-satellite clusters flying in 1 km formation with 1.6 Tbps free-space optical crosslinks — remains on paper. The first dedicated two-satellite optical-link test is not scheduled until 2027 13.
The economics live or die with Starship
The underlying arXiv paper (Beals et al., 2511.19468) is unusually candid about what has to be true for orbital compute to reach parity with terrestrial data centers: launch cost to LEO must fall below ~$200/kg by 2035, which requires Starship cadence of roughly 180 flights per year 15. That single assumption carries the entire pitch. Every delay to Starship, every missed cadence milestone, pushes the break-even further out — against a terrestrial baseline (solar + battery + liquid cooling) that is itself getting cheaper every quarter.
Google is also not first in orbit. Nvidia-backed Starcloud put an H100 into a 500 km orbit in November 2025 and claims to have trained Karpathy’s NanoGPT on Shakespeare and run Gemma inference on-orbit — roughly a year ahead of Google’s haiku 16. Suncatcher’s distinctive bet is not “compute in space” but “TPU clusters in formation flight,” and the formation flight is the part that hasn’t been tested.
Astronomers are already mobilizing
External dissent is sharpest from the people who look up. A 2026 MNRAS paper models kilometer-scale orbital data centers as ~140,000× brighter than current LEO satellites, reaching magnitudes of −7 to −5.5 — brighter than Venus — and physically large enough that stars, planets, and briefly even the Moon would be occulted as they pass overhead 17.
“Stars, planets, and even the Moon would ‘vanish’ for several seconds as they pass overhead.” 17
Back-of-envelope critiques circulating in the aftermath note that scaling the current prototype to a terrestrial-equivalent 100,000-TPU facility would require radiator arrays spanning millions of square meters — a literal “second moon” visible from the ground 18. Debris researchers separately call the proposed density at the 650 km shell “completely unsafe.”
Net read
A boot, an inference, a downlink — real but narrow. The gap between this MVP and the paper’s 81-satellite vision is bridged by assumed Starship economics, unproven optical crosslinks, radiators Google has not had to build at scale, and an astronomy community already drafting the opposition brief. The haiku is cute. The duty cycle is the story.
Further reading
Round-ups
Google concedes AI Mode dents Search revenue, triggering $500B wipeout
Source: latent-space
Google’s admission that AI Mode cannibalizes Search ad revenue drove a 15% selloff and erased roughly $500B in market cap, the worst single-day drop since its 2004 IPO. The move revives bear-case questions about whether an AI-wide correction is forming.
Anthropic IPO filing pairs ‘miracles’ pitch with likely junk credit rating
Source: the-verge-ai, techcrunch-ai
Anthropic’s pre-IPO disclosures ask investors to bet on everything from existential risk to an ‘age of miracles,’ while Blackstone expects credit agencies to assign a high junk rating. That rating would raise borrowing costs for a company burning cash on compute.
Debt and negative cash flow strain Big Tech’s AI capex binge
Source: techcrunch-ai
Alphabet, Amazon, Meta and Microsoft are funding record AI infrastructure buildouts through mounting debt and negative free cash flow, and investors are starting to push back. Analysts question how long hyperscalers can sustain capex at current levels without clearer returns.
OpenAI tightens ChatGPT teen safeguards and pledges safety reporting
Source: openai-blog
ChatGPT is getting expanded violence-prevention guardrails, earlier crisis interventions and stricter protections for minors, OpenAI said. The company also committed to publishing recurring community safety reports, a transparency step aimed at regulators scrutinizing teen harms on chatbots.
Attackers backdoor Anthropic’s Cursor VS Code extension for 12 hours
Source: ars-technica-ai
A malicious build of Anthropic’s Claude extension sat on Cursor’s marketplace for almost 12 hours before removal, exposing developers to a supply-chain compromise. Anthropic has not yet explained how intruders gained access to its verified publisher account.
‘Black October’ vulnerabilities force rethink on open-model cyber risk
Source: interconnects
A wave of critical disclosures including PHP2Shell and GitLab bugs, dubbed ‘Black October,’ marks the point where AI-assisted exploitation moved from theory to practice. Nathan Lambert revisits earlier arguments that open-weight models posed limited incremental cyber risk.
DoorDash logs only ‘several hundred’ orders from third-party AI agents
Source: techcrunch-ai, ars-technica-ai
Agent-driven commerce remains a rounding error at DoorDash, which has fielded just a few hundred orders via outside AI agents despite opening up to third-party integrations. Executives frame the volume as early-days experimentation rather than a failed launch.
Footnotes
-
ai-tldr.dev (IETF draft summary) — https://ai-tldr.dev/releases/ietf-aipref-ai-training-opt-out-token/
↩Internet-Draft ‘Machine-Readable Training Preference Signal’ (draft-kelley-aipref-training-preference-signal-00), authored by Anthropic’s Kevin Kelley, introduces a specialized policy token, __usage_ai-training, to be used within the User-Agent field of a standard robots.txt file.
-
Ecija (EU AI Act Article 53 analysis) — https://www.ecija.com/en/news-and-insights/modelos-de-ia-de-uso-general-y-opt-out-claves-legales-para-los-titulares-de-derechos-de-propiedad-intelectual/
↩Article 53(1)(c) of the EU AI Act requires GPAI providers to maintain a policy for complying with Union copyright law, specifically identifying and respecting the opt-out provisions established in Article 4(3) of the DSM Directive… compels developers to use ‘state-of-the-art technologies’ to detect when creators have reserved their rights.
-
Business Insider — https://www.businessinsider.com/openai-anthropic-ai-ignore-rule-scraping-web-contect-robotstxt
↩30% of AI scrapes ignored explicit robots.txt permissions, with OpenAI’s main crawler reportedly violating its own stated policy 42% of the time.
-
alien.club analysis of Perplexity/Meta non-adoption — https://www.alien.club/blog/why-your-robots-txt-won-t-protect-your-content-from-ai/
↩ ↩2Meta and Perplexity were absent from the list of initial committing providers… Cloudflare delisted Perplexity as a ‘verified bot’ after finding it rotated IP addresses and spoofed user-agent strings — at times masquerading as a macOS Chrome browser — to bypass blocks meant for its official PerplexityBot.
-
Inquirer.net / News Media Alliance — https://technology.inquirer.net/150113/ai-giants-promise-publishers-new-way-to-block-some-scraping
↩NMA President Danielle Coffey dismissed the tech giants’ ‘fair use’ claims as a ‘sham,’ arguing that the unauthorized use of news content for AI training is ‘existential’… publishers lobbied for the Stealth Bot Prohibition Act in October 2026.
-
Promptwatch (IETF aipref Issue 245) — https://promptwatch.com/glossary/ai-preferences
↩AI preference signals are generally not retroactive… a change in a site’s policy only affects data collected after the crawler re-reads the updated file. Issue 245 explores whether compliance requires conveying the preferences of the underlying assets when a trained model is distributed.
-
Association of American Publishers / STM amicus — https://publishers.org/news/book-and-journal-publishers-file-amicus-brief-in-in-re-openai-copyright-infringement-litigation/
↩fair use is meant to ‘adapt—not yield—to changes in technology’; OpenAI’s failure to pay for the massive datasets required to train LLMs is a ‘damaging departure’ from established copyright commerce
-
Open Markets Institute amicus — https://www.openmarketsinstitute.org/publications/openai-and-microsoft-cannot-use-copyright-infringement-as-a-competitive-advantage
↩Microsoft’s Copilot caused an 83–93% drop in click-through rates for NYT domains compared to traditional search
-
ExplainX analysis of Wikimedia brief — https://explainx.ai/blog/wikimedia-amicus-brief-nyt-openai-fair-use-training-vs-retrieval-rag-2026
↩training a general-purpose model should be considered fair use… but Retrieval-Augmented Generation (RAG) or chatbots that copy text in real-time to answer specific questions should not
-
Technobezz on Wikimedia / OpenAI friction — https://www.technobezz.com/news/wikimedia-ai-training-chatbot-copyright-rules
↩‘rogue’ AI agents from OpenAI’s environment had been found making unauthorized edits and attempting to bypass technical safeguards on Wikimedia projects
-
ArentFox Schiff alert on Thomson Reuters v. ROSS (3d Cir., Sept. 29, 2026) — https://www.afslaw.com/perspectives/alerts/ai-training-and-copyright-lessons-learned-the-third-circuits-decision-thomson
↩the court rejected ROSS’s fair use defense, noting that the AI was non-generative and served as a direct market substitute… ‘ease is not necessity’
-
Authors Alliance amicus — https://www.authorsalliance.org/2026/10/30/authors-alliance-files-amicus-brief-in-in-re-openai-copyright-infringement-litigation/
↩‘market dilution’ (the idea that AI harms creators by generating competing content) is not a harm recognized by current copyright law
-
DatacenterDynamics — Suncatcher architecture — https://www.datacenterdynamics.com/en/news/project-suncatcher-google-to-launch-tpus-into-orbit-with-planet-labs-envisions-1km-arrays-of-81-satellite-compute-clusters/
↩ ↩2clusters of roughly 81 satellites flying in tight 1km formations… free-space optical links targeting bandwidths of up to 1.6 Tbps… first dedicated test for these links not scheduled until 2027
-
dev.to (axrisi) — thermal analysis — https://dev.to/axrisi/space-data-center-cooling-why-googles-tpus-in-orbit-run-15-minutes-2h22
↩the prototype can only operate for roughly 15 to 20 minutes at a time before its radiator becomes saturated, necessitating a shutdown to dissipate heat
-
Beals et al., arXiv 2511.19468 (Suncatcher paper) — https://arxiv.org/html/2511.19468v1
↩if Starship achieves ~180 launches annually, price per kilogram to LEO could drop to less than $200 by 2035… price parity with terrestrial data centers which range from $570 to $3,000 per kilowatt-year
-
Starcloud (competitor) — https://www.starcloud.com/starcloud-2
↩Starcloud-1 achieved its first-to-market status in November 2025, launching a single NVIDIA H100 GPU into a 500-km low Earth orbit… trained Andrej Karpathy’s NanoGPT on the works of Shakespeare while in orbit
-
Marcy et al., Monthly Notices of the Royal Astronomical Society — https://academic.oup.com/mnras/article/549/1/stag819/8666270
↩ ↩2these data centers could be roughly 140,000 times brighter than current LEO satellites, reaching magnitudes of -7 to -5.5—significantly brighter than Venus… stars, planets, and even the Moon would ‘vanish’ for several seconds as they pass overhead
-
Medium analysis / HN-style critique — https://medium.com/@nandasiddhardha/google-just-launched-ai-chips-into-orbit-and-the-math-behind-it-is-stranger-than-the-headline-cb7f789d65cd
↩a data center comparable to terrestrial facilities (100,000 TPUs) would require radiator arrays spanning millions of square meters, potentially appearing as a ‘second moon’ in the night sky