From the decision to search, through how queries are built and what the search engine hands back, to which of it actually survives into the final, cited response — traced across four commercial platforms.
1Max Planck Institute for Software Systems
2Ruhr University Bochum
3Saarland University
4LIX, École Polytechnique
5Microsoft
6Seoul National University
ChatGPT
Claude
Grok
DeepSeekWeb‑calling rates range from 14% to 83% of identical prompts across models. Prompts an agent answers from parametric knowledge alone score higher on quality than prompts it sends to search — search helps, but doesn't fully close the gap.
Some agents fan out several queries per turn; others iterate one query deeply. The search engines behind them return few results (often <15) drawn from a narrow, platform‑specific set of preferred domains.
Cited URLs skew toward popular, high‑Tranco‑rank domains. A non‑trivial share of claims is supported only by uncited search results, and claims with no retrieval support at all score the lowest on factuality.
The paper follows a single user turn through four stages — this page is organized the same way. Every arrow below is a place where ChatGPT, Claude, Grok, and DeepSeek measurably part ways.
A verbose, conversational request — often with prior turns of context behind it.
The model judges whether its parametric knowledge is enough, or whether to reach for the web.
The prompt is compressed into one or more web queries, fanned out and/or refined across iterations.
Search results are synthesized with parametric knowledge into a cited, grounded answer.
Figure 1. The life cycle of agentic Web search — the same four‑stage frame the paper's Figure 2 uses to trace a prompt from decision through to a grounded response. Every arrow above is a point where ChatGPT, Claude, Grok, and DeepSeek measurably diverge, charted in the sections below.
ChatGPT
Claude
Grok
DeepSeek
— the same four marks and colors recur in every chart below
Each model saw the identical 1,000 sanitized prompts through its own API. GPT‑5.3‑chat searches for 1 in 7; Claude Sonnet 4.6 searches for more than 4 in 5.
GPT‑5.3‑chatClaude Sonnet 4.6Grok‑4.3DeepSeek‑v4‑flashFigure 2. Share of the same 1,000 invitro prompts that triggered a Web search, per platform. GPT‑5.3‑chat searches for roughly 1 in 7; Claude Sonnet 4.6 searches for more than 4 in 5 — nearly a six‑fold spread on identical prompts.
| Replay model ↓ / Developer prompt → | GPT‑5.3‑chat | GPT‑4.1‑mini | o4‑mini |
|---|---|---|---|
| o4‑mini | 478 | 640 | 634 |
| GPT‑4.1‑mini | 407 | 417 | 440 |
| GPT‑5.3‑chat | 140 | 145 | 169 |
Swapping in another model's harness moves the needle — o4‑mini drops from 634 to 478 calls under GPT‑5.3‑chat's conservative instructions — but never enough to close the gap. The model's own disposition still dominates the decision.
Figure 3. Web‑search calls out of 1,000 prompts when a GPT model is replayed under another GPT model's own developer‑prompt search instructions (ringed cells = a model under its native harness). Harness swaps shift the rate but never close the gap to the model's native behavior.
Claude fires one deep, iteratively‑refined query per turn. Grok fans out the most queries and racks up the most search results. Every engine returns far fewer pages than a plain Google search would.
ChatGPTClaudeGrokDeepSeekChatGPT, Grok, and DeepSeek issue parallel fan‑out queries within a turn; Claude issues one query per turn and instead goes deeper across more iterations — same query budget, opposite strategy.
Figure 4. Average number of Web queries issued per user prompt when search was invoked (invivo). Grok fans out the most queries per turn — nearly 4× ChatGPT's rate — while Claude deliberately issues one query and iterates instead.
ChatGPTClaudeGrokDeepSeekFor comparison, a typical open web search returns far more than 40 results — every one of these platform‑built engines is working from a deliberately narrow slate.
Figure 5. Average number of search‑result URLs surfaced per user prompt, per platform (invivo). Grok's engine returns the most (40.7); DeepSeek's the fewest (14.4) — a roughly 3× spread across platforms working from the same open web.
ChatGPTClaudeGrokDeepSeekreddit.com and youtube.com dominate ChatGPT, Grok, and DeepSeek results — and are completely absent from Claude's top domains, which instead lean toward developer and reference sites (github.com, learn.microsoft.com, stackoverflow.com).
ChatGPT — top‑10 domain concentrationGrok — top‑10 domain concentrationFigure 6. Top domains by share of returned search‑result URLs, per platform (invivo), with top‑10 domain concentration for ChatGPT and Grok under both real usage and controlled replay. reddit.com and youtube.com dominate three of the four platforms and are entirely absent from Claude's list.
Every response claim was checked against the URLs it cited, the URLs the search engine returned but the model chose not to cite, and — failing both — attributed to the model's own parametric memory.
ChatGPTClaudeGrokDeepSeekGrok returns the most search results of any platform (page above) yet cites almost none of them — a strikingly low 1.8% citation rate, versus 34.1% for DeepSeek.
Figure 7. Citation rate — the share of returned search‑result URLs that actually get cited in the response, per platform (invivo). Grok's 1.8% is an outlier against every other platform's double‑digit rate.
ChatGPTClaudeGrokDeepSeek| Platform | Associated citation | Other cited URL | Uncited search result | Ungrounded / parametric |
|---|---|---|---|---|
| ChatGPT | 37.3% | 25.1% | 15.4% | 22.1% |
| Claude | 37.0% | 16.3% | 25.2% | 21.5% |
| Grok | 8.6% | 9.0% | 52.7% | 29.7% |
| DeepSeek | 33.6% | 15.8% | 5.0% | 45.6% |
Grok's pattern stands out: over half of its grounded claims (52.7%) trace back to search results it retrieved but never cited — consistent with its 1.8% citation rate above. Ungrounded claims, wherever they occur, consistently score the lowest of any source on factuality in the paper's evaluation.
Figure 8. Where each response claim's supporting evidence actually comes from, per platform (invivo) — an associated citation, another cited URL elsewhere in the response, an uncited search result, or nothing retrieved at all. Grok's claims lean overwhelmingly on uncited search results; DeepSeek's lean overwhelmingly on unsupported, parametric knowledge.
invivo traces come from GDPR data donations by real platform users — nothing here was prompted for the study. invitro replays the same 1,000 sanitized prompts through each provider's API for an apples‑to‑apples comparison.
| Platform | Users | Conversations | Turns | Turns with web search |
|---|---|---|---|---|
| ChatGPT (OpenAI) | 310 | 143,730 | 690,754 | 42,273 |
| Claude (Anthropic) | 102 | 9,267 | 64,354 | 1,696 |
| Grok (xAI) | 100 | 9,005 | 53,840 | 3,004 |
| DeepSeek | 101 | 9,262 | 36,020 | 1,730 |
| Company | Model | API | Release | Knowledge cutoff |
|---|---|---|---|---|
| OpenAI | GPT‑5.3‑chat | Responses API | Mar 2026 | Aug 2025 |
| Anthropic | Claude Sonnet 4.6 | Messages API | Feb 2026 | Aug 2025 |
| xAI | Grok‑4.3 | OpenAI‑compatible Responses API | May 2026 | Dec 2025 |
| DeepSeek | DeepSeek‑v4‑flash | Anthropic‑compatible Messages API | Apr 2026 | Unknown |
@misc{amani2026characterizingwebsearchconversational,
title={Characterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responses},
author={Mahsa Amani and Seungeon Lee and Abhisek Dash and Asmaa El Fraihi and Yunah Jang and Elisabeth Kirsten and Qinyuan Wu and Krishna P. Gummadi and Manish Gupta and Abhilasha Ravichander and Muhammad Bilal Zafar and Soumi Das},
year={2026},
eprint={2609.19244},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2609.19244},
}