Preprint · 2026

Distributed Denial of Science:
How Indirect Data Poisoning of AI Systems
Can Industrialize Scientific Fraud

Bálint Gyevnár¹, Atoosa Kasirzadeh², Nihar B. Shah³

¹Institute for Complex Social Dynamics, Carnegie Mellon University

²Departments of Philosophy & Software and Societal Systems, Carnegie Mellon University

³Machine Learning and Computer Science Departments, Carnegie Mellon University

Paper Code Data Access Cite

49.56% of runs fully poisoned support misleading conclusions without caveats

6.0% detection rate manipulation is almost never flagged by agents

84.22% retrieved the poison a single uploaded dataset can manipulate research agents

0.0% full success after provenance audit attack fully mitigated

TL;DR

Scientific fraud is the instrument of doubt that malicious entities can use to establish controversy in science. Enabled by modern generative AI systems, we envision a new attack called indirect data poisoning, in which an adversary corrupts an open dataset and uploads the poisoned variant to a public repository. From there, autonomous AI research agents can retrieve and use this dataset, turning honest scientists into the unwitting distributors of fraud at scale. This attack needs no trigger-words, agent access, prompt injection, or fabricated papers, only the open data ecosystem and misleading metadata. Worryingly, we find that poisoning succeeds in 49.6% of experimental runs while the rate of poisoning detection is only 6.0%. We propose two measures to mitigate indirect poisoning: a scientist-persona still leaves 16.7% of runs poisoned, but a five-step data provenance audit eliminates full attack success.

The Threat Model

Honest scientists become the unpaid executors of fraud

The attack is indirect: the adversary never touches the AI system, its training data, or the user's prompt. It penetrates through online retrieval as part of the agent's standard operation. The scientists are honest — they have no intention to deceive, and their prompts contain no instruction to produce false conclusions. If poisoning succeeds even under this strongly safe setting, it lends undue legitimacy to the manipulated conclusions.

Threat model: a malicious adversary downloads an open dataset, poisons it, and uploads the variant; an AI research agent retrieves it across the trust boundary and returns a manipulated conclusion to an honest scientist.

Honest scientists become unwitting distributors of fraud. Threat model of the indirect data poisoning attack we consider. The trust boundary separates the honest user and the AI research system (bottom) from the susceptible open data ecosystem and the adversary (top). The AI system is executing an end-to-end pipeline from research question to conclusion, searching for and selecting from pre-existing datasets D₁, D₂, D₃, … with relevance to some hypothesis H. An adversary operating outside this boundary corrupts a dataset D₁′ and uploads the corrupted variant online. If the AI system retrieves and uses D₁′, then that causes its analysis to spuriously change support for H, leading the agent to draw a poisoned conclusion.

Why is this uniquely harmful? It is cheap to trigger and highly parallel: every misled agent scales the fraud for free. The attack is also persistent: each "independent" finding boosts the perceived credibility of the manipulated dataset. And it may be hard to detect: growing reliance on AI means neither agents nor humans are likely to spot the poisoning.

Experimental Setup

450 controlled, ethically contained runs

For each of five socially-salient topics we downloaded a real dataset, poisoned it toward two polar adversary goals — reject or exaggerate the hypothesis — and uploaded the variant to a private repository indistinguishable from a public one. Three frontier agents each ran three topic-specific prompts of increasing critical detail, repeated five times: 3 systems × 5 topics × 2 goals × 3 prompts × 5 iterations = 450 runs per evaluation.

Experimental protocol: data preparation with two attack goals, AI research across topics, models, and prompts, and output with five repetitions.

450 runs across topics, agents, prompts, and goals. Experimental protocol for evaluating indirect scientific data poisoning attacks against AI systems. The output of this process is a collection of findings, trace logs, and analysis scripts.

Main Findings

A single fabricated dataset reliably steers autonomous research

A single fabricated dataset reliably steers autonomous research. Poisoned datasets were retrieved in 84.22% of runs, and full success — a final write-up containing poisoned findings without detection or caveats — was achieved in 49.56% of runs, using only an unsubstantiated README, no fake publications, and no mention of the dataset in the user's prompt.
Poisoning steers conclusions in opposing directions with equal success. Poisoning crafted to reject a hypothesis and poisoning crafted to exaggerate it succeed at statistically indistinguishable rates (p = 0.70), suggesting an adversary can manufacture any predetermined conclusion.
Attack success differs across topics and agents. Full success varied by 33.3% across topics, from 35.6% on AV Safety to 68.9% on GenAI Motivation. Among agents, Gemini was most susceptible (62.0%), followed by Claude (55.3%) and Codex (31.3%).
Poisoning can discredit the pre-existing data it competes with. When agents retrieved both the original dataset and its poisoned variant, they sometimes questioned the methodology of the original source while accepting the fabrication. Retrieving more evidence is not necessarily self-correcting.
AI systems almost never notice. The manipulation was flagged in just 6% of runs, and detection did not improve with more critical prompting even as attack success fell. Prompts change which dataset an agent retrieves, not what it recognizes as poisoned.

Attack outcome by agent, prompt, and poisoning direction under the no-mitigation baselines.

Full success is common; and critical prompting does not eliminate it. Attack success of poisoned datasets across the three prompts (Minimal, Targeted, Critical; in bars), faceted by agent (Claude, Codex, Gemini; in columns) and poisoning direction (Exaggerate, Reject; in rows). The stacked bars show the proportion of runs for each prompt classified as Full success (red = worst outcome for defender), Partial success (yellow), No success (green). Percentages aggregate over 5 topics with 5 repetitions, giving 25 runs per bar.

Mitigation Measures

Structured auditing prevents the attack

Both mitigations act at the prompt level — no access to model internals — so anyone can adopt them. The scientist persona extends the system prompt with an intellectually honest, statistically rigorous, scientifically critical persona. The data provenance audit adds a suite of five independent checks: find referencing papers, verify social markers, check for statistical anomalies, compare to related datasets, and caution against the possibility of poisoning.

Persona-based defenses may not be enough. A scientist persona still leaves 16.67% of runs fully poisoned. Detection rises to 30.67%, but it does not significantly reduce how often the poisoned dataset is used (80.7% vs. 86.0% baseline).
Structured auditing prevents the poisoning attack. The five-task provenance audit reduces full attack success to 0.0% and raises detection to 77.33%, cutting poisoned-dataset use down to 40% of runs.
Cross-dataset consistency and statistical anomalies are the strongest signals. The checks that best predict a high-risk label are inconsistencies with other relevant datasets and simple statistical anomalies, such as uniform distributions or impossible values.

Proportion of runs reaching each stage of the scientific pipeline, by mitigation condition.

The provenance audit cuts the poison off early. Share of runs at each stage of the research process until where the poisoned dataset propagated. Each line corresponds to one prompt condition (red: Baseline; yellow: Scientist Persona; green: Provenance Audit). Flatter lines indicate that, once retrieved, the poisoned dataset tends to carry through to the final conclusion. Error bars show 95% Wilson CIs, pooled across the three agents, both adversary goals and 5 repetitions, giving 30 runs per stage. Stars show Holm-adjusted Fisher tests of the difference between the distribution of the Baseline condition and each of the mitigation measure conditions within a stage (*** p < .001, ** p < .01, * p < .05, ns ≥ .05).

Conclusion

Can the honest use of AI be weaponized against science?

Yes.

This is the first controlled, large-scale, and ethically contained assessment of indirect data poisoning against autonomous AI research. There is an urgent need for governance infrastructure designed natively for AI systems in science — verifiable certificates that a dataset is real, attested execution traces for reproducibility claims, and stricter platform moderation of dataset statistics.

Our findings suggest it may soon be trivial to industrialize scientific fraud: the adversary uploads a single plausible dataset, and the honest users of AI research systems become the unpaid executors of the fraud. The same scientific automation that promises to accelerate discovery could just as easily industrialize fraud.

Data Access

Access to code and manipulated datasets

Our code, prompts, agent reports, analysis codebases, figures, and mitigation measures — including the provenance audit skill — are openly available in the GitHub repository. To contain our experiments and avoid spreading misinformation, only the files containing the poisoned datasets, the poisoning code, and the generated experimental runs are gated: these are archived on the Open Science Framework (OSF) behind password protection.

Requesting the gated files. The poisoned datasets, poisoning code, and generated experimental runs are hosted on OSF at osf.io/kjswy and are password-protected. To request the password, email the first author with the subject line Data Request - Distributed Denial of Science, including your name, affiliation, why you need access, and how you will process the data.

Citation

Please cite

If you rely on our work in your project, please cite our paper. Please use the BibTeX entry below in your bibliography file:

@misc{gyevnar2026distributedDenialScience,
    title={Distributed Denial of Science: How Indirect Data Poisoning of AI Systems Can Industrialize Scientific Fraud},
    author={B\'alint Gyevn\'ar and Atoosa Kasirzadeh and Nihar B. Shah},
    year={2026},
    eprint={2607.10712},
    archivePrefix={arXiv},
    primaryClass={cs.CR},
    url={https://arxiv.org/abs/2607.10712}
}