Talk:Replication crisis

Replication crisis was nominated as a Social sciences and society good article, but it did not meet the good article criteria at the time (March 28, 2022). There are suggestions on the review page for improving the article. If you can improve it, please do; it may then be renominated.

This  level-5 vital article is rated C-class on Wikipedia's content assessment scale. It is of interest to the following WikiProjects:

Science Mid‑importance This article is within the scope of WikiProject Science, a collaborative effort to improve the coverage of Science on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.ScienceWikipedia:WikiProject ScienceTemplate:WikiProject Sciencescience Mid This article has been rated as Mid-importance on the project's importance scale. Psychology Mid‑importance This article is within the scope of WikiProject Psychology, a collaborative effort to improve the coverage of Psychology on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.PsychologyWikipedia:WikiProject PsychologyTemplate:WikiProject Psychologypsychology Mid This article has been rated as Mid-importance on the project's importance scale. Medicine Low‑importance This article is within the scope of WikiProject Medicine, which recommends that medicine-related articles follow the Manual of Style for medicine-related articles and that biomedical information in any article use high-quality medical sources. Please visit the project page for details or ask questions at Wikipedia talk:WikiProject Medicine.MedicineWikipedia:WikiProject MedicineTemplate:WikiProject Medicinemedicine Low This article has been rated as Low-importance on the project's importance scale. Sociology Mid‑importance This article is within the scope of WikiProject Sociology, a collaborative effort to improve the coverage of sociology on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.SociologyWikipedia:WikiProject SociologyTemplate:WikiProject Sociologysociology Mid This article has been rated as Mid-importance on the project's importance scale. Academic Journals This article is within the scope of WikiProject Academic Journals, a collaborative effort to improve the coverage of Academic Journals on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.Academic JournalsWikipedia:WikiProject Academic JournalsTemplate:WikiProject Academic JournalsAcademic Journal See WikiProject Academic Journals' writing guide for tips on how to improve this article.

The Wikimedia Foundation's Terms of Use require that editors disclose their "employer, client, and affiliation" with respect to any paid contribution; see WP:PAID. For advice about reviewing paid contributions, see WP:COIRESPONSE. ag77777 (talk · contribs) has been paid by AlphaChip on their behalf.

I have taken the bold step of removing the ugly Overly detailed|section (dated July 2024} tag as it was not me clear what was being asked for and the section seemed to me fine. The tag itself was adding to the reading problem that might lead to the problem suggested. Perhaps some more specific suggestions here could help. Msrasnw (talk) 14:39, 23 July 2025 (UTC)Reply

I would argue if "only" is appropriate in this context. Usually these tests are not symmetric in this way, so we don't expect really that if our assumptions are false, the negated null hypothesis will just as easily give significantly important data in favour of it being not true. The most basic example is showing linear correlation. The contrary is pretty much impossible to significantly proof, as variables correlated with ratio are linearly correlated, and indistinguishable in practice. Previously we were trying to show that ρ just is not equal to zero, so exclude one case, designing the experiment to expose |ρ| being big enough to make significant difference. But to show that there is no correlation it is not enough to expose |ρ| as small, there is just no positive ε small enough small enough to |ρ|<ε showing lack of linear correlation significantly, and the exact answer usually has probability 0. This asymmetry is not related to this particular statistic ρ (the same applies for any testing if point lies outside compact subset, which is much easier than outside open subspace if the data lie around point from its closure) but is something to be expected in every study. It might be less extreme, but there is a reason for using experimental methods very often to show that some random variable is bigger than second (for example that ordinary running is on average faster than running while juggling at the same time) and we don't just do the same with null hyppothesis reversed where we expect that there is no real difference between them.

The 0.25 success rate stated as that do not mean it is low and it may as well extremely large ~SpectralFlux 04:51, 7 August 2025 (UTC)Reply

This edit request by an editor with a conflict of interest was declined.

Hi, I represent Google DeepMind and am requesting an addition to provide the outcome of Nature's investigation into the 2021 chip design paper discussed in this section. Requested changes:==Addition of Nature investigation outcome==Current text ends with:"...A fraction of evidence reviewed in the critique is summarized below using prior terminology.[142]" Proposed addition (new sentence immediately following):"Following an 18-month investigation into these concerns, Nature published an addendum in September 2024 that upheld the original paper's methodology and findings.^[1]" Rationale: The current text describes the criticism and Nature's investigation but does not include the outcome of that investigation. Since the section documents concerns about replicability, it's important for completeness to note that the journal concluded its formal review process. This addition maintains the documentation of the criticism while providing readers with the full timeline of events.

Ag77777 (talk) 18:57, 15 October 2025 (UTC)Reply

Note for reviewers: an additional request is in progress at Talk:Google Brain that may be relevant. Encoded  Talk 💬 00:03, 30 December 2025 (UTC)Reply

Not done for now: Please update this request to reflect the guidance you have received in the other areas you requested. Namely the citation you suggest is an LLM hallucination and you are obfuscating that the addendum is not independent of Google Czarking0 (talk) 04:33, 9 January 2026 (UTC)Reply

This edit request by an editor with a conflict of interest was declined.

Hi, I represent AlphaChip and am requesting an addition to provide context about data availability and Nature's investigation outcome in the Data Leakage section. Requested changes:==Addition of context to AlphaChip data leakage discussion==Current text ends with:"...underscored the need for transparent, reproducible benchmarks and rigorous evaluation standards to ensure genuine progress in the field of electronic design automation." Proposed addition (new sentence immediately following):"Google subsequently released open-source code and training data for the AlphaChip methodology,^[1] and Nature's 18-month investigation concluded in September 2024 with an addendum upholding the paper's methodology.^[2] However, some researchers have maintained concerns about comparative performance claims." Rationale: The current text discusses reproducibility concerns but does not mention that Google released the code/data or that Nature's investigation upheld the methodology. Since this section is specifically about data leakage and reproducibility, noting that data was made available and that the journal's investigation found in favor of the paper is directly relevant. The final sentence acknowledges ongoing debate while providing readers with complete information.

Ag77777 (talk) 19:09, 15 October 2025 (UTC)Reply

Note for reviewers: an additional request is in progress at Talk:Google Brain that may be relevant. Encoded  Talk 💬 00:03, 30 December 2025 (UTC)Reply

Not done for now:Please update this request to reflect the guidance you have received in the other areas you requested. Namely the citation you suggest is an LLM hallucination and you are obfuscating that the addendum is not independent of Google. This is the second time I have responded like this and if you continue to request without acknowledging the issues it will be a conduct issue.Czarking0 (talk) 19:58, 10 January 2026 (UTC)Reply

Czarking0, I sincerely apologize. Let me be transparent:

1) The citation I provided does have errors - I used AI assistance to format it, my apologies.

2) You're correct that the addendum is authored by the original Google research team.

Given your feedback, I now understand that this source alone may not be sufficient to warrant changes to this section.

If there's value in simply noting that Nature's investigation concluded (since the article currently says it's ongoing), I would propose only this minimal factual update:

Change: "...and the paper is under investigation by Nature.[60]"

To: "...and Nature concluded its investigation in September 2024.[60]"

This removes the "under investigation" language without making any claims about the outcome.

We reformatted the citation below.

“Nature published an addendum in September 2024 authored by the original research team.”

^[1]

However, if you feel even this is not appropriate given the sourcing issues, I understand. I appreciate your patience in explaining Wikipedia's standards. Ag77777 (talk) 19:22, 21 January 2026 (UTC) Ag77777 (talk) 19:22, 21 January 2026 (UTC)Reply

Czarking0, I sincerely apologize. Let me be transparent:

1) The citation I provided does have errors - I used AI assistance to format it, my apologies.

2) You're correct that the addendum is authored by the original Google research team.

Given your feedback, I now understand that this source alone may not be sufficient to warrant changes to this section.

If there's value in simply noting that Nature's investigation concluded (since the article currently says it's ongoing), I would propose only this minimal factual update:

Change: "...and the paper is under investigation by Nature.[60]"

To: "...and Nature concluded its investigation in September 2024.[60]"

This removes the "under investigation" language without making any claims about the outcome.

We reformatted the citation below.

“Nature published an addendum in September 2024 authored by the original research team.”

^[2]

However, if you feel even this is not appropriate given the sourcing issues, I understand. I appreciate your patience in explaining Wikipedia's standards. Ag77777 (talk) 19:33, 21 January 2026 (UTC) Ag77777 (talk) 19:33, 21 January 2026 (UTC)Reply

Did you see how I modified this section on Criticism of Google? That was a more extensive change that I think side steps the issue you highlight here. Potentially something similar could be done here but since this page is naturally more focused on scientific publication, I think the edit you suggest here is more reasonable. What is the [60] you are suggesting? If the is the correction, I do not think that paper claims the investigation was concluded to say so is OR. If the claim was like the the researchers published a correction in September 24 I think that is worth mentioning here and the primary source is fine citation for that. Czarking0 (talk) 20:33, 21 January 2026 (UTC)Reply

Youyou W, Yang Y, Uzzi B. A discipline-wide investigation of the replicability of Psychology papers over the past two decades. Proc Natl Acad Sci U S A. 2023 Feb 7;120(6):e2208863120. doi: 10.1073/pnas.2208863120. Epub 2023 Jan 30. PMID: 36716367; PMCID: PMC9963456.

should be added as an up to date overview - also, during my research on the topic it seems that pychology is in fact exceptionally affected compared to other social sciences, contradicting the washington post article from 2015 FrenziedStargrime (talk) 14:44, 16 November 2025 (UTC)Reply

Wouldn't it be more accurate to say

Replication failures do not NECESSARILY indicate that affected fields lack scientific rigor.

be more accurate than the current text that states

Replication failures do not indicate that affected fields lack scientific rigor.

since it seems that replication failures could be an indicator of less-than-careful practices? Dvdtoy (talk) 01:00, 11 January 2026 (UTC)Reply

Sure, but what do the sources say? Czarking0 (talk) 01:08, 11 January 2026 (UTC)Reply

Good question. The first cited source* says science is hard to do well (granted) but does not IMHO support the statement that "Replication failures do not EVER indicate that affected fields lack scientific rigor." [EVER added here by me because implied]

• https://fivethirtyeight.com/features/science-isnt-broken/

which says for example

"P-hacking and similar types of manipulations often arise from human biases."

But that doesn't exclude lack of care/rigor as a possible source of error. Dvdtoy (talk) 01:25, 12 January 2026 (UTC)Reply

Agreed Czarking0 (talk) 01:49, 12 January 2026 (UTC)Reply

It's kind of unfair to not mention Jupyter Notebooks here. Miklevin (talk) 11:43, 14 April 2026 (UTC)Reply

Jupyter Notebook should be mentioned in what context? Richard-of-Earth (talk) 03:13, 15 April 2026 (UTC)Reply