Hidden-variable theory

This article is about a class of mechanics theories. For other uses, see Hidden variable.

In physics, a hidden-variable theory is a deterministic physical model which seeks to explain the probabilistic nature of quantum mechanics by introducing additional (possibly inaccessible) variables.

Indeterminacy of the state of a system previous to measurement is assumed to be a part of the mathematical formulation of quantum mechanics; moreover, bounds for indeterminacy can be expressed in a quantitative form by the Heisenberg uncertainty principle. Most hidden-variable theories are attempts to avoid this indeterminacy, but possibly at the expense of requiring that nonlocal interactions be allowed. One notable hidden-variable theory is the de Broglie–Bohm theory.

In 1935, Albert Einstein, Boris Podolsky, and Nathan Rosen in their EPR paper argued that quantum entanglement might indicate quantum mechanics is an incomplete description of reality.^[1][2] John Stewart Bell in 1964, in his eponymous theorem proved that correlations between particles under any local hidden variable theory must obey certain constraints. Subsequently, Bell test experiments have demonstrated broad violation of these constraints, ruling out such theories.^[3] Bell's theorem, however, does not rule out the possibility of nonlocal theories or superdeterminism; these therefore cannot be falsified by Bell tests.

Motivation

Macroscopic physics requires classical mechanics which allows accurate predictions of mechanical motion with reproducible, high precision. Quantum phenomena require quantum mechanics, which allows accurate predictions of statistical averages only. If quantum states had hidden-variables awaiting ingenious new measurement technologies, then the latter (statistical results) might be convertible to a form of the former (classical-mechanical motion).^[4]

Such a classical mechanics would eliminate unsettling characteristics of quantum theory like the uncertainty principle. More fundamentally however, a successful model of quantum phenomena with hidden variables implies quantum entities with intrinsic values independent of measurements. Existing quantum mechanics asserts that state properties can only be known after a measurement. As N. David Mermin puts it:

"It is a fundamental quantum doctrine that a measurement does not, in general, reveal a pre-existing value of the measured property. On the contrary, the outcome of a measurement is brought into being by the act of measurement itself..."^[5]

In other words, whereas a hidden-variable theory would imply intrinsic particle properties, in quantum mechanics an electron has no definite position and velocity to even be revealed.

History

"God does not play dice"

In June 1926, Max Born published a paper,^[6] in which he was the first to clearly enunciate the probabilistic interpretation of the quantum wave function, which had been introduced by Erwin Schrödinger earlier in the year. Born concluded the paper as follows:

Here the whole problem of determinism comes up. From the standpoint of our quantum mechanics there is no quantity which in any individual case causally fixes the consequence of the collision; but also experimentally we have so far no reason to believe that there are some inner properties of the atom which conditions a definite outcome for the collision. Ought we to hope later to discover such properties ... and determine them in individual cases? Or ought we to believe that the agreement of theory and experiment—as to the impossibility of prescribing conditions for a causal evolution—is a pre-established harmony founded on the nonexistence of such conditions? I myself am inclined to give up determinism in the world of atoms. But that is a philosophical question for which physical arguments alone are not decisive.

Born's interpretation of the wave function was criticized by Schrödinger, who had previously attempted to interpret it in real physical terms, but Albert Einstein's response became one of the earliest and most famous assertions that quantum mechanics is incomplete:

Quantum mechanics is very worthy of respect. But an inner voice tells me this is not the genuine article after all. The theory delivers much but it hardly brings us closer to the Old One's secret. In any event, I am convinced that He is not playing dice.^[7][8]

Niels Bohr reportedly replied to Einstein's later expression of this sentiment by advising him to "stop telling God what to do."^[9]

Early attempts at hidden-variable theories

Shortly after making his famous "God does not play dice" comment, Einstein attempted to formulate a deterministic counter proposal to quantum mechanics, presenting a paper at a meeting of the Academy of Sciences in Berlin, on 5 May 1927, titled "Bestimmt Schrödinger's Wellenmechanik die Bewegung eines Systems vollständig oder nur im Sinne der Statistik?" ("Does Schrödinger's wave mechanics determine the motion of a system completely or only in the statistical sense?").^[10][11] However, as the paper was being prepared for publication in the academy's journal, Einstein decided to withdraw it, possibly because he discovered that, contrary to his intention, his use of Schrödinger's field to guide localized particles allowed just the kind of non-local influences he intended to avoid.^[12]

At the Fifth Solvay Congress, held in Belgium in October 1927 and attended by all the major theoretical physicists of the era, Louis de Broglie presented his own version of a deterministic hidden-variable theory, apparently unaware of Einstein's aborted attempt earlier in the year. In his theory, every particle had an associated, hidden "pilot wave" which served to guide its trajectory through space. The theory was subject to criticism at the Congress, particularly by Wolfgang Pauli, which de Broglie did not adequately answer; de Broglie abandoned the theory shortly thereafter.

Declaration of completeness of quantum mechanics, and the Bohr–Einstein debates

Main article: Bohr–Einstein debates

Also at the Fifth Solvay Congress, Max Born and Werner Heisenberg made a presentation summarizing the recent tremendous theoretical development of quantum mechanics. At the conclusion of the presentation, they declared:

[W]hile we consider ... a quantum mechanical treatment of the electromagnetic field ... as not yet finished, we consider quantum mechanics to be a closed theory, whose fundamental physical and mathematical assumptions are no longer susceptible of any modification.... On the question of the 'validity of the law of causality' we have this opinion: as long as one takes into account only experiments that lie in the domain of our currently acquired physical and quantum mechanical experience, the assumption of indeterminism in principle, here taken as fundamental, agrees with experience.^[13]

Although there is no record of Einstein responding to Born and Heisenberg during the technical sessions of the Fifth Solvay Congress, he did challenge the completeness of quantum mechanics at various times. In his tribute article for Born's retirement he discussed the quantum representation of a macroscopic ball bouncing elastically between rigid barriers. He argues that such a quantum representation does not represent a specific ball, but "time ensemble of systems". As such the representation is correct, but incomplete because it does not represent the real individual macroscopic case.^[14] Einstein considered quantum mechanics incomplete "because the state function, in general, does not even describe the individual event/system".^[15]

Von Neumann's proof

John von Neumann in his 1932 book Mathematical Foundations of Quantum Mechanics had presented a proof that there could be no "hidden parameters" in quantum mechanics. The validity of von Neumann's proof was questioned by Grete Hermann in 1935, who found a flaw in the proof. The critical issue concerned averages over ensembles. Von Neumann assumed that a relation between the expected values of different observable quantities holds for each possible value of the "hidden parameters", rather than only for a statistical average over them.^[16][17] However Hermann's work went mostly unnoticed until its rediscovery by John Stewart Bell more than 30 years later.^[18][19]

The validity and definitiveness of von Neumann's proof were also questioned by Hans Reichenbach, and possibly in conversation though not in print by Albert Einstein. Reportedly, in a conversation circa 1938 with his assistants Peter Bergmann and Valentine Bargmann, Einstein pulled von Neumann's book off his shelf, pointed to the same assumption critiqued by Hermann and Bell, and asked why one should believe in it.^[20][21] Simon Kochen and Ernst Specker rejected von Neumann's key assumption as early as 1961, but did not publish a criticism of it until 1967.^[22]

EPR paradox

Main article: EPR paradox

Einstein argued that quantum mechanics could not be a complete theory of physical reality. He wrote,

Consider a mechanical system consisting of two partial systems A and B which interact with each other only during a limited time. Let the ψ function [i.e., wavefunction] before their interaction be given. Then the Schrödinger equation will furnish the ψ function after the interaction has taken place. Let us now determine the physical state of the partial system A as completely as possible by measurements. Then quantum mechanics allows us to determine the ψ function of the partial system B from the measurements made, and from the ψ function of the total system. This determination, however, gives a result which depends upon which of the physical quantities (observables) of A have been measured (for instance, coordinates or momenta). Since there can be only one physical state of B after the interaction which cannot reasonably be considered to depend on the particular measurement we perform on the system A separated from B it may be concluded that the ψ function is not unambiguously coordinated to the physical state. This coordination of several ψ functions to the same physical state of system B shows again that the ψ function cannot be interpreted as a (complete) description of a physical state of a single system.^[23]

Together with Boris Podolsky and Nathan Rosen, Einstein published a paper that gave a related but distinct argument against the completeness of quantum mechanics.^[24] They proposed a thought experiment involving a pair of particles prepared in what would later become known as an entangled state. Einstein, Podolsky, and Rosen pointed out that, in this state, if the position of the first particle were measured, the result of measuring the position of the second particle could be predicted. If instead the momentum of the first particle were measured, then the result of measuring the momentum of the second particle could be predicted. They argued that no action taken on the first particle could instantaneously affect the other, since this would involve information being transmitted faster than light, which is impossible according to the theory of relativity. They invoked a principle, later known as the "EPR criterion of reality", positing that: "If, without in any way disturbing a system, we can predict with certainty (i.e., with probability equal to unity) the value of a physical quantity, then there exists an element of reality corresponding to that quantity." From this, they inferred that the second particle must have a definite value of both position and of momentum prior to either quantity being measured. But quantum mechanics considers these two observables incompatible and thus does not associate simultaneous values for both to any system. Einstein, Podolsky, and Rosen therefore concluded that quantum theory does not provide a complete description of reality.^[25]

Bohr answered the Einstein–Podolsky–Rosen challenge as follows:

[The argument of] Einstein, Podolsky and Rosen contains an ambiguity as regards the meaning of the expression "without in any way disturbing a system." ... [E]ven at this stage [i.e., the measurement of, for example, a particle that is part of an entangled pair], there is essentially the question of an influence on the very conditions which define the possible types of predictions regarding the future behavior of the system. Since these conditions constitute an inherent element of the description of any phenomenon to which the term "physical reality" can be properly attached, we see that the argumentation of the mentioned authors does not justify their conclusion that quantum-mechanical description is essentially incomplete."^[26]

Bohr is here choosing to define a "physical reality" as limited to a phenomenon that is immediately observable by an arbitrarily chosen and explicitly specified technique, using his own special definition of the term 'phenomenon'. He wrote in 1948:

As a more appropriate way of expression, one may strongly advocate limitation of the use of the word phenomenon to refer exclusively to observations obtained under specified circumstances, including an account of the whole experiment.^[27][28]

This was, of course, in conflict with the EPR criterion of reality.

Bell's theorem

Main article: Bell's theorem

In 1964, John Stewart Bell showed through his famous theorem that if local hidden variables exist, certain experiments could be performed involving quantum entanglement where the result would satisfy a Bell inequality. If, on the other hand, statistical correlations resulting from quantum entanglement could not be explained by local hidden variables, the Bell inequality would be violated. Another no-go theorem concerning hidden-variable theories is the Kochen–Specker theorem.

Physicists such as Alain Aspect and Paul Kwiat have performed experiments that have found violations of these inequalities up to 242 standard deviations.^[29] This rules out local hidden-variable theories, but does not rule out non-local ones. Theoretically, there could be experimental problems that affect the validity of the experimental findings.

Gerard 't Hooft has disputed the validity of Bell's theorem on the basis of the superdeterminism loophole and proposed some ideas to construct local deterministic models.^[30][31]

Bohm's hidden-variable theory

Main article: de Broglie–Bohm theory

In 1952, David Bohm proposed a hidden variable theory. Bohm unknowingly rediscovered (and extended) the idea that Louis de Broglie's pilot wave theory had proposed in 1927 (and abandoned) – hence this theory is commonly called "de Broglie-Bohm theory". Assuming the validity of Bell's theorem, any deterministic hidden-variable theory that is consistent with quantum mechanics would have to be non-local, maintaining the existence of instantaneous or faster-than-light relations (correlations) between physically separated entities.

Bohm posited both the quantum particle, e.g. an electron, and a hidden 'guiding wave' that governs its motion. Thus, in this theory electrons are quite clearly particles. When a double-slit experiment is performed, the electron goes through either one of the slits. Also, the slit passed through is not random but is governed by the (hidden) pilot wave, resulting in the wave pattern that is observed.

In Bohm's interpretation, the (non-local) quantum potential constitutes an implicate (hidden) order which organizes a particle, and which may itself be the result of yet a further implicate order: a superimplicate order which organizes a field.^[32] Nowadays Bohm's theory is considered to be one of many interpretations of quantum mechanics. Some consider it the simplest theory to explain quantum phenomena.^[33] Nevertheless, it is a hidden-variable theory, and necessarily so.^[34] The major reference for Bohm's theory today is his book with Basil Hiley, published posthumously.^[35]

A possible weakness of Bohm's theory is that some (including Einstein, Pauli, and Heisenberg) feel that it looks contrived.^[36] (Indeed, Bohm thought this of his original formulation of the theory.^[37]) Bohm said he considered his theory to be unacceptable as a physical theory due to the guiding wave's existence in an abstract multi-dimensional configuration space, rather than three-dimensional space.^[37]

Recent developments

In August 2011, Roger Colbeck and Renato Renner published a proof that any extension of quantum mechanical theory, whether using hidden variables or otherwise, cannot provide a more accurate prediction of outcomes, assuming that observers can freely choose the measurement settings.^[38] Colbeck and Renner write: "In the present work, we have ... excluded the possibility that any extension of quantum theory (not necessarily in the form of local hidden variables) can help predict the outcomes of any measurement on any quantum state. In this sense, we show the following: under the assumption that measurement settings can be chosen freely, quantum theory really is complete".

In January 2013, Giancarlo Ghirardi and Raffaele Romano described a model which, "under a different free choice assumption [...] violates [the statement by Colbeck and Renner] for almost all states of a bipartite two-level system, in a possibly experimentally testable way".^[39]

See also

• Einstein's thought experiments
• Pusey–Barrett–Rudolph theorem
• Spekkens toy model

References

1. Einstein, A.; Podolsky, B.; Rosen, N. (1935). "Can Quantum-Mechanical Description of Physical Reality Be Considered Complete?". Physical Review. 47 (10): 777–780. Bibcode:1935PhRv...47..777E. doi:10.1103/PhysRev.47.777.
2. Genovese, M. (2005). "Research on hidden variable theories: A review of recent progresses". Physics Reports. 413 (6): 319–396. arXiv:quant-ph/0701071v1. Bibcode:2005PhR...413..319G. doi:10.1016/j.physrep.2005.03.003. S2CID 14833712. The debate whether Quantum Mechanics is a complete theory and probabilities have a non-epistemic character (i.e. nature is intrinsically probabilistic) or whether it is a statistical approximation of a deterministic theory and probabilities are due to our ignorance of some parameters (i.e. they are epistemic) dates to the beginning of the theory itself
3. Markoff, Jack (21 October 2015). "Sorry, Einstein. Quantum Study Suggests 'Spooky Action' Is Real". New York Times. Retrieved 21 October 2015.
4. Bell, John S. (1966-07-01). "On the Problem of Hidden Variables in Quantum Mechanics". Reviews of Modern Physics. 38 (3): 447–452. Bibcode:1966RvMP...38..447B. doi:10.1103/RevModPhys.38.447. ISSN 0034-6861. OSTI 1444158.
5. Mermin, N. David (1993-07-01). "Hidden variables and the two theorems of John Bell". Reviews of Modern Physics. 65 (3): 803–815. arXiv:1802.10119. Bibcode:1993RvMP...65..803M. doi:10.1103/RevModPhys.65.803. ISSN 0034-6861. S2CID 119546199.
6. Born, Max (1926). "Zur Quantenmechanik der Stoßvorgänge". Zeitschrift für Physik (in German). 37 (12): 863–867. Bibcode:1926ZPhy...37..863B. doi:10.1007/BF01397477. ISSN 1434-6001. S2CID 119896026.
7. The Collected Papers of Albert Einstein, Volume 15: The Berlin Years: Writings & Correspondence, June 1925-May 1927 (English Translation Supplement), p. 403
8. The Born–Einstein letters: correspondence between Albert Einstein and Max and Hedwig Born from 1916–1955, with commentaries by Max Born. Macmillan. 1971. p. 91.
9. This is a common paraphrasing. Bohr recollected his reply to Einstein at the 1927 Solvay Congress in his essay "Discussion with Einstein on Epistemological Problems in Atomic Physics", in Albert Einstein, Philosopher–Scientist, ed. Paul Arthur Shilpp, Harper, 1949, p. 211: "...in spite of all divergencies of approach and opinion, a most humorous spirit animated the discussions. On his side, Einstein mockingly asked us whether we could really believe that the providential authorities took recourse to dice-playing ("ob der liebe Gott würfelt"), to which I replied by pointing at the great caution, already called for by ancient thinkers, in ascribing attributes to Providence in everyday language." Werner Heisenberg, who also attended the congress, recalled the exchange in Encounters with Einstein, Princeton University Press, 1983, p. 117,: "But he [Einstein] still stood by his watchword, which he clothed in the words: 'God does not play at dice.' To which Bohr could only answer: 'But still, it cannot be for us to tell God, how he is to run the world.'"
10. The Collected Papers of Albert Einstein, Volume 15: The Berlin Years: Writings & Correspondence, June 1925-May 1927 (English Translation Supplement), p. 512
11. Albert Einstein Archives reel 2, item 100
12. Baggott, Jim (2011). The Quantum Story: A History in 40 Moments. New York: Oxford University Press. pp. 116–117. ISBN 978-0-19-956684-6.
13. Max Born and Werner Heisenberg, "Quantum mechanics", proceedings of the Fifth Solvay Congress.
14. Einstein, Albert (2011). "Elementary Considerations on the Interpretation of the Foundations of Quantum Mechanics". arXiv:1107.3701 [physics.hist-ph]. This paper, whose original title was "Elementare Uberlegungen zur Interpretation ¨ der Grundlagen der Quanten-Mechanik", has been translated from the German by Dileep Karanth, Department of Physics, University of Wisconsin-Parkside, Kenosha, USA
15. Ballentine, L. E. (1972-12-01). "Einstein's Interpretation of Quantum Mechanics". American Journal of Physics. 40 (12): 1763–1771. Bibcode:1972AmJPh..40.1763B. doi:10.1119/1.1987060. ISSN 0002-9505.
16. Jammer, Max (1974). The Philosophy of Quantum Mechanics. John Wiley and Sons. pp. 265–274. ISBN 0-471-43958-4.
17. Mermin, N. David; Schack, Rüdiger (September 2018). "Homer Nodded: Von Neumann's Surprising Oversight". Foundations of Physics. 48 (9): 1007–1020. arXiv:1805.10311. Bibcode:2018FoPh...48.1007M. doi:10.1007/s10701-018-0197-5. ISSN 0015-9018.
18. Hermann, G.: Die naturphilosophischen Grundlagen der Quantenmechanik (Auszug). Abhandlungen der Fries’schen Schule 6, 75–152 (1935). English translation: Chapter 15 of “Grete Hermann — Between physics and philosophy”, Elise Crull and Guido Bacciagaluppi, eds., Springer, 2016, 239- 278. [Volume 42 of Studies in History and Philosophy of Science]
19. Del Santo, Flavio (2022-01-02). "Beyond Method: The Diatribe Between Feyerabend and Popper Over the Foundations of Quantum Mechanics". International Studies in the Philosophy of Science. 35 (1): 5–22. arXiv:2108.13121. doi:10.1080/02698595.2022.2031430.
20. Reichenbach, Hans (1944). Philosophic Foundations of Quantum Mechanics. University of California Press. p. 14. OCLC 872622725.
21. Wick, David (1995). "Bell's Theorem". The Infamous Boundary: Seven Decades of Heresy in Quantum Physics. New York: Springer. p. 286. doi:10.1007/978-1-4612-4030-3_11. ISBN 978-0-387-94726-6.
22. Conway, John; Kochen, Simon (2002). "The Geometry of the Quantum Paradoxes". In Bertlmann, Reinhold A.; Zeilinger, Anton (eds.). Quantum [Un]speakables: From Bell to Quantum Information. Berlin: Springer. pp. 257–269. ISBN 3-540-42756-2. OCLC 49404213.
23. Einstein A (1936). "Physics and Reality". Journal of the Franklin Institute. 221.
24. Einstein, A.; Podolsky, B.; Rosen, N. (1935). "Can Quantum-Mechanical Description of Physical Reality Be Considered Complete?". Physical Review. 47 (10): 777–780. Bibcode:1935PhRv...47..777E. doi:10.1103/physrev.47.777.
25. Peres, Asher (2002). Quantum Theory: Concepts and Methods. Kluwer. p. 149.
26. Bohr N (1935). "Can Quantum-Mechanical Description of Physical Reality be Considered Complete?". Physical Review. 48 (8): 700. Bibcode:1935PhRv...48..696B. doi:10.1103/physrev.48.696.
27. Bohr N. (1948). "On the notions of causality and complementarity". Dialectica. 2 (3–4): 312–319 [317]. doi:10.1111/j.1746-8361.1948.tb00703.x.
28. Rosenfeld, L. (). 'Niels Bohr's contribution to epistemology', pp. 522–535 in Selected Papers of Léon Rosenfeld, Cohen, R.S., Stachel, J.J. (editors), D. Riedel, Dordrecht, ISBN 978-90-277-0652-2, p. 531: "Moreover, the complete definition of the phenomenon must essentially contain the indication of some permanent mark left upon a recording device which is part of the apparatus; only by thus envisaging the phenomenon as a closed event, terminated by a permanent record, can we do justice to the typical wholeness of the quantal processes."
29. Kwiat P. G.; et al. (1999). "Ultrabright source of polarization-entangled photons". Physical Review A. 60 (2): R773 – R776. arXiv:quant-ph/9810003. Bibcode:1999PhRvA..60..773K. doi:10.1103/physreva.60.r773. S2CID 16417960.
30. Gerard 't Hooft (2007). "The Free-Will Postulate in Quantum Mechanics". arXiv:quant-ph/0701097.
31. Gerard 't Hooft (2009). "Entangled quantum states in a local deterministic theory". arXiv:0908.3408 [quant-ph].
32. David Pratt: "David Bohm and the Implicate Order". Appeared in Sunrise magazine, February/March 1993, Theosophical University Press
33. Michael K.-H. Kiessling: "Misleading Signposts Along the de Broglie–Bohm Road to Quantum Mechanics", Foundations of Physics, volume 40, number 4, 2010, pp. 418–429 (abstract)
34. "While the testable predictions of Bohmian mechanics are isomorphic to standard Copenhagen quantum mechanics, its underlying hidden variables have to be, in principle, unobservable. If one could observe them, one would be able to take advantage of that and signal faster than light, which – according to the special theory of relativity – leads to physical temporal paradoxes." J. Kofler and A. Zeilinger, "Quantum Information and Randomness", European Review (2010), Vol. 18, No. 4, 469–480.
35. D. Bohm and B. J. Hiley, The Undivided Universe, Routledge, 1993, ISBN 0-415-06588-7.
36. Wayne C. Myrvold (2003). "On some early objections to Bohm's theory" (PDF). International Studies in the Philosophy of Science. 17 (1): 8–24. doi:10.1080/02698590305233. S2CID 10965929. Archived from the original on 2014-07-02.
37. David Bohm (1957). Causality and Chance in Modern Physics. Routledge & Kegan Paul and D. Van Nostrand. p. 110. ISBN 0-8122-1002-6.
38. Roger Colbeck; Renato Renner (2011). "No extension of quantum theory can have improved predictive power". Nature Communications. 2 (8): 411. arXiv:1005.5173. Bibcode:2011NatCo...2..411C. doi:10.1038/ncomms1416. PMC 3265370. PMID 21811240.
39. Giancarlo Ghirardi; Raffaele Romano (2013). "Onthological models predictively inequivalent to quantum theory". Physical Review Letters. 110 (17): 170404. arXiv:1301.2695. Bibcode:2013PhRvL.110q0404G. doi:10.1103/PhysRevLett.110.170404. PMID 23679689. S2CID 197479.

Bibliography

• Peres, Asher; Zurek, Wojciech (1982). "Is quantum theory universally valid?". American Journal of Physics. 50 (9): 807–810. Bibcode:1982AmJPh..50..807P. doi:10.1119/1.13086.
• Jammer, Max (1985). "The EPR Problem in Its Historical Development". In Lahti, P.; Mittelstaedt, P. (eds.). Symposium on the Foundations of Modern Physics: 50 years of the Einstein–Podolsky–Rosen Gedankenexperiment. Singapore: World Scientific. pp. 129–149.
• Fine, Arthur (1986). The Shaky Game: Einstein Realism and the Quantum Theory. Chicago: University of Chicago Press.