or, why life can be both random and destined

There’s a widespread delusion that life is a like a lottery, and that events in the real world are determined by chance. I notice it when people talk about the future using the language of statistics: what are the chances of X, e.g. getting rich, becoming famous, having an accident? The unique case of the individual is replaced with a vague idea about the population.

This delusion is very common: we see it in popular culture, politicians, journalists and advertisers use it. I hardly see anyone in the public eye oppose it – a notable exception being Peter Thiel in his recent book, From Zero to One.

Perhaps scientists share this attitude most of all. I encountered it at a recent popular science event in Berlin. A professional scientist at one of the local research institutes constructed an experiment “popcorn decay”. The aim of this experiment was to measure the popping of a population of popcorn kernels, and therefore show that popcorn popping was like radioactive decay, which is an intrinsically unpredicatable (i.e. random) process, and presumably therefore uncaused and meaningless. But it wasn’t really about popcorn – it was about everything in life. He wanted to convey an approach to thinking about all events in life – not just the popping of a popcorn.

What is randomness? What is probability?

What are we talking about when we say something is random? Basically it means that we cannot predict it. For example, when one flips a coin, it is very hard to predict how many times the coin will turn before it lands, and therefore one is completely ignorant about whether it turns out heads or tails.

The important thing to realise is that randomness is subjective. Just because you cannot predict an event, does not mean that nobody else can predict it, or that you could not predict it if you were given extra information or allowed to look at it in retrospect.

What is random for one person, may not be random for another. Imagine your dog barks some mornings for no apparent reason. On a given morning, you don’t know whether it will bark. This is random. Then you notice that the dog only barks when a certain postman delivers the mail. It is not random any more: you ask the postman for his schedule, and now you know when the dog is going to bark.

So, randomness means unpredictability. We call a variable random if we cannot predict it. It does not mean that it is not determined by some hidden cause or that it is meaningless. Only that we cannot predict it. Randomness is subjective. Randomness is not a physical property, it is a statement of ignorance.

The first person to make all this clear was Bayes. Bayesian statistics is a buzzword now (especially in this age of big data), so it is unfortunate that even most statisticians use him without really appreciating what he was talking about.

Probability is just a quantification of that ignorance. A probability of 50% means that one is completely ignorant about whether something will happen or not. So when the weather forecasters say 50% chance of rain, they are actually saying it might rain, and perhaps it often rains in circumstances that seem similar, but they are completely ignorant about whether it will rain or not.

Even famous scientists use the language of probability for deceptive purposes. Astronomer Royal, Prof Martin Rees caused a media sensation when he claimed in his book, that humanity has a 50% chance of extinction in this century. For most people, that conjures up a mental image of flipping a coin – heads: humanity lives, tails: humanity dies out. But it is not really like that. Most events in real life happen only once – and the 21^st century will only happen once. There is no meaningful analogy to flipping a coin. And Martin Rees is not privy to any secret knowledge about the dangers of human extinction that millions of other educated people have also studied. If he wanted to speak honestly, he would have said “I think there are real dangers facing us in the near future. I don’t know the future better than anyone else, but I feel completely unsure of whether humanity is going to survive this century.”

But what about quantum physics? Is there some level at which events are random in a physical sense, not just a subjective sense?

I studied physics at Oxford, which was a real traditional academic discipline, like almost no other subject. Of course we learned the Copenhagen interpretation of quantum physics, that a system is in a superposition of states until an observer “rolls the dice”, and the system randomly “collapses” into one or other state. This is of course famously illustrated by Schrödinger’s cat, which is killed if a radioactive nucleus decays. The fate of the cat is unknowable in advance, and its death is therefore somehow causeless and meaningless.

We learned that lots of people were once unhappy with this picture of things, and they came up with “hidden variable theories”, to reassure themselves that there was a hidden cause behind the cat’s fate. Einstein, was of course one of them, “God does not play dice,” he asserted. And we got a mental image of Einstein in the desert, like some biblical prophet, alone and mocked by his people. We learned that nowadays any self-respecting physicist ignores all that stuff. We got the vague impression that hidden variable theories were disproven by von Neumann. And even if they weren’t disproven, we should ignore them on the principle of Occam’s Razor.

As scientists we were also taught to think of all things objectively – that we would never learn the truth about an object if we failed to remove ourselves from the picture. Although we learned probability theory and the physics of stochastic (random) processes, the understanding that we gained was flawed. Random processes cannot be seen completely objectively, because randomness itself is a subjective experience.

The Copenhagen interpretation inspired the “many worlds” interpretation. In some ways, many worlds is a logical extension of the Copenhagen interpretation. It basically holds that everything in this world is without cause – that for every effect, the opposite effect might just as well have occurred, because both exist in parallel worlds that diverged from each other when the event took place.

All this is vastly removed from our everyday experience. Of course in real life too, many events have an uncertain outcome. I recently founded a start-up company with a couple of others in Berlin. Now, whether a start-up company succeeds or not is uncertain, even to its founders. However, the probability of it succeeding is perhaps conceptually different to the probability of a nucleus decaying. In a nuclear decay, the probability of decay in a given period is the same for all observers with access to a table of half-lives. However, in judging whether a company will succeed, all observers are not equal. My co-founders, investors and myself have privileged information. Other industry insiders and competitors may have other privileged information. Each observer’s understanding and judgement of the company, the service, the competitive situation and the industry varies immensely. Every observer may assign his own subjective probability of the company succeeding. Several years from now, when the fate of the company is clear, we may retrospectively analyse the history of the company and understand why it succeeded or failed. So at no point is there a roll of the dice. Eventually when we look back, we will understand the fate of the company, and conclude that given the environment and our actions, there could not have been the opposite outcome. The universe never divides into two parallel universes. There is only the gradual transition from unknown to known.

Randomness does not mean uncaused (an event can be unpredictable, but still have causes) – the de Broglie-Bohm theory and Grete Hermann

I believe that the understanding of probability and randomness which is normally taught to scientists is actually rather shallow. I think a much deeper view is that offered by Bayesian statistics: randomness is a property of the observer (ignorance), not a property of the system. When I first heard about Bayes, I thought that quantum mechanics must be the exception. So, shortly after I left academic science, I was fascinated to discover that there was an accepted interpretation of quantum mechanics – the de Broglie-Bohm interpretation – in which probability is once again a property of the observer and not of the system, and therefore compatible with Bayesian probability theory.

My understanding of de Broglie-Bohm is that a particle has both a definite position and a definite momentum. However its trajectory is determined not by Newton’s equation of motion, but by an equation of motion that depends on the wavefunction. The de Broglie-Bohm theory of the double slit experiment can be appreciated in a single figure:

Thus the “randomness” is from initial conditions, not from collapse of the wavefunction. It can therefore be called a “non-local” hidden variable theory, as unlike the “local” hidden variable theories. It is not disproven by Bell’s theorem. In other words, there’s no getting away from the wavefunction, but the wavefunction is deterministic: there’s no roll of dice.

But the trouble with the de Broglie-Bohm theory is that, even if it formally just a flavour of canonical quantum mechanics, it’s mathematically rather complicated, and therefore fails the test of Occam’s razor. When I cast my mind back to the electron-in-trap or the hydrogen atom, I wouldn’t dream of applying the de Broglie-Bohm theory. It’s just a more complicated version of the same quantum mechanics.

Recently I heard of Grete Hermann’s work. In the 1920s, Hermann disproved von Neumann’s theorem on hidden variables, as Bell did 20 years later, although her work remained obscure. Hermann also argued that although hidden variable theories are not forbidden, they are actually unnecessary for causality in quantum mechanics. The key is to make the difference between two concepts as we talk about randomness: causation and prediction. So it was impossible to predict the fate of Schrödinger’s cat, because that depended on the decay of a nucleus, which was physically unpredictable. But that doesn’t mean it was not caused: on the contrary, in quantum mechanics, the measurement itself gives us the deterministic cause. It is always possible to explain an event with classical physics after the fact, even if it is impossible before the fact.

Conclusions

1. Life is both random (hard for us to predict) and destined (there is no effect without cause)

I must admit that I have an ideological agenda in this. I want to believe that life is not “random”. All events in a person’s life happen only once: the fact that we can’t predict the future is because of our ignorance, not because the future is meaningless. And although we can learn a lot from statistics of a population, it does not show us the secrets of our own lives.

2. Proving the individuality of popcorn

All this gave me inspiration for an experiment to follow “popcorn decay”. Instead of showing that popcorn popping (like nuclei decay) is random, I would like to investigate whether popcorn popping can be predicted. Using as reproducible a heating method as I can find, and using a single variety of popcorn from a single batch, I’ll also measure everything that’s easy to measure about each kernel before popping it: weight, dimensions, density, color, defects, etc, and see how much of the variation in popping time can be explained by these parameters, using simple machine learning – i.e. multivariate linear regression.

Afterword: the meaning of a popcorn’s life

A criticism of all this discussion, normally given to me by scientists, is that all this is of no practical value. Popcorn individuality is therefore to be scorned in the same way that scientists are supposed to scorn all discussions of philosophy or faith. Belief in “truth” is useless – all we need is a practical theory, not “truth”.

Regarding the practical point of view, I have an additional perspective. I have been out of academic science now for 6 years, and I’m a start-up entrepreneur. I can say from experience that belief is very important practically for an entrepreneur: I have to have complete faith in the purpose of what I’m doing, because if I don’t, nobody else will, not investors, not colleagues, not customers. But isn’t doing science also an enterprise? Looking back, I feel that doing science requires conviction, even if the character of natural science itself is only to offer observable explanations, and not a deeper truth.