Author Archives: ssr

Ramadan predictions for the next 200 years

Prediction of Ramadan dates 2015 – 2215

In a previous post, I described a strikingly robust and elegant prediction for the Islamic calendar:

The new Islamic Lunar month begins at sunset of the day when the conjunction occurs before 12:00 Noon UTC

This is enough to predict/stansdardize Ramadan dates hundreds of years into the future (inshallah), using a high-accuracy ephemeris such as the NASA JPL DE430/431 and some data analysis software. Full results in CSV format can be downloaded at the link below. In addition, the table below contains the Ramadan start and end dates.

Download: Islamic calendar prediction 2015-2215 (CSV format)

 

Gregorian date Islamic month begins at sunset
17 June 2015 Ramadan
16 July 2015 Shawwal
05 June 2016 Ramadan
04 July 2016 Shawwal
26 May 2017 Ramadan
24 June 2017 Shawwal
15 May 2018 Ramadan
14 June 2018 Shawwal
05 May 2019 Ramadan
03 June 2019 Shawwal
23 April 2020 Ramadan
23 May 2020 Shawwal
12 April 2021 Ramadan
12 May 2021 Shawwal
01 April 2022 Ramadan
01 May 2022 Shawwal
22 March 2023 Ramadan
20 April 2023 Shawwal
10 March 2024 Ramadan
09 April 2024 Shawwal
28 February 2025 Ramadan
29 March 2025 Shawwal
17 February 2026 Ramadan
19 March 2026 Shawwal
07 February 2027 Ramadan
08 March 2027 Shawwal
27 January 2028 Ramadan
26 February 2028 Shawwal
15 January 2029 Ramadan
14 February 2029 Shawwal
04 January 2030 Ramadan
03 February 2030 Shawwal
25 December 2030 Ramadan
23 January 2031 Shawwal
14 December 2031 Ramadan
13 January 2032 Shawwal
03 December 2032 Ramadan
01 January 2033 Shawwal
22 November 2033 Ramadan
22 December 2033 Shawwal
11 November 2034 Ramadan
11 December 2034 Shawwal
31 October 2035 Ramadan
30 November 2035 Shawwal
19 October 2036 Ramadan
18 November 2036 Shawwal
09 October 2037 Ramadan
07 November 2037 Shawwal
29 September 2038 Ramadan
28 October 2038 Shawwal
18 September 2039 Ramadan
18 October 2039 Shawwal
06 September 2040 Ramadan
06 October 2040 Shawwal
27 August 2041 Ramadan
25 September 2041 Shawwal
16 August 2042 Ramadan
14 September 2042 Shawwal
05 August 2043 Ramadan
03 September 2043 Shawwal
25 July 2044 Ramadan
23 August 2044 Shawwal
14 July 2045 Ramadan
13 August 2045 Shawwal
04 July 2046 Ramadan
02 August 2046 Shawwal
23 June 2047 Ramadan
23 July 2047 Shawwal
12 June 2048 Ramadan
11 July 2048 Shawwal
01 June 2049 Ramadan
30 June 2049 Shawwal
21 May 2050 Ramadan
19 June 2050 Shawwal
10 May 2051 Ramadan
09 June 2051 Shawwal
29 April 2052 Ramadan
28 May 2052 Shawwal
19 April 2053 Ramadan
18 May 2053 Shawwal
08 April 2054 Ramadan
08 May 2054 Shawwal
28 March 2055 Ramadan
27 April 2055 Shawwal
16 March 2056 Ramadan
15 April 2056 Shawwal
05 March 2057 Ramadan
04 April 2057 Shawwal
23 February 2058 Ramadan
24 March 2058 Shawwal
12 February 2059 Ramadan
14 March 2059 Shawwal
02 February 2060 Ramadan
03 March 2060 Shawwal
22 January 2061 Ramadan
20 February 2061 Shawwal
11 January 2062 Ramadan
09 February 2062 Shawwal
31 December 2062 Ramadan
29 January 2063 Shawwal
20 December 2063 Ramadan
18 January 2064 Shawwal
08 December 2064 Ramadan
07 January 2065 Shawwal
28 November 2065 Ramadan
27 December 2065 Shawwal
18 November 2066 Ramadan
17 December 2066 Shawwal
07 November 2067 Ramadan
07 December 2067 Shawwal
26 October 2068 Ramadan
25 November 2068 Shawwal
15 October 2069 Ramadan
14 November 2069 Shawwal
04 October 2070 Ramadan
03 November 2070 Shawwal
24 September 2071 Ramadan
23 October 2071 Shawwal
12 September 2072 Ramadan
12 October 2072 Shawwal
02 September 2073 Ramadan
01 October 2073 Shawwal
23 August 2074 Ramadan
21 September 2074 Shawwal
12 August 2075 Ramadan
10 September 2075 Shawwal
31 July 2076 Ramadan
29 August 2076 Shawwal
20 July 2077 Ramadan
18 August 2077 Shawwal
09 July 2078 Ramadan
08 August 2078 Shawwal
29 June 2079 Ramadan
28 July 2079 Shawwal
18 June 2080 Ramadan
17 July 2080 Shawwal
07 June 2081 Ramadan
07 July 2081 Shawwal
28 May 2082 Ramadan
26 June 2082 Shawwal
17 May 2083 Ramadan
15 June 2083 Shawwal
05 May 2084 Ramadan
03 June 2084 Shawwal
24 April 2085 Ramadan
24 May 2085 Shawwal
14 April 2086 Ramadan
13 May 2086 Shawwal
03 April 2087 Ramadan
03 May 2087 Shawwal
23 March 2088 Ramadan
21 April 2088 Shawwal
12 March 2089 Ramadan
11 April 2089 Shawwal
01 March 2090 Ramadan
31 March 2090 Shawwal
18 February 2091 Ramadan
20 March 2091 Shawwal
08 February 2092 Ramadan
08 March 2092 Shawwal
27 January 2093 Ramadan
26 February 2093 Shawwal
17 January 2094 Ramadan
15 February 2094 Shawwal
06 January 2095 Ramadan
05 February 2095 Shawwal
27 December 2095 Ramadan
25 January 2096 Shawwal
15 December 2096 Ramadan
14 January 2097 Shawwal
04 December 2097 Ramadan
03 January 2098 Shawwal
23 November 2098 Ramadan
23 December 2098 Shawwal
13 November 2099 Ramadan
12 December 2099 Shawwal
02 November 2100 Ramadan
02 December 2100 Shawwal
23 October 2101 Ramadan
21 November 2101 Shawwal
12 October 2102 Ramadan
11 November 2102 Shawwal
02 October 2103 Ramadan
31 October 2103 Shawwal
20 September 2104 Ramadan
19 October 2104 Shawwal
09 September 2105 Ramadan
08 October 2105 Shawwal
29 August 2106 Ramadan
28 September 2106 Shawwal
19 August 2107 Ramadan
17 September 2107 Shawwal
07 August 2108 Ramadan
06 September 2108 Shawwal
28 July 2109 Ramadan
26 August 2109 Shawwal
17 July 2110 Ramadan
16 August 2110 Shawwal
06 July 2111 Ramadan
05 August 2111 Shawwal
24 June 2112 Ramadan
24 July 2112 Shawwal
14 June 2113 Ramadan
13 July 2113 Shawwal
03 June 2114 Ramadan
03 July 2114 Shawwal
24 May 2115 Ramadan
22 June 2115 Shawwal
13 May 2116 Ramadan
11 June 2116 Shawwal
02 May 2117 Ramadan
01 June 2117 Shawwal
21 April 2118 Ramadan
21 May 2118 Shawwal
10 April 2119 Ramadan
10 May 2119 Shawwal
30 March 2120 Ramadan
28 April 2120 Shawwal
19 March 2121 Ramadan
18 April 2121 Shawwal
09 March 2122 Ramadan
07 April 2122 Shawwal
26 February 2123 Ramadan
28 March 2123 Shawwal
16 February 2124 Ramadan
16 March 2124 Shawwal
04 February 2125 Ramadan
05 March 2125 Shawwal
24 January 2126 Ramadan
22 February 2126 Shawwal
13 January 2127 Ramadan
11 February 2127 Shawwal
02 January 2128 Ramadan
01 February 2128 Shawwal
22 December 2128 Ramadan
20 January 2129 Shawwal
12 December 2129 Ramadan
10 January 2130 Shawwal
01 December 2130 Ramadan
31 December 2130 Shawwal
20 November 2131 Ramadan
20 December 2131 Shawwal
08 November 2132 Ramadan
08 December 2132 Shawwal
28 October 2133 Ramadan
27 November 2133 Shawwal
18 October 2134 Ramadan
16 November 2134 Shawwal
07 October 2135 Ramadan
06 November 2135 Shawwal
26 September 2136 Ramadan
26 October 2136 Shawwal
15 September 2137 Ramadan
15 October 2137 Shawwal
05 September 2138 Ramadan
04 October 2138 Shawwal
25 August 2139 Ramadan
23 September 2139 Shawwal
13 August 2140 Ramadan
11 September 2140 Shawwal
02 August 2141 Ramadan
01 September 2141 Shawwal
23 July 2142 Ramadan
21 August 2142 Shawwal
13 July 2143 Ramadan
11 August 2143 Shawwal
01 July 2144 Ramadan
31 July 2144 Shawwal
20 June 2145 Ramadan
20 July 2145 Shawwal
09 June 2146 Ramadan
09 July 2146 Shawwal
29 May 2147 Ramadan
28 June 2147 Shawwal
18 May 2148 Ramadan
16 June 2148 Shawwal
08 May 2149 Ramadan
06 June 2149 Shawwal
27 April 2150 Ramadan
27 May 2150 Shawwal
17 April 2151 Ramadan
16 May 2151 Shawwal
05 April 2152 Ramadan
05 May 2152 Shawwal
25 March 2153 Ramadan
24 April 2153 Shawwal
14 March 2154 Ramadan
13 April 2154 Shawwal
04 March 2155 Ramadan
02 April 2155 Shawwal
21 February 2156 Ramadan
22 March 2156 Shawwal
10 February 2157 Ramadan
11 March 2157 Shawwal
30 January 2158 Ramadan
01 March 2158 Shawwal
20 January 2159 Ramadan
18 February 2159 Shawwal
09 January 2160 Ramadan
08 February 2160 Shawwal
28 December 2160 Ramadan
26 January 2161 Shawwal
17 December 2161 Ramadan
16 January 2162 Shawwal
06 December 2162 Ramadan
05 January 2163 Shawwal
26 November 2163 Ramadan
26 December 2163 Shawwal
15 November 2164 Ramadan
14 December 2164 Shawwal
04 November 2165 Ramadan
04 December 2165 Shawwal
24 October 2166 Ramadan
23 November 2166 Shawwal
13 October 2167 Ramadan
12 November 2167 Shawwal
01 October 2168 Ramadan
31 October 2168 Shawwal
21 September 2169 Ramadan
20 October 2169 Shawwal
10 September 2170 Ramadan
10 October 2170 Shawwal
31 August 2171 Ramadan
30 September 2171 Shawwal
20 August 2172 Ramadan
18 September 2172 Shawwal
09 August 2173 Ramadan
07 September 2173 Shawwal
29 July 2174 Ramadan
27 August 2174 Shawwal
18 July 2175 Ramadan
17 August 2175 Shawwal
07 July 2176 Ramadan
05 August 2176 Shawwal
26 June 2177 Ramadan
26 July 2177 Shawwal
16 June 2178 Ramadan
15 July 2178 Shawwal
05 June 2179 Ramadan
05 July 2179 Shawwal
25 May 2180 Ramadan
23 June 2180 Shawwal
14 May 2181 Ramadan
12 June 2181 Shawwal
03 May 2182 Ramadan
01 June 2182 Shawwal
22 April 2183 Ramadan
22 May 2183 Shawwal
11 April 2184 Ramadan
10 May 2184 Shawwal
01 April 2185 Ramadan
30 April 2185 Shawwal
21 March 2186 Ramadan
19 April 2186 Shawwal
10 March 2187 Ramadan
09 April 2187 Shawwal
27 February 2188 Ramadan
28 March 2188 Shawwal
15 February 2189 Ramadan
17 March 2189 Shawwal
05 February 2190 Ramadan
06 March 2190 Shawwal
25 January 2191 Ramadan
24 February 2191 Shawwal
15 January 2192 Ramadan
13 February 2192 Shawwal
03 January 2193 Ramadan
02 February 2193 Shawwal
24 December 2193 Ramadan
22 January 2194 Shawwal
13 December 2194 Ramadan
11 January 2195 Shawwal
02 December 2195 Ramadan
31 December 2195 Shawwal
20 November 2196 Ramadan
20 December 2196 Shawwal
10 November 2197 Ramadan
09 December 2197 Shawwal
30 October 2198 Ramadan
29 November 2198 Shawwal
20 October 2199 Ramadan
18 November 2199 Shawwal
09 October 2200 Ramadan
08 November 2200 Shawwal
28 September 2201 Ramadan
28 October 2201 Shawwal
17 September 2202 Ramadan
17 October 2202 Shawwal
07 September 2203 Ramadan
06 October 2203 Shawwal
26 August 2204 Ramadan
24 September 2204 Shawwal
16 August 2205 Ramadan
14 September 2205 Shawwal
06 August 2206 Ramadan
04 September 2206 Shawwal
26 July 2207 Ramadan
24 August 2207 Shawwal
14 July 2208 Ramadan
12 August 2208 Shawwal
03 July 2209 Ramadan
02 August 2209 Shawwal
22 June 2210 Ramadan
22 July 2210 Shawwal
12 June 2211 Ramadan
11 July 2211 Shawwal
01 June 2212 Ramadan
30 June 2212 Shawwal
21 May 2213 Ramadan
20 June 2213 Shawwal
11 May 2214 Ramadan
09 June 2214 Shawwal
30 April 2215 Ramadan
29 May 2215 Shawwal

 

How to predict the Islamic calendar with a high-accuracy ephemeris

Like most Muslims around the world, my daily life is measured by a combination of the Gregorian Calendar and Coordinated Universal Time (UTC) with time zones. There is only one exception: every year I fast in Ramadan, the dates of which are not standardized. All over the world, Muslim countries and Islamic institutions appoint their own arbiters for the dates of Ramadan, and the result is that:

  1. The 1st of Ramadan and Eid al-Fitr (1st of Shawwal) are unpredictable. This is inconvenient.
  2. We end up starting and finishing Ramadan on different days (spread over about 4 days). This, I feel, is a greater shame, as I would like to be able to share the fast with my family in different countries and with Muslims locally, wherever I am.

So I studied this matter, with the aim of predicting standard dates for Ramadan in the future. I leave it to the reader to choose whether to consider this as a standardization or a prediction of Ramadan. The method and results are below.

New moon: conjunction and sighting, Islamic law and conventional practice

Most Muslim scholars consider that the new month of the Islamic calendar begins with the sighting of the new moon. Astronomically, the new moon is the conjunction of the moon and the sun: in terms of astronomical coordinates, the conjunction occurs when the right ascension of the moon equals the right ascension of the sun. But the trouble with the conjunction is that it is invisible: the moon is aligned with the sun, and sets simultaneously with the sun. It only becomes visible later to the naked eye, typically 18 hours later, as a thin crescent.

Telescope image of the new moon, perhaps about a day old

 

As is recorded in the Hadith, “Allah’s Messenger (peace be upon him) said: Whenever you sight the new moon (of the month of Ramadan) observe the fast, and when you sight it (the new moon of Shawwal) break it, and if the sky is cloudy for you, then observe the fast for thirty days.”

This is taken by most Muslim scholars today as a basis for the opinion that one must wait for the sighting of the moon, and it is wrong to base the calendar on standards or calculations. On the other hand, there have been many standards of the Islamic calendar historically. Note that the Qur’an gives an impression which is quite different from the conventional opinion:

The sun and the moon [move] by precise calculation (Qur’an, Surat Ar-Raĥmān 55:5)

and

It is He who made the sun a shining light and the moon a derived light and determined for it phases – that you may know the number of years and account [of time]. Allah has not created this except in truth. He details the signs for a people who know  (Qur’an, Surat Yūnus 10:5)

This is a complicated subject fraught with disagreements, so I refer the reader to much better articles on the Islamic perspectives:

My view is that, as long as we wait for the sighting of the moon (or follow other people who claim to have sighted the moon), we will never start and finish Ramadan on the same day. Consider the following:

  1. The world is round and there are different time zones. Suppose the moon is sighted in Mecca (or further east, say in Indonesia) after sunset. However, at the same moment, in Hawaii (or North America), the sun has already risen, and Muslims have started the fast. Should they then break their fast immediately, or wait for the following sunset? Even if all Muslims around the world followed a single human authority on the moon sighting, this means that they would still break their fasts on separate days.
  2. Humans will never agree on the moon sighting, even in the same location, under clear skies, because the moon sighting depends on one’s eyesight. At the start and end of Ramadan, people compete with each other to be the first to see the new moon, with the result that 1st Ramadan and 1st Shawwal are declared while it is actually still impossible for 99% of ordinary people to see the new moon with the naked eye.

How to standardize the Islamic calendar so that all Muslims can fast on the same dates?

I recent read a proposed standardization of Islamic calendar dates, which I find to be strikingly simple, elegant and robust:

The new Islamic Lunar month begins at sunset of the day when the conjunction occurs before 12:00 Noon UTC.

This standardization was originally proposed by the Fiqh Council of North America in 2006. However, they later abandoned it because they were roundly criticized by traditionalists. So, let’s neglect their role in it, and consider it only on its own merits.

Firstly, why “12 Noon Coordinated Universal Time (UTC)”? The advantage of UTC is that it is an astronomical standard, also known as GMT. At this moment, the whole world is on the same the same day of the week and the same date in the Gregorian calendar. This would not be the case if we considered another place or point in time: sunset in Mecca is not a standard time. Sunset in Mecca varies in time of day, and at that moment of sunset, midnight has already passed in the Far East, which is then on the next day of the Gregorian calendar.

Secondly, why “conjunction before Noon”? The result of this is that somewhere on the Earth’s surface there will probably be a sighting of the moon on the same date in the Gregorian calendar. It also corresponds pretty well with the actual dates that are declared.

Hypothetically, if we consider this proposal, we can see how easy it is to predict Ramadan in the future.

Ephemeris: the motion of the heavenly bodies

Three institutions in the world publish a high-accuracy ephemeris. These are: the NASA Jet Propulsion Laboratory’s DE (Development Ephemeris) series ephemerides, the Paris Observatory (Intégrateur Numérique Planétaire de l’Observatoire de Paris, or INPOP) and the Institute of Applied Astronomy in St. Petersburg (Ephemerides Planets-Moon, or EPM). I am no expert on these models, but see here for a comparison. Of course, nobody knows the future except Allah, and we should say “inshallah” even for a high-accuracy planetary ephemeris. However, the ayats from the Qur’an quoted above, still encourage us to think of the motion of heavenly bodies as standards in time. The JPL Ephemeris is believed to predict the moon position to an accuracy within about 20 m in the coming century.

Predicting the Islamic calendar with the NASA JPL Development Ephemeris

So hypothetically, let’s see how easy it is to predict the Islamic calendar with NASA’s JPL Development Ephemeris. The JPL maintains a website (and telnet service) Horizons, which uses the latest version of their model DE430/DE431. Try the following:

  1. Visit http://ssd.jpl.nasa.gov/horizons.cgi or use telnet to ssd.jpl.nasa.gov, port 6775
  2. Select the following options to retrieve the positions (Right Ascension) of the Sun and Moon at 12:00 Noon UTC in astronomical coordinates.
    Ephemeris Type: OBSERVER
    Target Body: Sun [Sol] [10] and Moon [Luna] 301
    Observer Location: Geocentric [500]
    Time Span: Start=2015-06-15 12:00, Stop=2215-07-20 12:00, Step=1 d
    Table Settings: QUANTITIES=2 (RA and DEC); angle format=DEG; extra precision=YES; CSV format=YES

Horizons allows you to download the data in CSV format by FTP. (One can also use other astronomical software that employs a high-accuracy ephemeris.) Then take the difference between the RA of the moon and RA of the sun. Whenever the difference RA(moon) – RA(sun) passes through zero, the conjunction has occurred. If therefore, at 12:00 Noon UTC the difference is positive, a new moon has been born before noon, and that day can be counted as the new day of the Islamic calendar, according to the standard.

Islamic calendar prediction 2015 – 2215 using JPL DE + Matlab

Plotting the difference RA(moon) – RA(sun) at 12:00 Noon UTC each day produces the following graph. Where RA(moon) – RA(sun) is negative, the moon is waning, where RA(moon) – RA(sun) is positive, the moon is waxing. the first noon where RA(moon) – RA(sun) is positive is counted as the first day of the new Islamic month. Below is the prediction for Ramadan 1437 AH: begin on 5 June 2016 (sunset) and end on 4 July 2016 (sunset), in all time zones.

Islamic calendar Ramadan 2016

 

You too can predict the Islamic calendar using Matlab and the Horizons output. First download the data for the sun and moon and unzip. Then follow the instructions below.

Download JPL Horizons Sun and Moon RA and DEC every day at noon 2015-2215 (zipped CSV files)

 

 

1. Import Horizons output

Run the following Matlab code to import the Horizons data files into Matlab variables ra_sun and ra_moon:

%% import data
delimiter = ',';
startRow = 35;
endRow = 73083;
formatSpec = '%s%s%s%s%s%[^\n\r]';

filename = 'luna wld20518.16.txt';
fileID = fopen(filename,'r');
textscan(fileID, '%[^\n\r]', startRow-1, 'ReturnOnError', false);
dataArray = textscan(fileID, formatSpec, endRow-startRow+1, 'Delimiter', delimiter, 'ReturnOnError', false);
fclose(fileID);
date = datetime(dataArray{1});
ra_moon = str2num(char(dataArray{4}));
dec_moon = str2num(char(dataArray{5}));

filename = 'sun wld20518.15.txt';
fileID = fopen(filename,'r');
textscan(fileID, '%[^\n\r]', startRow-1, 'ReturnOnError', false);
dataArray = textscan(fileID, formatSpec, endRow-startRow+1, 'Delimiter', delimiter, 'ReturnOnError', false);
fclose(fileID);
date = datetime(dataArray{1});
ra_sun = str2num(char(dataArray{4}));
dec_sun = str2num(char(dataArray{5}));

2. Calculate relative right ascension, new moon dates and Islamic month order

ra_rel = angle(exp(1i*(ra_moon-ra_sun)/180*pi))/pi*180;

waxing = sign(double(ra_rel>=0));
moonborn = find([0;diff(waxing)]>0);
moonfull = find([0;diff(waxing)]<0);
Islamic_month_begins_at_sunset_of_Gregorian_date = date(moonborn);
ra_rel(moonfull) = NaN;
months = {'Muharram', 'Safar', 'Rabi'' I', 'Rabi'' II', 'Jumada I', 'Jumada II', 'Rajab', 'Sha''ban', 'Ramadan', 'Shawwal', 'Dhu al-Qa''dah', 'Dhu al-Hijjah'}';
newmonth = mod((1:length(Islamic_month_begins_at_sunset_of_Gregorian_date))+7,12)+1;
Islamic_month = months(newmonth);

3. Plot calendar

figure
plot(date,ra_rel)
hold on
plot(Islamic_month_begins_at_sunset_of_Gregorian_date,zeros(size(moonborn)),'sk')
ylabel('RA(moon) - RA(sun)')
grid on

4. Write output table (CSV)

monthtable = table(Islamic_month_begins_at_sunset_of_Gregorian_date, Islamic_month);
writetable(monthtable, 'Islamic_months_2015-2215.csv')

Full results

Full results in my separate post.

Random Business School case generator

Viewing the world through the limited lens of HBS

During my MBA year, we were made to read lots of case studies. The largest single source of these case studies was, of course, Harvard Business School. After reading a few of these cases, my classmates and I started to notice that they always sounded the same, whatever the subject matter and nature of the business that they dealt with. I soon started to wonder if it would be possible to write a random case generator and sell it to Harvard Business School. After all, selling case studies to other business schools is undoubtedly a significant money-earner for HBS. However, as one of my classmates quipped, they clearly already have a random case generator.

Anyway, just to make the point, I decided to write my own “Random Business School” case generator, (inspired by the Postmodernism Generator):

http://random-business-school.samsons.co/

The similarity to the HBS cases is intentional. Enjoy!

The Random Business School case generator is a Ruby on Rails web app. Credits are due to Succubus, a random text generator in Ruby.

 

Popcorn popping: prediction and causality

Free will vs. determinism

When heated to 200°C, popcorn kernels tend to pop at random times between about 40 s and 2 minutes. But what is this “randomness”? My co-worker Jesús MB and I had different points of view. Jesús previously ran a popcorn decay experiment to make the link between popcorn and radioactive nuclei. The decay of a nucleus is a physically unpredictable process, according to the laws of quantum physics. However, unpredictable does not necessarily mean uncaused. As I argued in a previous post, even quantum mechanics allows causality.

Now, a popcorn is a macroscopic object containing a hierarchical biological structure, mainly composed of starch granules, a little moisture, and a hard impermeable shell. Surely there must be a cause of its popping time (at well-defined temperature) that we can find by studying its physical properties? Moreover, maybe there are measurable characteristics of the unpopped popcorn, that will allow us to predict the popping time, and eliminate the randomness wholly or partially?

Jesús and I got together at the Berlin Science Hack Lab to put these questions to an experimental test. Thanks also to Narek B who joined us on the day to help with the experiments.

Popping individual popcorn kernels and predicting the popping time

The idea was to use a hot air blower as shown below, to pop individual kernels in a reproducible manner. We first measured the height, weight and photograph each kernel, before popping it and recording the popping time. This information allows us to calculate the volume, density and 2D shape of the kernel also.

Furthermore we photographed each kernel after popping. This may give us more information on the physical properties of the kernel which was not practical to measure before popping.

It look a whole day to collect 36 data points (and one kernel did not pop). Then we used a simple machine learning technique: multiple linear regression (MLR) with related methods to try to predict popping time from the physical data.

machine

Popcorn machine: consists of a hot air blower blowing into an aluminium drum. The machine reaches a steady state temperature of 200 C within about 2 minutes. The popcorns bounce around in the hot air, sampling an average temperature in the drum.

 

What do the photos of popcorn tell us?

Below are the images of a few unpopped kernels. They differ slightly in shape: some are more or less rounded or symmetrical. They differ slightly in size and mass. We know that the vast majority of kernels split at the yellow round end, not the pointy white end.

image12 image11 image10

image9 image8 image7

However there is more variation after the pop. Most popcorns look like an “octopus” with 3 or 4 legs, but some have a “blocky” or “blotchy” appearence. See photos below.

image3

“Octopus”: the kernel splits symmetrically, and the hot starch solution expands in a smooth round bulge. 3 or 4 sections of the shell peel back, like a banana peel or legs of an octopus.

image5

“Blocky”: the expanding starch solution is not always uniform. Sometimes cracks appear, leading to the “blocky” appearance.

image4

“Blotchy”: sometimes the shell (mainly at the pointy white end) disintegrates too, leading to a blotchy appearence.

Multiple linear regression with cross-validation: mass, height, area, volume and density don’t help us predict popping time

The first model correlates the mass, height, area, volume and density of the kernel against its popping time. We used “leave-one-out” cross-validation to ensure that we were not cheating ourselves: each data point n was predicted using a model calibrated on all the other data excluding point n). The model is poor as shown below. R-squared is 0.05. The standard error of the prediction is 25 s, which is indeed worse than if we just used the mean popping time as a prediction (error 22 s).

image2

MLR model of popping time based on mass, height, area, volume and density of the kernel. No predictive value.

2D Kernel shape also doesn’t predict popping time

We photographed and traced all kernel outlines, rotated and centered them, and made a Fourier decomposition of their shapes to be able to use the shape (at low spatial frequencies) in our model:

image13

Popcorn kernel outlines

image14

Fourier decomposition (and reconstruction) of a kernel outline (up to circular frequency = 5)

Unfortunately this was also not helpful in making a useful prediction. We added Fourier components up to circular frequency = 5 (i.e. 11 components: 5 sine, 5 cosine and 1 constant) to the predictor data above. The extra data necessitates some more sophistication: we used instead a principal components analysis followed by MLR (PCA-MLR) with cross-validation. 6 components worked best. However, the prediction is still useless: a standard error of 23 seconds shows this model is better than the earlier model, but still worse than just the mean.

image16

image15

PCA-MLR model based on mass, height, area and 2D spatial information. Still no useful prediction.

Some “after the fact” information is correlated to popping time

Inspection of the popping data shows that, in general, “blocky” and “blotchy” take longer to pop than “octopus” individuals. The PCA-MLR model (3 PCs with cross-validation) is shown below, to which the popped shape data has been added to the mass, height, area, volume, density data. This is our only model that shows a useful prediction (standard error of 19 s) compared to the mean (22 s).

image6

“After the fact” model: “blocky” and “blotchy” take longer to pop than “octopus” individuals. The PCA-MLR model shows an improved prediction than the mean.

Conclusion: popcorn popping time is caused but unpredictable

When we started this experiment, we hoped that we would be able to predict popping time from measurements on the unpopped kernels. However, the only useful measurement that we found came from a measurement on the popped kernel (its shape classification), which is of course impossible to measure before popping. So this counts as a causation but not a prediction!

This experiment was just for fun, of course. But it is interesting that the analogy to quantum mechanics and radioactive nuclei has held true. We have found a (partial) cause of the popping time, but no way of predicting it!

The story of sciencebite: an internet marketplace for scientific expertise

Sciencebite was an internet startup in operation from April 2014 to March 2015, which aimed to provide an open marketplace for scientific expertise to industrial R&D. I led the startup from its conception; we won the support of a prominent angel investor in Berlin: the team numbered five people by the time it closed. Although we did not succeed in realising a successful marketplace, we learned much through building and launching two web applications that reached many scientists and R&D-led companies. The aim of this article is to share some of these learnings: especially about the ways in which scientists are bound to their conventional working practices, despite a desire to advance their careers in an open way.

The Sciencebite hypothesis: an open marketplace that would add value to the interface between industrial R&D and scientific experts

We started with the ideal of founding an “open expertise” platform that incentivized scientists to promote themselves based on their expertise, and take R&D out of secretive corporate relationships and onto the open internet. Until now there is little open exchange on the internet between industrial R&D and scientific expertise. The best examples are the “open innovation” crowdsourcing platforms, including InnoCentive and NineSigma, which are well established but serve only a tiny proportion of the world’s R&D needs. Rather, the main exchange between R&D and scientific expertise, which includes the job market, is fairly secretive.

A great source of inspiration for us was the recent trend of open internet platforms for other professions, such as StackExchange for programming knowhow, and Elance for software / web design freelancing.  We admired the way software engineers could build their careers, make connections, and even sell their services using online tools, by being open about their expertise and contributing to open-source projects. Was there any reason why this culture should not also work for scientific R&D?

The following figure shows sciencebite’s space as we see it. A great variety of scientific expertise is used in industrial R&D, and there are many successful examples of open internet marketplaces for different professions. However there is little in the way of an open internet connection between scientific expertise and industrial R&D (X on the diagram). On the other hand, there are many successful service companies providing R&D expertise secretively (Y on the diagram), and there are successful open platforms providing open sharing in academic science like ResearchGate, Mendeley and Academia, but do not provide expertise to industry (Z on the diagram).

hypthesis

Our first product iteration: an idealistic communication tool for scientists to share knowhow, consult privately and refer colleagues

Our first product was full of idealism and hope for better communication between scientists. Launched July 2014 (private beta), the web application allowed users (mainly from Industry)  to find scientists, ask them questions, and invite them to share publicly, consult privately (under our standard NDA and Service Contract) or refer colleagues for a share of the benefit. A sketch is shown in Appendix A. We built the platform using the current best practices for rapidly prototyping a web app: a Ruby-on-Rails back-end, Bootstrap front-end, hosted on Heroku, and connecting to external identity services (LinkedIn), external data sources (Pubmed and BASE), and an external search service (based on Solr), and email based workflows.

We launched this product to hundreds of relevant target users recruited mainly through our extended network of R&D professionals (applied scientists, engineers, project managers and executives in R&D). A considerable number of users came from our TechCrunch Disrupt San Francisco exposure, the related Wall Street Journal Germany article and subsequent events in Berlin and Europe where we took part (e.g. WebSummit Dublin, Slush Helsinki). We also reached a variety of potential experts for one-on-one interviews, including academic scientists, employees in R&D companies and independent consultants.

The results were that a small number of our industrial contacts soon placed requests with us, to source expertise for immediate problems that they had. We found that, when we had these requests, it was easy to recruit junior scientists who were willing to consult. However, senior scientists or established consultants tended not to respond. The other functions we built were trickier to get right: few scientists made referrals and none shared knowhow openly on our site. We discovered many assumptions about our users were incorrect, with regard to their working practices and the user experience. For a start, we embraced the use open identities through LinkedIn as a way of verifying our user base, however most target customers wanted to deal with experts anonymously – therefore our use of open identities was a serious discouragement to many corporate R&D users. On the other side, scientists were not motivated to share publicly because they would have all the exposure of a journal article, and none of the academic benefits. We also found that users took too much care choosing scientists, and too little care to ask a good question. Regarding the search functionality, we thought that a minimal viable product would be to search open academic data with Solr, a state-of-the-art open source search engine. Unfortunately, this was not enough, as it did not give us a competitive advantage on Pubmed / Google Scholar, even though ours focused on people rather than papers. It was particularly problematic to deal with all the kinds of search terms that users threw at us, which were very diverse: from discipline expertise (e.g. microbiology) to full questions (e.g. how to estimate lifetime of a plastic part under stress?) or specific keywords (e.g. molecular drug targets).

Following our participation in the Google LaunchPad Berlin in October 2014, we reviewed our assumptions by conducting additional persona interviews with 5 target groups: Junior Scientists, R&D Managers, PhDs, PostDocs and Freelance Consultants.

We shifted our focus to where we found the traction – the need for short consultations with experts, and depreciate the other features. Our literature search function was replaced with a curated database of scientists that we recruited. (We planned to bring back the literature search when we had more resources to improve the search technology and data sources.) From the persona interviews, we decided to target specifically R&D managers on the industry side (most direct need out of our users in industry), and specifically junior scientists on the expert side (most available for flexible consulting, easiest to recruit). Instead of approaching all industries, we decided to focus on biotech (large investment in R&D, multidisciplinary, very scientific).

Our second product iteration: a freelance consulting marketplace, for scientists to promote their profiles along with their availability for consulting

Thus came our second product, launched January 2015.  This product was more realistic and based on what our target users had asked for: a two-sided marketplace for expertise consultations. A sketch is shown in Appendix B. Scientists could register to present their profiles and availability (on our standard contract). Companies could browse our scientists, send them messages and request a consultation. We built a signup process to import profile data from Mendeley and LinkedIn, and profiles that emphasized a scientist’s skills.

We launched an outreach campaign, to assess the expertise needs of business owners and chief scientists at innovative biotech startups and SMEs, in Germany, Europe and the US. This was a great success – around 35 executives gave us in-depth interviews, and we offered each of them publicity in the form of a feature on our blog which we decided to focus on the biotech scene. We also got into Wired Germany’s feature “15 Ideas for a Better World” in February 2015. We experimented with social media advertising, leading to successful campaigns on Facebook and LinkedIn to recruit experts. We also recruited scientists and executives at academic conferences and trade shows – these events were a high investment for the number of users, but they allowed us high-quality face-to-face feedback. We attempted to form university partnerships via several tech transfer / career offices of German universities, and joined BioDeutschland (the industry association for German biotech) to promote ourselves to its member companies.

Our earlier successes with finding companies to request consultations were unfortunately not repeated, perhaps because the new contacts were from cold contact. Biotech startups all said they would find expertise within their networks (and not pay for it), and none placed a request with us. Large companies said the same and commented that they invest in managing their expertise networks, and therefore doubted our service would work for large companies.  Our social media ads showed many scientists to be curious about this type of work, and enthused by our idea, but focusing on freelance consulting was not an option for them – they were much more worried about finding permanent jobs. Our newly recruited scientists differed widely in how much data they would put on their profiles – only a few wrote a lot about their skills. However, none were contacted by target companies, either via our own contacts or organically from their profile as an internet landing page. Content marketing on our blog resulted in bursts of visits, with around 5% of visits converted into use of website (i.e. search on web app).

Many scientists showed they were unsure about whether they were even allowed to do paid work outside their research institution; contacts at university commercialization offices also expressed these concerns. A couple of companies reported that they do use flexible labor from junior scientists, but not as short consultations, rather as occasional project work lasting weeks or months. University commercialisation / career offices showed polite interest in our idea but opposed their own PhDs consulting without their involvement (fear of IP being shared, legal regime not authorizing such consulting engagements, etc.)

Our main learning was that, unfortunately, the market for junior scientist skills on a consultation basis is very limited. Junior scientists were difficult for us to sell to biotech companies, because these companies already have practices of acquiring knowhow, and what we offered them was not powerful enough to make them change their practices.

Variation of the second product: a freelance jobs marketplace

Without developing a new product, we investigated whether our platform could be adapted for jobs, particularly freelance/flexible jobs, i.e. engagements lasting from a few weeks to a few months. This idea was marketed from February 2015. We set up meetings with several HR managers in large and medium-sized German biotech companies, who were very open to discussing the market with us. The HR managers were generally friendly to the idea of an open marketplace for scientific labor. However, they were unconcerned with freelance jobs. Most said that, whenever they have an opening, they have no problem having scientists come to them. We also targeted recruiters who had recently posted entry-level scientific jobs. The response from this group was disappointing: despite being lower in the corporate hierarchy, they were very difficult to reach, probably because they are used to being chased by job applicants and staffing agents. The few we reached were unenthusiastic, because the balance of power is already in their favour, they did not need to actively look for candidates. Moreover, there were many criteria they wished to see in a candidate outside what was provided in our profiles. Some feedback we got from scientists suggested their availability for flexible jobs was likely to be scarcer than for short consultations.

We were forced to acknowledge that there simply isn’t a significant market for short term scientific jobs in R&D, and the current Sciencebite profiles were not sufficient to convince companies to try hiring junior scientists in this way.

Conclusion – we did not succeed in developing  a model that connects industrial clients to scientific expertise in a viable way

After careful review of alternative options, we have to admit we have exhausted the product options available to us without finding a model that connects industrial clients to scientific experts in a viable way. Each model has met roadblocks that are at present difficult to overcome.

Model Roadblock
Freelance marketplace insignificant market for consultations or flexible scientific work in R&D
Expertise search insignificant market for expertise of junior scientists (and far behind competitors as a general expertise search platform)
Q&A (open knowhow) little incentive for scientists to share outside academic system; inhibition to ask open questions in R&D
Promotional platforms unclear competitive advantage of the current Sciencebite profiles, no particular evidence that developing a better profile would be profitable
Crowdsourcing far behind competitors, no particular inspiration for original product
Networking platforms
Job marketplaces

We believe that the failure of our initiatives cannot be attributed to execution. We worked marvelously as a team, built and launched the products in a relatively short time, and promoted them to many people. Even if we had reached 10x as many target companies; even if we had spoken to 10x as many people within those companies; even if we had recruited 100x as many scientists, or scientists only from top institutions; even if our web application had behaved slicker; even if there had been more ways for scientists to import their data; even if we had had higher search rankings, etc., it is difficult to imagine that the market reaction to our products would be any different.

Hypothetical technological advances could perhaps have given us a viable product: e.g. a radical new search technology or data source, but these were not the vision of the business – it was more about a marketplace than a technology.

Likewise, we cannot blame lack of finance. Our pre-seed fund has sufficed for us to build two web applications, and try all the marketing and business models that we conceived. If we had succeeded in raising a seed round, we could have built more product features and a slicker product, and scaled up the marketing, but it would not have helped with the roadblocks above.

So we could not prove the Sciencebite hypothesis because (1) the product vision was insufficient, (2) the market that we are going after hardly exists, and (3) we have not brought a radical advance that is able to quickly change R&D working practices.

Outlook: there is hope

Nevertheless, there are signs of hope. We discovered a universal interest from R&D professionals about connecting openly – many people told us that they can’t imagine why our concept did not exist already. There was a widespread need for expertise among companies, despite the skepticism that high quality knowhow could be provided via a website. We found a genuine desire among scientists to sign up, share their skills, and a curiosity that they could earn money this way. I believe that someday, someone will figure out a way to build a platform that caters for these needs, whether it is through a technology advance or an amazing insight about scientists’ behavior. I look forward to that day.

A. First product: share, refer and consult

1. Users were invited to enter a search query to find relevant scientists.

image2

2. The user could select one or more scientists to invite an answer.

image3

3. The recipient had three options: share publicly, creating a public page linked to their profile for self-promotion…

image4

… or advise privately in the Sciencebite project room, which provided standard contracts and payment services for R&D consulting…

image5

… or refer a colleague and take a share of the rewards if that colleague earned money through Sciencebite.

image6

B. Second product: two-sided marketplace

1. Users could search for a scientist.

image7

2. Scientists could present their profiles and availability for consulting.

image8

3. A user could request consultations from a scientist and buy their time.

image9

How do scientists share on academic social networks like ResearchGate?

A data analysis of user habits shows open sharing is mostly limited to publications; very few scientists are liberal with knowhow

A key aim of Sciencebite is to create a open platform for scientific expertise online. At the moment, Sciencebite is much smaller than the academic social networks, academia.edu, researchgate.net and mendeley.com. So I wondered how scientists behave on these networks, how they are promoting their identities, publications and expertise, and how they are connecting with each other.

Without wishing to comment on the competitive positions of the three largest academic networks, in this part of the world – Berlin – where I am based, ResearchGate is the one that we hear about the most, and with a claimed user count of > 6m academics from whitelisted institutions, it has certainly achieved an amazing penetration of the world’s academics. Speaking personally as someone with a background in academic science, I can confirm that every scientist that I know now seems to have a ResearchGate profile, although many do not actively engage with it.

So how are scientists behaving online, taking ResearchGate as the source of data? I wrote a script to browse ResearchGate profiles at random, and ran it twice, to look at the change over a year: firstly in November 2013 (sampling 3028 profiles), and secondly in February 2015 (sampling 3407 profiles). The sample represents some 0.1% of ResearchGate profiles, and leads to some pretty interesting conclusions about how scientists behave on the new internet platforms.

Scientists are increasingly sharing their professional identities online

ResearchGate has 6m users as of January 2015, and is currently growing at around 10k users per day. A growing minority of users share their professional identity on the site (Figure 1): 36% shared a profile picture in 2013, 43% in 2015. Fewer have updated their profiles with their current positions, but this too is growing: 7% in 2013 and 24% in 2015. It’s interesting that fewer have filled in a current position than uploaded a profile picture, which perhaps reflects that many users register with Facebook, or have left academia since joining ResearchGate. As of February 2015, 18% have both uploaded a profile picture and filled in their current position. If we consider this as a definition of an active user, we can estimate ResearchGate’s active user base as 1.1m.

Figure 1: identity sharing on ResearchGate

Figure 1: identity sharing on ResearchGate

 

A growing number of scientists are sharing full texts of their publications

We see an impressive engagement with sharing full texts of publications online (Figure 2). Some 41% of ResearchGate users had uploaded at least one full text by Nov 2013, growing to 80% by Feb 2015. Considering the rapid growth of ResearchGate signups in this time, it reflects an impressive rate of sharing among new users.

Figure 2: publication uploading on ResearchGate has acheived an impressive penetration.

Figure 2: publication uploading on ResearchGate has achieved an amazing penetration of the academic community. Now 80% of RG users have uploaded at least one full text.

It’s remarkable that so many more users have shared their publications (~3m) than filled in their profile information (~1m). However, given the importance of publications in the academic career path, it is perhaps less surprising: academics find it more important to share publications than to display their current position. This would surely be the opposite of user behavior on mainstream professional networks like LinkedIn.

In addition to the sampling of users, I sampled 1000 publications randomly on ResearchGate, looking at the relationship between publication date and sharing. In the past 3 years, a great number of new publications have been listed on ResearchGate, reflecting its deep penetration of the scientific community. We can also see that new publications are increasingly shared on ResearchGate – almost half of new publications in the past three years are uploaded (Figure 3).

fig3-publication-sharing-on-rg

Figure 3: the growth in shared full texts on ResearchGate reflects the amazing penetration of the academic community in recent years. Now almost 50% of new publications are shared openly as full text.

 

Almost nobody is using ResearchGate’s Open Reviews

ResearchGate launched a new feature in March 2014: Open Reviews, for scientists to openly peer-review each others’ papers after publication, for quality, originality, reproducibility, etc. Unfortunately almost nobody is using it. Only 4 users in my sample of 3407 have given an Open Review, and none came back to give a second.

The reason for this low adoption is probably that scientists get little career advancement from an Open Review. If they give a negative review, they may make enemies, and they gain none of the influence over a journal that they do in the traditional peer review system. Moreover, if they give a positive review, it does not help their publication and citation record, which are their main criteria to advance their careers.

 

Sharing of knowhow and advice is still lacking

Far fewer users engage with the Q&A forum on ResearchGate, in comparison with their sharing of publications (Figure 4). In November 2013, only 6.3% had ever asked, commented on, or answered a question. Even of those who had engaged with Q&A, very few were repeat users: 2.3% had ever asked a question, only 0.56% had come back to ask a second question, and only one in the entire sample could be really classified as an active user by asking more than 10 questions.

The situation had not improved much by February 2015. 6.6% had even asked/commented/answered. 2.8% had ever asked, and 1.1% had ever come back for a second. No users in the sample had asked more than 10 questions.

Why do so few scientists request and share practical knowhow and advice online? Perhaps it is the secretive culture of research, and the importance of publications versus other types of dissemination. Perhaps it is also because of the highly specialized nature of knowhow in science – valuable knowhow is often much more specialized than in other professions, for example in software development, where programmers enjoy a culture of sharing all kinds of practical information, through open source software, blogs and Q&A sites.

 

Figure 4: knowhow sharing on ResearchGate is still minimal. Only a very small proportion of users use the Q&A, and this proportion has changed little in 2013 - 2015

Figure 4: knowhow sharing on ResearchGate is still minimal. Only a very small proportion of users use the Q&A, and this proportion has changed little in 2013 – 2015

Conclusion: scientists are changing some of their habits, but only as far as the academic career path allows them

Great numbers of scientists are now sharing online, and in the past couple of years, it has become common to share publications openly on academic social networks like ResearchGate. However, not many scientists have changed their behavior beyond this – other types of sharing are still lacking, In particular, sharing of expertise and knowhow is still minimal.

I believe this represents a great challenge and opportunity to scientists who are interested in working differently. The trend of online sharing among scientists still overwhelmingly reflects the traditional academic career path, because it so strongly anchored to journal publications and citations. For all the frustrations of collaborating and making progress in science, there are perhaps still a great number of undiscovered solutions that would allow us to work together better.

The individuality of popcorn

or, why life can be both random and destined

There’s a widespread delusion that life is a like a lottery, and that events in the real world are determined by chance. I notice it when people talk about the future using the language of statistics: what are the chances of X, e.g. getting rich, becoming famous, having an accident? The unique case of the individual is replaced with a vague idea about the population.

This delusion is very common: we see it in popular culture, politicians, journalists and advertisers use it. I hardly see anyone in the public eye oppose it – a notable exception being Peter Thiel in his recent book, From Zero to One.

Perhaps scientists share this attitude most of all. I encountered it at a recent popular science event in Berlin. A professional scientist at one of the local research institutes constructed an experiment “popcorn decay”. The aim of this experiment was to measure the popping of a population of popcorn kernels, and therefore show that popcorn popping was like radioactive decay, which is an intrinsically unpredicatable (i.e. random) process, and presumably therefore uncaused and meaningless. But it wasn’t really about popcorn – it was about everything in life. He wanted to convey an approach to thinking about all events in life – not just the popping of a popcorn.

Every one counts

What is randomness? What is probability?

What are we talking about when we say something is random? Basically it means that we cannot predict it. For example, when one flips a coin, it is very hard to predict how many times the coin will turn before it lands, and therefore one is completely ignorant about whether it turns out heads or tails.

The important thing to realise is that randomness is subjective. Just because you cannot predict an event, does not mean that nobody else can predict it, or that you could not predict it if you were given extra information or allowed to look at it in retrospect.

What is random for one person, may not be random for another. Imagine your dog barks some mornings for no apparent reason. On a given morning, you don’t know whether it will bark. This is random. Then you notice that the dog only barks when a certain postman delivers the mail. It is not random any more: you ask the postman for his schedule, and now you know when the dog is going to bark.

So, randomness means unpredictability. We call a variable random if we cannot predict it. It does not mean that it is not determined by some hidden cause or that it is meaningless. Only that we cannot predict it. Randomness is subjective. Randomness is not a physical property, it is a statement of ignorance.

The first person to make all this clear was Bayes. Bayesian statistics is a buzzword now (especially in this age of big data), so it is unfortunate that even most statisticians use him without really appreciating what he was talking about.

Probability is just a quantification of that ignorance. A probability of 50% means that one is completely ignorant about whether something will happen or not. So when the weather forecasters say 50% chance of rain, they are actually saying it might rain, and perhaps it often rains in circumstances that seem similar, but they are completely ignorant about whether it will rain or not.

Even famous scientists use the language of probability for deceptive purposes. Astronomer Royal, Prof Martin Rees caused a media sensation when he claimed in his book, that humanity has a 50% chance of extinction in this century. For most people, that conjures up a mental image of flipping a coin – heads: humanity lives, tails: humanity dies out. But it is not really like that. Most events in real life happen only once – and the 21st century will only happen once. There is no meaningful analogy to flipping a coin. And Martin Rees is not privy to any secret knowledge about the dangers of human extinction that millions of other educated people have also studied. If he wanted to speak honestly, he would have said “I think there are real dangers facing us in the near future. I don’t know the future better than anyone else, but I feel completely unsure of whether humanity is going to survive this century.”

But what about quantum physics? Is there some level at which events are random in a physical sense, not just a subjective sense?

I studied physics at Oxford, which was a real traditional academic discipline, like almost no other subject. Of course we learned the Copenhagen interpretation of quantum physics, that a system is in a superposition of states until an observer “rolls the dice”, and the system randomly “collapses” into one or other state. This is of course famously illustrated by Schrödinger’s cat, which is killed if a radioactive nucleus decays. The fate of the cat is unknowable in advance, and its death is therefore somehow causeless and meaningless.

We learned that lots of people were once unhappy with this picture of things, and they came up with “hidden variable theories”, to reassure themselves that there was a hidden cause behind the cat’s fate. Einstein, was of course one of them, “God does not play dice,” he asserted. And we got a mental image of Einstein in the desert, like some biblical prophet, alone and mocked by his people. We learned that nowadays any self-respecting physicist ignores all that stuff. We got the vague impression that hidden variable theories were disproven by von Neumann. And even if they weren’t disproven, we should ignore them on the principle of Occam’s Razor.

As scientists we were also taught to think of all things objectively – that we would never learn the truth about an object if we failed to remove ourselves from the picture. Although we learned probability theory and the physics of stochastic (random) processes, the understanding that we gained was flawed. Random processes cannot be seen completely objectively, because randomness itself is a subjective experience.

The Copenhagen interpretation inspired the “many worlds” interpretation. In some ways, many worlds is a logical extension of the Copenhagen interpretation. It basically holds that everything in this world is without cause – that for every effect, the opposite effect might just as well have occurred, because both exist in parallel worlds that diverged from each other when the event took place.

All this is vastly removed from our everyday experience. Of course in real life too, many events have an uncertain outcome. I recently founded a start-up company with a couple of others in Berlin. Now, whether a start-up company succeeds or not is uncertain, even to its founders. However, the probability of it succeeding is perhaps conceptually different to the probability of a nucleus decaying. In a nuclear decay, the probability of decay in a given period is the same for all observers with access to a table of half-lives. However, in judging whether a company will succeed, all observers are not equal. My co-founders, investors and myself have privileged information. Other industry insiders and competitors may have other privileged information. Each observer’s understanding and judgement of the company, the service, the competitive situation and the industry varies immensely. Every observer may assign his own subjective probability of the company succeeding. Several years from now, when the fate of the company is clear, we may retrospectively analyse the history of the company and understand why it succeeded or failed. So at no point is there a roll of the dice. Eventually when we look back, we will understand the fate of the company, and conclude that given the environment and our actions, there could not have been the opposite outcome. The universe never divides into two parallel universes. There is only the gradual transition from unknown to known.

Randomness does not mean uncaused (an event can be unpredictable, but still have causes) – the de Broglie-Bohm theory and Grete Hermann

I believe that the understanding of probability and randomness which is normally taught to scientists is actually rather shallow. I think a much deeper view is that offered by Bayesian statistics: randomness is a property of the observer (ignorance), not a property of the system. When I first heard about Bayes, I thought that quantum mechanics must be the exception. So, shortly after I left academic science, I was fascinated to discover that there was an accepted interpretation of quantum mechanics – the de Broglie-Bohm interpretation – in which probability is once again a property of the observer and not of the system, and therefore compatible with Bayesian probability theory.

My understanding of de Broglie-Bohm is that a particle has both a definite position and a definite momentum. However its trajectory is determined not by Newton’s equation of motion, but by an equation of motion that depends on the wavefunction. The de Broglie-Bohm theory of the double slit experiment can be appreciated in a single figure:

The double-slit diffraction pattern according to the de Broglie-Bohm interpretation. The particle always has a definite position and momentum, and moves deterministically according to a function of the wave function. However its initial conditions are unknowable.

Thus the “randomness” is from initial conditions, not from collapse of the wavefunction. It can therefore be called a “non-local” hidden variable theory, as unlike the “local” hidden variable theories. It is not disproven by Bell’s theorem. In other words, there’s no getting away from the wavefunction, but the wavefunction is deterministic: there’s no roll of dice.

But the trouble with the de Broglie-Bohm theory is that, even if it formally just a flavour of canonical quantum mechanics, it’s mathematically rather complicated, and therefore fails the test of Occam’s razor. When I cast my mind back to the electron-in-trap or the hydrogen atom, I wouldn’t dream of applying the de Broglie-Bohm theory. It’s just a more complicated version of the same quantum mechanics.

Recently I heard of Grete Hermann’s work. In the 1920s, Hermann disproved von Neumann’s theorem on hidden variables, as Bell did 20 years later, although her work remained obscure. Hermann also argued that although hidden variable theories are not forbidden, they are actually unnecessary for causality in quantum mechanics. The key is to make the difference between two concepts as we talk about randomness: causation and prediction. So it was impossible to predict the fate of Schrödinger’s cat, because that depended on the decay of a nucleus, which was physically unpredictable. But that doesn’t mean it was not caused: on the contrary, in quantum mechanics, the measurement itself gives us the deterministic cause. It is always possible to explain an event with classical physics after the fact, even if it is impossible before the fact.

Conclusions

1. Life is both random (hard for us to predict) and destined (there is no effect without cause)

I must admit that I have an ideological agenda in this. I want to believe that life is not “random”. All events in a person’s life happen only once: the fact that we can’t predict the future is because of our ignorance, not because the future is meaningless. And although we can learn a lot from statistics of a population, it does not show us the secrets of our own lives.

2. Proving the individuality of popcorn

All this gave me inspiration for an experiment to follow “popcorn decay”. Instead of showing that popcorn popping (like nuclei decay) is random, I would like to investigate whether popcorn popping can be predicted. Using as reproducible a heating method as I can find, and using a single variety of popcorn from a single batch, I’ll also measure everything that’s easy to measure about each kernel before popping it: weight, dimensions, density, color, defects, etc, and see how much of the variation in popping time can be explained by these parameters, using simple machine learning – i.e. multivariate linear regression.

Afterword: the meaning of a popcorn’s life

A criticism of all this discussion, normally given to me by scientists, is that all this is of no practical value. Popcorn individuality is therefore to be scorned in the same way that scientists are supposed to scorn all discussions of philosophy or faith. Belief in “truth” is useless – all we need is a practical theory, not “truth”.

Regarding the practical point of view, I have an additional perspective. I have been out of academic science now for 6 years, and I’m a start-up entrepreneur. I can say from experience that belief is very important practically for an entrepreneur: I have to have complete faith in the purpose of what I’m doing, because if I don’t, nobody else will, not investors, not colleagues, not customers. But isn’t doing science also an enterprise? Looking back, I feel that doing science requires conviction, even if the character of natural science itself is only to offer observable explanations, and not a deeper truth.