Connectom

Powered by 🌱Roam Garden

Fooled by Randomness

Summary:: (in progress)

There are professions with low randomness (e.g. dentist) that can only be successful with skill; and professions with high randomness (e.g. day trader) that could see someone unskilled get successful purely based on luck and variance.

People often mistake luck and variance for skill, i.e. survivor-ship bias.

The absence of evidence is not evidence of absence. Just because we have only encountered white swans so far, doesn’t mean that all swans are white. It only takes one black swan to prove the hypotheses wrong.

The scientific method can only assess falsifiable hypotheses. Good hypotheses can be proven wrong quickly, like by a single black swan.

Theory of evolution postulates survival of the fittest, not the best. Sometimes things that survive the longest is the sub-optimal choice, e.g. QWERTY keyboard.

Nature and life are non-linear. We can build a sandcastle with a million grains of sand, yet adding another would bring the whole thing down.

The human mind is not evolved to handle sophisticated statistics and randomness. We depend on heuristics, e.g. availability heuristics where we value things that are easily visible more.

The heuristics cause us to attribute the wrong thing as a cause. e.g. our success is caused by our skills, but failure is bad luck

We are driven by emotion which helps us to make a quick decision, even though it is sub-optimal. Emotion influences our rational thinking rather than the other way around.

It is a lot easier to find ways to avoid triggering emotion rather than try to fight it; e.g. checking share portfolio once a month instead of daily to avoid impulse sale.

Hindsight bias is another big one that we need to be aware. Past events always seem predictable and easier to handle than they were at the time. We naturally find patterns where there were none.

Overfitting is when we try to find pattern, rule or formula to fit historical data too much that it becomes less able to predict future events.

We need to account for an unexpected event (black swan event). e.g. stock market crash and climate change.

Look for both chance and magnitude of an event. e.g. 99.9% chance of winning $1 with 0.1% chance of losing $10,000 looks good on paper, but the expected outcome is a loss of ~$9.

Too often we only looked at the chance and not enough of the magnitude. Rare events could be catastrophic.

Embrace beautiful randomness, e.g. poetry, art.

Stoicism philosophy is a great way to deal with randomness. We do not control randomness, but we manage our reaction to it. Expect it, don’t complain or blame. Face variance with dignity and wisdom.

Reading daily news takes a lot of effort for little reward. Therefore we should stop doing it. If the news is essential, it will travel through the grapevine to reach you somehow.

Same goes for checking portfolio every minute. We might be affected by small variance and let our emotion take over.

Temp:

Faire un couplet sur luck

Our brain sees the world as less random than it actually is. It is as if there were two planets: the one in which we actually live and the one, considerably more deterministic, on which people are convinced we live.

Our mind is not equipped with the adequate machinery to handle probabilities. Past events always look less random than they were because an event that has already taken place has 100% probability, i.e., certainty. When you look at the past, the past will always be deterministic, since only one single observation took place. Our mind will interpret most events not with the preceding ones in mind, but the following ones. Things are always obvious after the fact.

People think that there is a story when there is none. Psychologists call this overestimation of what one knew at the time of the event due to subsequent information the hindsight bias, the “I knew it all along” effect." Bias

One cannot judge a performance in any given field (war, politics, medicine, investments) by the results, but by the costs of the alternative (i.e., if history played out in a different way)

When I wrote Fooled by Randomness, which argues—a relative of this message—that we have a tendency to underestimate the role of randomness in human affairs, summarized as “it is more random than you think,” the message in the media became “it’s all random” or “it’s all dumb luck,” an illustration of the Procrustean bed that changes by reducing.

We are fooled by randomness.

Highlights::

Preface

Our brain sees the world as less, far less, random that it actually is.

The “fat tails,” that wild brand of uncertainty that causes large deviations (rare events explain more and more of the world we live in, but at the same time remain as counterintuitive to us as they were to our ancestors). Fat Tails

It is as if there were two planets: the one in which we actually live and the one, considerably more deterministic, on which people are convinced we live. It is as simple as that: Past events will always look less random than they were (it is called the hindsight bias). I would listen to someone’s discussion of his own past realizing that much of what he was saying was just backfit explanations concocted ex post by his deluded mind. Bias

Living in the real world may be painful particularly if one finds statements more informative about the people making them than the intended message.

You can mispredict everything for all your life yet think that you will get it right next time.

Purge our minds of the recent tradition of intellectual certainties.

It certainly takes bravery to remain skeptical; it takes inordinate courage to introspect, to confront oneself, to accept one’s limitations.

Probability is principally a branch of applied skepticism. Probability

"Probability is not a mere computation of odds on the dice or more complicated variants; it is the acceptance of the lack of certainty in our knowledge and the development of methods for dealing with our ignorance. Outside of textbooks and casinos, probability almost never presents itself as a mathematical problem or a brain teaser. Mother nature does not tell you how many holes there are on the roulette table, nor does she deliver problems in a textbook way (in the real world one has to guess the problem more than the solution)"Computation

There is no such thing as bad publicity: Some people manage to promote your work by insulting it. Advertising

I have to confess that I never felt really particularly directly of service to anyone being a trader (except myself); it felt elevating and useful being an essayist.

It is more random than we think” rather than “it is all random.

Of course chance favors the prepared! Chance

One needs to go out and buy a lottery ticket in order to win. Does it mean that the work involved in the trip to the store caused the winning?

Assume that good qualities cause success; based on that assumption, even though it seems intuitively correct to think so, the fact that every intelligent, hardworking, persevering person becomes successful does not imply that every successful person is necessarily an intelligent, hardworking, persevering person.

Luck is democratic and hits everyone regardless of original skills. Luck

Clearly risk taking is necessary for large success—but it is also necessary for failure. Risk

Strangely, one needs to use the press to communicate the message that the press is toxic. Media

People think that there is a story when there is none.

Most journalists do not take things too seriously: After all, this business of journalism is about pure entertainment, not a search for truth, particularly when it comes to radio and television. The trick is to stay away from those who do not seem to know that they are just entertainers and actually believe that they are thinkers. Media

Acknowledgments for the Updated Second Edition

I doubt that many mistakes are left; however, those that remain are mine.

Prologue

This book is about luck disguised and perceived as nonluck (that is, skills) and, more generally, randomness disguised and perceived as non-randomness (that is, determinism). It manifests itself in the shape of the lucky fool, defined as a person who benefited from a disproportionate share of luck but attributes his success to some other, generally very precise, reason. Luck

Just as one day some primitive tribesman scratched his nose, saw rain falling, and developed an elaborate method of scratching his nose to bring on the much-needed rain, we link economic prosperity to some rate cut by the Federal Reserve Board.

The confusion between noise and meaning, that is, between a randomly constructed arrangement and a precisely intended message.

Some people play the game too seriously; they are paid to read too much into things.

The growth in available information has been exceeded only by the expansion of noise. Information

Our mind is not equipped with the adequate machinery to handle probabilities; such infirmity even strikes the expert.

Basically this category would include those who think that the cure for obesity is to inform people that they should be healthy. On the other hand there is the Tragic Vision of humankind that believes in the existence of inherent limitations and flaws in the way we think and act and requires an acknowledgment of this fact as a basis for any individual and collective action.

Delivering advice assumes that our cognitive apparatus rather than our emotional machinery exerts some meaningful control over our actions. We will see how modern behavioral science shows this to be completely untrue.

There are those who think that there are easy clear-cut answers and those who don’t think that simplification is possible without severe distortion. The Map Is Not The Territory

PART I

Which came with the help of luck could be taken away by luck (and often rapidly and unexpectedly at that). Luck

Things that come with little help from luck are more resistant to randomness. Luck

It does not matter how frequently something succeeds if failure is too costly to bear.

Chapter One

Mild success can be explainable by skills and labor. Wild success is attributable to variance. Variance

Intellectual contempt does not control personal envy.

Psychologists have shown that most people prefer to make $70,000 when others around them are making $60,000 than to make $80,000 when others around them are making $90,000. Economics, schmeconomics, it is all pecking order, he thought. No such analysis could prevent him from assessing his condition in an absolute rather than a relative way. Happier is not the one who has a lot of money, but the one who has more than his neighbor does. Tweet

This high-yield market resembles a nap on a railway track. Metaphor

The big house is simply a loan. Appearance

Lucky fools do not bear the slightest suspicion that they may be lucky fools—by definition, they do not know that they belong to such a category. They will act as if they deserved the money. Their strings of successes will inject them with so much serotonin (or some similar substance) that they will even fool themselves about their ability to outperform markets (our hormonal system does not know whether our successes depend on randomness). Luck

Arguably, in expectation, a dentist is considerably richer than the rock musician who is driven in a pink Rolls Royce.

For one cannot consider a profession without taking into account the average of the people who enter it, not the sample of those who have succeeded in it."

The idea of taking into account both the observed and unobserved possible outcomes sounds like lunacy. For most people, probability is about what may happen in the future, not events in the observed past; an event that has already taken place has 100% probability, i.e., certainty. Probability

Chapter Two

I start with the platitude that one cannot judge a performance in any given field (war, politics, medicine, investments) by the results, but by the costs of the alternative (i.e., if history played out in a different way). Such substitute courses of events are called alternative histories. Clearly, the quality of a decision cannot be solely judged based on its outcome, but such a point seems to be voiced only by people who fail (those who succeed attribute their success to the quality of their decision). Decision

The public observes the external signs of wealth without even having a glimpse at the source (we call such source the generator). Consider the possibility that the Russian roulette winner would be used as a role model by his family, friends, and neighbors.

The reader can see my unusual notion of alternative accounting: $10 million earned through Russian roulette does not have the same value as $10 million earned through the diligent and artful practice of dentistry. They are the same, can buy the same goods, except that one’s dependence on randomness is greater than the other. Randomness

Leibniz’ idea of possible worlds. For Leibniz, God’s mind included an infinity of possible worlds, of which he selected just one. These nonselected worlds are worlds of possibilities, and the one in which I am breathing and writing these lines is just one of them that happened to have been executed.

In physics, there is the many-world interpretation in quantum mechanics (associated with the works of Hugh Everett in 1957) which considers that the universe branches out treelike at each juncture; what we are living now is only one of these many worlds. Physics

Reality is far more vicious than Russian roulette. First, it delivers the fatal bullet rather infrequently, like a revolver that would have hundreds, even thousands, of chambers instead of six. After a few dozen tries, one forgets about the existence of a bullet, under a numbing false sense of security. Metaphor

Second, unlike a well-defined, precise game like Russian roulette, where the risks are visible to anyone capable of multiplying and dividing by six, one does not observe the barrel of reality.

There are some (though very few) who will call you to express their gratitude and thank you for having protected them from the events that did not take place.

We flipped a coin to see who was going to pay for the meal. I lost and paid. He was about to thank me when he abruptly stopped and said that he paid for half of it probabilistically.

Realism can be punishing. Probabilistic skepticism is worse.

Heroes are heroes because they are heroic in behavior, not because they won or lost.

Such tendency to make and unmake prophets based on the fate of the roulette wheel is symptomatic of our ingrained inability to cope with the complex structure of randomness prevailing in the modern world.

I do not dispute that arguments should be simplified to their maximum potential; but people often confuse complex ideas that cannot be simplified into a media-friendly statement as symptomatic of a confused mind. Simplicity

People do not like to insure against something abstract; the risk that merits their attention is always something vivid.

The consequences are not trivial: It means that rational thinking has little, very little, to do with risk avoidance. Much of what rational thinking seems to do is rationalize one’s actions by fitting some logic to them. Rationality

This is one of the many reasons that journalism may be the greatest plague we face today—as the world becomes more and more complicated and our minds are trained for more and more simplification. Media Information

Like a doctor torn between the two types of errors, the false positive (telling the patient he has cancer when in fact he does not) and the false negative (telling the patient he is healthy when in fact he has cancer), they need to balance their existence with the fact that they inherently need some margin of error in their business.

From the standpoint of an institution, the existence of a risk manager has less to do with actual risk reduction than it has to do with the impression of risk reduction. Philosophers since Hume and modern psychologists have been studying the concept of epiphenomenalism, or when one has the illusion of cause-and-effect. Does the compass move the boat? By “watching” your risks, are you effectively reducing them or are you giving yourself the feeling that you are doing your duty?

Chapter Three

Mathematics is principally a tool to meditate, rather than to compute.

Monte Carlo methods, in brief, consist of creating artificial history. MonteCarlo

The invisible histories have a scientific name, alternative sample paths, a name borrowed from the field of mathematics of probability called stochastic processes. The notion of path, as opposed to outcome, indicates that it is not a mere MBA-style scenario analysis, but the examination of a sequence of scenarios along the course of time. We are not just concerned with where a bird can end up tomorrow night, but rather with all the various places it can possibly visit during the time interval. MonteCarlo

Stochastic processes refer to the dynamics of events unfolding with the course of time. Stochastic is a fancy Greek name for random. This branch of probability concerns itself with the study of the evolution of successive random events —one could call it the mathematics of history. The key about a process is that it has time in it. Stochastic MonteCarlo History

One can generate thousands, perhaps millions, of random sample paths, and look at the prevalent characteristics of some of their features. The assistance of the computer is instrumental in such studies. The glamorous reference to Monte Carlo indicates the metaphor of simulating the random events in the manner of a virtual casino. MonteCarlo

With no mathematical literacy we can launch a Monte Carlo simulation of an eighteen-year-old Christian Lebanese successively playing Russian roulette for a given sum, and see how many of these attempts result in enrichment, or how long it takes on average before he hits the obituary. We can change the barrel to contain 500 holes, a matter that would decrease the probability of death, and see the results. Monte Carlo simulation methods were pioneered in martial physics in the Los Alamos laboratory during the A-bomb preparation. They became popular in financial mathematics in the 1980s, particularly in the theories of the random walk of asset prices. MonteCarlo

I became addicted to the various Monte Carlo engines, which I taught myself to build, thrilled to feel that I was generating history, a Demiurgus. It can be electrifying to generate virtual histories and watch the dispersion between the various results. Such dispersion is indicative of the degree of resistance to randomness. MonteCarlo History

My aim, as a pure amateur fleeing the boredom of business life, was merely to develop intuitions for these events.

The evolutionary way of thinking taught by my Monte Carlo simulators. MonteCarlo Evolution Darwin

I have two ways of learning from history: from the past, by reading the elders; and from the future, thanks to my Monte Carlo toy. MonteCarlo History

Every man believes himself to be quite different, a matter that amplifies the “why me?” shock upon a diagnosis.

Somehow all respect we may have for history does not translate well into our treatment of the present. History

Things are always obvious after the fact.

When you look at the past, the past will always be deterministic, since only one single observation took place. Our mind will interpret most events not with the preceding ones in mind, but the following ones.

Psychologists call this overestimation of what one knew at the time of the event due to subsequent information the hindsight bias, the “I knew it all along” effect. Bias

A mistake is not something to be determined after the fact, but in the light of the information until that point.

Unlike many “hard” sciences, history cannot lend itself to experimentation. History

Ergodicity. It means, roughly, that (under certain conditions) very long sample paths would end up resembling each other. The properties of a very, very long sample path would be similar to the Monte Carlo properties of an average of shorter ones. Ergodicity MonteCarlo

The janitor who won the lottery, if he lived one thousand years, cannot be expected to win more lotteries. Those who were unlucky in life in spite of their skills would eventually rise. The lucky fool might have benefited from some luck in life; over the longer run he would slowly converge to the state of a less-lucky idiot. Each one would revert to his long-term properties. Ergodicity

Aside from the decorum of ancient thought as opposed to the coarseness of fresh ink, I have spent some time phrasing the idea in the mathematics of evolutionary arguments and conditional probability. For an idea to have survived so long across so many cycles is indicative of its relative fitness. Noise, at least some noise, was filtered out. Evolution Ideas

It is optimal for someone, when in doubt, to systematically reject the new idea, information, or method.

People tend to infer that because some inventions have revolutionized our lives that inventions are good to endorse and we should favor the new over the old. I hold the opposite view.

The problem with information is not that it is diverting and generally useless, but that it is toxic. Information

People do not realize that the media is paid to get your attention. For a journalist, silence rarely surpasses any word. Media

The fools of randomness.

Like a health club membership taken out to satisfy a New Year’s resolution, people often think that it will surely be the next batch of news that will really make a difference to their understanding of things. Metaphor Media

Much of the thinking about the negative value of information on society in general was sparked by Robert Shiller. Information

If a stock price is the estimated value of “something” (say the discounted cash flows from a corporation), then market prices are way too volatile in relation to tangible manifestations of that “something” (he used dividends as proxy). Prices swing more than the fundamentals they are supposed to reflect, they visibly overreact by being too high at times (when their price overshoots the good news or when they go up without any marked reason) or too low at others. Finance

The defender of the dogmas of modern finance and efficient markets started a fund that took advantage of market inefficiencies! It is as if the Pope converted to Islam. Metaphor

The ratio of undistilled information to distilled is rising, saturating markets. The elder’s messages need not be delivered to you as imminent news. Information Media

Journalism goes to what can capture our attention, with adequate sound bites. Media

Under regime switching, as we will see in Chapter 5, it will be unclear who is actually the fittest, and those who will survive are not necessarily those who appear to be the fittest. Curiously, it will be the oldest, simply because older people have been exposed longer to the rare event and can be, convincingly, more resistant to it.

The wise man listens to meaning; the fool only gets the noise. The modern Greek poet C. P. Cavafy wrote a piece in 1915 after Philostratus’ adage “For the gods perceive things in the future, ordinary people things in the present, but the wise perceive things about to happen.”

Being emotional, he feels a pang with every loss, as it shows in red on his screen. He feels some pleasure when the performance is positive, but not in equivalent amount as the pain experienced when the performance is negative. At the end of every day the dentist will be emotionally drained. A minute-by-minute examination of his performance means that each day (assuming eight hours per day) he will have 241 pleasurable minutes against 239 unpleasurable ones. These amount to 60,688 and 60,271, respectively, per year. Now realize that if the unpleasurable minute is worse in reverse pleasure than the pleasurable minute is in pleasure terms, then the dentist incurs a large deficit when examining his performance at a high frequency. Loss

Over a short time increment, one observes the variability of the portfolio, not the returns. In other words, one sees the variance, little else. Variance

If you think that you can control your emotions, think that some people also believe that they can control their heartbeat or hair growth. Emotions Metaphor

The same methodology can explain why the news (the high scale) is full of noise and why history (the low scale) is largely stripped of it (though fraught with interpretation problems). This explains why I prefer not to read the newspaper (outside of the obituary), why I never chitchat about markets, and, when in a trading room, I frequent the mathematicians and the secretaries, not the traders. It explains why it is better to read The New Yorker on Mondays than The Wall Street Journal every morning (from the standpoint of frequency, aside from the massive gap in intellectual class between the two publications).

Finally, this explains why people who look too closely at randomness burn out, their emotions drained by the series of pangs they experience. Regardless of what people claim, a negative pang is not offset by a positive one (some psychologists estimate the negative effect for an average loss to be up to 2.5 the magnitude of a positive one); it will lead to an emotional deficit.

My sole advantage in life is that I know some of my weaknesses, mostly that I am incapable of taming my emotions facing news and incapable of seeing a performance with a clear head. Silence is far better.

Chapter Four

a statement could fall only into two categories: deductive, like “2 +2 =4,” i.e., incontrovertibly flowing from a precisely defined axiomatic framework (here the rules of arithmetic), or inductive, i.e., verifiable in some manner (experience, statistics, etc.), like “it rains in Spain” or “New Yorkers are generally rude.”

Rhetoric can be constructed randomly, but not genuine scientific knowledge.

What is the Turing test? The brilliant British mathematician, eccentric, and computer pioneer Alan Turing came up with the following test: A computer can be said to be intelligent if it can (on average) fool a human into mistaking it for another human. The converse should be true. A human can be said to be unintelligent if we can replicate his speech by a computer, which we know is unintelligent, and fool a human into believing that it was written by a human.

Yiddish saying: “If I am going to be forced to eat pork, it better be of the best kind.” If I am going to be fooled by randomness, it better be of the beautiful (and harmless) kind.

Chapter Five

A trader buys and sells (he may sell what he does not own and buy it back later, hopefully making a profit in a decline; this is called “shorting”).

A trader’s mental construction should direct him to do precisely what other people do not do.

the firehouse effect. He had observed that firemen with much downtime who talk to each other for too long come to agree on many things that an outside, impartial observer would find ludicrous (they develop political ideas that are very similar).

Let us remember that economists are evaluated on how intelligent they sound, not on a scientific measure of their knowledge of reality.

at any point in time, the richest traders are often the worst traders. This, I will call the cross-sectional problem:

At a given time in the market, the most successful traders are likely to be those that are best fit to the latest cycle.

Following the incident, John regarded himself “ruined”; yet his net worth is still close to $1 million, which could be the envy of more than 99.9% of the inhabitants of our planet. Yet there is a difference between a wealth level reached from above and a wealth reached from

below. The road from $16 million to $1 million is not as pleasant as the one from 0 to $1 million.

There is absolutely nothing wrong with investing “for the long haul,” provided one does not mix it with short-term trading—it is just that many people become long-term investors after they lose money, postponing their decision to sell as part of their denial.

An overestimation of the accuracy of their beliefs in some measure

Loyalty to ideas is not a good thing for traders, scientists—or anyone.

When the losses occurred there was no clear acceptance of what had happened. The price on the screen lost its reality in favor of some abstract “value.”

Darwinian fitness applies to species developing over a very long time, not observed over a short term—time aggregation eliminates much of the effects of randomness

we do not live in a world where things “converge” continuously toward betterment. Nor do things in life move continuously at all.

in the very small, particles jump (discretely) between states; they do not slide between them.

This is not what evolution means; on average, animals will be fit, but not every single one of them, and not at all times. Just as an animal could have survived because its sample path was lucky, the “best” operators in a given business can come from a subset of operators who survived because of overfitness to a sample path—a sample path that was free of the evolutionary rare event. One vicious attribute is that the longer these animals can go without encountering the rare event, the more vulnerable they will be to it. We said that should one extend time to infinity, then, by ergodicity, that event will happen with certainty—the species will be wiped out!

Chapter Six

On the surface, my statement here may seem to contradict earlier discussions, where I blame people for not learning enough from history. The problem is that we read too much into shallow recent history, with statements like “this has never happened before,” but not from history in general (things that never happened before in one area tend eventually to happen). In other words, history teaches us that things that never happened before do happen. It can teach us a lot outside of the narrowly defined time series; the broader the look, the better the lesson. In other words, history teaches us to avoid the brand of naive empiricism that consists of learning from casual historical facts.

By generalization, I started to label a rare event as any behavior where the adage “beware of calm waters” can hold. Popular wisdom often warns of the old neighbor who appears to remain courtly and reserved, the model of an excellent citizen, until you see his picture in the national paper as a deranged killer who went on a rampage. Until then, he was not known to have committed any transgression. There was no way to predict that such pathological behavior could emanate from such a nice person. I associate rare events with any misunderstanding of the risks derived from a narrow interpretation of past time series.

Rare events are always unexpected, otherwise they would not occur.

Whenever there is asymmetry in outcomes, the average survival has nothing to do with the median survival.

Asymmetric odds means that probabilities are not 50% for each event, but that the probability on one side is higher than the probability on the other. Asymmetric outcomes mean that the payoffs are not equal.

Why do they confuse probability and expectation, that is, probability and probability times the payoff? Mainly because much of people’s schooling comes from examples in symmetric environments, like a coin toss, where such a difference does not matter. In fact, the so-called bell curve that seems to have found universal use in society is entirely symmetric.

Alas, investors and businesses are not paid in probabilities; they are paid in dollars. Accordingly, it is not how likely an event is to happen that matters, it is how much is made when it happens that should be the consideration. How frequent the profit is irrelevant; it is the magnitude of the outcome that counts.

I try to make money infrequently, as infrequently as possible, simply because I believe that rare events are not fairly valued, and that the rarer the event, the more undervalued it will be in price.

someone with nineteenth-century training under Laplace would be able to solve the equations, called differential equations, or,

equivalently, equations of motion—since we are studying the dynamics of an entity whose position depends on time.

Psychologists recently found out that people tend to be sensitive to the presence or absence of a given stimulus rather than its magnitude. This implies that a loss is first perceived as just a loss, with further implications later. The same with profits. The agent would prefer the number of losses to be low and the number of gains to be high, rather than optimizing the total performance.

In the markets, there is a category of traders who have inverse rare events, for whom volatility is often a bearer of good news. These traders lose money frequently, but in small amounts, and make money rarely, but in large amounts. I call them crisis hunters. I am happy to be one of them.

the more information you have, the more you are confident about the outcome. Now the problem: by how much? Common statistical method is based on the steady augmentation of the confidence level, in nonlinear proportion to the number of observations. That is, for an n times increase in the sample size, we increase our knowledge by the square root of n.

This is called “the problem of stationarity.” Think of an urn that is hollow at the bottom. As I am sampling from it, and without my being aware of it, some mischievous child is adding balls of one color or another. My inference thus becomes insignificant.

What if vicious children were changing the composition of the urn? In other words, what if things have changed?

I wondered whether the time series reflecting the activity of people now dead or retired should matter for predicting the future.

There is no point searching for patterns that are available to everyone with a brokerage account; once detected, they would be self-canceling.

Chapter Seven

In his Treatise on Human Nature, the Scots philosopher David Hume posed the issue in the following way (as rephrased in the now famous black swan problem by John Stuart Mill): No amount of observations of white swans can allow the inference that all swans are white, but the observation of a single black swan is sufficient to refute that conclusion.

For instance, a conventional-wisdom, empirical style statement like

automobile accidents happen closer to home

can be tested by taking the average distance between the accident and the domicile of the driver (if, say, about 20% of accidents happen within a twelve-mile radius). However, one needs to be careful in the interpretation; a naive reader of the result would tell you that you are more likely to have an accident if you drive in your neighborhood than if you did so in remote places, which is a typical example of naive empiricism. Why? Because accidents may happen closer to home simply because people spend their time driving close to home

refute a conjecture, never to affirm it. For instance, the statement

The market never goes down 20% in a given three-month period

can be tested but is completely meaningless if verified. I can quantitatively reject the proposition by finding counterexamples, but it is not possible for me to accept it simply because, in the past, the market never went down 20% in any three-month period (you cannot easily make the logical leap from “has never gone down” to “never goes down”).

You can more safely use the data to reject than to confirm hypotheses. Why? Consider the following statements:

Statement A: No swan is black, because I looked at four thousand swans and found none.

Statement B: Not all swans are white.

the problem of interpreting past data literally, without methodology or logic:

I have just completed a thorough statistical examination of the life of President Bush. For fifty-eight years, close to 21,000 observations, he did not die once. I can hence pronounce him as immortal, with a high degree of statistical significance.

I can use data to disprove a proposition, never to prove one. I can use history to

I cannot logically make statement A, no matter how many successive white swans I may have observed in my life and may observe in the future (except, of course, if I am given the privilege of observing with certainty all available swans). It is, however, possible to make Statement B merely by finding one single counterexample. Indeed, Statement A was disproved by the discovery of Australia, as it led to the sighting of the Cygnus atratus, a swan variety that was jet black! The reader will see a hint of Popper’s ideas, as there is a strong asymmetry between the two statements; and, furthermore, such asymmetry lies in the foundations of knowledge.

a small chance of large loss and a large chance of a small win

If you engaged in a Russian roulette–type strategy with a low probability of large loss, one that bankrupts you every several years, you are likely to show up as the winner in almost all samples—except in the year when you are dead.

stay away from people of a competitive nature, as they have a tendency to commoditize and reduce the world to categories

To conclude, extreme empiricism, competitiveness, and an absence of logical structure to one’s inference can be a quite explosive combination.

I was at the age when one felt like one needed to read everything, which prevented one from making contemplative stops.

Why is a theory never right? Because we will never know if all the swans are white

A theory cannot be verified.

It can only be provisionally accepted.

Newtonian physics is scientific because it allowed us to falsify it, as we know that it is wrong, while astrology is not because it does not offer conditions under which we could reject it.

the matter of knowledge and discovery is not so much in dealing with what we know as in dealing with what we do not know

An open society is one in which no permanent truth is held to exist; this would allow counter-ideas to emerge.

we like to emit logical and rational ideas but we do not necessarily enjoy this execution

Memory in humans is a large machine to make inductive inferences. Think of memories: What is easier to remember, a collection of random facts glued together, or a story, something that offers a series of logical links? Causality is easier to commit to memory. Our brain would have less work to do in order to retain the information. The size is smaller. What is induction exactly? Induction is going from plenty of particulars to the general. It is very handy, as the general takes much less room in one’s memory than a collection of particulars. The effect of such compression is the reduction in the degree of detected randomness.

I want to take the best of what the past can give me without its dangers.

They trade on ideas based on some observation (that includes past history) but, like the Popperian scientists, they make sure that the costs of being wrong are limited (and their probability is not derived from past data).

PART II

I do not deny that if someone performed better than the crowd in the past, there is a presumption of his ability to do better in the future. But the presumption might be weak, very weak, to the point of being useless in decision making. Why? Because it all depends on two factors: The randomness content of his profession and the number of monkeys in operation.

The initial sample size matters greatly.

If one puts an infinite number of monkeys in front of (strongly built) typewriters, and lets them clap away, there is a certainty that one of them would come out with an exact version of the Iliad.

The greater the number of businessmen, the

greater the likelihood of one of them performing in a stellar manner just by luck.

There are other aspects to the monkeys problem; in real life the other monkeys are not countable, let alone visible. They are hidden away, as one sees only the winners—it is natural for those who failed to vanish completely. Accordingly, one sees the survivors, and only the survivors, which imparts such a mistaken perception of the odds.

We do not respond to probability, but to society’s assessment of it

Chapter Eight

the costs on his wife, Janet, are monstrous. Why? Because of their relative nonsuccess—as geographically defined by their Park Avenue neighborhood.

a very common illustration of the emotional effect of survivorship bias. Janet feels that her husband is a failure, by comparison, but she is miscomputing the probabilities in a gross manner—she is using the wrong distribution to derive a rank.

as compared to his co-op neighbors, he is at the bottom! Why? Because he chose to live among the people who have been successful, in an area that excludes failure. In other words, those who have failed do not show up in the sample, thus making him look as if he were not doing well at all. By living on Park Avenue, one does not have exposure to the losers, one only sees the winners.

Aside from the misperception of one’s performance, there is a social treadmill effect: You get rich, move to rich neighborhoods, then become poor again. To that add the psychological treadmill effect; you get used to wealth and revert to a set point of satisfaction. This problem of some people never really getting to feel satisfied by wealth (beyond a given point) has been the subject of technical discussions on happiness.

the deformations of chance in life

There is no solace to be found from reasoning

we should remember that becoming rich is a purely selfish act, not a social one. The virtue of capitalism is that society can take advantage of people’s greed rather than their benevolence, but there is no need to, in addition, extol such greed as a moral

money spent bears no fruit (except the enjoyment of the spender)

we are trained to take advantage of the information that is lying in front of our eyes, ignoring the information that we do not see.

In a nutshell, the survivorship bias implies that the highest performing realization will be the most visible. Why? Because the losers do not show up.

Optimism, it is said, is predictive of success. Predictive? It can also be predictive of failure. Optimistic people certainly take more risks as they are overconfident about the odds; those who win show up among the rich and famous, others fail and disappear from the analyses.

Chapter Nine

I can practically make the same statement about anyone operating in the physical world, or in a business in which the degree of randomness is low. But there is a problem in anything related to the business world.

it is harder to fry an egg than buy and sell

As the reader now knows, the fund manager can expect to be heckled by me during the presentation, particularly if he does not exhibit the minimum of humility and self-doubt that I would expect from someone practicing randomness.

Machiavelli ascribed to luck at least a 50% role in life (the rest was cunning and bravura), and that was before the creation of modern markets.

And the following year, should he stop outperforming (recall that his odds of having a good year have stayed at 50%) they would start laying blame, finding fault with the relaxation in his work ethics, or his dissipated lifestyle. They will find something he did before when he was successful that he has subsequently stopped doing, and attribute his failure to that. The truth will be, however, that he simply ran out of luck.

The first counterintuitive point is that a population entirely composed of bad managers will produce a small amount of great track records.

The second counterintuitive point is that the expectation of the maximum of track records, with which we are concerned, depends more on the size of the initial sample than on the individual odds per manager.

A result is that in real life, the larger the deviation from the norm, the larger the probability of it coming from luck rather than skills

the properties of ergodicity, namely, that time will eliminate the annoying effects of randomness.

Remember that nobody accepts randomness in his own success, only his failure.

The information that a person derived some profits in the past, just by itself, is neither meaningful nor relevant. We need to know the size of the population from which he came.

adverse selection. Judging an investment that comes to you requires more stringent standards than judging an investment you seek, owing to such selection bias.

“It’s a small world!” is often uttered with surprise. But these are not improbable occurrences—the world is much larger than we think. It is just that we are not truly testing for the odds of having an encounter with one specific person, in a specific location at a specific time. Rather, we are simply testing for any encounter, with any person we have ever met in the past, and in any place we will visit during the period concerned.

I am fitting the rule on the data. This activity is called data snooping. The more I try, the more I am likely, by mere luck, to find a rule that worked on past data. A random series will always present some detectable pattern.

there is no scientific evidence that can convince them more potently than a sincere and emotional testimonial.

The late astronomer Carl Sagan, a devoted promoter of scientific thinking and an obsessive enemy of nonscience, examined the cures from cancer that resulted from a visit to Lourdes in France, where people were healed by simple contact with the holy waters, and found out the interesting fact that, of the total cancer patients who visited the place, the cure rate was, if anything, lower than the statistical one for spontaneous remissions.

Philosophers of statistics call this the reference case problem to explain that there is no true attainable randomness in practice, only in theory.

There may be great information in the fact that nothing took place. As Sherlock Holmes noted in the Silver Blaze case—the curious thing was that the dog did not bark.

Chapter Ten

It could be said that the last grain of sand is responsible for the destruction of the entire structure. What we are witnessing here is a nonlinear effect resulting from a linear force exerted on an object. A very small additional input, here the grain of sand, caused a disproportionate result, namely the destruction of my starter Tower of Babel.

This is called a path dependent outcome

The Polya process can be presented as follows: Assume an urn initially containing equal quantities of black and red balls. You are to guess each time which color you will pull out before you make the draw. Here the game is rigged. Unlike a conventional urn, the

probability of guessing correctly depends on past success, as you get better or worse at guessing depending on past performance.

reality rarely gives us the privilege of a satisfying linear positive progression: You may study for a year and learn nothing, then, unless you are disheartened by the empty

results and give up, something will come to you in a flash.

Most people give up before the rewards.

Imagine a donkey equally hungry and thirsty placed at exactly equal distance from sources of food and water.

In such a framework, he would die of both thirst and hunger

The reader no doubt has played a version of Buridan’s donkey, by “flipping a coin” to break some of the minor stalemates in life where one lets randomness help with the decision process.

It is better to have a handful of enthusiastic advocates than hordes of people who appreciate your work—better to be loved by a dozen than liked by the hundreds.

Too much success is the enemy (think of the punishment meted out on the rich and famous); too much failure is demoralizing. I would like the option of having neither.

Chapter Eleven

The fact that your mind cannot retain and use everything you know at once is the cause of such biases. One central aspect of a heuristic is that it is blind to reasoning.

When you take a gamble, do you say: “My net worth will end up at $99,000 or $101,500 after the gamble” or do you say “I lose $1,000 or make $1,500?” Your attitude toward the risks and rewards of the gamble will vary according to whether you look at your net worth or changes in it. But in fact in real life you will be put in situations where you will only look at your changes. The fact that the losses hurt more than the gains, and differently, makes your accumulated performance, that is, your total wealth, less relevant than the last change in it.

This dependence on the local rather than the global status (coupled with the effect of the losses hitting harder than the gains) has an impact on your perception of well-being.

mathematical expectation, i.e., the probabilities of each payoff multiplied by the dollar values at stake (50% multiplied by 0 and 50% multiplied by $2,000 =$1,000). Can you imagine (that is visualize, not compute mathematically) the value being $1,000? We can conjure up one and only one state at a given time, i.e., either 0 or $2,000. Left to our own devices, we are likely to bet in an irrational way, as one of the states would dominate the picture—the fear of ending with nothing or the excitement of an extra $1,000.

Somehow the news reached his body before his mind.

Now the reader might wonder about the mathematical difference between a 28% chance of death and a 72% chance of survival over the next five years. Clearly, there is none, but we are not made for mathematics. In Nero’s mind a 28% chance of death meant the image of himself dead, and thoughts of the cumbersome details of his funeral. A 72% chance of survival put him in a cheerful mood; his mind was planning the result of a cured Nero skiing in the Alps. At no point during his ordeal did Nero think of himself as 72% alive and 28% dead.

The problem with thinking is that it causes you to develop illusions.

“Satisficing” was his idea (the melding together of satisfy and suffice): You stop when you get a near-satisfactory solution. Otherwise it may take you an eternity to reach the smallest conclusion or perform the smallest act. We are therefore rational, but in a limited way: “boundedly rational.”

Herbert Simon

His idea is that if we were to optimize at every step in life, then it would cost us an infinite amount of time and energy. Accordingly, there has to be in us an approximation process that stops somewhere.

Kahneman and Tversky went in a completely different direction than Simon and started figuring out rules in humans that did not make them rational—but things went beyond the shortcut. For them, these rules, which are called heuristics, were not merely a simplification of rational models, but were different in methodology and category. They called them “quick and dirty” heuristics. There is a dirty part: These shortcuts came with side effects, these effects being the biases

A normative science (clearly a self-contradictory concept) offers prescriptive teachings; it studies how things should be.

a positive science, which is based on how people actually are observed to behave

Normative economics is like religion without the aesthetics.

people were flawed rather than imperfect

when something is in relation to something else, that something else can be manipulated. Psychologists call this effect of comparing to a given reference anchoring.

The availability heuristic

The simulation heuristic

the practice of estimating the frequency of an event according to the ease with which instances of the event can be recalled.

t corresponds to counterfactual thinking: Imagine what might have happened had you not missed your train

Our brains are made for fitness not for truth

They believe (roughly) that we have three brains: The very old one, the reptilian brain that dictates heartbeat and that we share with all animals; the limbic brain center of emotions that we share with mammals; and the neocortex, or cognitive brain, that distinguishes primates and humans

One cannot make a decision without emotion.

we need a shortcut; emotions are there to prevent us from temporizing.

It seems that the emotions are the ones doing the job. Psychologists call them “lubricants of reason.

we feel emotions (limbic brain) then find an explanation (neocortex)

our modern society is ruled by probability (because of the explosion in information), while important decisions are made without the smallest regard for its basic laws.

plight of a man, Joseph K., who is arrested for a mysterious and unexplained reason, hit a spot as it was written before we heard of the methods of the “scientific” totalitarian regimes. It projected a scary future of mankind wrapped in absurd self-feeding bureaucracies, with spontaneously emerging rules subjected to the internal logic of the bureaucracy. It spawned an entire “literature of the absurd”

It is easy for a society to live without mathematics—or traders to trade without quantitative methods—when the space of possible outcomes is one-dimensional. One-dimensional means that we are looking at one sole variable, not a collection of separate events.

They confuse the expected value and the most likely scenario

The option is simply the weighted average of the possible states the asset can take.

What is less unpleasant: to lose 100 times $1 or lose once $100? Clearly the second: Our sensitivity to losses decreases. So a trading policy that makes $1 a day for a long time then loses them all is actually pleasant from a hedonic standpoint, although it does not make sense economically.

People overvalue their knowledge and underestimate the probability of their being wrong.

What these journalists confuse is the unconditional and conditional life expectancy. At birth, your unconditional life expectancy may be seventy-three years. But as you advance in age and do not die, your life expectancy increases along with your life. Why? Because other people, by dying, have taken your spot in the statistics, for expectation means average.

Significance: How did I decide that it was perfect noise? Take a simple analogy. If you engage in a mountain bicycle race with a friend across Siberia and, a month later, beat him by one single second, you clearly cannot quite boast that you are faster than him. You might have been helped by something, or it can be just plain randomness, nothing else. That second is not in itself significant enough for someone to draw conclusions. I would not write in my pre-bedtime diary: Cyclist A is better than cyclist B because he is fed with spinach whereas cyclist B has a diet rich in tofu. The reason I am making this inference is because he beat him by 1.3 seconds in a 3,000 mile race.

PART III

The epiphany I had in my career in randomness came when I understood that I was not intelligent enough, nor strong enough, to even try to fight my emotions. Besides, I believe that I need my emotions to formulate my ideas and get the energy to execute them.

If my brain can tell the difference between noise and signal, my heart cannot.

Wax in my ears.

Wittgenstein’s ruler: Unless you have confidence in the ruler’s reliability, if you use a ruler to measure a table you may also be using the table to measure the ruler. The less you trust the ruler‘s reliability (in probability called the prior), the more information you are getting about the ruler and the less about the table. Wittgenstein's Ruler

Unless the source of the statement has extremely high qualifications, the statement will be more revealing of the author than the information intended by him.

A compliment is always pleasant, regardless of its authorship—something manipulators know rather well.

Chapter Twelve

felix qui po¨tuit cognoscere causas (happy is he who understands what is behind things)

we are not made to view things as independent from each other. When viewing two events A and B, it is hard not to assume that A causes B, B causes A, or both cause each other. Our bias is immediately to establish a causal link.

The Greek philosopher Pyrrho, who advocated a life of equanimity and indifference, was criticized for failing to keep his composure during a critical circumstance (he was chased by an ox). His answer was that he found it sometimes difficult to rid himself of his humanity.

I cannot help it, but I am emotional and derive most of my energy from my emotions. So the solution does not reside in taming my heart.

Most of us know pretty much how we should behave. It is the execution that is the problem, not the absence of knowledge.

Chapter Thirteen

Monsieur de Norpois was not lying. He had just forgotten. One forgets rather quickly what one has not thought about with depth, what has been dictated to you by imitation, by the passions surrounding you. These change, and with them so do your memories. Even more than diplomats, politicians do not remember opinions they had at some point in their lives and their fibbings are more attributable to an excess of ambition than a lack of memory.

They are totally free from their past actions. Every day is a clean slate.

probability is not about the odds, but about the belief in the existence of an alternative outcome, cause, or motive.

preferred to be guided by probability than allege with certainty

Beliefs are said to be path dependent if the sequence of ideas is such that the first one dominates.

An academic who became famous for espousing an opinion is not going to voice anything that can possibly devalue his own past work and kill years of investment.

this is a ten sigma

My lesson from Soros is to start every meeting at my boutique by convincing everyone that we are a bunch of idiots who know nothing and are mistake-prone, but happen to be endowed with the rare privilege of knowing it.

You attribute your successes to skills, but your failures to randomness.

A scientist may be forced to act like a cheap defense lawyer rather than a pure seeker of the truth.

Chapter Fourteen

Start stressing personal elegance at your next misfortune. Exhibit sapere vivere (“know how to live”) in all circumstances.

Dress at your best on your execution day (shave carefully); try to leave a good impression on the death squad by standing erect and proud. Try not to play victim when diagnosed with cancer (hide it from others and only share the information with the doctor—it will avert the platitudes and nobody will treat you like a victim worthy of their pity; in addition, the dignified attitude will make both defeat and victory feel equally heroic). Be extremely courteous to your assistant when you lose money (instead of taking it out on him as many of the traders whom I scorn routinely do). Try not to blame others for your fate, even if they deserve blame. Never exhibit any self-pity, even if your significant other bolts with the handsome ski instructor or the younger aspiring model. Do not complain. If you suffer from a benign version of the “attitude problem,” like one of my childhood friends, do not start playing nice guy if your business dries up (he sent a heroic e-mail to his colleagues informing them “less[…]

There is nothing wrong and undignified with emotions—we are cut to have them. What is wrong is not following the heroic or, at least, the dignified path. That is what stoicism truly means. It is the attempt by man to get even with probability.

The stoic is a person who combines the qualities of wisdom, upright dealing, and courage. The stoic will thus be immune from life’s gyrations as he will be superior to the wounds from some of life’s dirty tricks.

Postscript

consider the difference between judging on process and judging on results. Lower-ranking persons in the enterprise are judged on both process and results—in fact, owing to the repetitive aspect of their efforts, their process converges rapidly to results. But top management is only paid on result—no matter the process. There seems to be no such thing as a foolish decision if it results in profits. “Money talks,” we are often told. The rest is supposed to be philosophy.

Now take a peek inside the chief executive suite. Clearly, the decisions there are not repeatable. CEOs take a small number of large decisions, more like the person walking into the casino with a single million-dollar bet. External factors, such as the environment, play a considerably larger role than with the cook. The link between the skill of the CEO and the results of the company are tenuous.

Actually, wouldn’t it be better if the length of movies were kept a secret?

THE INVERSE SKILLS PROBLEM

But CEOs are not entrepreneurs. As a matter a fact, they are often empty suits.

There is an asymmetry, as these executives have almost nothing to lose.

economics is a narrative discipline, and explanations are easy to fit retrospectively

we need to drill into the heads of those who measure the contribution of executives that what they see is not necessarily what is there.

research on happiness shows that those who live under a self-imposed pressure to be optimal in their enjoyment of things suffer a measure of distress.

The difference between satisficers and optimizers raises a few questions. We know that people of a happy disposition tend to be of the satisficing kind, with a set idea of what they want in life and an ability to stop upon gaining satisfaction. Their goals and desires do not move along with the experiences. They do not tend to experience the internal treadmill effects of constantly trying to improve on their consumption of goods by seeking higher and higher levels of sophistication. In other words, they are neither avaricious nor insatiable. An optimizer, by comparison, is the kind of person who will uproot himself and change his official residence just to reduce his tax bill by a few percentage points. (You would think that the entire point of a higher income is to be free to choose where to live; in fact it seems, for these people, wealth causes them to increase their dependence!) Getting rich results in his seeing flaws in the goods and services he buys. The coffee is not warm enough. The cook no longer deserves the three stars given to him by the Michelin guide (he will write to the editors).The table is too[…]

I am convinced that we are not made for clear-cut, well-delineated schedules. We are made to live like firemen, with downtime for lounging and meditating between calls, under the protection of protective uncertainty.

To go from Munich to Milan, I picked the seven-and-a-half-hour train instead of the plane, which no self-respecting businessperson would do on a weekday, and am enjoying an air unpolluted by persons squeezed by life.

At the limit, you can decide whether to be (relatively) poor, but free of your time, or rich but as dependent as a slave.

A mild degree of unpredictability in your behavior can help you to protect yourself in situations of conflict. Say you always have the same threshold of reactions. You take a set level of abuse, say seventeen insulting remarks per week, before getting into a rage and punching the eighteenth offender in the nose. Such predictability will allow people to take advantage of you up to that well-known trigger point and stop there. But if you randomize your trigger point, sometimes overreacting at the slightest joke, people will not know in advance how far they can push you. The same applies to governments in conflicts: They need to convince their adversaries that they are crazy enough to sometimes overreact to a small peccadillo. Even the magnitude of their reaction should be hard to foretell. Unpredictability is a strong deterrent.

We favor the visible, the embedded, the personal, the narrated, and the tangible; we scorn the abstract.

A Trip to the Library: Notes and Reading Recommendations

Problem 1: Suppose that I have a pack of cards, each of which has a letter written on one side and a number written on the other side. Suppose in addition that I claim that the following rule is true: If a card has a vowel on one side, then it has an even number on the other side. Imagine that I now show you four cards from the pack: E 6 K 9. Which card or cards should you turn over in order to decide whether the rule is true or false?

Problem 2: You are a bartender in a town where the legal age for drinking is twenty-one and feel responsible for the violations of the rules. You are confronted with the following situations and would have to ask the patron to show you either his age or what he is drinking. Which of the four patrons would you have to question?

1, drinking beer; 2, over twenty-one; 3, drinking Coke; 4, under 21.

While the two problems are identical (it is clear that you need to check only the first and last of the four cases) the majority of the population gets the first[…]

“Evolution has no foresight. It lacks the long-term vision of drug company management. A species can’t raise venture capital to pay its bills while its research team . . . Each species has to stay biologically profitable every generation, or else it goes extinct. Species always have cashflow problems that prohibit speculative investments in their future. More to the point, every gene underlying every potential innovation has to yield higher evolutionary payoffs than competing genes, or it will disappear before the innovation evolves any further. This makes it hard to explain innovations.”

problem and ease in the second show evidence of a cheater detection module—just consider that we adapted to the enforcement of cooperative tasks and are quick at identifying free riders.

Referenced in

Fooled by Randomness