The waterfilling of fate

The term waterfilling is familiar to information theorists as a well known solution to a famous optimization problem.

The idea is simple. If you had to carry your stuff on donkeys from point A to point B, you would be better off piling more stuff on the stronger donkeys. 'Waterfilling' tells you exactly how much to pile on which donkey so as to maximize the total weight carried.

Now comes the fate part. Some days are luckier than others. And you cannot have missed the army of psychics, palmists, astrologers, tarot-card readers, numerologists... who claim to know much about your fate. The business of prediction is so heavily loaded with confident pretense and deception that it has hardly any credibility. But imagine if there were a way to predict, with 51% correctness the value of an unbiased coin toss, a coin that you could bet on consistently. Or if a man could distinguish between times when his judgment is sound versus when it is not (a catch-22 situation). Impossible? Who knows. We willingly accept that any two massive objects attract each other because we have read it in a book. The world was crawling with people before Newton, but hardly anybody would have agreed. Indeed, of the 1.8 million years that man has walked this earth, almost all were spent believing that the earth is flat. We humans, as a species, are not very imaginative, nor particularly open to new ideas. What we know about nature is astonishing, if you think about it, yet perfectly mundane because we read it up for grades.

What if we could waterfill fate? Uncertainty is an enemy that we try to defend against. What if we could befriend it?

“There is a tide in the affairs of men, Which taken at the flood, leads on to fortune. Omitted, all the voyage of their life is bound in shallows and in miseries. On such a full sea are we now afloat. And we must take the current when it serves, or lose our ventures.” - William Shakespeare

How can I know for sure that I know?

Learning is an elusive thing. The other day I was reading an interview of Robert Gallager, a pioneering information theorist. The man had a really deep understanding of information theory, and he frankly says that it took him a lot of time to understand some really simple things... well, things that we tell each other are simple. My conjecture is that very few people understand the "simple things", but most of us do a pretty good job of keeping up the pretense.

For people in positions like mine, learning is an unending process. In my experience, learning by reading alone leaves too many gaps. Whenever I try writing what I think I understand, these gaps immediately become visible. Implementing, when possible, further enhances understanding. A person who has implemented the BCJR algorithm knows a lot more about it than a person who has read about it.

How to test the depth of understanding? Two tests are useful. One, you should be able to solve any problem based on the content. Two, try explaining it to a person (or an imaginary person) who does not know the content, but has the background to understand it. Both are time-consuming exercises. Gallager claimed that he enjoyed writing his book (which was an exercise in making material that he knew accessible) more than he enjoyed writing his papers. This is probably a trait to be found in every great teacher, but I believe that a penchant for simple, solid understanding can also go a long way into making a person a great researcher.

I think I know...

Sometimes we know, other times we think we know. There is a big class of people who cannot distinguish these two. To grow in knowledge, one must keep up the habit of entertaining doubt. To grow in power, however, the opposite is helpful.

Knowledge and confidence

Bertrand Russel said in many different ways that knowledge and confidence are inversely related. Although a lot of knowledge can lead to genuine confidence, the amount that we know is tiny as compared to what is there to be known. A person immersed in the process of learning is acutely aware of the sea of ignorance that we are swimming in.

As a rule of thumb, therefore, when I begin to feel that I know much, it is usually a sign that I have been drifting away from learning. It is then time to go back to reading, a pen, and a piece of clean white paper.

fair experiments

I was reading "The only guide to a winning investment strategy you'll ever need" by Larry E. Swedroe. A good book. I recommend reading it.

It started off well. Along the way, it got a little wordy, a little repetitive, but quite OK. Then came one story that disappointed me. In his attempt to deprecate technicians who try to predict future stock trends he came up with the following. A group of technicians were given randomly generated charts that were posed as stock price charts. The technicians made their predictions, and then it was disclosed that the stock trends had been randomly generated. HA HA. The author believes that this example will convince one and all that technical analysis is futile.

Perhaps it is true that it is impossible to make accurate predictions based on past stock price charts. BUT, the experiment does not show that. In fact, this prank is not an experiment. To give real stock price charts to the technicians and to compare predictions with what actually happens would have been an experiment.

A random number generator is capable of generating possibly any pattern. For example, I could have a random image generator that produces 2-D color pictures. Such an image generator is certainly capable of producing the Mona Lisa given enough attempts. Now, suppose that it does generate an image that looks like the Mona Lisa. I take it to an art expert and ask him what painting it looks like, and get the obvious answer. Then I reveal that the image was in fact randomly generated. HA HA.

Does it show that the art expert is an idiot? On the contrary, the person conducting such an experiment certainly is. The person is even insecure, scared of being proved wrong, so much so that he will take no chances, will ask no questions, will receive no answers, lest he receive the correct answer.

There are people with scientific temper, and then there are people who don't know the difference.

look before you leap

Good intentions are important, yet never sufficient to ensure that the end will be good. In the end, consequences matter. Think of the end before beginning. Sometimes the end is unpredictable, yet often we simply blind ourselves by confusing the intended outcome with the likely (sometimes inevitable) outcome. There will be no dearth of people telling you exactly where you went wrong, after you have gone wrong. Never indulge in post-analysis but for building your own foresight.