The Signal and The Noise by Nate Silver

作者 Nate Silver 擅長於統計分析  曾利用工作之餘自行開發出分析棒球球員能力的工具  有點像魔球(Money Ball)  另外他也喜歡玩德州撲克  後來也曾辭掉工作而從事職業撲克  不過因一些法規改變導致而到更專業的網站賭博而虧了些錢後  就放棄以賭博為生的日子  他也做一些美國大選的預測  由於有相當的準確性  因此目前在紐約時報有一專欄 專門分析美國大選

他的新書 The Signal and the Noise 主旨在於說明有哪些事是比較容易做預測的  哪些較不容易  書中有提到  選舉  棒球  籃球運動彩券  西洋棋  天氣  地震  撲克  股市  傳染病  經濟 恐怖活動等等

書中提到信號和雜訊常混在一起  所以如果誤把雜訊當信號 則會影響到預測的準確性

另一方面很多科學太強調簡化的觀念  簡單的模式固然容易懂  但有些行為是較複雜的  如果用這簡單的模式來預測就會不切實際

書中也提到一個簡單的機率分析概念<貝氏定理>  最通常的應用在於醫學的檢測上面  例如若 癌症基因檢測為陽性  那真的有癌細胞的可能性有多少  這跟此癌症在人口的比率有直接的關係  簡單的說明如下

人口中可分為 有癌症基因 與 無癌症基因 的兩族群

有癌症基因檢驗出陽性的機率為 X

無癌症基因檢驗出陽性的機率為 Y

那檢驗出陽性且有癌症基因的機率為

X * 有癌症基因的人口比率 / (  X * 有癌症基因的人口比率 + Y * 無癌症基因的人口比率 )

以大腸癌為例  假設大腸癌發生率約為每10萬人有307.4人  X 假設為 99%  Y 假設為 5%

檢驗出陽性且有癌症基因的機率為  0.003074*0.99 / (0.003074*0.99 + (1-0.003074)*0.05)
約為 5.75%

所以即使 X 高達 99%  實際上 驗出陽性且有癌症基因的機率僅為 5.75%

榮總有一癌症篩檢快報

　　癌症 - 陽性人數 - 確診癌病(比率)
子宮頸癌 - 104 - 56(53.9%)
　　乳癌 - 630 - 26(4.1%)
　直腸癌 - 303 - 19(6.3%)
　口腔癌 - 201 - 4(2%)

由此可知即使篩檢陽性的正確性很高也並非意味著確診的比率會很高

不過如果把發生率提高十倍  把 307.4 改為 3074
則 0.03074*0.99 / (0.03074*0.99 + (1-0.03074)*0.05) = 38.57%

由此可看出癌症的人口發生率與確診的比率有相當大的關係

* * *
有一個簡單的觀念可以加深大家對<貝氏定理>的印象  例如下雨地上會濕 X=100%   那如果地上濕了 下雨的機率有多少呢  顯然不會是100%  也不會相當高

地上濕了是因為下雨的機率   跟 { 下雨的機率 與 不下雨地上會濕的機率(Y) 還有 X } 有關

* * *
有些時候事件的發生率並無法掌握得很好  就必須用推估的  然後根據結果再回去修正  例如先假設一癌症發生率去推估確診比率  再用實際的確診比率去修正假設的癌症發生率  這樣循環下去就會得到更接近真實的資料

這也意味著一個模型通常僅是真實世界的一個簡化的版本  經過不斷的修正會慢慢接近真實的世界  當然也必須謹慎的選用資料  避免把雜訊當成正確的信號來處理

書中還提到很多有關於預測與賭博的趣事  有興趣的人不妨參考看看

The Signal and the Noise: Why So Many Predictions Fail-but Some Don't

Is X related to Y?

As we are investors and speculators, we always want to forecast the future, so we would like to find correlations between Xs and Y.  Xs are any kind of historical data that could come from macro-economic indicators, weather reports, fundamental analysis, technical indicators, and almost anything that you can think of.  Y is what we want to forecast, most are like stock prices, prices of commodities, or market indices.

So what we usually do is to find the "correlation" between X and Y, defined as corr(X,Y).  Basically we can find all the corr(Xi,Y).  So if corr(Xi,Y) is closer to 1 or -1, we might think there is some kind of correlation between X and Y, assuming you have enough samples of data points.  What if corr(Xi,Y) is almost zero?  Is there no correlation at all? or no relationships at all?

When we exam all the corr(Xi,Y), we tempt to pick those value closer to 1 or -1, and use those Xi to do more advanced analysis.  We used to neglect the Xi, whose corr(Xi,Y) is close to zero.  Is this approach valid?

Let's take a look at a simple example, although corr(Xi,Y) is close to zero, actually Xi is "correlated" to Y, or the causation does exist.

Let's say, Y=XOR(X1,X2), X1 and X2 are random numbers of -1 and 1 with probability prob(-1)=prob(1)=0.5.  corr(X1,Y) and corr(X2,Y) are both zero.  But the relationship between (X1,X2) and Y does exist.

Most people who know XOR know how this trick works, but in real life, it's hard to imagine how this could be applied to data analysis.

A weird thing could happen is like that, when we try to find all the possible Xs:=X to forecast Y, and we define Y as f(X), f could be any kind of function, and we calculate corr( f(X),Y).  We want to find the best f, such that f makes corr(f(X),Y) close to 1.  So we can use f(X) to do some forecast.  As we know, X might not be complete, there might be some Xu not observed, or ignored due to corr(Xu,Y) is close to zero.  And corr(fu(X),Y) might be close to zero for some fu, either.  But actually Y=XOR(Xu,fu(X)).  If we ignore Xu or fu, there is no chance to find this relationship.

It means we have to pay attentions to those Xu and fu, even corr(Xu,Y) or corr(fu(X),Y) is close to zero.  So having no correlation(individually) could mean having correlation(communally)?

How many Xu and fu are there to check?

$OEXA200R Market Indicator

An article by John F. Carlucci on dshort.com mentioned a market indicator using OEX stocks above the 200 day moving average, $OEXA200R, with the other indicators as buy/sell signals for the general US market.  The introduction of the methodology is here.

You can find the daily update of the $OEXA200R by Michael E. Pitre.

I created and monitored the OEX portfolio at finance.yahoo.com.  For S&P500 at current level 1380, if it drops another 3.5+%,  bulls should be cautious according to the methodology mentioned before.