I’m just finishing reading “Trading and Exchanges” (Larry Harris) which is a 600 page textbook. My overall conclusion is that statements like “even experts can’t beat the market” are HUGELY oversimplifying what’s actually going on in the market.

On the market you have fund managers, which are usually what people mean when they say experts, but all fund managers put together comprise a large fraction of the market. This makes them the average by construction and obviously they can’t beat it(*). However, you also have speculators, timers, value-traders, momentum-traders, gamblers, brokers, specialists, dealers, prop trading companies, etc.

(*) If you say that fund managers can’t beat the average, you’re effectively saying that class of high school graduates can’t get a grade that’s higher than the average of the class. This is true for the average: The average of the class can’t be higher than the average of the class. However, individual students can and do beat the average.

There are all kinds of different motivations and behaviors in the market. In particular a large part of the market action is utilitarian and not intended to beat it, e.g. saving, lending, and hedging (harvests, commodity costs, interest risk). This means that a lot of money is invested because IT HAS TO BE, not because it’s profitable. E.g. someone needs to purchase 500,000 or 5M shares of GE simply because it’s in their fund-charter. Not because it’s profitable.

Often experts (mind you, those experts are generally college-graduates produced in huge numbers from the same school learning the same things and talking to each other everyday and in many ways thinking the same ideas, not necessarily creative independent geniuses—”expert” simply means knowing the regulations; “creative genius” may be a sufficient criterion but it’s not required) are so regulated or need to manage such large sums of money that they can’t profit from the inefficiencies in the market. Sometimes they act to create inefficiencies. It’s worthwhile to keep in mind that there are someone who actually invests to make the markets efficient. Markets are not magically efficient.

Anyway, a couple of points …

1) Some people can actually beat the market. However, I think most of us who do that are quite interested in the market and think about it a lot. This thinking adds value to the investment process and thus we profit. None of us are professionals but all of us know at least as much about the market as someone with a degree in business.

2) Knowledge isn’t everything. You also need to act correctly. E.g. you may be able to aim and fire a gun, but could you still do it while taking fire. Can you follow your strategy when you’re losing money?

Given (1) and (2) “civilians” can beat the market.
See http://www.people.hbs.edu/jcoval/Papers/persist.pdf for a survey of 100,000+ individual retail accounts. The top decile beat the market by 8%/year consistently. Conversely, the bottom decile lost money consistently.

3) Some people know that they don’t know enough about the markets but they do know enough to identify people who know. (“I don’t know much about carpentry myself but I know enough to tell a good carpenter from a bad carpenter”). They should pick a money manager that beats the market. Warren Buffett is an example. You can t-test his performance and show it’s extremely unlikely that he’s just being lucky. Even the “out of 100,000 random investors, someone is bound to be that lucky”-argument doesn’t hold. He’s simply that good.

4) Some (many?) people know that they don’t know enough the market and they also know that they can’t identify a manager who does know. (“I don’t know anything about carpentry and I also don’t know how to identify a good carpenter”). These people should by all means realize this about themselves and pick index funds. In particular, while they should rationally go with e.g. Warren Buffett, they know they’ll jump ship if their manager has a bad year. However, they also know they won’t jump off the market. (Of course some will do even that. They’re beyond help.) I would not count this as a “defeat” in the “I’m not smart enough”-sense. It may simply be that investing is uninteresting or that they have so little savings relative to their earned income that they would be better off spending their time improving their earned income instead of their investment income. This is the case for most people (but not for ERE).

Given (3) and (4) the best generic cover-your-ass advice-for-everybody is to buy and hold broad market index funds. (“If you don’t know or trust your driving, you should take the bus.”) The only danger here is that index funds typically concentrate on the equity markets which is just a small sliver of the total market (you also have currency, land, commodities, and bonds). In particular, it is an asset class that’s done particularly well, trend-wise, from 1983-2001 which is why “uninformed investors” are still going woot woot woot about it(*). I suspect that after 10 more years of flat markets, equities aren’t going to look so hot anymore.

(*) I think to a large extent the current recommendation of index funds is not much different that the widespread recommendation of money market funds in the late 70s/early 80s.

By the way, most people start naturally in (4). If they’re interested and capable they might move to (3). Then to (2) and then to (1). I think this is a good progression. I think it is a mistake to believe you can just jump into group (1) by reading a couple of books.

PS: I think it’s somewhat problematic that pension-risk has been shifted to the individual in a defined contribution sense rather than as a defined benefit sense. Indexes are clearly dangerous as they are being used far more by large traders (like eMinis/ES) than as a pension-vehicle. It’s no good that “civilians” can lose 50% of their retirement funds just because Wall Street got overexcited.