by Judith Curry
How can we cope with becoming more hyper-specialized and fragmented in our personal knowledge bases while at the same time being exposed to too much relevant formation on the internet for any of us to learn and process?
Michael Smithson provides a modest proposal for addressing this challenge: We should all become expert about experts and expertise. That is, we should develop meta-expertise.
Climate Etc. has had several previous posts on expertise, focused on the politics of expertise:
The blog ignorance and uncertainty has an interesting new post “Expertise on expertise.” Some excerpts:
We can’t know everything, but knowing an expert when we see one, being able to tell the difference between an expert and an impostor, and knowing what it takes to become an expert can guide our search for assistance in all things about which we’re ignorant. A meta-expert should:
- Know the broad parameters of and requirements for attaining expertise;
- Be able to distinguish a genuine expert from a pretender or a charlatan;
- Know whether expertise is and when it is not attainable in a given domain;
- Possess effective criteria for evaluating expertise, within reasonable limits; and
- Be aware of the limitations of specialized expertise.
That said, the Wikipedia entry also raises a potentially vexing point, namely that “expertise” may come down to merely a matter of consensus, often dictated by the self-same “experts.”
What are the requirements for attaining deep expertise? Two popular criteria are talent and deliberative practice. Re deliberate practice, a much-discussed rule of thumb is the “10,000 hour rule.”
The 10K rule can be a useful guide but there’s an important caveat. It may be a necessary but it is by no means a sufficient condition for guaranteeing deep expertise. At least three other conditions have to be met: Deliberative and effective practice in a domain where deep expertise is attainable.
Back to the caveats. First, no deliberation makes practice useless. Having spent approximately 8 hours every day sleeping for the past 61 years (178,120 hours) hasn’t made me an expert on sleep. Likewise, deliberative but ineffective practice methods deny us top-level expertise. Early studies of Morse Code experts demonstrated that mere deliberative practice did not guarantee best performance results; specific training regimes were required instead.
There are, at least, some domains that are deeply complex where “experts” perform no better then less trained individuals or simple algorithms. In Philip Tetlock’s 2005 book on so-called “expert” predictions, he finds that many so-called experts perform no better than chance in predicting political events, financial trends, and so on.
What can explain the absence of deep expertise in these instances? Tetlock attributes experts’ poor performance to two factors, among others: Hyperspecialization and overconfidence. “We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly,” he reports. Tetlock also claims that cognitive style counts: “Foxes” tend to outperform “hedgehogs.” These terms are taken from Isaiah Berlin’s popular essay: Foxes know a little about lots of things, whereas hedgehogs know one big thing.
Another contributing factor may be a lack of meta-cognitive insight on the part of the experts. . . . The disquieting implication of these findings is that domain expertise doesn’t include meta-cognitive expertise.
Finally, here are a few tests that can be used to evaluate the “experts” in your life:
- Credentials: Does the expert possess credentials that have involved testable criteria for demonstrating proficiency?
- Walking the walk: Is the expert an active practitioner in their domain (versus being a critic or a commentator)?
- Overconfidence: Ask your expert to make yes-no predictions in their domain of expertise, and before any of these predictions can be tested ask them to estimate the percentage of time they’re going to be correct. Compare that estimate with the resulting percentage correct. If their estimate was too high then your expert may suffer from over-confidence.
- Confirmation bias: We’re all prone to this, but some more so than others. Is your expert reasonably open to evidence or viewpoints contrary to their own views?
- Hedgehog-Fox test: Tetlock found that Foxes were better-calibrated and more able to entertain self-disconfirming counterfactuals than hedgehogs, but allowed that hedgehogs can occasionally be “stunningly right” in a way that foxes cannot. Is your expert a fox or a hedgehog?
- Willingness to own up to error: Bad luck is a far more popular explanation for being wrong than good luck is for being right. Is your expert balanced, i.e., equally critical, when assessing their own successes and failures?
- Tetlock attributes experts’ poor performance to two factors, among others: Hyperspecialization and overconfidence.
- domain expertise doesn’t include meta-cognitive expertise.