Wednesday, February 1, 2012

Human Organizational Incompetence

Reading this post on the consulting industry, and the dozens of supporting comments, set in motion a train of thought that made me realize just how bad humans are at organizing ourselves and coordinating our activities.

I'd recommend reading the whole thing, but if you can't, here's a key quote,

My guess is that most intellectuals underestimate just how dysfunctional most firms are. Firms often have big obvious misallocations of resources, where lots of folks in the firm know about the problems and workable solutions. The main issue is that many highest status folks in the firm resist such changes, as they correctly see that their status will be lowered if they embrace such solutions.

I can recall dozens of case studies with this pattern, and it was true of the firm I used to work at. Managers insist on doing things that literally destroy the organization, because it gives them a feeling of power and control. A typical example is keeping information secret from employees. Most people who have been part of a large organization have similar experiences. Decisions are made based on office politics rather than scientific judgment or business efficiency. This problem seems inherent in human nature; it happens whenever people try to organize themselves to do something.

More often than not, the business consultants are simply there to bypass office politics and give a stamp of approval to what any thoughtful observer knows needs to be done. CEO's and shareholders are willing to pay millions of dollars for this service, which suggests that office politics costs the economy billions of dollars each year.

But economists know that the failures of human-led organization are far deeper. One of the things that we have to do in every introductory class is to explain how the price system works and what it is good for. It has repeatedly been shown that a system of market prices allocates resources far more effectively than central planning in almost every case. Whenever there is something that prevents the price system from working, the result is chaos and mismanagement.

A naive observer of human behavior might think that a human planner would be strictly better than a price system. After all, a price system is one of many possible allocation mechanisms a planner could use. When the price system works, the planner can use it, and when it does not, than a different system can be used. But things rarely work out that way. Central planning, where humans can choose from a menu of options, usually does worse than blind adherence to a simple price system, because of the inherent psychological desires of the human race that cause power to corrupt decision-makers.

Economists always sell the price system as an amazing model of emergent order, focusing on its elegance and benefits. But objectively, allocation via market prices is a remarkably blind, dumb, simple, stupid mechanism. Its flaws are obvious and well-documented: externalities, large income differences, information assymetries, irrational short-term desires, and an inability to accurately measure value can can all cause it to fail.

Our inability as humans to consistently do better than this flawed and senseless mechanism is truly amazing. Human discretion reliably causes dysfunctional outcomes that are far worse than the problems generated by markets. It is one of our more fundamental failures as a species.

A well-programmed computer would not have either of these problems. It would be strictly better than markets, using a price system when appropriate and using something else otherwise. It could capture information not available from market prices, such as social media chatter and patterns of network connections among people. It could measure and test people and products to find ideal fits. It would not be selfish, or play politics, or lie, or be distracted by games of power and signaling. It could do a better job than either human management or markets of aggregating information and using that information to make decisions.

We are already seeing glimpses of this. Think about the recommendation engines of Netflix and Amazon. They already do a better job of predicting what products you will enjoy than most people who know you. They certainly do a better job than market prices. A high price is only a signal of quality if your tastes are very similar to everyone else's and there is a reliable connection between inputs and output quality. Neither of these is typically true of entertainment. The decision of how to allocate your entertainment time is best done not by other humans, and not by market prices, but by a cleverly programmed computer.

I predict that, sometime in the next few decades, a private equity company will program an intelligent-allocation computer and use it to replace the management of companies it buys out. There will be many experiments, and many of them will fail. But when they get it right, a lot of dysfunctional human-led firms will be quickly replaced by machine-directed enterprises who make production and hiring decisions that coordinate human activity better than humans can.

No comments: