The Unaccountability Machine

Quote

Systems don’t have motivations, so they don’t have hidden motivations. If the system consistently produces a particular outcome, then that’s its purpose. But on the other hand, systems don’t make mistakes.

Unaccountability Machine is a branch non-fiction book on decision making, systems, and a history of cybernetics. Its central thesis is that accountability, particularly in large business/systems, has been transferred from people to policies, he calls this an Accountability Sink. Along the way Davies explores the history Cybernetics and it’s arguably father, Stafford Beer. Through this lens on decision making, he explores the pathologies in decision making that lead to large systems “making” decisions that no individual within the source would endorse (see Schipol squirrel shredding incident).

Key Ideas

• Large systems “do” things that may be counter to what every individual within the system may want. Examples: Fox News spreading 2020 disinformation and the Schipol Squirrel Shredding incident

• Large systems (economies, large corporations, etc.) are not comprehensible, or at least explainable, and so accountability transfers from an individual running it to the system itself.

  • This complexity rises not just from the number of components (people, business-units, etc.) but all of the combinations of the interactions between units

  • This complexity eventually means that people have to treat the system as a whole, as a black box.

    Dan Davies ¹

    An organisation does things, and it systematically does some things rather than others. But that’s as far as it goes. Systems don’t make mistakes – if they do something, that’s their purpose. But it also works the other way round. Systems don’t have inner desires, so they don’t do things intentionally either … It’s not what you’d call an attractive view of the world – most of us are just cogs in a machine, working on what’s in front of us while the big picture is determined elsewhere. It’s particularly unattractive if you’re a CEO or head of government, and you feel like you ought to be making the decisions. But it feels comprehensible, in tune with the way things seem to keep turning out. Among other things, it might provide some sort of explanation of why it is that public inquiries into major scandals always seem to be so unsatisfactory.

• There is a failure model within the viable systems model and management by exception which is that transmission on exceptions through the various systems take time. The solution here is to have signals that cut through the recursive viable systems (red-handle signals).

• Accounting systems and metrics provide decision-making signals, but are flawed models and so it is often the role of middle management to manipulate the financial reports to provide the correct timely and actionable information. This can be a problem if this error correction is not enough or if the middle manager distort the information for their own purposes.

• You arrive at a situation where you have compounding blind spots: economists are building models that ignore bits of reality on a macro scale while managers work off of inflexible accounting systems. Managers are additionally overwhelmed by complexity, academics in management are too weak and divided while consultants are telling their clients what the want to hear. This leads to a model that drifts from reality and the system (incentive and fail safes) seeks to preserve the status quo rather than adjust to reality.

• With the rise of AI, decisions will be even less attributable, and we must get comfortable with attributing things to the system rather than a person. There must be a way to communicate with the system though.

• The complexity of the system (at any level) will continue to increase so we need to continue to evolve the systems to handle more variety.

• The populist movements across the world were a “red handle” feedback between the governed to highest level of the system that life had become “too complex”, there was too much input and they no longer felt they had autonomy or control over their lives. The people leading the populist movements promised a return to simplicity.

• Debt is a control technology that shifts the power from the managerial class to the investor class.

  Quote

  If the market is the brain of the capitalist system, its nerves and muscles are made out of the debt relationship

  • Debt reduces the information down to “Is the interest rate worth the time and risk?” and “Will I get paid back?”. These are the only questions an investor has to answer.
  • Debt becomes the over-riding signal and concern for the operators of a company

• The managerial class lost the struggle with the capitalist / investor class comprehensively. High level executives were aligned to the investors through stock options and lower level managers were cowed by threat of redundancy.

• Outsourcing broke the information processing and regulatory systems of business by creating boundaries between key parts of the operation and management structure.

  Quote

  The global economic system had gradually removed vital parts of its own brain; informal and rich communication channels within organisations were replaced with formal and contractual links which often crossed time zones. Intelligence functions that were meant to consider the future were swamped with debt, so that they couldn’t pay attention to anything that didn’t generate cash for the next payment. Because it was reducing its capacity, the corporate system with its managers and investors was no longer able to absorb volatility and shocks; the economy had to be reorganised so the working class performed this function. It looked like the final triumph of capital, and it was.

• Davies builds up to his conclusion and solutions and I encourage you to read the book, however for my purposes

  • Businesses must not work as single metric optimizers, paperclip maximizers, they must begin to make decisions again that are more nuanced.
  • In order for businesses to have a less myopic view, Davies argues that private equity leveraged buy outs must be reformed to take incentives away from loading companies up with debt. When a company decision making is not short circuited by debt, the resource bargain between capital and management can re-stabilize. Management can focus on viability and the future. A shift away from profit-maximization (debt service) will allow the managerial class to focus on customers and employees.
  • A system that only maximizes a single objective will go sideways, and therefore a maximizer needs a higher-level system watching over it. Therefore, economists can not be in charge.

Secondary Ideas

• Models are the basis of assumptions and simplifications and so often they are engineered to reach their conclusion.
  • On top of this, the people who design the model often select the data collected for validating metrics.
  • “Everyone who has put together a business plan knows that if you can’t fudge the key assumptions to justify the decision your boss wants to make, you don’t know enough about the business.”
• “Variety” of a system is the number of states a system can exist in. The “principle of requisite variety” states that in order to be “regulator” of a system you need an equal amount of variety as the system.
• Policies are created to reduce the amount of time making decisions. When you set a policy you have to build a mechanism for handling exceptional cases or be confident that you have handled all cases.
• Explainability is a major concern with AI, who is responsible when a driver-less car kills someone? However, this idea can be extended to corporations, if we are concerned about being able understand or attribute actions for an AI, why are we not equally concerned about inscrutable actions of a corporation.
• Flow of information, or lack there of, through a system can create pathologies. For example, ataxia within a biological system.
• The polycrisis has it’s roots in dysfunction within issues of identity within a system.
• Information is only relevant when it arrives in a timely fashion and form that can be acted upon. Otherwise, information is just data.
• The idea of the market is a computing machine ( “An artificial intelligence”) that reaches the optimal solution is another form of abdicating accountability to the system. This shows up in phrases like “the invisible hand” or “market efficiency”)
• Economics often flattens time and uncertainty into simplified models.
• Ricardian Vice: make a model of the economy, stripping away complexity by making simplifying assumptions, show that the conclusion follows from the assumptions. Assume that the conclusion has been proved in the in the real model. This is a flaw in economic modeling.
• Managers are the front-line (System 1) of the larger economic system that economists seek to be higher level organizers of.
• Accounting is the language of management.
• “General accepted accounting principles” are a convention that serves as an accountability sink for decision making in cost allocations and management decision making.
• Management consultants are an accountability sink but they also serve as a cross-hierarchy communication channel transmitting and then selling ideas to upper management.
  • Management consultants are often telling people obvious generic things that they do not want to hear, so their is tendency to tell clients what they
• Phrases like management by objectives are actual just ways of saying the business should be restructured in a way that it doesn’t generate complexity faster than it can be managed
• Information Theory
  • Wiener’s Version: information was represented by negative entropy
  • Claude Shannon’s: “information was taken taken as representing the extent to which a sequence was unpredictable (whether it constituted ‘news’)”
• Milton Friedman, rise of neoliberalism, and the singular focus on profit maximization.

Quote

The principle of diminishing accountability: Unless conscious steps are taken to prevent it from doing so, any organisation in a modern industrial society will tend to restructure itself so as to reduce the amount of personal responsibility attributable to its actions. This tendency will continue until crisis results.

Davies argues the above principle is too simple, you don’t want to have a lot of mid-manager variability in process, you also do see responsive changes to policy in major events.

Cite

1. Davies, D. The Unaccountability Machine: Why Big Systems Make Terrible Decisions – and How the World Lost Its Mind. (Profile Books Ltd, 2024).

Metadata

Title:: The Unaccountability Machine: Why Big Systems Make Terrible Decisions – and How the World Lost Its Mind Year:: 2024 Publisher:: Profile Books Ltd ISBN:: 978-1-78816-954-7

1. Davies, D. The Unaccountability Machine: Why Big Systems Make Terrible Decisions – and How the World Lost Its Mind. (Profile Books Ltd, 2024).