Complexity: Darwin’s Gate for Organizations
Bottom Line Up Front
Organizations must evolve.
In today’s complex environment, organizations must adopt structures and processes that enable them to be adaptive and dynamic. To achieve this, they must confront the limitations of traditional hierarchies and embrace flatter, more decentralized structures, with an emphasis on smaller action units (individuals and teams) with clearly defined interfaces that allow rapid and transient collaboration and partnering both within and across organizations.
“In the long history of humankind (and animal kind, too) those who learned to collaborate and improvise most effectively have prevailed.” — Charles Darwin
- Centralized vs. Decentralized: Centralized organizations evolved to achieve large-scale coherence in action, allowing for a collective impact greater than that which an individual alone could achieve. However, centralized organizations are limited in their response by the capacity of the “centralized” point—the person or position directing action—to process information.
- Establishing Decentralized Organizations: Today’s informational and technological environment tends to overwhelm centralized organizations, leading to the need for decentralized organization to survive and thrive.
We offer several recommendations for enabling decentralized organizations:
- Deferring to those closest to the problem: increasing an organization’s ability to process information
- From departments to teams: positioning to better respond to complexity
- Designing for interface: enabling organizational flexibility
- Empowering generalists and fostering personal growth: for individual adaptability to match organizational
- Leveraging modeling and simulation to extend the visible horizon: identifying and creating custom solutions
Discussion
Our recent reports have touched on a variety of aspects of the changing informational and technological environment. The New Information Environment (May 2017) detailed how information has evolved, leading to a massive proliferation of information that is fast, connected and networked, and how advanced analytics (e.g. Artificial Intelligence, Machine Learning) can assist organizations in becoming more agile and adaptive by filtering through the noise. The Pending Technological Upheaval (May 2018) delved deeper into strategies that can be adopted to explore and exploit emerging technologies which introduce deep uncertainty into our ability to forecast, namely Active Design and Evolutionary Innovation.
In between those two reports, Active Policy Design (September 2017) explored the impacts of today’s information ecosystem and its enabling technologies on government policy making, and some necessary organizational responses to grapple with the complexity these generate. The piece focused on the need for organizations to support Active Policy Design, deploying iterative design cycles
that engage all stakeholders in order to overcome the challenges of achieving desired outcomes in a dynamic and uncertain environment. However, as this new information and technological environment represents a tectonic shift in how success is achieved, a deeper examination of what this means for organizations is warranted.
Navigating Complexity Without Complication
A complex environment can only be reliably navigated and responded to by an organization that matches its complexity. This general statement was made mathematically precise in the mid-20th century by the cyberneticist W. Ross Ashby. Termed “Ashby’s Law of Requisite Variety,” it helps explain why we see such richness and complexity in living systems — the real-world environment organisms must deal with to survive and thrive presents a huge number of challenges and uncertainties. When an organism does not adequately manage the complexity of the environment, natural selection happens. Organizations, like organisms, must also respond to complexity to survive and thrive.
A complex organization should not be confused with a complicated one. In fact, the policies and processes of a complex organization from the perspective of any individual can, and should, be rather simple. The idea is to set conditions such that the appropriate level of complexity of the organization as a whole can emerge from the interaction and collaboration of the individuals and teams that make it up. Complication is an enemy in any environment, and will only exacerbate the challenges that complex environments pose. Complication occurs when we use the wrong kind of organization for a given problem set or environment.
In order to understand how an organization can be complex without being complicated, we must consider the structure of information- ow and decision-making in different kinds of organizations, and what challenges different structures are effective and ineffective at addressing.
Centralized Organization
One of the purposes of human organizations is to generate coherent behavior across a large number of individuals in order to achieve large-scale effects that would not be possible if each were acting independently. Roman Armies present a classical example of leveraging such large-scale coherence to devastating effect. Often, hundreds or thousands of individuals would move together in tight formations “as one,” potentially overwhelming opponents by sheer mass. Soldiers were able to move together by obeying the commands of a designated officer making tactical decisions about the group’s movements.
In order to achieve large-scale coherence like a Roman Army, decision-making must be centralized to an individual or small number of individuals. Those individuals who are “in charge” are responsible for understanding the current situation, making decisions about how the whole organization should respond, and relaying those decisions to those who will carry out their orders. Those individuals are expected to obey those orders closely, otherwise the coherent behavior will break down. Those carrying out the orders are extremely constrained in their ability to make decisions freely — they are not autonomous, but rather locked in to the movement of the group.
This is the philosophy of “command and control,” and it naturally leads to a hierarchical organizational structure. On the top of the hierarchy, decision-makers take in the big picture, and give guidance and instruction to their subordinates, who each in turn do the same.
In order for information to flow across a strict hierarchy, it must move up and down the “chain of command,” rendering much information sharing indirect, high-friction, or non-existent. As the information flows “converge” as they move up the hierarchy, individuals at each level must simplify their understanding of what is happening at the level(s) below them. When orders or instructions are projected down the hierarchy, the behavior of those carrying out the orders can collectively be no more complex than the imagination of the individual who designs the orders. These are unavoidable consequences of the fact that every individual, no matter how brilliant, is bandwidth- limited.
The more layers of hierarchy, the more simplified the decision-makers’ understanding of the situation necessarily becomes, and the number of local details that must be ignored when individuals carry out orders swells dramatically. When the environmental complexity becomes too great, a hierarchy cannot respond effectively — the centralized decision process breaks down due to oversimplification.
Decentralized Organization
In contrast to centralized organizations, those that are decentralized spread decision-making around. In a decentralized organization, individuals make decisions based on their own local vantage point rather than simply carrying out orders. They should have some understanding of how their decisions t into the big picture, but importantly there is no need for any individual to have a complete understanding of the entire organization — this frees the organization to become more complex than any individual within it.
Whereas centralized organizations must deploy hierarchical structures, decentralized ones are better understood as dynamic networks. The nodes in the network may represent individuals, teams, or whole organizations that are connected to each other in some way, for example through collaborations, partnerships, providing services, or client-customer engagements. Unlike a hierarchy, there is often no clear ‘up’ or ‘down’ in the organizational structure — connections are ‘lateral’ and the organization is ‘ at’.
The network itself is dynamic in that the pattern of connections changes over time. When connections between nodes are no longer needed they can disconnect, and when new connections are needed they can be forged.
Decentralized systems are often characterized as ‘bottom-up’ because most of the decisions are made and actions taken locally, while the behavior of the entire organization emerges as a combination of these local decisions and actions. In bottom-up systems, individual agents are able to decide what resources they need to accomplish their tasks and seek them out, rather than needs being predicted and planned for in a ‘top-down’ design process.
A striking example of a mismatch between what top-down designers imagine and what emerges from bottom-up decisions can be seen in the common example of a well-worn walking path across a lawn that is favored by pedestrians over the intended walkway that is paved. In an interesting case study, the University of Maryland, being aware of this common outcome, did not install any pathways into its McKeldin Mall until organic pathways emerged from people traversing it — only then did they pave them to make them more permanent.
The same lesson holds in organizations — it is often best to let individuals and teams decide for themselves who can best help them solve their problems, and allow them to go to them directly. It is too much to ask of an individual to understand the nuances of the needs and challenges across an organization. When those taking action are at least semi-autonomous and able to make their own decisions, organizational creativity can be unleashed and the organization can surpass the complexity of any individual.
Challenges of Decentralization
Decentralization requires a shift in the way we think about how work gets done and the recognition that the most relevant information is often more readily available at the level of execution. One of the biggest challenges is simply unlearning the tendency to apply a “command and control” mindset where it is insufficient or inappropriate. It is often in the best interest of an organization to grant autonomy, making room for creative and innovative problem-solving. The “optimization” and “efficiency” that centralization promises can be seductive, but if applied in the wrong problem space, a centralized system will prove more costly than a decentralized one that can respond effectively to complexity. Unintended and unanticipated consequences stemming from a complexity mis-match result in unexpected costs, including those associated with necessary re-organization.
There are legitimate challenges beyond mindset to overcome in a decentralized setting. For instance, how can we maintain alignment of various players towards a common goal? When there is conflict or disagreement, who has the final say? How do individuals decide what is important and ensure their efforts contribute to creating value? Below, we offer some guidance and recommendations to overcome some of these challenges.
Finally, it should be noted that in general organizations are not “purely” centralized or decentralized. The binary comparison is intended for conceptual clarity, but not to suggest that there is no middle- ground. For instance, one might observe a network of hierarchical units, or a system that is generally hierarchical may contain lateral connections, both within and across levels, sometimes referred to as a “heterarchy.” One organizational structure is not inherently better or worse than another — but the structure must match the problems being faced in order to grapple with them successfully. However, most operating environments today demand that organizations become less centralized if they hope to thrive amidst increasing complexity.
Ashby’s law cannot be sidestepped — the organization must match the complexity of the environment if it is to persist.
Recommendations for Decentralized Organizations
Generating organizations that leverage decentralized structures and processes in order to grapple with complexity requires a different way of thinking than for hierarchical, top-down structures. In a top-down system, the “org-chart” is designed, in a bottom-up system, it emerges organically from the interactions of the participants.
Below are some general recommendations and guiding principles to achieve a functional organization with decentralized characteristics, including:
- Defer to those closest to the problem
- Teams, not departments
- Design for interface
- Empower generalists and foster personal growth
- Leverage Modeling and Simulation
Defer to Those Closest to the Problem
One of the main advantages of decentralized decision-making is the ability to defer to those who are intimately familiar with a problem or challenge. They are often sensitive to relevant variables that would be infeasible for a central decision-maker to track or be aware of.
The economist Friedrich Hayek identified precisely this quality of decentralized systems as one of the main strengths of market-based economies — decentralized price-setting and spending enables those with the best knowledge to make decisions. In contrast, centrally planned economies, such as those observed in Soviet Russia or present-day Venezuela suffer precisely from being overwhelmed by complexity.
While organizations are not economies, the same principle of distributing decision-making applies when challenges are too complex for a central controller to manage. Local, expert knowledge is invaluable and should be leveraged, mitigating informational bottlenecks and allowing for local adaptations to relevant challenges that may be invisible from the vantage point of a distant decision-maker.
Teams, Not Departments
Hierarchical systems are typically organized into departments. For instance, an accounting department that is responsible for maintaining the books, a marketing department responsible for advertising, a data science department for analyzing relevant datasets, etc.. In decentralized organizations grappling with complex problems, organizing instead around teams is often advantageous. Teams combine complementary skills and expertise of individuals directly, fostering organic information ow among members in a way that is difficult across departmental organizations. Consider as an example the difference between (1) three teams each composed of an accountant, a marketing expert, and a data scientist, and (2) three departments which house respectively those same three accountants, marketing experts, and data scientists. In this example, both organizations are composed of the same people, but will have very different collective capabilities.
Emphasizing teams rather than departments also encourages people to work together in smaller groups. Small groups have advantages over large ones in their ability to quickly and flexibly make decisions and take coordinated action. Working in small teams sets the stage for active design processes that are fast, iterative, and agile, which would be infeasible in larger groups due to coordination demands.
Design for Interface
Being deliberate and explicit about the form and function of interfaces is a key enabler of decentralized organization. Recall that decentralized organizations can be represented as dynamic networks: sets of ‘nodes’ linked together in some way. The properties and behaviors of the whole network come about as a result of the internal processes and capabilities of each node, and the way they are connected to one another. The connections put the capabilities of a node into contexts where they are useful in some way to the other nodes — that is, they create value.
In order for nodes to connect to exchange value, a means of connection must be established — an interface. Too often, interface is an afterthought. Emails are exchanged, perhaps with important attachments, phone calls are made and text messages are sent; things are scattered and dif cult to retrieve. This adds friction to collaboration and coordination, preventing potential from being realized.
For decentralized networks to function effectively, attention must be given to the interfaces that form connections between nodes. This means establishing channels for exchange, and making expectations about exchanges explicit.
The advantage to treating the interface of components in a deliberate and explicit way has been recognized in technical settings where systems whose internal processes are evolving in time need to interact reliably. This can be seen for instance in the ubiquity of APIs (application programming interfaces) that allow external code to interact with software via an explicit interface. This stabilizes interactions even while both the software and external code evolve internally. However, less attention has been given to how this advantage translates to non-technical, human-mediated settings, but many of the same considerations apply.
Advantages to designing for interface include:
- Sets expectations for exchanges of information and materials
- Reduces friction of coordination
- Makes roles and responsibilities clear
- Allows for the evolution of internal processes of nodes while maintaining their utility in the network
- Can enable spatiotemporally asynchronous collaboration
- Makes components readily composable and re-configurable — key to innovation
Organizational interfaces may come in the form of off-the-shelf solutions (e.g. Trello or Slack for project management), or may be constructed in custom ways to meet special requirements. There is no single right interface, and interfaces themselves should be expected to evolve in time, albeit more slowly than other processes. What is crucial, however, is to give explicit attention to the form, function, and discoverability of interfaces to enable bottom-up processes and interactions.
Empower Generalists and Foster Personal Growth
Hierarchical organizations emphasize the importance of the specialist. Each component has limited roles and responsibilities, and serves the organization best by being effective and efficient at them, not spending time building other skills or knowledge.
In a decentralized organization grappling with a dynamic environment, roles and responsibilities necessarily shift as the organization adapts to novel situations and problems. This creates a niche for generalists to serve as adaptive agents who help re-configure the organization for various problems. Moreover, being a leader is no longer a position, but a role that gets played by different individuals at different times. It also makes clear the organizational value of continued professional development and intellectual growth. The decentralized organization is much more dependent on the uniqueness of individuals and their ability to complement one another.
Leveraging Modeling and Simulation
While the structure, policies, and processes of organizations are typically designed with particular problems in mind, unintended consequences typically reveal themselves in time. Simply, it is dif cult- to-impossible to mentally envision all of the effects of such decisions, as they are myriad. However, we can expand our visible horizon when considering organizational policy or process changes through the careful use of appropriate modeling and simulation paradigms.
Of particular utility to organizations is the so-called agent-based modeling (ABM) paradigm. ABM allows one to model a system based on the known units (e.g. individuals), interactions (e.g. report filings), and environmental constraints (e.g. organizational construct) to discover macroscopic consequences that may emerge. Often, these consequences are non-obvious from other types of analyses, or when possible futures are merely imagined.
ABM and modeling and simulation generally are not to be used as a crystal ball, but when used appropriately can help to illuminate current problem points, catalyze organizational evolution, and avoid otherwise dif cult-to-foresee pitfalls.
Conclusion
A large variety of organizational constructs are possible, with no “one size fits all” solution. However, moving from centralized to decentralized organizations is often necessary to grapple with complexity, especially in today’s uncertain and dynamic environment. The successful implementation of an organization with decentralized qualities depends on a shift in thinking and execution: from designing an organizational structure in a top-down process, to enabling it to emerge from a bottom-up process in response to complex and dynamic problem sets.
Neptune Advisory
202–827–0277
www.neptuneasc.com
415 8th Street SE Washington, DC 20003
Scott Ellison scott@neptuneasc.com
Alex Kugajevsky alex@neptuneasc.com
Patrick McCarthy, CFA patrick@neptuneasc.com
Joe Norman, PhD jnorman@neptuneasc.com
David Schopler schop@neptuneasc.com
