It seems that wicked problems are not unique to public policy.  In a wonderful book published in 1990 called “Wicked Problems, Righteous Solutions,”[2]  Peter DeGrace and Leslie Hulet Stahl pointed out that many of the systems problems facing software developers have all the characteristics of wicked problems.  Judge for yourself.

Wicked Problems

Wicked problems, according to Horst and Webber, have ten characteristics:[1]

1. There is no definitive formulation of a wicked problem. 
   Formulating the problem and the solution are essentially the same thing.  Each attempt at creating a solution changes the understanding of the problem.

2. Wicked problems have no stopping rule.
   Since you cannot define the problem, it is difficult to tell when it is resolved.  The problem solving process ends when resources are depleted, stakeholders loose interest or political realities change.

3. Solutions to wicked problems are not true-or-false but good-or-bad.
   Since there are no unambiguous criteria for deciding if the problem is resolved, getting all stakeholders to agree that a resolution is ‘good enough’ can be a challenge.

4. There is no immediate and no ultimate test of a solution to a wicked problem.
   Solutions to wicked problems generate waves of consequences, and it is impossible to know how all of the consequences will eventually play out.

5. Every implemented solution to a wicked problem has consequences.
   Once the web site is published or the new customer service package goes live, you can’t take back what was on-line or revert to the former customer database.

6. Wicked problems do not have a well-described set of potential solutions.
   Various stakeholders will have differing views of acceptable solutions.  It is a matter of judgment as to when enough potential solutions have emerged and which should be pursued.

7. Every wicked problem is essentially unique.
   There are no ‘classes’ of solutions that can be applied to a specific case.  “Part of the art of dealing with wicked problems is the art of not knowing too early what type of solution to apply.”¹

8. Every wicked problem can be considered a symptom of another problem.
   A wicked problem is a set of interlocking issues and constraints which change over time, embedded in a dynamic social context.

9. The causes of a wicked problem can be explained in numerous ways. 
   There are many stakeholders who will have various and changing ideas about what might be a problem, what might be causing it, and how to resolve it.

10. The planner (designer) has no right to be wrong.
    A scientist is expected to formulate hypothesis, which may or may not be supportable by evidence.  A designer doesn’t have such a luxury, they are expected to get things right.

Recognizing Wicked Problems

What kind of problems are wicked problems?  Here are some examples:

1. Locating a new freeway or homeless shelter. 
2. Optimizing all the features on a new model car.     
3. Deciding on the best way to re-engineer a business process. 

Wicked problems arise when an organization must deal with something new, with change, and when multiple stakeholders have different ideas about how the change should take place. 

How might you identify a wicked problem?   The thing to look for is divergence.  If requirements are volatile, constraints keep changing, stakeholders can’t agree and the target is constantly moving, in all likelihood, you are dealing with a wicked problem.  If considerable time and effort has been spent, but there isn’t much to show for it, there is probably a wicked problem lurking somewhere.   If business process reengineering is involved, there is a good chance of encountering a wicked problem.

Tame Problems

According to Rittel and Webber, the opposite of a wicked problem is a ‘tame’ problem.  Tame problems may be quite complex, but the lend themselves to analysis and solution by known techniques.  A traditional linear processes is sufficient to produce a workable solution to a tame problem in an acceptable period time, and it is clear when a solution has been reached. 

It is possible, but not advisable, to pretend that a wicked problem is a tame problem.  This makes it easy to address the well-formulated problem with standard techniques.  In time, however, the wicked problem will surface as changed constraints, volatile requirements, or stakeholder resistance.  If a problem was truly a wicked problem in the first place, treating it like a tame problem before it is actually tamed is a recipe for disaster.

How to Handle Wicked Problems

The most fundamental rule for handling wicked problems is that they must not be treated like tame problems.  To quote Rittel and Webber:  “The classical systems approach … is based on the assumption that a … project can be organized into distinct phases:  ‘understand the problems’, ‘gather information,’ ‘synthesize information…,’ ‘work out solutions’ and the like.  For wicked problems, however, this type of scheme does not work.  One cannot understand the problem without knowing about its context; one cannot meaningfully search for information without the orientation of a solution concept, one cannot first understand, then solve.”[1]

The appropriate way to tackle wicked problems is to discuss them. Consensus emerges through the process of laying out alternative understandings of the problem, competing interests, priorities and constraints. The application of more formal analysis tools is impossible before the problem can be articulated in a concise, agreed upon, well-bounded manner.  In other words, the problem must first be tamed.

Wicked Projects

Wicked projects arise when a project is organized to tackle a wicked problems as if it were a tame problem.  Another candidate for a wicked project is a ‘Death March Project’ (as defined by Yourdon in [3], this means a mission-critical project with less than half the time and resources necessary to do the job).  Think about it.  Death March Projects are often caused when volatile requirements, changing constraints and stakeholder disagreements meet up against immovable deadlines.   They are frequently a symptoms of management's unwillingness to tackle wicked problems.

Failure to recognize wicked projects has given Software Development a bad name.  A 1994 Standish Group Report[4] found, for example, that about a third of software development projects get canceled and half do not meet their original cost projections.  Some have taken this to indicate that the state of software development is in disarray.  However, it can also be read as strong evidence that there are a large number of wicked software development projects out there, trying to address wicked problems with the wrong approach.

A typical management reaction to a failed software development project is to conclude that the organization is immature and to aim for more maturity.  This usually means imposing more requirements documentation, more analysis, more planning and tracking against the plan.  Managers feel that more use of the classic project management processes will avert future disasters.  If the failed project was addressing a tame problem, this approach will probably be beneficial.

However, classic project management practices simply do not work for wicked projects.  In fact, referring again to Rittel and Webber, attempting to baseline requirements and then use an analytical approach to reach a solution is a recipe for disaster with wicked problems.  These problems are resolved through discussion, consensus, iterations, and accepting change as a normal part of the process. 

How to Deal with a Wicked Project

Step 1:  Recognize that this is a Wicked Project.

The first step to combating most systemic problems is to recognize them – admit that they are there.  If the project has to satisfy stakeholders who do not agree on fundamental issues surrounding the project, then it is definitely a wicked project.  It’s fine to say that there should be an executive sponsor or that the customers should speak with one voice.  What if they don’t?  What if they pretend to agree, but deep down, they really don’t?   This is a wicked project and everyone would be better off if it was recognized as such.

Step 2:  See if the Project can be Tamed.

In recent Harvard Business Review[5], Robert Herbold gave an excellent example of taming a wicked project.  As new COO of Microsoft in 1994, he realized that he had to impose centralized financial, purchasing and human resource systems on staunchly independent Microsoft business units.  Here’s how he tamed the problem:

1.     Executive Support.  Bill Gates made it clear to all divisions that anyone not reporting results through the new system would be deemed to have no results at all.    Steve Ballmer told balking general managers:  “We define the measures.  You put all your creative juices into growing them.”

2.     Clear problem Definition. All central systems were implemented with off-the-shelf software which dumped data into a data warehouse.  There were few system decisions to be made beyond which package to use and how to configure it.  This was accomplished by a small group of technical and business experts because, according to Herbold, “you can get the job done in the time it takes a cross-functional committee to decide on its goals.”

3.      Separation of Concerns / Reduction of Stakeholders. Business units had web-based access to the data warehouse for each system, and they could get at the data however they wanted to, but they did not have access to the packaged software.  Thus business unit managers had no stake in the packaged software, and where they did have an interest, which was data access, they had a high degree of control and flexibility.  This had the effect of reducing the number of stakeholders for the central system to a very few.

A project is not easily tamed at a low level, although a component-based architecture helps to reduce stakeholders and thus tame the problem. 

The main approach for taming a wicked project is to obtain appropriate executive support.  This means that the sponsor must recognize and resolve wicked problems.  If the sponsor cannot resolve conflicts and the customer does not speak with one voice, the problem has not been tamed.

Step 3.  Use Adaptive Processes

Wicked problems are resolved through discussion, consensus, iterations, and accepting change as a normal part of the process.  There is nothing that difficult about an adaptive process.  It has been used successfully in World Class new product development organizations for decades.  Most public policy is developed this way.  Most business policy is developed this way.  Most marketing is done with adaptive processes. All sophisticated process control systems use adaptive control techniques.   

There is nothing wrong with using adaptive processes to solve wicked software development problems.  In fact, it if the problem cannot be tamed, this is the only ‘good’ choice.

There is intense pressure not to use adaptive processes for software development, because they are not considered ‘mature’ or ‘predictable’ or ‘controllable’.  In fact, empirical control techniques are just a successful at controlling processes as defined control techniques, when used for the right kind of problems.  Since classic processes have such a poor track record in controlling wicked projects, it is time to give adaptive processes a chance.  If the project is a wicked one, the chances of success will be significantly improved.

Scrum

Perhaps the best adaptive project management approach for wicked software development projects is Scrum.   This Rugby term was first used by Takeuchi and Nonaka in 1986[6] to describe the way in which world class companies develop new products.  Considering that a new product development project is by its nature a wicked problem, the best new product development processes present a good model for solving other wicked problems.

Ken Schwaber and Mike Beedle describe the use of Scrum for software development projects in their book Agile Software Development with Scurm[7].  The important thing about Scrum is that it forces incremental action which creates a basis for stakeholder dialog and project feedback.  A scrum project collects stakeholder input in a feature list called the Backlog.  Each month, the development team starts at the top of the Backlog and selects as many of the top priority features as they think can develop in a month.  The team is then left alone for a month, when the result is demonstrated to the stakeholders.  This provides a basis for rethinking the backlog features and priorities.  The stakeholders are allowed to modify and reprioritize the backlog, after which the team selects it’s next month’s work from the top of the list .

Scrum provides a way for the development team to make regular progress even if the problem is not well understood.  At the same time, it provides a method for stake holders to discuss the problem and reach consensus.  It provides a way for solutions to emerge, as is necessary for the resolution of wicked problems.

At the same time, the Scrum process provides a high degree of predictability.  Each line on the Backlog is estimated, and the estimates are added to create an overall project completion estimate. After three months, a graph of the Backlog estimate against time is a highly reliable indicator of the project’s progress and estimated completion date.   Features may be added or subtracted from the Backlog monthly to adjust for constraints as well as changing stakeholder interests.  The trend in the Backlog estimate is a reliable indication of whether the project is converging or diverging. 

Conclusion

If software development projects have not responded well to traditional project management practices, the fault may lie in the attempt to apply inappropriate methodologies to software development projects.  When projects must deal with conflicts in stakeholder requirements and changes in management constraints, an adaptive process is far more likely to succeed than traditional methodologies.  At minimum, adaptive project management practices will quickly determine if a project will converge on a solution, and thus provides actionable data for project portfolio management. 

_____________________________

1.    Rittel, H., and M. Webber; “Dilemmas in a General Theory of Planning” pp 155-169, Policy Sciences, Vol. 4, Elsevier Scientific Publishing Company, Inc., Amsterdam, 1973.

2.    DeGrace, P., and L. H. Stahl. Wicked Problems, Righteous Solutions. Prentice Hall, Yourdon Press, 1990.

3.    Yourdon, Edward; Death March Projects, the Complete Software Developer's Guide to Surviving ‘Mission Impossible’ Projects.  Prentice Hall, 1999, 1997

4.    The Standish Group.  “Charting the Sea of Information Technology – Chaos.”  The Standish Group International, 1994.

5.    Herbold, Robert J.; “Inside Microsoft:  Balancing Creativity and Discipline”, Harvard Business Review, January 2002.

6.    Takeuchi, H., and I. Nonaka. “The New New Product Development Game” Harvard Business Review, January-February 1986

7.    Schwaber, Ken, and Mike Beedle. Agile Software Development with SCRUM. Prentice Hall, 2001