We are celebrating this year the tenth anniversary of the Agile Manifesto. As Agility has become popular in software development organizations, you can see more and more material about the fact that Agile has “lost its soul”. People would be more “doing Agile”, that is following “blindly” some routines, than “being Agile”. I would define Agile as the human vision software development: we accept uncertainty but we can aim at providing value developing software through collaboration. Plan-driven approaches, often grouped under the “Waterfall” name, are the engineering vision of software development: people know what they want and we can mostly accurately estimate how to get there and deliver the final product following a detailed plan.
Individuals are the main pillar of the Agile culture as “individuals and interactions” are present in the first of the Agile manifesto values. As the issues in plan-driven project could be attributed to a lack of consideration of human factors, the problems in Agile project will rather be of the over-reliance on individuals. Individuals are the most important success factor in software development projects, but they are also the most important failure factor, due to the inherent weaknesses of our human condition. Trying to blame the “Waterfall” is a nice way to say that every project could succeed, but things might be a little bit more complicated than this.
Agile requires that developers are self-disciplined and emphasize team achievement over their individual interest. However these objectives are not easy to achieve, because many developer are psychological introverted and don’t have strong communication skills. Additionally, most of our programming training has a “lone programmer” bias. There aren’t a lot of opportunities to evolve in teams with external users in most software development curriculums.
The product owner is a second pillar for the success of agile project. In every project, it is difficult to achieve meaningful results if the users are not able to explicit and prioritize requirements, or worse are not very concerned by the outcome of the project. These issues can be sometimes “neutralized” when the project members have good domain knowledge, but this is not always the case. As a developer, Agile or not, you are rarely going to tell to your managers or customers that your user contact is unqualified. You do your work and cash the money. In many organizations, end users will also see their involvement in IT projects as a negative addition to their current workload and they tend to minimize it. You need to find Agile users on the other side of the interaction value.
Finally, it is difficult to operate in an Agile way within an organization that doesn’t necessarily shares or implements the value of the Agile Manifesto. Trust and openness are important factors if you need to give more weight on “individuals and interactions”. There are still many political games at the management level that can prevent Agile to be applied in optimal conditions because of the control and predictability requirements of most large companies. Culture has also an impact in Agile adoption and Michael Sahota has produced a very interesting work on culture and Agile, using the Schneider model.
The emergence of Agile approaches ten years ago had a positive impact on a software domain that was often more concerned with processes and resources than individuals. At that time, the word was spreading that Agile was the key to every project success. Surveys were showing improved productivity, quality, customer satisfaction and success rate for Agile projects. Those who failed were just not enough Agile to deserve success. It might be also that Agile has been meeting now the wall of the Pareto law: converting the 20% of more favorable people and project contexts might have created 80% of the improvements. As an interesting point, the same survey conducted by Scott Ambler in 2007 and 2010 seems to show that the success rate of Agile project has dropped. And as a funny side note to these surveys, an “iterative” (but not agile) category has been added between the two surveys after Scott was hired by IBM and develop a better sensitivity to RUP perspective of software development ;o) And Iterative seems to be as successful as Agile in these surveys. Do we really need this Agile thing?
Improving the success rate of software development activity will always face the limits of our human condition and cultural influences. Occidental manufacturers have witness the efficient Japanese car manufacturers models for years, but I don’t think there was a widespread adoption of these practices in their own factories. We might consider ourselves “rational”, but we are more often that we think looking for silver bullets to solve our problems. Agile is not a silver bullet and if we have to retain only one principle of Agile is that it is important to respond to change. We are still going to see a lot of failure context and we can just tray to mitigate this the best we can.