Agile development is more culture than process
Why thinking of agile as culture and not just process explains resistance and difficulty in teaching and learning the approach
Culture, not process
I’d been working with Josh and lots of others at his company for many months aiding with their agile adoption. Finally, things were beginning to go smoothly. The team was productive and happy – and we were making good progress on the product we were building. One day when passing in the hall, Josh said to me:
“I finally figured something out. Agile development is a culture, not a process.”
Josh is responsible for user experience at company I work with regularly. Like a lot of UX people, he was concerned about adopting an agile process since it didn’t seem to value the attention and time spent understanding users, reflecting on their needs, and iteratively working through possible product solutions. Agile approaches seemed to want to dive in, leap quickly to a possible solution, and begin its construction.
Josh is an interaction designer, one of the best. He places great value in understanding his users. You could say it’s part of his design culture. For agile to succeed in his organization, especially in Josh’s eyes, we couldn’t abandon that value. The agile process we’re using there today retains that value, and a number of other values held by design culture. The team has also adopted new values that came in with agile culture – notably valuing visible working software. The result is a process that accommodates all these values – or at least works very hard to try to.
Culture is process
Sitting with my friend Jonathan at lunch last week, we were talking about the process changes he felt like he was being cornered into making. He was adding more teams, and teams were growing. Things needed to change. Jonathan was justifiably concerned that all the new process being added would erode at the fluid communication and teamwork he’d worked so hard to promote. “How do you keep those things in your process?” He wondered. After talking a bit we decided that those things weren’t really process points – rather they were part of his company’s culture. These were the things he and others in his company valued.
Once in a while I feel particularly wise. It’s fleeting, and sometimes I get a “false positive” – stupid ideas that sound wise. But I said this, and it sounded wise enough to Jonathan.
“Culture is process. Identify your culture and promote that.”
Now, that statement “culture is process” may not make sense to you – so I’ll tell you why it makes sense to me.
As a member of a culture you value particular things, communicate in particular ways, and I believe participating in a culture often implies a particular process. Here’s what I mean. To participate in American culture, we’re all aware of this ideal process flow to achieve success:
- Go to school and work hard
- Go to a good college
- Get a good job
- Get married
- Buy a house
- Have 2.5 children
- Save for retirement
- Climb the career ladder
- Retire to a warmer climate with golf course
- Spoil grandchildren while reminding your children of all their bad behavior when they were this age
Now that’s the ideal process – which of course no one follows exactly, and some folks dislike. I suspect many of you recognized the process as you read it, although I’m not sure of any place it’s written down. We just know it. I’m not a fan of this particular process. I often work hard to deviate from the process – but often find myself comparing my current performance with the ideal American life process performance.
Processes vary by area of the country, social status, and likely lots of other factors that sociologists are aware of. Moving to other countries will cause the process to vary. For instance, folks I know from India might replace the “retire to a warmer climate” step with “Move in with eldest son’s family.” I’m sure there’s lots of other variations in the Ideal Indian Life process. But, I believe that behind that "process" step lies a cultural value enforced by norms or expectations of others in that culture.
My big point here is that while culture doesn't dictate a precise process, cultural values supported by teaching structures embedded in cultures and enforcement structures such as norms and mores all lead to a commonly understood process.
"I'm not buying it."
At the coffee shop discussing this with my friend Alistair, he flatly disagreed on the point that culture is process.
"I agree that agile is a culture. But, I don't buy the idea that culture generates process."
Now Dr. Cockburn does hold a doctorate in software methodology, so you should believe him. But, I've chosen to suspend my judgment on the issue and pretend for the next couple of months that agile development isn't a process, but a culture that generates sort of a "process prototype." For a long time I've had a trouble believing that agile development is a process anyway.
Assertion: business and software development cultural values motivate common software process
Let’s bring this back to software development and its smaller, more local culture.
I’ll assert that in traditional software development individual skill and competency is highly valued. It’s important to be demonstrably good at your job. I’ll assert the best way to demonstrate competence is to understand what’s required of you in your organization, and to perform it flawlessly. An ideal process for that might assign individuals specific work-products to create, give them time to create the work products, then judge individual's success on the quality of that work product.
Since it takes lots of people and lots of work products to build software, we might create a “project plan” work product that strings the creation of all these other work products over time into a dependent sequence. This may be starting to sound a bit familiar.
Since individual competence is valued, each person strives to create their work product on time with high quality. Happily if we’re unable to do that, it’s easy to blame others upstream for deficiencies in the work products they constructed and you leveraged. And, for the person who created the “project plan” work product, to prove that was high quality, they’ll need your work to go exactly as they predicted. In fact, their value in the traditional software development culture is based on it. You not finishing your task on time affects their perceived competence.
Lots of dysfunctional process emerges from us valuing personal competence at the expense of overall group success. I suspect we carried that concern for personal competence with us from our primary and secondary education where no one's report card reflected the grade of the entire class. At least I don’t remember getting a poor grade in class when the rest of the class did poorly. On the contrary, when they did poorly, it just meant that I was likely to be comparatively better. Life is graded on a curve.
There are a number of other things valued in traditional business and software development culture that are likely at odds with best process outcomes. It's worth pondering a bit.
We can't escape our values
Winston Royce first drew what we now call the waterfall model in 1970. You'll find when you look up the term "waterfall" that Winston is often referenced. He didn’t draw it as an example of how we should develop software – but as an example of how not to do software development. That common waterfall model is one of the first figures in a long paper followed by many other figures. The figure immediately following the simple waterfall is one showing feedback loops. Winston explains that "as each step progresses and the design is further detailed, there is an iteration with the preceding and succeeding steps but rarely with the more remote steps in the sequence." He follows that model with this quote:
I believe in this concept, but the implementation described above is risky and invites failure.
Winston goes on to point out that it's not until further downstream when we test working software that we really see problems - the sorts of problems that cause us to go all the way back to requirements - do not pass go, do not collect two hundred dollars. Winston didn't actually say that last bit about the $200. But he did go on to describe a much more sophisticated process than the waterfall.
I honestly don’t think anyone really believes a waterfall-style process works. The guy who’s credited with first drawing it in 1970 didn’t. So why then does it keep emerging as the “right” process in so many organizations for so many years?
I’ve irreverently referred to waterfall development as a “CYA Process.” The coolest thing about a waterfall process is that it allows me personally to succeed, to demonstrate skill and competence, while the end result of the process is a dismal failure. Cleverly built into a waterfall process are a variety of scapegoating mechanisms that allow us to blame other people, or outside influences for failure. Given that important value of demonstrating personal competence, and keeping me personally safe, a waterfall process seems like an ideal process choice.
"It's not my problem. The hole is in their side of the boat."
—John Brewer (the first person I heard say this.)
I suspect there are other values that motivate poor process. I'd love to hear from you. What seems to be valuable in your organization that seems to result in detrimental process or behavior?
Teach culture first, then process and techniques
After working through the cultural discussion with other agile colleagues, we were left with the realization that we spend a fair bit of time teaching process and techniques, when it's the culture that matters most.
Contrary cultural values may lead people to use their new found agile process and techniques for evil instead of good.
Given that we understand that agile development is a culture, if we were teaching culture, how would we do that?
This line of though lead me to a little digging around into the aspects of culture.
Every culture has its own specific language – and agile language is pretty rich.
There's on old joke told about England and the United States – they're often described as two countries separated by a common language."
Agile language sounds quite a bit like traditional software development language. So much so that those new to agile may hear the words and think they understand.
There's a number of agile homonyms words that look and sound the same as traditional software development terms – but have differences in meaning in an agile context. For example:
Now, in a culture of individual responsibility and blame, software estimation is a pretty tense hand-wringing affair. In an agile context, estimation - or what I feel better calling "sizing" - is actually a pretty fun and useful activity. There's a long list of other words used in agile that don't mean what you might thing if you don't come from agile culture including word like: requirement, test, customer, working software, deliverable, team, meeting, and lots of others.
And if agile homonyms weren't problem enough, there's tons of new vocabulary that cause those new to agile to be as lost as anyone might feel in a foreign country. And, may cause reasonable people to ask "we're doing software development here – why do we need to redefine all the terminology we already had?"
- Product Owner
- Scrum – the process and the daily…
- Story Point
- Burn-down chart
- Product Backlog
- Planning Game
- Yesterday's Weather
Stories, heros, myths, legends, and jokes
Stop a person you know who's been involved with agile development for a while. The old-timers can tell you where the term Scrum came from, what the C3 project was, and about the meeting of 17 at a ski resort in Utah. These are stories – many of them evolving to the status of myth due to frequent retelling.
I've often thought of agile as a cult of personality – except without all the bad political implications. There's certainly a long list of colorful personalities – all of them intelligent, charismatic, and compelling. Some of agile's heroes include: Kent Beck, Ward Cunningham, Martin Fowler, Alistair Cockburn, Ken Schwaber, Jeff Sutherland, Dave Thomas not to be confused with Big Dave Thomas, Uncle Bob Martin, Mike Cohn, Mary and Tom Poppendiek, and I could carry this list out for a long time. Each of these folks have made valuable contributions to the big agile story. We often explain agile by describing these people, their ideas, and their approaches to doing things. Speaking for myself, I often tell second hand stories I've heard from others - sometimes wondering how accurate they are… that's the myth part.
And, there's even a joke or two embedded in common agile culture. As well as the commonly observed trend for agile teams to build up their own supply of stupid jokes.
Values, norms, folkways, laws, taboos
We can't talk about agile without talking about values. The agile manifesto itself is a statement about what that group of creators valued – not about process. The supporting 12 principles are littered with phrases for things that agile culture values – such as:
- continuous delivery of valuable software,
- welcome changing requirements,
- satisfy the customer through early and continuous delivery,
- business people and developers must work together daily,
- build projects around motivated individuals,
and I could go on and on about those too.
All of these value statements describe what's important to agile culture, and we can see how they likely have process implications… but we can't see exactly what the process is.
Recently Brian Marick, in a keynote talk at the Better Software Agile Practices conference, discussed what the Agile Manifesto left out. He focused on new emergent values - things agile people care about that didn't get mentioned. Brian noted: skill, discipline, ease, and joy (who can get enough joy?).
I was lucky enough to join him on the stage where I added "purpose" and "angst" to the set of values he suggested. Purpose, because it's important to me that I'm helping to build software I know will ultimately help people, and "angst" because I keep my edge by always being a bit dissatisfied with the status quo. Anxiousness keeps me aware and keeps me looking for opportunities to make things better - which I value in others as well.
Agile projects enforce their values with established norms. Agile folks have even come up with some slang used to dig at people who don't look like they're respecting agile values. For instance:
- Agile values doing only what's necessary. If an agile developer catches another developer over-complicating design, especially to include functionality not yet needed he might say "YAGNI" - short for "You aint gunna need it." Kind of an update of the popular KISS - keep it simple stupid.
- That simple design ideal is enforced with the mantra "Do the simplest thing that could possibly work."
- Agile values not trying to be too predictive - to minimize up front design in favor of short planning and execution cycles. If your up front planning or analysis starts to look to heavy, you might be accused of "BDUF" - or "big design up front" - which is bad.
The agile community as a whole will frown on teams that don't work together, and don't deliver working software, Frowning and discouragement is as rigorous as agile culture gets with enforcement - although an agilista's Zen slap can sting your ego a bit.
Beliefs, attitudes, assumptions, mental models, symbols
Agile practitioners have a large body of beliefs bottled up in their story and terminology. Agile folks don't believe they can effectively predict the future, or estimate development time. Many agile folks believe in emergent architecture, and in growing software incrementally.
Mental models like the simplified scrum framework can be sketched quickly by lots of agile folks. The two loops often called the scrum snowman model have been turned into a simple symbol that's the logo for the ScrumAlliance.
Rituals, rites, ceremonies, and celebrations
Daily standups or daily scrums, planning meetings, pair programming, and product demonstrations are all commonly taught "ceremonies." Healthy agile teams treat them as celebrations, and invent lots more celebrations. I think it's an excuse to overeat.
Certified Scrum Master has become a bit of a rite of passage that the ScrumAlliance has tried to add a bit more ceremony to by adding additional testing and subsequent rites of passage like Certified Scrum Practitioner, and Certified Scrum Coach. So much for the secret handshake. I remember back in 2001 when it was a bit of an Extreme Programming right of passage to have attended an XP Immersion. At one of the first XP conferences people proudly wore their Ward number on their shirt to indicate how close they'd been to pair programming with Ward Cunningham. Sorta like the 6 degrees of separation with Kevin Bacon, a Ward number of "1" meant you'd paired with Ward, a "2" meant you'd paired with someone who's paired with Ward. I have a "2." That's a story about an agile hero and a bit of a right of passage.
And finally, cultures develop their own technology and artifacts. Agile teams use burn down charts, task boards, and story cards. And a large number of tool vendors have created software products to support agile development.
The long pondering on culture, and the closer look at aspects of culture have caused me to look closer at the way I teach agile development – and anything to do with software development.
1. Underscore agile values that motivate practice
I find it more necessary to highlight and underscore the values that motivate the process or technique I'm speaking about.
2. Identify organization values that compete with agile values
When I sense concern from someone about a particular agile process point or practice, I work to identify the agile value the process supports and to identify what's valuable to the person with the concern. I'm looking for a conflict of values.
3. Be sensitive to culture shock
I'm more sensitive to culture shock. The terminology, stories, and ritual that agile is chocked full of are bound to make anyone feel a bit uncomfortable. In particular, people who've spend a great number of years working in software development may be feeling something they haven't felt for many years – uncomfortable doing their own jobs. They may feel like someone has drugged them, put them on an airplane, and dropped them off on a foreign country without their consent. They may have incorrectly assumed they were just learning a new process.
After pushing this out last night, I immediately received a few great pieces of feedback. (thanks twitter) And, as a result, I've made a couple changes. (thanks iteration)
One notable changes comes in the way I've described Winston Royce's characterization of waterfall. The quote above is correct, and due to an interesting layout choice in the original paper, it happens to be adjacent to the popular drawing of the waterfall model itself. Download the Royce paper and see for yourself. But, the quote is actually about the second waterfall model the one with feedback loops. Ironically many organizations omit those feedback loops. But Royce's quote about "believing in this concept" applies to the idea of getting feedback - not the straight forward waterfall. Who would believe in that concept?
My friend Alistiar pointed out my dramatization wasn't entirely fair. And, he's right.
Also immediately after posting this I received a nice note from Philippe Kruchten. Not only is Philippe one of the sharpest people around, he was one of the creators of the Unified Process. And, before agilistas get too crazy, note that Philippe is a leader in the agile community in Vancouver B.C where he now lives. Philippe extracted a bit from his July 2007 article: "Voyage in the Agile Memeplex". In this extract Philippe shows that understanding agile as a culture isn't a new idea in fact I'm likely the one just catching on here.
Agility as a culture
My first claim was that agility is not a technology, a science or product but a culture. I realize that I set myself up with an even more difficult task, that of defining 'culture.' Hofstede  defined culture as "the collective programming of the mind which distinguishes the member of one group or category of people from another" (p.5). Spencer-Oatey  defines culture as "a fuzzy set of attitudes, beliefs, behavioural norms, and basic assumptions and values that are shared by a group of people, and that influence each member's behaviour and his/her interpretations of the 'meaning' of other people's behaviour". One of the implications of this definition has to do with the function of culture. More specifically, besides being an influencing factor on people's behaviours, culture also affects how people interpret others' behaviour. This interpretive function of culture is especially important in intercultural situations as the behavioural norms of one culture may be misunderstood or regarded as incongruous by other cultural groups.
Some of this can be observed in the agile world, where people say or write things like "this is very waterfall" when often they have never seen a waterfall project. For example a waterfall project may have customers on site and do continuous integration, unlike what agilistas may claim. Often they are not aiming at inherent properties to "the waterfall", but more characteristics of bad projects. So "waterfall" becomes synonymous of "the other culture," a scapegoat, a strawman, a piñata.
A culture can be rich and varied, and described in many of its forms. But as expatriates can attest, it is very difficult to acquire, to learn another culture, because many of the values it relies on are embedded in our brains during a long educational process, taking years from a very young age. The same applies to agility; since there is not a clear underlying science, a well-packaged body of knowledge, much of it is really transmitted by oral tradition and by imitation, i.e., learning to do something by seeing it being done.
(3) G. Hofstede, Culture and organizations—Software of the mind. New York: McGraw-Hill, 1997.
(4) H. Spencer-Oatey, Culturally Speaking: Managing Rapport through Talk across Cultures. New York: Cassel, 2000.
Finally, my real reason for pondering this and writing thoughts out were to help me answer the question: "If I were teaching agile development, and understood it to be culture, and not process, how would I go about it?" I'm increasingly concerned by what one colleague described as the "carpet bombing" of agile and in particular Scrum training offerings - now worldwide. I'm concerned that both teachers and students think that they're teaching a process - when I think it's something quite different.
Thanks everyone for your feedback and comments.