Agile For Executives

The Drug of Choice

Scott — Tue, 26 Feb 2008 01:01:28 +0000

There is an addiction in the software development industry. It is as real as any physiological dependency. It is the obsession with “certainty” and extensive up front design and definition.

Some believe that, to a large extent, we can plan an entire software development project and execute to plan. That we can deliver what we say we will deliver, when we say we will deliver it, within budget and quality objectives. The Project Management Institute is a neighborhood pusher of this drug.

Inject yourself with this toxic drug and you detach yourself from reality.

Our dependency takes many forms. Excessive analysis and requirements gathering, work breakdown structures, task breakdowns, resource assignment and management, PERT and Gantt charts. Each promulgates the deception that we know, or can figure out, with reasonable certainty, what we are doing.

Worse, each takes a good amount of time to perform. This chews up the market window on tasks that are likely to be difficult to analyze, and then reworked at a later date. To compound the dysfunction, as rework takes place, documents and artifacts must be updated to keep from being misleading or flat wrong. My experience suggests that once a developer does encounter an error or two in a requirements document, they abandon it and go straight to the product owner with questions. This renders the entire requirements document unviable.

I believe new software development projects are not conducive to advanced work and task breakdown for an entire project. Unproven technology is, well, unproven. Competitors merge, and change the market landscape with new offerings. Access to resources changes and new technologies can render new products obsolete before delivery. Simply put, too many things can (and will) go wrong!

The answer is to take smaller bites. Cyclically develop your software and progressively elaborate your product. Define only those requirements that must be implemented now. Defer requirements that are less important until their time is at hand.

One of my favorite agile analogies is by comparison to a game of golf. Several good discussions exist in the blog-o-sphere:

     http://trailridgeconsulting.com/blog/?cat=14
     http://parlezuml.com/blog/?postid=18
     http://parlezuml.com/blog/?postid=419

     Can you plan an entire golf game? Only at a high level.
     Can you execute your game to plan? Probably not.
     Does it make sense to reassess your situation after each shot: Absolutely!
     Does it make sense to re-plan based upon the current situation: Absolutely!

I suggest that excessive planning does not well serve the goal of delivering a new software product. A more reasoned approach abandons the illusion of full project definition and focuses on just-in-time clarification, dynamic resource allocation, short cycle development, and rapid feedback loops.

Agile software development can help you kick the Big-Design-Up-Front habit.

Figure, Ground, Focus

Scott — Fri, 08 Feb 2008 16:03:41 +0000

When a painter draws a portrait on a canvas, she not only defines the figure, but also implicitly defines the ground. The ground is all but the figure itself. In addition, the figure represents the foreground concept; the ground represents the less important contextual setting. Analogously, the software development process can be viewed as figure within ground. How so?

To answer this question, we reach back in time and across the disciplines to our cousins in lean manufacturing. Pioneered by Taiichi Ohno and Shigeo Shingo of Toyota, Value Stream Mapping is a visualization tool used to identify, demonstrate, and decrease waste.

In its simplest form, a value stream map is the decomposition of a software development or manufacturing process into each of its component steps. Each step is then classed a value-add step or as waste. By analogy, the process is the canvas, the figure correlates to the value-add steps, and the ground is the remaining waste.

Here is a simplistic diagram. For real world examples, see Mary Poppendieck’s Value Stream Maps Examples.pdf on the Lean Software Development forum.

Within software development, as in manufacturing, such decomposition allows one to begin to focus on specific portions of the process with an eye to optimization. But unlike a static painting, software development is a temporal activity. As such, we must also consider transitions between the parts of the process in our analysis. This leaves us with three considerations:

Reduce waste
Increase value
Smooth flow (remove bottlenecks)

If we can do this, our value stream delivers more value in less time. But to succeed, each of these parts must be addressed and optimized.

Reducing waste

But how do we reduce waste in a software development process? What is waste? Again, we turn to lean manufacturing and its seven wastes to help us understand. It’s beyond the scope of this article (and will be the subject of others) to correlate these concepts in detail, but I do include a brief italicized example of a related software development process in a similar waste class. The seven:

Defects (Bugs, testing)
Over-production (Over-engineering a requirement)
Transportation (Moving intermediate requirements/code products around)
Waiting (One team waiting on another, e.g. test waiting on development)
Inventory (requirements not yet implemented and delivered)
Motion (people actions: phone calls, tool administration, meetings)
Over-processing (adding unneeded features, goldplating)

Agile software development practices address and reduce each of these wastes. Waterfall projects that introduce serialized functional stages, with signoffs and handoffs at each stage create extra transportation, waiting, inventory, and motion wastes. In addition, because it takes so long to deliver value in traditionally managed projects, there is a strong tendency to over-produce and over-process requirements.

Increase value

As noted, certain steps in the software development process actually create tangible value. For example, the work of collaboratively defining and understanding a feature is such a step, as is the act of authoring the code, or physically building the deliverable product.

Value Engineering is the tool of choice to address optimization of the value-add steps of a process. Value is function divided by cost. Increase function or reduce cost and you increase value. Value engineering involves looking at a component of the product and asking three questions:

Can I eliminate the part? (ie: remove over-production and over-processing)
Can I combine the part’s function with that of another part?
Can I provide the same function at less cost?

Agile software development consists of a number of value engineering practices. As one example, there is a strong emphasis on engineering support of automated unit and customer acceptance testing in agile projects. Contrary to what you might think, a primary value of automated tests is not quality per se, but the vital role it plays in support of value engineering. There are at least two ways in which this is so:

Writing automated acceptance tests before writing code helps engineers understand requirements faster than reading a document, coding, then testing. The value-add step of understanding requirements happens faster because understanding and generating tests occur in parallel up front, each reinforcing the other.
Refactoring is a value engineering process that enhances the architecture or improves the execution sequence of code without changing the results. Test results are essential to insure that refactoring changes were completed in a completely transparent fashion.

Smooth Flow

Like a magnifying glass focuses attention on one part of the printed page, an organization must focus improvement efforts on specific foreground and background value steam steps to deliver more value in our software products. A kind of organizational dyslexia occurs if we are unable to do this:

Loss of focus between what contributes value in a project and what doesn’t results in the same sort of organizational confusion illustrated in wonderful imagery by Scott Kim above.

This leaves one question to be answered. We can separate value from waste with value stream analysis, but what tools help us focus on the right steps in the system for improvement?

In one of the most unpretentious and important books of our time Eliyahu Goldratt laid out the Theory of Constraints in “The Goal“.

The five steps of the Theory of Constraints (TOC) are:

Identify the constraint
Decide how to exploit the constraint
Subordinate all other processes to the above decision
Elevate the constraint
Repeat steps 1 through 5

Repeated application of TOC focuses attention on process bottlenecks. Focusing attention at each “most critical” step, the loop is repeated indefinitely; reducing or removing each waste or value engineering each value-add. At some point, only minor bottlenecks exist and parts (software requirements) flow smoothly and quickly through the system.

Condensed Soup

Figure and ground equals value-add and waste, the process is the canvas. Use value stream mapping to identify the value-adds and wastes. Use the theory of constraints to identify and focus attention on process tasks that need optimization. Identify and decrease waste by removing the seven wastes. Value engineer the value-add portions.

This is the work of lean and agile software development. Simple eh?

The Theater of War

Scott — Sat, 02 Feb 2008 05:08:55 +0000

How appropriate to begin a discussion of agile development with the analogy of war and the metaphor of theater!

It is too easy to compare battlegrounds and warfare to new software development projects. Many of the very same forces are at play in each. Clouds of uncertainty and risk pervade each. Advances in technology can radically alter the balance of power. Flexibility and adaptability are necessary for survival. Great skill is necessary to successfully prosecute the campaign. Disinformation is a weapon in each endeavor. Good intelligence can turn the tide. Campaigns are begun with insufficient information. Execution is of paramount importance. I could go on, but I leave it to you to reap the value of extending this analogy.

If the characteristics of warfare are similar to that of new software development, then perhaps we, in the software development community, can also learn from its accumulated knowledge. After all, great minds have been thinking about the tactical and strategic execution of war for over three millennium.

One of those minds is Colonel John R. Boyd. Little known, inside the military or outside, he was one of our great tactical thinkers. A fighter pilot early in his career, he later developed fundamental principles of fighter maneuverability that influenced the design of the F16. In addition, he carefully studied warfare.

Boyd never wrote a book, but he constructed many presentations. Often they were four hours long and hundreds of slides. One such was his “Patterns of Warfare” presentation. In this exhaustive study of human conflict he first put forth the idea of the OODA loop.

OODA is an acronym for Observe - Orient - Decide - Act. Boyd maintains that the adversary that can competently execute the OODA loop the fastest will emerge victorious. Getting better at executing your OODA loop means cycle speed AND accuracy increase. (Does this strike you as similar to the Toyota Production System where quality and speed increase together?)

But getting better at the OODA loop is all about vision, teamwork, delegation, commitment, trust and ability. These concepts recur frequently in agile thinking.

Steve Adolph preceded me in recognizing the similarities between Boyd’s writing and agile software engineering. I highly recommend his article: What Lessons Can the Agile Community Learn from a Maverick Fighter Pilot? He also has an interesting podcast presented at a less technical level entitled “Agility in the Art of War … a dissertation“.

When reading Steve’s article, pay close attention to Boyd’s analysis of the German Blitzkrieg and consider that in light of the principles of Scrum and especially XP. I believe you will appreciate the similarities.

There is great value in a review of Colonel Boyd’s discourses on military strategy for those trying to understand agile. For agile is nothing more than the expression of practices in software development, that have been successful in manufacturing by Toyota and in warfare long, long, before that (e.g. The Art of War, Sun Tzu, 2nd Century BC).

Finally, I’ll leave you with one final imperative from Colonel Boyd:

“People, Ideas, and Hardware, in that order!”

To me, this statement succinctly embodies the agile philosophy.

An Introduction - Symbolic representation (Part 4 of 4)

Scott — Sun, 27 Jan 2008 15:24:05 +0000

Please grant me one last word before I begin these chronicles in earnest. This pertains to the style or approach I will take in my writing.

One of my favorite authors is a cognitive scientist name Douglas Hofstadter. His writings often focus on what it means to think, reason and understand.

In his excellent book “Fluid Concepts and Creative Analogies“, he discusses the importance of metaphor and analogy in learning. I agree.

Everything we learn, we grasp because we are able to understand it in the context of knowledge we already possess. Our minds are symbolic representational systems that work at the most basic level by this mechanism. For example, when I say the word “dog”, your mind conjures a representative image of a certain type of small furry creature. If I say “cat” it fabricates a different, but in some ways similar, image. If I were to describe to you an animal that I saw at a zoo for the first time, these ingrained concepts might be useful to me and very helpful to your understanding. A picture might be even better, but again, photographs are nothing more that analogous artifacts of a physical event or scene.

Analogy and metaphor are valuable tools that can improve one’s understanding of why Agile is a better way to develop new software. I intend to use them heavily in my writing. My hope is that I can map every major point I make symbolically to another concept you understand well.

I encourage feedback regarding my success in achieving this goal.

An Introduction - The Agile Alternative (Part 3 of 4)

Scott — Wed, 23 Jan 2008 21:25:30 +0000

Iterative development methodologies, commonly called Agile, seem to me to be a better way to develop new software.

There are many reasons why I think this is so. I begin this blog to set out and explore those reasons with you. But now is the time for grand overviews and not detailed justifications. To that end, I will focus the next few paragraphs on some of the benefits of agile development without paying too much attention to supporting detail. That detail with emerge from my stream of weekly blog postings.

No matter what you have heard, Agile is a disciplined approach to software development. There are practices that may seem, at first blush, to be quite chaotic and undisciplined, but even those practices play a very fundamental role in the development cycle. Most are designed specifically to accommodate rapid change. If you hear a comment like: “Scrum is a justification for letting developers do anything they want”, then you know you are dealing with an organization that exercises no discipline within their agile process.

Agile projects, properly executed, produce products valued by customers, on time and on budget. The product produced may not bear close resemblance to the original conception, but the difference will consist wholly of value added by the customer herself.

Agile projects track at a sustainable pace. Schedules based upon thumb in the wind estimates are gone, replaced by projections based upon demonstrated capacity to deliver. Three factors make a sustainable pace of productivity possible:

Estimates come directly from empirical measurement and are thus reasonable.

Those estimates always fall into a normal distribution of accuracy.

Engineering teams commit (ie: exercise discipline) to deliver.

Agile projects do not react to change; they seek out and proactively manage change into the project. The origin of the change matters not: market change, technology change, changes of resources, corporate priorities, or anything else. Agile projects combine the knowledge gained from frequent health checks with the best thinking of the here and now to mandate a project course correction based upon new information. This is a far, far cry from a Project Management Institute Change Control Board, whose purpose is to protect a project from the chaos of change.

Agile projects have a more sensible social structure. Deliverables cease to pass, relay style, from one functional silo in an organization to the next. Instead, a cross-functional team is formed. That team, working together, creates the final deliverable. Engineers may write test scripts. QA test engineers may help the technical writers with the help system. Focus and accountability is removed from the individual and moved to the team in an agile project.

Another aspect of Agile projects I really like is the de-emphasis on formal communications. Formal communications are necessary in functionally structured organizations where requirements, design documents, test plans and the like are passed from group to group for review. Writing, accuracy, assumptions, interpretations and maintenance of such documents are always problematic. But small, co-located agile teams solve the same problem by stressing personal interaction. Stakeholders sit down together and, side by side, develop the requirements, code, tests, acceptance criteria and other components. Notice that everyone from product owners to test analysts are involved in this process. Also note that the deliverable is code and not a document. Working code, not a document, is the measure of success. Agile cuts out the work of the middleman.

As soon as I close this blog entry out, I will think of other reasons why I think agile is a better way to develop new software. But, in the spirit of agile, I’ll focus on the most important things first. I believe I’ve named most of those important reasons.

I’m not an agile purist. I believe that waterfall methodologies can be appropriate for certain types of software projects (software platform ports come to mind). I think agile is a management tool like many other project management tools. It just happens to be very well suited for new product development and/or major new release work.

In addition, there isn’t one way to do agile. One of the common characteristics of all agile flavors (and there are many) is that they are self inspecting and self adapting. The result being that, an agile project in one organization may not look like an agile project in another. The differences that exist between the various hues of agile thought and even within a specific implementation, spawn passionate discussions within this vital community.

It is my privilege that I get to explore these topics with you, my reader. But if writing helps me clarify my thinking, dialog helps me expand it. Don’t be a stranger. Tell me about a favorite blog. Turn me onto new ideas. Challenge my viewpoint. I welcome the interaction.

(to be continued)

An Introduction - Trials and Tribulations (Part 2 of 4)

Scott — Wed, 23 Jan 2008 21:25:09 +0000

For quite some time now, I’ve been pressured to begin blogging. I resisted. Too much work. Writing is too hard. To now I’ve had a surfeit of reasons not to begin. But I’ve never demurred because I felt that I had nothing to contribute. Quite the contrary, I think about software development every day.The issues that interest me are organizational in nature. How does one organize teams of contributors, properly motivate them, facilitate working communication between them, and apply technology to build an efficient development process that turns out products of high quality.

My journey to find the answer to this question began some ten years ago when I became a certified PMP (through the Project Management Institute). After ten years of architecting, engineering and building automated test systems I became disenchanted with testing as a means to quality. The works of Edwards Deming and Philip Crosby made an impression upon me and I began to study quality assurance in its many forms. I was hooked.

For almost seven years I toed the PMI line. I believed in waterfall. I believed that if one just strived to understand requirements a little bit better that everything would be ok. That change control boards could work if they exercised the right kind of discipline. That the right product could be delivered on schedule. But it didn’t happen.

After a number of projects went south, I faced myself one morning in the mirror and asked myself a hard question: Was I a bad project manager? To answer that question, I went outside of myself. I looked around at the success rates of other project managers that I truly respected.

What I discovered was that their projects largely were troubled also. I began to arrive at the conclusion that the system was flawed. The way we were taught to execute software development projects is fundamentally unsound. There is too much information. Markets change too quickly. Employee turnover rates are high. Technology changes too rapidly. It’s all too much to plan. The rate of change is too high to manage with a change control board.

Enter Agile development.

(to be continued)

An Introduction - Ego (Part 1 of 4)

Scott — Wed, 23 Jan 2008 21:24:03 +0000

To begin, please allow me to get the absolute basics out of the way.

My name is Scott W. Killen. I live in Austin Texas. I’ve been married for right at 30 years to the love of my life. My kids are grown, productive, members of our society and both make me very proud.

I’ve been a software engineer, technical lead, architect, project manager, PMO director, and software development process engineer at one time or another during my career. The curious can check out my resume.

I’ve spent the last several years of my life studying, promoting and implementing agile software development. I’m a CSM and Founder and President of Agile Austin.

(to be continued)