Over the past several months I’ve had a recurring conversation with various large, enterprise organizations transitioning from traditional approaches to more agile methods.  

The topic of this conversation has been a discomfort with User Stories, and a desire to maintain their investment in Use Cases.  

These organizations come to this conversation hesitantly, but steeled for battle - convinced that I am going to try and disuade them from their ‘un-agile’ ways and insist that they adhere to agile ‘best practices’.

They’ve been generally surprised by my response, so I thought I’d share it here:

‘I actually really like Use Cases; though I tend to use them a little bit differently than it sounds like you are.  I actually combine them with User Stories.  But before I tell you how I do that, let me ask you a question -

Why do you want to keep Use Cases?’

‘Why’ is always a tough question, so we ramble around a bit, but we ultimately get to the point where we say that the Use Cases have 2 primary purposes:

1. They provide a reference for all the details of what we’re doing while building and testing
2. They provide a history of what was done, and how the system behaves

At which point I tell them, “that’s perfect, because that’s pretty much what I’ve used them for too.  What I find all too often is that organizations have a 3rd, hidden purpose for Use Cases - they provide an alternative to talking to each other…”

They look at me kind of funny and I continue - 

“See, in those organizations they view ‘good’ requirements documentation - whether in Use Case format or otherwise - as a stack of paper that they can drop on a developer’s desk, so that the developer doesn’t need to talk to anyone.  If the developer does need to have a conversation with the analyst (or, God forbid, the customer) then that’s a sign of bad or less than ideal requirements documentation - that we didn’t really specify them in enough detail…”

Usually there are a few knowing smiles and nodding heads at this point, so I continue -

“So let me tell you how I’ve used Use Cases.  

I start with a list of Use Case titles - basically a Use Case Catalog.  They generally look something like:

 ‘As a <User Type> I need to <Activity> so that <Business Value>.’  

We called that our Initial Backlog.

Now, those probably look a lot like User Story right?”

They nod

“Well, yes and no actually.”

They then squint at me.

“See that’s just a Story CARD, which isn’t really a story - it’s a ‘reminder to have a conversation’ - which is where the STORY comes from.

We have those conversations on a just-in-time basis of progressive elaboration that we called Backlog Grooming.

Now obviously we don’t want to keep having the same conversations over and over again, so we need to capture the results somehow, so we can refer back to them to know what we’ve discussed; and so we can update them if subsequent discussions lead to changes.  

For some teams, the best way to capture those details is with activity diagrams, state diagrams and wireframes.  For others its a bulleted set of Acceptance Criteria and notes.  And for others it might be Use Cases Scenarios with a data dictionary, business rules and other attributes…  
In fact, it might be any combination of those things based on the story and the preferences of the team.

I’ve worked with lots of teams that used the Use Case format as their means of capturing the results of conversations.

And many of those teams were obligated to provide requirements documentation as part of their product - for regulatory or governance purposes, or as an artifact for support and maintenance.

So, for those teams, we made having an up-to-date set of Use Cases part of our Definition of Done.  Having up-to-date Use Cases allowed us to keep our product Potentially Deployable - which included meeting regulatory and governance documentation needs.”

Being agile doesn’t necessarily mean discarding everything you know.  And it certainly doesn’t mean blind adherence to a set of tools and techniques approved by the agile police.  

Being agile means holding to a set of values and principles that ultimately come down to customer, value, feedback, quality, transparency and sustainability.  

Being agile means questioning, experimenting, inspecting and adapting, and using what works.

This post has been a long time coming.  I’ve started and restarted it at least a half-dozen times.  

Quality - there may be no more multi-faceted and powerful attribute in successful software development.  Quality is central to everything we do and seek.    

• Higher Quality leads to greater Productivity, throughput and velocity
• Higher Quality leads to increased Responsiveness, reduced cycle-times, shorter lead-times
• Higher Quality leads to improved Customer Satisfaction, Employee Satisfaction
• Higher Quality leads to better Predictability, reduced risk, improved decision making

Or at least that’s my hypothesis…

And that hypothesis is widely shared amongst the agile and product development communities.  We’ve developed numerous principles, practices and techniques intended to improve quality:  Test Driven Development; Continuous Integration; Automated Build and Deploy; Pair Programming; Customer Demos; Behavior Driven Development; Acceptance Test Driven Development; and Set-based Design techniques are all at least partially focused on yielding quality improvements.

But quality can’t simply be viewed as a set of tools and techniques - independent variables/levers which we hypothesize will lead to improved business outcomes.  Quality is also a business outcome unto itself.  

This series emphasizes the need to focus on business outcomes (success) 1st - methods and practices 2nd.  So, putting aside the methods and good practice assumptions of agile, and focusing solely on the business outcome of improved quality:

QUALITY = FEWER DEFECTS IN PRODUCTION

We apply agile quality practices and techniques, because we believe that doing so will yield improved business outcomes - Quality (fewer defects in production) being one of those outcomes - along with Productivity; Predictability; Responsiveness; Customer and Employee Satisfaction.

Large, manual, end-of-cycle execution of formal testing by an independent QA organization is also a method aimed at improving these business outcomes.  I hypothesize that it is less effective than alternative agile techniques.  But I don’t take that on faith, and neither should you.  We must test our hypothesis.

HOW DO WE MEASURE QUALITY?

There are innumerable quality metrics that have been devised over the years - each with its own strengths and weaknesses.  In my experience, it’s important to keep metrics simple, and to not let great become the enemy of good enough.  In other words, if you have a metric that does a good job of providing insight into the quality of your product/solution, and is simple to collect and interpret; that is likely better than chasing after a metric that will do a great job, but would be more complicated.

For my part, I’ve had success over the years using a couple relatively simple metrics:

• DEFECT DENSITY - # Defects / KLOC
• DEFECT ARRIVAL - # Defects Identified / Month

WHAT DO WE CONSIDER A DEFECT?

In both cases I include only defects in the production system.  

Measuring defects found and eliminated during the development cycle may be useful for managing your development and quality processes.  But from a business outcomes perspective our focus is reducing the # of defects that make it to production - not making assumptions about how or when to achieve that.

NOT ALL DEFECTS ARE CREATED EQUAL

Any good metric should drive in more questions than answers.  I find it useful to tag Defects with information about Type and Severity, so that we can consider some of those questions more deeply.

• Our Defect Density is high; but our Severity 1 & 2 density is low.  What is the impact on other outcomes (Productivity; Customer Satisfaction; etc…) if we were to invest in reducing our low severity defects?
• Our Defect Arrival is very high immediately following a major release.  But the defects are mostly Type = Usability.  Why are our customers having such a tough time using our new features; and how can we ease them through the learning curve? 

You may have some hypotheses based on these questions.  Perhaps those hypotheses involve application or improved use of agile tools and techniques.  What experiments would you run to prove or disprove your hypothesis?  What new questions will those results yield?

In my bag right now:

• Agile Software Requirements: lean requirements practices for teams, programs and the enterprise; Dean Leffingwell
• The Knowing-Doing Gap: how smart companies turn knowledge into action; Jeffrey Pfeffer & Robert I. Sutton
• Strategy and the Fat Smoker: doing what’s obvious but not easy; David Maister

Essential Topics

• Agile Software Development with Scrum; Ken Schwaber and Mike Beedle
• Extreme Programming Explained; Kent Beck
• Kanban: Successful Evolutionary Change for your Technology Business; David Anderson
• Kanban & Scrum: Making the Most of Both; Henrik Kniberg & Mattias Skarin
• Scrum and XP from the Trenches; Henrik Kniberg (download from infoq.com

Product Owner

• Agile Product Management with Scrum: Creating Products that Customers Love; Roman Pichler
• User Stories Explained; Mike Cohn
• The Principles of Product Development Flow: 2nd Generation Lean Product Development; Donald Reinersten

ScrumMaster

• Collaboration Explained; Jean Tabaka
• Agile Retrospectives: Making Good Teams Great; Esther Derby and Diana Larsen
• Coaching Agile Teams: A Companion for ScrumMasters, Agile Coaches, and Project Managers in Transition; Lyssa Adkins
• Agile Project Management with Scrum; Ken Schwaber
• Agile Estimating and Planning; Mike Cohn

Delivery Team

• Agile Testing: A Practical Guide for Testers and Agile Teams; Lisa Crispin and Janet Gregory
• The Art of Agile Development; James Shore and Shane Warden
• Refactoring: Improving the Design of Existing Code; Kent Beck
• Working Effectively with Legacy Code; Martin Feathers
• Test Driven Development: By Example; Kent Beck
• Clean Code; A handbook of Agile Software Craftsmanship; Bob Martin

Managers & Executives

• The Business Value of Agile Software Methods:  Maximizing ROI with Just-in-time Processes and Documentation; David F. Rico
• Agile and Iterative Development:  A Manager’s Guide; Craig Larman
• Management 3.0 Leading Agile Developers, Developing Agile Leaders; Jurgen Appelo
• MacGregor; Arthur Elliott Carlisle (download)

Large Scale & Enterprise Agile

• Scaling Lean & Agile Development:  Thinking and Organizational Tools for Large-Scale Scrum; Craig Larman and Bas Vodde
• Practices for Scaling Lean & Agile Development: Large, Multisite, and Offshore Product Development with Large-Scale Scrum; Craig Larman and Bas Vodde
• Agile Software Requirements: Lean Requirement Practices for Teams, Programs and the Enterprise; Dean Leffingwell
• Scaling Software Agility:  Best Practices for Large Enterprises; Dean Leffingwell

Organizational Change

• Switch: How to Change Things When Change is Hard; Chip Heath and Dan Heath
• Drive: The Surprising Truth About What Motivates Us; Daniel H. Pink
• Leading Change; John P. Kotter
• Abolish Performance Appraisals: Why They Backfire and What to Do Instead; Tom Coens & Mark Jenkins M.D.
• Succeeding with Agile:  Software Development Using Scrum; Mike Cohn

Advanced Topics and Theory

• The Principles of Product Development Flow: 2nd Generation Lean Product Development; Donald Reinersten
• The Goal: A Process of Ongoing Improvement; Eliyahu M. Goldratt
• The Human Side of the Enterprise; Douglas MacGregor
• The New Economics; W. Edwards Deming
• Scoring a Whole in One; Edward Martin Baker
• Understanding Variation: The Key to Managing Chaos; Donald J. Wheeler
• The Black Swan: The Impact of the Highly Improbable; Nassim Nicholas Taleb

For many leaders, increasing the productivity of the development organization is their primary and over-riding goal.  ‘Doing more with less’ is a mantra they preach to their teams continuously, and which colors every decision they make.

Yet few have a clear and consistent definition of productivity, or an effective means of measuring it. 

Productivity:  a relative measure of the efficiency of a system in converting inputs into useful outputs.

Productivity: the ratio of the real value of outputs to the combined input of labour and capital.

Productivity: a measure relating a quantity and quality of output to the inputs required to produce it.

So in simplest terms:

Productivity = Valuable Outputs / Costly Inputs

Be Careful What you Wish For

While I am a ardent believer in the value of metrics, it is important to keep in mind the potential for unintended consequences.  Individual and group behaviors will evolve to meet your measures - not necessarily your goals.  So ensure that the measures you use promote the behaviors you desire.

Most documented productivity metrics in software development define a unit of output as either a KLOC (thousand lines of code) or as a Function Point.  Both have significant disadvantages.

KLOCs are generally favored for the simple reason that they’re unambiguous and easy to measure.  However, they can unintentionally promote poor design and coding practices.  Elegant, efficient, maintainable software takes more time to write, and takes fewer lines of code.  Therefore, a KLOC metric inadvertently creates a disincentive for building high quality software, and rewards poorly thought through designs and shoddy craftsmanship.

While Function Points don’t necessarily promote poor design and craftsmanship they have their own unique challenge - they are famously difficult and expensive to calculate and measure.

Perhaps even more importantly, both KLOCs and Function Points ignore the core aspect of ‘Value’ inherent in our definition of output - and therefore productivity. 

It has been stated that 64% of the features and function in the typical software system are rarely or never used (Standish 2002).  Calling the code that delivers these features and functions “productive” may be a mischaracterization. 

Similarly, those features and functions that ARE commonly used, are generally not of uniform value.  The Pareto Principle would suggest that 80% of the value is delivered by 20% of the effort.  Yet KLOC and Function Point based metrics treat all features and functions (and code) delivered as interchangeable.  Which promotes focusing on the easiest and lowest technical risk; rather than the most valuable, most innovative and greatest business risk…  

Output is a measure of the value delivered, not the effort expended.

A number of people in the Agile Community have written about an alternative unit of output measure - Value Points.  While the simplicity and value focus of this model is appealing, it has challenges when scaled beyond a few teams.  In order to be meaningful at the organizational level you must normalize the relative value point scale across teams and programs - which can be difficult and expensive. 

Also, the Value Point approach does not easily translate to initiatives and/or divisions that may not be delivering in an Agile manner.  Having a common measure of output, and therefore productivity, is critical to measuring the impact of your agile investment.

So, the approach we recommend is to associate a percentage of the total value of a given initiative to each Minimally Marketable Feature (MMF) or production release.  These percentages can then be applied to a any monetary business justifications (ROI, NPV, Discounted Cashflow, etc…) to arrive at an expected dollar value realized from each release.

Hence,

Productivity = Total Value Realized (delivered to production) /  Total Cost of production (labor)

Using this approach, your organization (be it a scrum team, a multi-team agile program, a waterfall project, or even an entire product development group) base-line their productivity for a period of time, and monitor its change over time.

Example:

The division has 3 initiative in progress:

• Initiative A:  Total Expected Value $5MM; being delivered by a scrum team with an iteration run-rate of $70,000.
• Initiative B:  Total Expected Value of $25MM; being delivered 5 scrum teams with a combined iteration run-rate of $400,000.
• Initiative C:  Total Expected Value of $50MM; being delivered according to a project plan and resource matrix that charges $2.5MM to the project in the 1st quarter.

In Q1:

• Initiative A: Released to production monthly and delivered a total of 60% of Expected Value; or $3MM.  25% of the backlog has been burned-down in terms of story points.  Total Cost $210,000.  PRODUCTIVITY = 14.29
• Initiative B: Released to production quarterly and delivered a total of 65% of Expected Value; or $16.25MM.  35% of the backlog has been burned-down in terms of story points.  Total Cost $1.2MM.  PRODUCTIVITY = 13.54
• Initiative C: Completed requirements definition and is 50% done with Detailed Design, and delivered 0% of Total Expected Value.  Total Cost $1.2MM.  PRODUCTIVITY = 0.

                       *undelivered work is WIP, and therefore not yet productive.

Aggregate Division Productivity for Q1 = 7.38

As you can see, it would be relatively straight forward to predict Q2 productivity - at the initiative as well as division levels - by assessing the various product roadmaps and traditional project plans.

Those projections could then be used to drive discussion about trade-offs on where to allocate limited capacity and maximize productivity.  To staff and fund initiatives where productive potential is high, and to cancel successful projects whose greatest productive potential has already been harvested.

To inform intelligent business decisions - which is WHY we’re measure outcomes.

Next Up:  QUALITY

     “How do we measure the success of agile?” 

It’s one of the most common questions we hear from senior leaders.  And its a critically important one for agile evangelists working to justify the organization’s investment in agile, and maintain momentum as other priorities compete for leadership attention.

The agile community’s typical response to this question has been some form of an agile maturity assessment - such as the Nokia Test.  

These tools are clear, easy to use, and can be extremely effective in helping organizations assess their adherence to good agile practices.  Yet, when used in isolation, they can leave senior leaders unfulfilled - and miss an important opportunity for aligning agile to fundamental objectives. 

For business leader, the question isn’t how well are we doing agile?  The question is how well is agile doing for us?  What impact is agile having on business results?

This is the question senior leaders really want answered. 

My response has typically been:

     “How did you measure your impact on business results before Agile?”

Which is generally met with awkward silence and a muttered admission of:

     “not very well.”

The conversation then moves toward why measurement is expensive, why you can’t show progress if you don’t have a baseline, and why you need to be very careful about what and how you measure - lest you create unhealthy behaviors and unintended consequences. 

Someone ultimately quotes Einstein, we nod our heads thoughtfully, and finally move on to other topics. Crisis averted!

But the question remains -

     “How do we measure the success of agile?”

and, if Agile Success = Business Success, then the real question is:

     “How do we measure business success?”

Which is the question we’re beginning this blog series to address.

While every organization will have their own unique objectives and priorities, most can be encapsulated as some combination of these:

• Productivity
• Quality
• Time-to-Market
• Responsiveness
• Customer Satisfaction
• Employee Satisfaction
• Predictability

In the posts to follow we will examine each of these Business Outcomes and look at:

• How can we measure the business outcome?
• What agile practices are effective levers in improving the business outcome?
• How can we measure our agile levers as leading indicators of improved business outcomes?

1st up - PRODUCTIVITY

Several weeks ago, I made a pact with my Rally colleague Ken Clyne to finally begin blogging in 2011. 

At the time, I expected that I’d write about agile product development - what it is to “be agile”, the relative merits of different frameworks, various tips and tricks, and basically share my experiences leading agile transformations.  

After all, it’s what I do - it’s what I know.

But then I came across this great TED Talk: “The Golden Circle”, by Simon Sinek. 

“People don’t buy What you do, they buy Why you do it”…  Its amazing what happens when you start asking the right questions.

Agile isn’t the ‘Why’ - at least not for me.  Agile coaching is ‘What’ I do.  Why do I do it?  What do I believe?

First, I believe in LEADERSHIP.

Leadership tends to get a bad rap in some corners of the agile community.  After all, agile is about self-managed, empowered teams and the wisdom of the crowd.  Potential leaders are too often equated with traditional, autocratic management - slow, bureaucratic and in-humane - so they’re effectively told to ‘just stay out of the team’s way’ - and that’s unfortunate.

Truly enlightened leadership is the key to high-performing teams.  Real leadership unleashes the potential of people; transforms them into a team; inspires their passions and focuses their energies. 

And I believe in RESULTS.

When we invest our time, energy and passion we expect to achieve something.  To realize meaningful results.  Results matter.  Results can mean more than making money (though it almost always includes that) - delighting customers, being first to market, and creating a great work environment may all be objectives for you, your team and your organization.  Are you achieving results?

No matter the beauty of your process or the philosophical purity of your approach; if it doesn’t yield results it’s a well executed failure.

Too often, results seem to get lost in process maturity and methodology dogma - and that’s certainly not unique to the agile community.

So…  Why Leading Results?

• Because I believe that enlightened leadership is the key to unleashing the potential of high performing teams that achieve results beyond the imagining of their individual members…

• I help unleash this potential by coaching people on how to lead effectively, from any position within the organization…

• I just happen to coach lean/agile principles and practices…

• Would you like to buy some?