You are here:>Disciplined Learning

Disciplined Learning

How “Learn Early, Learn Often” Takes Us Beyond Risk Reduction

Dr. Alistair Cockurn
Humans and Technology
Technical Report TR-2013.01
Feb. 2013,
©2013 Alistair Cockburn

alias: Learn Early, Learn Often
alias: DAKA2 (and see also DAKA)

The beautifully laid out PDF of this article, as prepared for CrossTalk Magazine, is available at:

Apply Figure 1 and you have everything, including Lean Startups, Collaborative Agile Marketing, feature thinning, “Trim the Tail”, Deliberate Discovery and risk-based project management [ER, GW, AC-ft, AC-ttt, DN, W-rm].

knowledge aquisition curve.png
Figure 1: Disciplined knowledge acquisition during product development

Although Figure 1 shows the desired path of learning, most development and marketing looks more like Figure 2:

Late learning curve.png
Figure 2: The typical “late-learning” sequence of product development.

To get acquainted with the idea, let’s first walk through Figure 2, the typical “late-learning” sequence:

The project starts with a great idea, high hopes, and no consideration that any of the ideas might be wrong. The charter is drawn up, the team(s) assembled, and people start working energetically. They work hard, talking with each other, learning about their assignment, about the design, about each other.

They need to learn four things:
1. How to work together, and indeed, whether these people can get this job done at all.
2. Where their technical ideas are flawed.
3. How much it will cost to develop. And finally:
4. Whether they are even building the right thing.

When do they learn these four things?

#1 they learn along the way. At some point they learn whether the people they assembled have the skills needed to get the job done or they got the staffing wrong. They also learn how to work together, becoming a “jelled” or high-performing team [TdM], if they are lucky.
#2 they all-too-often learn toward the end, when they integrate the pieces designed by the different teams, and learn what mistaken assumptions they made about each others’ work (hence the famous quip: “the first 90% of the project takes the first 90% of the time, the last 10% of the project takes the next 90% of the time”).
#3 they learn at the end, when all the work is done. There are typically so many surprises at the end that no cost estimate is meaningful until the final bill is turned in.
#4 they learn when it is all over late: when the customer does or does not buy the product. Unfortunately, this, the most important to learn, is learned last.

My purpose in this article is to describe how some project teams manage to avoid the bulk of these errors (late surprises catch even the best teams). These project teams enjoy not only greater success, but something incredible and rare:

the ability for the business to decide whether to deliver:
– earlier with small things trimmed out in order to hit a desired delivery moment, or
– later, with those small things improved for a more refined, higher quality experience.

No established method, agile or otherwise, allows that choice at the moment.

This disciplined knowledge acquisition approach is the evolution of risk management. Where risk management approaches ask

“What could prevent us from delivering?”

disciplined knowledge acquisition asks

“How can we get the best possible results from our efforts?”

Let’s turn our attention back to Figure 1, repeated below:

knowledge aquisition curve.png
Figure 1 (repeated): Disciplined knowledge acquisition during product development

The magic trick is to apply the “broken” strategy in very small stages. The problem is not that learning doesn’t happen in the ordinary approach, the problem is that the learning comes at integration and again at sales, i.e., late. These are often the first moments when the team members are forced to put their ideas together, and the first time that the customer, user or buyer gets to see the new system.

What we will do is to apply the concept of “breaking things” at very small scales, and take them in the preferred order:

Q1. How can we learn what should get built?
Q2. How can we learn how much it will cost (time, money, people)?
Q3. How can we accelerate the team learning how to work together?
Q4. How soon can we correct the mistaken assumptions in the design?

The result is an amazingly effective development plan, quite different from what is recommended in standard agile development, and quite disorienting for the newcomer.

The discussion is simpler if we look at these in reverse order:

Q4. Correct mistaken design decisions

This category includes recovering from all technical errors, such as

  • choosing technology that doesn’t work as advertised,
  • the inevitable omissions and mistakes in design, and
  • mistakes due to people not talking to each other, causing out of date and mistaken assumptions about each others’ work.

The way to do this is the simple, if sometimes difficult practice of “micro-incremental” development:

  • Integrate early, integrate often

The best developers integrate every hour, more often if possible. (Less expert developers explain why they can’t integrate that often; the expert developers find ways to integrate anyway.)

When designing, people make decisions often at the rate of several per minute or one every several minutes (note: these are approximate, not measured rates). The mistakes in those decisions are based on mistakes in their memories, in their conversations, in their thoughts. Left untested, the mistakes build on each other, becoming increasingly more expensive to correct. Thus, finding mistakes sooner is important.

The “walking skeleton” strategy [AC-ws] is the start of the micro-incremental sequence. The basic architecture is constructed supporting the smallest connections through the architectural elements, allowing the team to discover the first round of surprises in the technologies they are using, and celebrate success once they have the pieces working (also part of Q3, “Learn to work together”).

Once the system is thinly connected, each team adds onto the system, integrating every hour or so, and possibly refactoring the skeleton itself. Since they are integrating so frequently, they will discover their mistaken assumptions and misunderstood communications much more quickly. As a side benefit, they are less likely to change the same part of the design at the same time, and so they do not need to check out and hold onto separate copies of the design, making integration easier, faster, and less error prone.

Micro-incremental development with frequent integration does not address a second category of technical mistakes: system performance, including load time, speed and scaling. They are addressed through early use of the walking skeleton and

  • Spikes [WC-c2spike]

A spike is a small, disposable piece of work created to explicitly address the question, “Will this possibly work?”

The difference between a spike and ordinary incremental development is that ordinary incremental development is conducted using full production conventions, with the assumption that the work will be used in the final product; whereas the spike work will absolutely not be used in the final product, is throwaway work, and therefore will be done in the most rapid and effective manner possible with the sole purpose of addressing the question at hand.

With a walking skeleton at hand and spike solution work available as a strategy, questions about the loading, throughput, and scaled performance of new technology and of the final, integrated system can be moved forward in the timeline so they can be addressed early(-ier) in the project, and not left to the very end.

There is a final category of technical surprises, those that might seem impossible to move forward in the schedule. An example is the final conversion of the database from one form to another.

The strategy to use here is this:

  • Story splitting [AC-abws]

Split a story into a spike piece and a production piece. Move the spike piece forward in the schedule in order to learn what problems lie underneath it and how to deliver it effectively; leave the actual implementation late in the schedule, but with the added confidence of the information that the spike delivered.

Here is an example of story splitting:

Our website was going to be converted from Wikipedia markup format to Textile format. Part-way through the project, we suddenly discovered that none of us was expert at language manipulation and really knew how to do this conversion. We stopped other work in order to set into place a trusted way to do that conversion. It took less than two hours to locate an open-source framework, install it, try it out, modify it, and convert a few sample pages of the site. With that in place, we went back to our regularly scheduled work, knowing that when it came time to convert the database, it would be a straightforward task.

Summarizing, the three strategies for disciplined learning on technical issues are:

  • Micro-incremental development
  • Spikes
  • Story splitting

I will introduce one, final, slight adjustment to these strategies in the section, “Creating a plan.”

Q3. Correct staffing and learn to work together

This category of learning includes the social aspects, from hiring to creating what Tom DeMarco and Tim Lister called a “jelled team” [TdM-jt].

Ordinary incremental development and increment delivery helps to flush out as quickly as possible whether the people on the team have the skills they advertised when getting hired.

However, failure to deliver sometimes is due, not to the people being not correct for the assignment, but to them not having learned how to work together. This is remedied through the strategies

  • “Early victory” [AC-ev]
  • “Walking Skeleton” [AC-ws]
  • “Simplest first, worst second” [AC-sfws]

The first strategy is based on the work of sociologists [KW], showing that achieving results helps people come to trust each other more, raises morale and helps them perform better.

The development strategy “Walking Skeleton” calls for a thin thread through the system under design to be constructed. It delivers an early, real victory to the team and to the sponsors. Originally a technical design strategy, the Walking Skeleton idea can be applied to developing and deploying new organizational processes, products into work flow, technical architecture, and even the development of individual system features.

The third strategy, “simplest-first, worst second” builds on “early victory” in a way contrary to the current common recommendation in the agile development world. The advice there, to build “highest business value first”, makes good sense once the team is functioning well, social risks have been reduced, and the team is capable and confident of being able to deliver whatever is of the highest business value (delivering something of high business value is, indeed, a victory the team can celebrate).

However, many conversations need to take place before the team has ironed out the types of conflicts and skill gaps that are likely to be present. For this reason, a useful strategy is to build something real, but very simple, so that you can encounter both social and technical surprises early, in good time to fix them.

Q2. Learn how much it will cost

The most difficult to apply set of strategies surrounds discovering how much a development project will cost. These strategies are difficult to apply because they cost money, and time, and deliver nothing but knowledge, and so test the patience of everyone involved, from the sponsor to the developers. As with all strategies, there are times when these are not needed, and others when these are badly needed.

  • “Core samples”
  • “Microcosm” [AC-rrp]

The following story is one I heard Tim Lister give at a conference:

A man wanting a pool built in his back yard calls in three contractors to present estimates. The third contractor, instead of presenting an estimate, tells the homeowner he will need to drill a core sample in the ground, and will charge the man for that.
The homeowner complains, saying that the first two contractors didn’t charge him for core sampling.
The contractor responds that he has no idea how the first two contractors could submit a bid, since they don’t know what sorts of rock layer lies under the lawn, but he couldn’t possibly put in a bid without having that information.
The homeowner is equally puzzled, but now comfortable with with the third contractor, whom he hires for the work.

One can do the same with a development project, identifying and isolating particular parts of the system the development of which is not completely obvious, and developing very small elements of the system within those areas, in order to understand how easy or how difficult the work will really be, and to identify what sorts of surprises might lurk below the surface. Carefully selecting such “core samples” allows the project team to develop a more reliable cost-, time-, and resource estimate for the project.

Core sampling is the miniature version of the more general “Microcosm” strategy, in which a mini-project is run for the sole purpose of establishing a sound estimate. A full Microcosm project can be set up to test the productivity of a new development team (think off-shoring, in particular), as well as to test the learning speed of staff with new technologies, and to benchmark the productivity of expert versus ordinary or new developers.

Whereas a core sample effort is intended to take hours to days, a full Microcosm project may take weeks to carry out, and should therefore only be used for large development efforts.

Q1. Learn what should get built

Finally, we get to the most important question needing to be answered: Will people like, buy and use what we’re building? Normally, this question gets answered when it is all too late. Recently, however, strategies have come into fairly common usage that move this learning process forward. The strategies are fairly simply, but as with all the others, require discipline, patience, and a willingness to change based on what is learned.

  • Paper prototyping
  • Ambassador user
  • Early delivery
  • Empty delivery and Manual Delivery [ER]

Paper prototyping [W-pp] and related strategies involve nothing more complicated than putting a mockup of the product into the hands of the consumer, who reacts to these early design thoughts. Prepared at low cost, early in the development cycle, these prototypes allow the development team to change their minds about how to proceed.

An “ambassador user” is a friendly user to whom the team can deliver an incomplete but growing product (using, of course, incremental development). This user can usually break the system within moments, or give valuable feedback from his or her limited perspective. The difference between the “ambassador user” and “paper prototyping” is that the ambassador user is encountering the actual system as it grows, not a mockup of the system.

“Early delivery” is a full deployment of the system, but with reduced capabilities. The intention is to learn, first of all, what is incorrect with the product as envisioned, but possibly more significantly, how the presence of the product in its final environment will change the requirements for what should be built in the first place. This strategy is built on the recognition that once people start using a system, their habits and needs change, often in unpredictable ways. Delivering a thinned version of the system early allows the development team to gather new input and adjust the priorities on what should be developed.

The most interesting strategies to emerge in the last decade are Empty and Manual Delivery (my terms for them), both legitimized by the “lean startup” movement [ER].

“Empty Delivery” is epitomized by the “parrot cage” story [BBC]. In brief:

The company wanting to find a new market, saw that there were many searches for parrot cages, but not many paid ads supplying them. So they created an empty site that advertised parrot cages, but had no delivery capability.

Over time they adjusted the web site to fit demand, introduced suppliers, and eventually moved into the business fully.

It is practical to use the “Empty Delivery” strategy for online products, where, initially, all you care about is whether anyone clicks on a link or accesses a feature. Measuring these clicks, you can develop those features that are drawing the most attention, and evolve the system in the direction of maximum draw.

“Manual Delivery” is described for several companies in Eric Ries’ book, The Lean Startup. Companies initially spend far more money than they eventually could afford to, doing work, even delivering products, manually, for the simple reason that manual procedures can be set up and changed for very little cost. Delivering manually, the designers can change the product offering on every purchase, evolving to what the customer base indicates is really desired.

All of these strategies, while requiring inventiveness and discipline, allow the development team to move this most important category of learning far forward in the development cycle shifting the question from

Will people like, buy and use what we’re building?


What changes should we make so that it is?

Creating a plan

In the light of these disciplined knowledge acquisition strategies, the creation of a project plan is rather different than before.

The old-style, and still frequently used, project plan is a list of the tasks to be performed to construct the system. An effective, rapid and low cost planning method of this type is “Blitz Planning” [AC-bp]. This conventional project plan still provides a good starting point, serving to peek ahead at what is involved in developing the system and delivering approximate size and difficulty for the effort.

The more recent, agile-style project plan is simply a list of all the features or user stories [W-us] to be built in a single list, usually with the highest business value item placed at the top of the list (see above for the reasons this may not be the best order to place them in).

The knowledge-acquisition approach, on the contrary, starts with a brainstorming of the four categories of knowledge needing to be acquired, along with the starting set of features to be delivered, making five categories of possible work efforts. These are artfully interleaved into a sequence of work assignments designed to reduce risk, deliver crucial information, and develop product capability in an “optimal” way.

You don’t have to be very alert as a reader to see the words “artful” and “optimal” there. The quality of the plan is quite plainly sensitive to the ability of the planners to brainstorm, list, identify and sort the possible risks, the learning opportunities, and the possibilities for income in their future. In everyday life, the project plan will need to be updated periodically as lessons are learned and new risks and opportunities spotted. Even with the best intentions and work, the team is quite likely to get surprised late in the game by something they had not considered or detected.

I mentioned at the end of Q4, above, there is one twist to mention in creating the project plan. That twist is that there are not just two parts to each feature (the risk element that gets reduced and the actual delivery part), there are three:

  • The risk element
  • The main value part
  • The polishing and glossing that makes the feature “wonderful”, the “tail”.

Not every feature is of equal value to the buyers and users. Arrange for the minimum set of features to be at an “adequate” level of delivery in plenty of time before delivery (if possible), then spend the last time polishing and glossing those feature that are more important than the others [JP-ttt].

This leads straight into the “trim the tail” result.

Reaping the Benefits: Early Income & Trimming the Tail

Developing with disciplined knowledge acquision well delivers two benefits:

  • Early income
  • Trimming the tail

Early income is the obvious benefit and is well presented in Software by Numbers [MD]. A project can become self-funding if it is delivered to paying users part-way through its development, thus lowering the load on the sponsors.

Less obvious but equally valuable is the ability to “trim the tail,” not developing aspects of the system that are less valuable. This is a bit less obvious, and deserves a more description. Here is the shortest example, to give the idea:

When you are opening a new hotel, it may not be necessary to shine the doorknobs before opening to the public. If it is necessary to have shined doorknobs for the guests, it is probably not necessary that all of the doorknobs need be shined.

Four aspects of a system might be trimmed back:

  • Complete features
  • Feature details
  • Usage quality
  • Internal quality

The first is obvious: For an early delivery, your (e.g.) car might not need a sun roof. The first iPads delivered did not not contain telephone capability.

The second is less obvious: Given that your car must have all of the basics of a car (such as brakes), it might not need fancy brakes with (for example) an antilock braking system. A computer system might require searching capability, but not include auto-completion or auto-correction ability. In the case of the iPhone, it would have been possible to have delivered it without the sophisticated scrolling, bouncing visuals.

The third is related to the second: Really smooth and easy to use features take a lot of work, with a lot of revision. it is possible to prepare a delivery-ready version of a feature that is intermediate in usability, ready to deliver earlier.

The fourth has to do with defect rates and internal design quality. The question is how much internal quality is needed for the delivery in question.

If, and this is the important if, development has proceeded incrementally, attending to the knowledge management areas, then the sponsoring, management, and development teams have the luxury of being able to deliver

  • early, with reduced features or quality;
  • on time, with either full or reduced quality, depending on where development stands at that time; or
  • later, with enriched features or quality;

at the choice of the sponsors!

Under usual project circumstances, these choices are not available, the only choices being to delay or work overtime. The “trim the tail” option is available only for those who have worked in this more disciplined fashion.

The trim-the-tail strategy is one of the few strategies equally available to very small and very large projects, fixed-price and floating-price projects. Here are three examples, taken from real projects:

Small, floating-price project: A new web site development involving only two people, the web site owner and the programmer. After several months of open-ended work, the web site owner wanted the site delivered “soon”, and trimmed the tail back aggressively and repeatedly until something much smaller than expected but still suitable was deployed.
Small, fixed-price project: The company in question always bid small, fixed-price contracts of three- to six-months, involving three to eight people. As usual, the bids were aggressive and the teams typically ended late, missing the deadline or scope, with resulting overtime from the developers and penalties at the end of the contract. Jeff Patton [JP] worked in the manner described in this article, leaving the least important features to the end, and deliberately thinning the less critical features, so that when the contract period ended, it was clear to the customers that they had gotten most of what they wanted. This produced the least overtime, the smallest penalties, the highest customer satisfaction and the greatest likelihood of receiving a follow-on contract.
Very large development project: A company with several thousand developers in several countries, working on a product line with multiple variations, applications and releases. Under normal circumstances, when they call for a full integration on a particular date, every team starts to work overtime and jockey for position not to be the one most behind schedule. The integration date keeps getting slipped back as team after team fails to complete their work on time. Using the trim-the-tail approach, each team would have in place the essential elements needed for the integration, with only tail elements left unfinished. For delivery, management would be in position to deliver slightly less, on time, or slightly more, a bit later.

Final Summary

Four knowledge areas in which development project suffer are

  • What is the right thing to build.
  • How much it will cost to develop.
  • Whether adequate people are on the team, and how they best work together.
  • Where their technical ideas are flawed.

The disciplined knowledge acquisition approach consists of identifying, early and continuously, small experiments and small slices of deliverable work, so that the development team gains the knowledge in these four areas in time to maximize the effectiveness of the product delivered.

Some strategies described to accomplish this include:

  • Integrate early, integrate often
  • Early victory
  • Core samples
  • Microcosm
  • Paper prototyping
  • Walking Skeleton
  • Simplest first, worst second
  • Story splitting
  • Spikes
  • Ambassador user
  • Early delivery
  • Empty delivery and Manual delivery
  • Trim the tail

It is exciting to find a baseline strategy that applies to projects of such different sizes and natures as just outlined. The disciplined knowledge acquisition approach is, as mentioned at the start, not for the faint of heart — it requires discipline, inventiveness and constant correction. The payoff is the ability to get a development team working together, discover what is needed in time to delivery it, delivery it early in order to create a self-funding project, and trim the tail at the end to meet inelastic deadlines.


[AC-sfws] Alistair Cockburn, Crystal Clear: A Human-Powered Methodology for Small Teams, Addison-Wesley, 2005. Also online at
[BBC] “Searching the internet’s long tail and finding parrot cages”,
[ER] Eric Reis, The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses, Crown Business, 2011.
[JP] Jeff Patton, “Unfixing the Fixed Scope Project: Using Agile Methodologies to Create Flexibility in Project Scope”, in Agile Development Conference 2003, Proceedings of the Conference on Agile Development, 2003, ACM Press. Available online through a Google docs search.
[KW] Karl Weick, The Social Psychology of Organizing, McGraw-Hill Humanities/Social Sciences/Languages; 2nd edition, 1979.
[MD] Mark Denne and Jane Cleland-Huang. Software by Numbers: Low-Risk, High-Return Development. Prentice-Hall, 2003.
[TdM-jt] Tom DeMarco and Timothy Lister, Peopleware: Productive Projects and Teams. New York: Dorset House Publishing Co., 1987.

Known Uses

The Lean Startup field is the best current arena in which to find examples. A super example is this one:

One Comment

  1. astrograph October 1, 2013 at 9:18 am - Reply

    The diagrams are intriguing, but what is actually plotted on the vertical axis? Knowledge gained during the projects progress? To me this doesn’t really map to business value which seems to be flattening out during the later stages.

    A lot can be learned by a team (exploring new frameworks) or an organisation without actually creating business value, or bringing the project nearer to completion. I would like to see some elaboration on how and under which circumstances knowledge and business value (or completeness of the product) correlate.

    Also in a multi-Team multi project environment early integration can be a danger to the overall success of a program consisting of multiple projects. How can early integration and the concept of a stable trunk be reconciled? One would need multiple stable “trunks” and multiple integration teams to allow early integration of work-in-progress sub projects.

    -by astrograph on 10/1/2013 at 9:18 AM

Leave A Comment

4 + 1 =

This site uses Akismet to reduce spam. Learn how your comment data is processed.