Why won't Agile work in Government?

In a Computer Weekly blog, lawyer Alistair Maughan argues that "Agile will fail GovIT".  The blog was also referenced in the DSDM group on Linkedin.

Mr Maughan puts his case well and succinctly, in his original post and in his response to comments on the CW blog.  Quite reasonably, he does recognise that the current process is not very good – but it is the natural outcome of the constraints of UK and EU legal, and historic behavioural, constraints.

I have frequently quoted McCracken and Jackson (as increasingly have others) from 1982, presciently objecting at the conference setting out the software development lifecycle: “systems requirements cannot ever be stated fully in advance, not even in principle, because the user doesn't know them in advance--not even in principle”.  IT projects, indeed all change projects, are voyages of discovery.  We know more about what we want at the end of a project than we could ever have done at the beginning – it not’s hindsight, it’s learning from experience, it’s the way life works, to argue otherwise is to ignore the human condition.

In his blog, Mr Maughan highlights four specific areas where Agile fails.  Let me address each in turn.  In his subsequent comment, it becomes clearer that the key is not that Agile is inappropriate for government, but that the change from current practice, beliefs, and behaviours is a far greater obstacle than the concept of Agile itself.  The goal therefore for proponents of Agile in government is to plot a path that overcomes the obstacles, and allows Agile to succeed.

1. Knowledge of costs up front.  For an government investment in IT, the cost should be in proportion to the business benefit, and should be limited by the business case at Main Gate. 

Agile provides a mechanism, prioritisation, to contain function to the available cost.  Agile has a strong principle of containing cost and varying detailed requirements, discarding those that offer least value. 

It is however critical that the overall budget is realistic - estimating is hard for large scale government IT (whatever approach is adopted).  Suppliers, when bidding for work, must have confidence that they can deliver a sufficient solution for the money, and have a process to contain cost to the budget.  Agile will only work within a fixed price if the value of the contract exceeds the cost a minimal useable subset of the requirements, sufficient to deliver the business case.

There is much experience that suppliers are still learning the skills required to manage an agile contract to a fixed price, notably in managing the daily negotiations of features, and in the significant hygiene factors for government such as security.  The move to cloud and enterprise architectures which provide many of the hygiene factors as services to all applications within the architecture should reduce the exposure of individual projects to such overheads, but government IT is always likely to be more expensive than similar functions for commercial organisations.

The standard approach has no such negotiation mechanism, and costs from change controls (sic) frequently raise costs beyond the original business case. 

In summary, Agile can be done fixed price, against a defined outcome, typically a satisfactory working system, but ideally a new way of working, delivering the business case.

2. Open procurement - as anyone who has run an open procurement knows, anyone bidding in an open procurement will commit to meeting all the requirements at the highest cost they think will win the business.  The less scrupulous will then work out how to extend the funding to meet the real cost of delivering the stated requirements.  As the project progresses, new requirements emerge.  There is therefore an ongoing negotiation throughout any IT contract, wherein the client and supplier independently better understand the gaps in the contracted requirements, and seek a compromise within their respective constraints.  Sometimes, amicable negotiations are successful, sometimes formal changes controls have to be funded, sometimes it finishes up in court.

It's another part of the broken system, but we're stuck with it.

Agile explicitly recognises the reality of on-going negotiations.  Many client staff quickly recognise the reality, and the common sense intrinsic in Agile.  Mr Maughan may be over estimating the training and refocusing required to adopt agile thinking.  Adoption of Agile is always incremental with pathfinders developing locally appropriate ways of working, and propagating best practice around their organisations.

In my experience, however, many government IT suppliers struggle with the openness, and adequate management controls needed to negotiate the contractual detail in daily/weekly meetings, in order to stay within budget. 

Moreover, (somewhat scurrilously) I have long suspected that suppliers are now so accustomed to the change control mechanism associated with fixed price government contracts, that managers are (implicitly?) rewarded for "developing the account" through change controls.  Suppliers, and many clients, expect the value of a contract to increase, often significantly, above the original value, and suppliers act to fulfil the expectation.

Agile at least gives the client side some procedural leverage against the change control, and highlights very early, in the first few iterations, whether the supplier is in charge of the process/project.

EU procurement need not preclude Agile approaches.  Clients need not, and indeed should not,  explicitly demand agile management;  after all, all capable bid teams will bid an agile approach, regardless of actual capability.  Clients can however value agile approaches on the basis that the approach is more likely to deliver a satisfactory outcome within the available budget, with less risk of overshoot.  Price will always be a factor, but risk of failure needs to be factored.  Evaluation teams no longer automatically select the cheapest bid.  The differentiator should always be value for money, taking account of the risks.

Once it becomes clear that government values agile working, then the suppliers will respond.  Some will become more agile; some will resist, any way they can, by undermining agile thinking and approaches.  It may take many years, but the government market can become more agile friendly.

3.  Insufficient remedy.  I agree: T&M projects place the risk on the client to manage the supplier’s work.  Similarly, (you'd think this is obvious, but it always comes as surprise) heavily specified requirements place the risk on the authors and approvers of the specification, not on the supplier – suppliers are given leverage by the inevitable errors and omissions in any specification.  Consequently government projects spend precious time and money reviewing and revising requirements, at great cost, often delaying the project beyond it’s window of opportunity.  Agile attempts to address the E&O risk at source, by correcting errors and omissions within the project process, rather than as exceptions to the process.

Remedy is only available therefore from specifying a business related outcome – a working solution, a new way of working, a demonstrably more efficient process.  Moreover the outcome must be specified at a sufficiently high level to minimise errors and omissions. 

Government and industry have seldom succeeded in agreeing such outcome based contracts, with adequate remedy.  Industry goes to considerable length to avoid taking the risk, and government is seldom robust in it’s position, often agreeing to fault in specification, resulting in additional expenditure on change controls.

We recently completed a 3 year programme with a dozen statements of work, each of which had a couple of paragraphs defining an outcome, negotiated fixed prices, and penalties.  The risk was on the supplier, who had to pay significantly for under-estimating tasks they had supposedly fully analysed.  Interestingly, we and the client wanted an Agile approach, to which the supplier initially agreed, but then traded a fixed price for a fixed specification, but eventually had to become agile to meet the contractual deadlines, the old “agile last mile”.

Remedy is feasible within an Agile project, but it requires a different approach.

4.  Public Sector management structures - decisions going up instead of down.  I have to concede there are issues with accountability, not only in government but also across commerce and industry.  Managers have more pressure and less time to take considered decisions - less confidence in their own position discourages proper delegation of powers and decisions. 

We have a long tradition in Britain of raising a decision to a level where the arguments can be heard by a neutral authority, whether a judge or a senior civil servant.  This tradition however disempowers managers, and creates a dysfunctional management, where the main objective is to hide bad news from superiors. 

Political pressure from ministers for rapid action, and a desperate desire from the press, if not always the public, for a blame game, creates a climate of fear, leads to Sir Humphrey avoiding all accountability for anything except the most glorious success.  On the other hand, many projects are declared a success, regardless of how they compare to orginal business case, which should be, simply because expediency demands a success.

There is a public sector management structure that is fully functional, and very agile:  the military in theatre.  Rules of engagement are promulgated to empower commanders on the ground/in the air/at sea, so that in the fog of war, decisions can be taken, regardless of losses of personnel, communications failures, or changes of circumstance.  It’s disappointing but understandable when service men and women return to Main Building that their decision making becomes more tentative and circumspect. 

Having under-secretaries making decisions about the details of IT systems has added perspective, but seldom improved the quality of the details of the solution.  Absolutely, those accountable must take their duties seriously:  they must properly delegate, not abdicate.  The troops, and civil servants, on the ground must be assured that they have the confidence of their masters when they take decisions, and know how to take the decisions that their masters would take. 
 
Al Goerner of ValTech concluded that "Agile will not long survive in a non-Agile environment".  Few would dispute that most of government is a "non-Agile environment", which explains many of the Agile failures. 

If Agile is to succeed, there must be an cultural change - government itself must become more Agile.  A wholesale cultural change programme across government would take many years.  But each Agile project can, as all good Agile projects do, engage with all stakeholders in a cultural change to enable proper governance, and decision making. By being successful, each Agile project will inch the monolith forward.  Agile isn't the whole anwer, but it can contribute to the change.

Introducing Agile may be the "nudge" to start the curltural change.  With the current government trying to devolve decisions to the appropriate level, Agile should be (part of) the right answer at the right time.

To summarise, adopting Agile in government has still many obstacles to overcome, but Agile offers a significant improvement on the current approach.  With all the constraints on government spending, the case for a flexible approach to delivering value early is surely enough to win the day.

System Error - Procuring Agile

There has been a fair of amount of comment on LinkedIn (although sadly not in the mainstream press) about the Institute of Government's report System Error.

In summary, the report recommends two big ideas, platform and agile.

The platform thread seems to be following the pervasive movement towards the cloud.  The report however substantially ignores security issues, which are often a major additional cost factor in government computing.  It's not good if a bank loses a laptop, or has it's firewall breached, or exposes customer data over the internet.  But when MOD loses a laptop, or a CD, or USB disk, the press have a field day.  While amateur hackers enjoy targetting Microsoft or Sony, state sponsored hackers focus on the big government departments, agencies, and government contractors, for whose security of course the taxpayer pays. Few then correlate the security risk with the cost of making Government devices and information secure.  Any government platform will have to be demonstrably secure, with robust (expensive) processes to maintain security.

Of more interest to the Agile community is the recommendation to adopt agile methods more widely.

Cornwell consultants have been trying to get various government departments to procure more projects using agile methods (usually DSDM, since it seemed most likely to be be approved by OGC) for at least 15 years, with some success.  In general, we were restricted to small-ish projects by the entrenched belief that agile doesn't scale.

We found that while the business sponsors got "agile", procurement often resisted because agile contradicted their standard risk management models.  Finding agile IT suppliers able to work in government was also problematic.

Contracting was always a problem. As soon as the project got near EU procurement, the need for a precise specification and a fixed price took over, and it became harder to get through the gates.  Moreover the time and effort required to bid often made it harder for smaller suppliers to stay in the race.  The usual suspects, with professional bids teams, could always write a compliant bid, and find some colateral to show how agile someone somewhere in the business had been.

What to do?  We've learned never to require "agile" - any salesman worth his commission knows to say "If you want agile, then we're agile".  But when the going gets tough, the tough revert to type, stop collaborating, and go waterfall.  The suppliers' strengths and weaknesses are only revealed late in the project, when stage payments may have already been paid, and it is too late to recover. 

Better by far, to frame the requirement such that an agile response will be a better solution, and assess who has the more agile approach.  Also take up references early, to assess whether the team proposed has indeed been agile.

The contract for agile may not be as difficult as it might first appear.  Agile aims to produce an outcome, not just software. The outcome will include business change, the establishment of new ways of working, not just delivery of software.  Can the contract not be for the desired outcome, stated simply in a few short paragraphs, and not in a requirements/features list?  There must be some criteria to inform acceptance, but again at a high level.  A key differentiator is whether/how the proposals address the need to resolve the detail, what kind of project process they propose.  The risk of delivery is placed, correctly, with the supplier, who works to his strengths, manages his weaknesses, and has to deliver the desired outcome.

Sounds easy, doesn't it?  It's just not often been done up till now.

Lost in translation?

When starting some major building work on our house, we found it a challenge to visualise the outcome of the project.  We got an architect to draw up our ideas, which he presented as a series of drawings, required for planning consent.  It wasn’t easy but we could work our way through the drawings, correct mistakes and omissions, so that before we appointed a builder, we had a high degree of confidence in what the builders would create. 

I recently worked with some people in the shipping industry who buy new ships.  They also use technical drawings, but often commission scale models of new vessels.  Sometimes computer animations are produced  from the CAD drawings to enable the customer to walk through and fly around the ship.  Similarly when businesses commission new commercial properties, or new products, or any designed object, there are drawings and models for documenting the object, which is reasonably comprehensible to the customer.

These drawings and models can be viewed as a language through which all the parties can communicate.

Critically, this common language gives two major benefits:  

• Everyone involved in the project shares a high degree of common understanding of the desired outcome

• at a relatively early stage in the process, and before too much money has been spent, the customer can confidently approve the major part of the expenditure

Key thought:  the key risks of variation of time, cost and quality are reduced to an acceptable level, very early in the process, at the expense of only a small fraction of the potential total cost, because there is a common language.  The customers may not be able to write in the language, e.g. create the drawings, but the customer can understand the language sufficiently well to make informed decisions.


There is a further advantage that the drawings provide a rigorous specification of the new product, that can be verified for integrity, whether by experts like structural engineers, or by computer software, further ensuring the quality of the eventual outcome.

In computing, and to some extent in business change, we have struggled to find a language to express the detail of business processes and computer programs, before we require the customer to sign off the bulk of the budget for the project.  How often have we found that none of the customers have fully understood the specification (in whatever form) and have proceeded on blind trust?  I use the term "quality of sign off" to discuss whether the customer really understands what they are signing up to.  The number of failed projects indicates that the IT industry has not been able reliably to reduce to an acceptable level the major risks of an IT project early enough in the process.

Nevertheless, we persist with a governance and contractual model from building and engineering:  don’t proceed until you build a model that the customer can sign off – based on the analysis that the cost of errors increases throughout the project.  Moreover, the common reaction to failure of one project is to screw down the contract, by putting more detail into the models and specifications, which is of course of unknown and unverified accuracy.

We must conclude that the IT industry, blessed with some of the most innovative and stunning intellects, has not yet found a language to model a system to customers.  After 50 years, should we now presume that the problem is in fact  fundamentally insoluble, that even if we had a language, the customer simply cannot envisage a complex program or process?  Is it time to ask "will we ever?"  Instead of trying to find a "good enough" language to communicate a system, should we not be looking at the problem more fundamentally, and find a completely different approach?

So, why do we need specifications?  To reduce, and ideally minimise, the risk of developing the wrong solution, customers need to know what they are going to get for their money.    (We also need the specification for the contract between customer and developer, but let's not get legal here.)   What is the earliest point at which the customer can understand the nature of the solution?  In many cases, customers only really understand software when they use it, and often the customer has to run a full critical cycle, like year end, stock take, or Xmas, to understand the capability, and the limitations.

We have to conclude therefore that the only effective language to communicate a software design is the software itself. But even testing the software is less than effective, with many functional issues only understood from live running.

But the working software represents the end of the build cycle, when all the money has been spent.  Any deficiency must therefore increase the cost, and delay the realisation of benefits.  If the test is using the software, how can we bring forward that experience without actually building the software? 

In Agile, we prefer “working software over comprehensive documentation” (Agile Manifesto).   The Agile approach is to build first the least necessary software, or minimal useable subset, in order to let customers test and use the software, as part of the joint customer/builder learning journey.  That software will need improvement and enhancement, but at least some of the software will be good enough to use from the first release.  The build team will understand more about the customers’ needs, and the complexity of the eventual solution, so they can better understand the emerging requirements.  Then they can build additional features, with greater confidence, of outcome, cost and time. 

In summary, when commissioning software, we cannot achieve our governance goal of minimising the whole project risk by agreeing on a specification, no matter how comprehensive, before we start to build.  The best we can achieve is to reduce the scale of commitment, and therefore risk, by using iterative development and incremental deployment, so that customers are able to understand parts of the solution and gradually envisage the whole.  


That still leaves the basic governance of most organisations faced with an unpalatable catch-22:  can't proceed until until the outcome is specified; can't specify the outcome until the project is started - a subject for much future discussion.