Our Dysfunctional Industry, Part 7

In the previous entry I introduced the first three Project Scope Axioms. They are as follows:

1. It is impossible to fully estimate feature complexity in objective terms.

2. Because it is impossible to objectively estimate complexity fully and thus anticipate all contingencies, an estimate of project scope will, in practice, always be an underestimate.

3. Project scope will expand to consume available resources, time and budget.

The practical implication of these axioms--that all projects will expand to fill an underestimated scope--is as follows:

Unanticipated contingencies cannot not be adequately addressed in the original time and budget estimated.

This is Project Scope Axiom 4, and simply tells us what we already now know: that the delivery of a feature-complete, high-quality project on time and on budget is not possible. But all is not lost! With a shift in expectations, we can still reach our win conditions and promote healthier relationships between clients and agencies. Once we recognize the tradeoffs involved in building any software system we can set our metrics for success accordingly. Owing to these tradeoffs, every project has three potential outcomes:

1. Feature complete; exceeds available resources

2. Feature incomplete; meets available resources

3. Feature incomplete; exceeds available resources

The unlisted fourth outcome, feature-complete and meets available resources, is the outcome I’ve just spent the last 5,000 words or so explaining isn’t possible. Of these outcomes, the first two can be considered successful, while the third is failure. As it stands, the vast majority of projects, at least in my experience, end up in the third, failure category. This is completely avoidable, but again requires a different and in some ways scarier approach.

Here’s another way of looking at it:

I believe we should focus our energies on achieving the second listed outcome: feature-incomplete; meets available resources. I want to focus on this outcome because “feature-completeness” is a nebulous concept, and extremely difficult to measure in practice, meanwhile we can readily measure when a project has met or exceeded available resources (usually time and/or money). This might sound crazy, since not being feature complete surely means the project has failed, right? Nope! What we’re trying to accomplish is not to reach feature-completeness, but rather to properly match project scope to available resources. This is in contrast to what we usually do, which is to attempt to cram a predefined scope into available resources. In order to properly match project scope to available resources, we must adequately plan for the appropriate feature set. To make this easier, at the outset of every project we should allocate extra time toward extensive planning and scoping exercises and cut as many features as possible right off the bat. What features should and shouldn’t be cut is a delicate conversation between the leadership teams on both sides, but the aim should be to execute at a higher level on the smallest number of features possible (the set of features that meet the minimum win conditions set forth by the stakeholders).

Now we’ve reached a critical point. When we talk about extensive scoping exercises, cutting features, etc, we’re making assumptions about the nature of the project, assumptions with may not necessarily hold. If you’ve involved yourself in software delivery for any appreciable length of time, you’re probably aware of the term “agile,” or more commonly, “Agile,” with a capital ‘A’. The Agile movement started when a group of pre-eminent figures in the software industry got together and wrote the Agile Manifesto in 2001. The Agile Manifesto is a set of principles by which software developers and product stakeholders might finally come together to develop software systems of the highest quality and sing Kumbaya. At this point I will not ascribe any further intentions to the creators of the Agile Manifesto and instead will focus on how agile project planning and development is practiced in the real world.

Agile is commonly seen as a response to earlier SDLC (software or systems development life cycle) models, in particular to what we now often-derisively call “waterfall,” an SDLC model by which a software project would be more extensively planned in advance with each discipline completing its work in distinct and separate phases (requirements gathering, design, development, testing, etc), delivering working software only toward the end, leaving little or no room for revision of features or collaboration either with other teams, disciplines, or with the client. With the advent of agile, waterfall (and, to a lesser extent, incremental and spiral) became a catch-all term for the reason why we were all so bad at developing quality software systems with limited resources. We used to do waterfall, and that’s why we never got it right. Now we’re agile, so that means we’ll get it right (except, of course, when we don’t).

Most, if not all, software development agencies call themselves “agile development shops.” The term is ubiquitous and as a result of its pervasiveness has become increasingly misused. When an agency brands itself “agile,” what they’re saying is not that they believe in and attempt to hold to a set of principles that will help them deliver better products, but rather that they fulfill whatever expectations clients have for what “agile” means to those clients. I know this is true because I could walk into any agency in the world right now and within a few minutes find a waterfall process model being used for an allegedly “agile” project on someone’s laptop or whiteboard.

When an agency says “we do agile development,” often what they’re saying is that they eschew extensive up-front implementation planning in favor of drafting a loose set of “requirements” against a set of designs or documentation, against which fluid phases of development will take place and the results continually offered up to the client for feedback and changes, and this process continues until enough features are developed or money runs out. Here’s the thing, though: this is not agile. The fact that you rushed headlong into developing software before understanding what you were aiming for and continually solicited and utilized client feedback throughout the design and development process does not by itself constitute agile development.

You might be thinking this is the part where I declare that everyone is bad at agile and my way is the right way to do agile, but it’s not. In fact, I’d prefer it if we stopped using the term altogether unless it’s in reference to specific principles that we’re following. The scenario described above is not agile. At best, it is iterative. We take a stab at a design, we get feedback, we revise, we get feedback, etc. We take a stab at requirements, we get feedback, we revise, we get feedback, etc. We take a stab at implementation, we get feedback, we revise, you see the point. 

Iterative development, in a strict sense, has nothing to do, necessarily, with an agile development process, and literally everyone does iterative development to some degree. A waterfall development process can accommodate iterative development just as can an agile one. The problem is that dysfunction sets in when we try to use the wrong tools to solve the wrong problems. Even so, a true agile process, one that closely adheres to the principles set forth in the Agile Manifesto, is simply not the right tool for every project. An agile process is more appropriate for projects with qualities such as the following:

• unknown, poorly-defined, flexible, or conflicting requirements

• dedicated, small, autonomous, co-located teams 

• close collaboration between cross-functional teams (e.g., design and development)

• close collaboration between client and agency

• flexible timeline

• flexible budget

If even one of these attributes does not apply to a given project, a true agile process may not be (and probably isn’t!) the right choice. Does that mean we must revert to a pure waterfall process? Certainly not. A colleague of mine once characterized the dichotomy between waterfall vs agile like this: With waterfall you are buying features; with agile you are buying resources. Additionally, there is an entire spectrum of options between the two often ignored, therefore we have a range of choices for how we conduct any given project, and we should be willing to adapt our process to the nature of the project. Is it appropriate to impose a strict set of rules and processes on a project? Sometimes, yes! The key is identifying what processes are appropriate to use in a given situation. There is no One True Process, and clients and agencies should work together to determine the best way to run a project based on the specific needs of the project and the tools and resources available. 

I’d like to conclude this series by reiterating the main points of what I’ve covered. We established that the software industry as a whole (and specifically web software in the client-agency sphere) is highly dysfunctional as a result of distorted expectations and a fundamental misunderstanding of software development as a mechanical and predictable endeavor and not the creative and chaotic endeavor it actually is. 

As a result of flawed client expectations with regards to how we can (or rather, can’t) adequately assess ahead of time the effort and cost involved in any software initiative, and agency complicity in fulfilling those flawed expectations, incentives on both sides become twisted around: clients want to know how much time and money will be required to implement some system without paying the necessary costs to acquire this knowledge, and agencies give out estimates on building that system which have no bearing on reality due to a consequent lack of adequate research and discovery and/or the presence of a budget or time constraint. Clients unsurprisingly then expect the perfunctory estimate received from the agency to be adhered to despite the utter impossibility of anticipating all contingencies, and agencies are forced to either lose money, cut vital features, or push the client for money and time they may not have, both of which result in unsatisfactory results and, worse, a loss of trust between both parties.

To address these issues, I have proposed the following: an extensive requirements-gathering effort should be undertaken before any estimates are given out regarding time or money, even in the cases of ostensibly “agile” projects. If the project has unknown or a poorly-defined end state, we must insist on a flexible timeline or budget, ideally both. If we are working under a fixed budget or timeline, then we must recognize that feature-completeness is not possible, and the goal of the project is instead to do as much work as possible until the budget is exhausted or the deadline reached, and it is necessary to prioritize and cut features. It is possible for this to result in a product that is incomplete and unusable when resources have run out (one of the things “Agile” was meant to address but frequently in practice does not), which means a certain amount of trust is required on the part of the client toward the agency, and the agency must strive to earn and keep that trust by being honest and forthright in all their efforts. 

Our goal should be transparency and trust between agency and client, and the only way to reach this goal is to be realistic and well-informed about the facts of software development, and to be willing to pay the price for that knowledge.

Additionally, we must be willing and able to recognize the characteristics of each project as a unique set of needs for which a unique set of processes will be required. This is the topic that will be the focus of a future series. Thanks for reading!

Our Dysfunctional Industry, Part 6

In the previous entry, we explored the inescapable tradeoffs we face in building any software system. For convenience, I’m going to re-insert the diagram that summarizes some of the major tradeoffs:

We see in this picture the unfortunate reality that we can’t have everything we want in a finite world. Never will there be a project that is delivered on time, on budget, feature complete and of high quality. It is, simply, not possible. Why is it not possible? Here’s why:

It is impossible to fully estimate feature complexity in objective terms.

I call this “Project Scope Axiom 1.” When considering the complexity of a given feature, the tools we have for estimating that complexity are very limited. We typically use estimates of hours or relative complexity, but producing those numbers is a haphazard process at best. Estimates of hours are dependent not only on the complexity of the task itself, but also on the skill of the team or individual implementing it. Relative complexity scores attempt to take this into account by measuring the complexity of a particular task vs other tasks, but this leaves us with determining what these numbers actually mean, and leave us with accounting for things we do not yet know. What actually makes a particular task more complex than another? We could say the time it takes, but that forces us back into judging developer skill, which is a whole other mess of uncertainty. I intend to go into this in more depth in a later series, but for now let’s concede that we don’t have a reliable, objective measure of feature (and therefore project) complexity, which leads us directly to a corollary of Axiom 1:

Because it is impossible to objectively estimate complexity fully and thus anticipate all contingencies, an estimate of project scope will, in practice, always be an underestimate.

This is Project Scope Axiom 2, which is another way of saying that we cannot account for what we don’t yet know. The effects of Axiom 2 can be limited with more knowledge as the better we understand a feature or a project’s goals, the less of an impact unknowns can have. Nevertheless, because knowledge is always limited we cannot avoid underestimating. What if we resort to artificially padding our estimates to account for these unknowns, as we usually do? Turns out that doesn’t really help either, for two reasons: said padding is almost always done with an eye toward the available budget, which means it still has nothing to do with the actual requirements of the project, and Project Scope Axiom 3:

Project scope will expand to consume available resources, time and budget.

I think we probably all recognize this as intuitively true. Another way of stating this would be: the more time we have to do something the more time it will take to do it. In practice, artificially padding an underestimate simply results in a larger-but-still-inadequate estimate. What happens is that by one method or another, the scope of the project will expand to meet the padding. People will take longer to make decisions, developers will take longer to get something done, more meetings will be scheduled and, most dangerous of all, features will be added.

(Before we go on, let’s quickly define what we mean by “project scope.” Project scope is used here both descriptively and prescriptively. That is, scope tells us both what the project requires in terms of time, budget, and features, and we can actively change the scope to better fit the available resources.)

The first three Project Scope Axioms seem to have put us in an odd position. I’ve just said that it’s impossible to correctly estimate project scope, but recommended we do it anyway. Then I said project scope will change to consume available resources, which would seem to render an estimate, no matter how accurate, potentially useless.

Yes, it’s true, we cannot perfectly estimate project scope, but we do it anyway, not just to give ourselves a reference point and an idea of cost and timeline, but also to uncover as much as possible about the project so that we can make more informed decisions and reduce the deleterious effect of unanticipated complications on timeline, budget and quality. And yes, it’s true, project scope has an insidious tendency to expand to consume available resources, but this is precisely why it’s so important to take as much time and expend as much effort as we can to accurately estimate the scope. The closer project scope matches available resources, the more likely it is that we will get what we want done with the resources we have available, and the more likely everyone involved is to consider the project a success in the end. That is therefore our goal.

So how do we accurately estimate project scope? The precise, technical methodology is a story for another day, but there are a number of strategic decisions we can make that will help us. Here are two straightforward ways we can help ourselves reach more accurate estimates:

1. reduce features

2. do more planning

These are deceptively simple ideas, but they are vastly underutilized. Clients rarely want to reduce features unless absolutely necessary, and agencies are eager to please. Clients are usually eager to get something in hand as quickly as possible, and agencies are champing at the bit to get usable software out the door. Reducing features means objectively less project complexity and a smaller number of things that we can’t anticipate, and more planning means we can better anticipate potential complications and lessen the impact of contingencies. Ideally, we would do both. We would more extensively plan fewer features, as our objective is to narrow what can go wrong as much as possible. These activities help us to limit scope and estimate scope, respectively.

Neither of these things will come naturally. The kind of planning I am talking about is far more extensive and involved than the typical scoping exercises agencies engage in, both in the bidding process and as part of the initial project planning stages. Clients and agencies would need to be willing to spend more time and money on understanding the true requirements and needs of a project before committing to anything, and this requires a fundamental change in expectations about how projects are proposed, bid, and planned.

Along with this comes a recommendation that we reduce features. This isn’t the same as realizing, halfway into the project, that we’re not going to make the deadline so we start desperately pushing features into a post-launch phase. I’m suggesting instead that one of the very first explicit exercises we undertake in any new project is to cut as many features as possible. This is rarely done because the implicit, default choice is to cut quality rather than features, which is exactly what happens at the outset of every project--but no one ever likes to actually bring it up! I’m suggesting, and firmly believe, that we will be better served overall with fewer features of higher quality.

I understand that it might sound a little crazy for anyone to engage with a new client or partner for a new initiative and immediately ask for money to put together an estimate and then suggest cutting things out, but here’s the thing: we’re going to end up doing it anyway! The advantage to this strange approach is that by doing it at the beginning we aren’t assuming the costs of research, scoping, and discovery of features that may not (need to) be implemented, nor the frustration of discovering months later that we’re going to lose something we wanted to have, and what we do have will be of higher quality, which will greatly reduce costs for adding features later on as we seek to reach feature-completeness.

This requires a major shift in thinking. We’ll talk more about that in the next entry. If you were wondering when I would finally break out the A-word (*gasp* Agile), wonder no more.

Our Dysfunctional Industry, Part 5

In the previous entry of this series I made the case that agencies routinely underestimate project costs, commit to more than is possible within those constraints, and that clients expect them to do so. Wew, lad! That’s a pretty bold assertion. Why would clients expect this? Surely clients aren’t actively hoping for or expecting the project to go poorly? Surely not all clients are trying to pull one over on agencies?

Well, no. They aren’t. But what they are doing leads directly to fractured relationships and poor results. As discussed in previous entries, the very act of soliciting an estimate for a project before enough information is gathered for that estimate to be usefully accurate has two major consequences: it materially changes the end result into something different from the expected result while setting the expectation that the expected result can be achieved cheaper than it actually can. We can sum up this idea with the following principle:

You will always get less than you want for your budget.

This may seem obvious, but let me state it another way:

You can get what you want but not for the budget you want, or you can stick to your budget but not get everything you want.

This principle helps us to better understand why clients expect agencies to underestimate project costs and commit to more than they can possibly do for a given amount of time and money: because clients do not like to be told that they can’t get everything they want for their available time and budget. Meanwhile agencies, for fear of missing out on a lucrative contract, are willing to take the risk of telling clients what they want to hear in direct violation of the principle stated above. That’s not to say agencies never tell clients they can’t get everything they want. They very frequently do, but usually not until after they’ve already committed to doing more than they can with the resources on hand, either by producing an inaccurate estimate or by under-planning, or both.

So, if we accept this as true (which I think I’ve made a solid case for), then what does it mean for us in practice? It means that client-agency relationships based on these dysfunctional behaviors inevitably decay as trust is lost. Trust is lost because along each commonly measured axis of project success (timeline, budget, and feature-completeness), something unavoidably and unexpectedly falls severely short. Of course, something will always fall short in any project because nothing is ever perfect, but falling short becomes a real problem when it’s unexpected and when the severity is great. Misjudging the true costs of a project by overcommitting and overpromising results in both unexpected and severe shortfalls.

How do we stop overcommitting and overpromising? We start by paying the price to learn as much as possible about the project before we commit and before we discuss any numbers. That means we approach each new project from two angles: figure out how much we think it will cost to deliver the project feature-complete, and figure out how many features we think we can fit into the available budget and timeline. This process is much more involved than the superficial scoping exercises most agencies undertake as part of the bidding and sales process. It will take significant time and money, but avoiding it is burying our heads in the sand. Refusing to arm yourself with this knowledge is like taking $100 into the grocery store and then filling your cart with your eyes closed. We can do better. We must do better.

Am I saying that if we undertake an extensive, expensive scoping exercise before we commit to a project we’ll be able to achieve our desired results on time and on budget? Well, that’s a more complicated question, and one we’ll be discussing as we go. Right now, it suffices to say we will get much closer to doing so, and we’ll maintain much healthier relationships between clients and agencies in the process.

Now that we’ve set all that out, let’s discuss another principle that follows directly from the above:

On time, on budget, and feature-complete is impossible.

This might seem like a bit of an overstatement, but remember: what you can afford and what you need aren’t necessarily related (and in fact very probably aren’t). A simple diagram (you’ve probably seen this before, or a variation on the theme) illustrates this: 

Your project is represented by a dot placed somewhere in the triangle. I don’t want to be too reductive as this hardly captures nuance, but it does show the essence of the idea, which is that moving toward one corner necessarily takes us further away from one or both of the others. For example, the closer we want to be to an on-time delivery, the further away we will be to coming in on budget and/or being feature-complete. Why is this? The lines of the triangle represent all available resources, and these resources are finite--that is, moving toward one corner necessarily takes us further away from the others.

This picture, however, is incomplete. There are more dimensions to project success--or what’s commonly referred to as the “win conditions”--than time, money, and features. This diagram revises the previous diagram and brings in one such added dimension I consider to be crucial to project success:

This diagram pairs up aspects of project success along the horizontal or vertical axes to depict the forces that work directly against each other. We see here again that being on time means we’re further away from being on budget, but now we’ve added the notion that prioritizing features directly takes us further away from “higher quality,” the new dimension. Of course, as before, adding features also takes us further away from being on time and on budget, but more obliquely--that is, we can’t readily quantify how much from each in comparable terms due to other factors (team size, project complexity, etc), so to simplify things we say adding features takes away from on time and on budget equally. On the other hand, we can more readily quantify the relationship between features and quality--we can say that adding more features has a direct adverse effect on quality, even when accounting for team size and project complexity. For completeness, here’s a diagram that tries to incorporate all those aspects of a project, which I proudly call “The Dismal Diamond of Project Perfection.”

Here we see that complexity increases as we add features, which reduces quality, and adding to team size gets us closer to being on time and/or allows us to add features but gets us further from being on budget and can affect quality by adding complexity. In the real world things are obviously more complex, but it gets closer to capturing reality, most importantly the reality that there are difficult tradeoffs in every project that we cannot avoid.

Most of these aspects are self-explanatory, but what about “quality”? Quality represents a software system that costs more or less money and time to maintain. Low quality code has the knock-on effect of consuming an outsized share of available resources the longer the project exists. It is often said that only 20% of the final cost of a software system is in implementation, with the remaining 80% in maintenance. Think of low quality code like rust on a car. Once it takes hold, it can spread and corrode everything around it, and the longer it’s left alone the more damage it does and the costlier it will be to fix, to the point where it can destroy the system (the point at which it becomes costlier to maintain or fix than the value it offers). Low quality code is unavoidable under any constraints of time, budget, and feature-completeness. It will happen, and we want to minimize it.

At this point you might be asking “Why can’t we just implement features with high quality from the start?” Unfortunately, software development simply doesn’t work that way. There is a concept in software development called “technical debt.” Technical debt is a cost incurred by the very act of creating software. It is, in essence, the sum total of deficiencies (bugs, poor optimization, poorly-documented code, poorly-written code, code resulting from inadequate understanding or planning, etc) in a software system that can only be ameliorated by extra effort above and beyond what it took to simply “get it to work.” Even the most talented software developers produce technical debt. Just like financial debt, we pay a steep price by imposing inadequate time and budgetary constraints on software development, a price that must eventually be paid back if the system is to work as intended indefinitely and without spiraling upkeep costs. You can think of technical debt as a function of the ratio of project complexity versus timeline: the more complex a piece of functionality is implemented in a relatively short time, the more technical debt there will be. Here’s an example of what happens when we neglect quality:

It was once reported on Hacker News that the Oracle Database source code consists of 25 million lines of code. Changing even one line of code breaks thousands of tests. It takes days to understand what a single line of code does. It takes weeks to implement the fix before submitting it for compilation. It takes, in many cases, months to see the fix through to production. It can take up to two years to add small new features. This code is the result of decades of work under difficult deadlines without, obviously, paying back nearly enough technical debt.

Imagine the explosive costs of such a system in terms of dollars and opportunity. Imagine how much time is wasted over the course of a year. This might be an extreme example, but quality is not optional if you want a functional, affordable software system. Striving for feature-completeness at the expense of quality (again, an unavoidable tradeoff) is an extremely dangerous game to play.

Well, this is a dire view of things, isn’t it? How do we fix it all? Can we fix it? I’ll discuss that in the next entry.

Our Dysfunctional Industry, Part 4

As discussed earlier, we’ve established that the real-world costs of a project and the project’s complexity are directly linked and don’t necessarily change on the basis of any given estimate. That is to say, the presence or lack of an estimate doesn’t materially change the intrinsic complexity of a project as originally conceived. Instead, an estimate tends to have a causal effect on the expectations of what it will take to accomplish a project, which means that the quality and nature of the end result can be affected by an estimate. In other words, the thing that ends up in the client’s hand at the end of the project can (and usually is) materially different from the original expectation. Therefore, a bad estimate can directly cause a bad product. Let’s examine this a bit more closely.

The mere presence of budgets and estimates in relation to an unknown and fundamentally unknowable thing like how long it will take to produce a software system causes all subsequent conversations and decisions to be made around those initial pieces of information even if they have no relevant relationship to the system we are intending to build. That is to say, if I were to propose to build a software system and gave an initial estimate of $500,000, we would wind up with a different end result than if I had given an initial estimate of $1,000,000, not because the software system proposed has somehow become more or less complex, but because we have set expectations about it before we know enough about what it will actually take to build. The project itself, both the way it is run and thus the resulting software system, will be different simply because I threw out different numbers at the start. Don’t believe me? Well, you should. This is an established psychological phenomenon known as “anchoring,” and has applications in everything from salary negotiations to buying real estate. These numbers matter, and getting them wrong has a real cost.

Let’s look at it from a different angle. When a project is initially sketched out and proposed, it doesn’t actually exist. Let’s revisit the “Dream Home Architect” scenario for a moment. In this scenario, the dream home in your head is just that: a dream.

In one hand, you have a budget, an amount of money you can spend. In the other, you have the idea, your ideal dream home. It is virtually impossible for you to correlate the two without outside help. There are aspects to your dream home that you haven’t considered. There are things you are hoping to get that simply may not be possible. There are features you think you want that you may not actually need, or that may cost far more than you realize. You, the client, rely on the expertise of the architect and builder to tell you what’s possible and what’s not, what they recommend and what they don’t, and, finally, how much everything will cost.

Your plan, of course, is to get everything you can for the money you have to spend. So with that in mind, let’s see what happens in two different hypothetical situations:

• Situation 1: You begin the conversation with the architect by describing your dream home. You do not discuss a budget.

• Situation 2: You begin the conversation with the architect by stating exactly how much you can spend, and then you begin describing your dream home. 

Now, consider two questions:

1. Which situation will result in an estimate that more closely reflects the actual costs to build the home as you have described it?

2. Which situation will result in a house that more closely reflects the dream home as you have described it?

You can see clearly here that we will get two completely different answers to these questions based on only one difference: the introduction of a budget. 

In the first situation we should expect to receive a cost estimate that is likely to more closely reflect the actual costs of building your dream home as described. This is because the architect will be focused on your description without reference to an overall budget. This means that the estimate, while reflecting a reality more accurate and faithful to your ideal dream home, may be wildly out of proportion to your budget.

In the second situation we will very likely receive an estimate that hews much more closely to your budget, because the architect will be sure to keep your budget in mind. But what of the house? A conscientious architect will show you the compromises that had to be made to fit your dream home into the available resources, which means the house, while accurate and faithful to the budget, will likely be significantly out of sync with your original dream home vision.

In an earlier entry of this series, I mentioned that, as a client, receiving an estimate for a project that lines up well with your stated budget should immediately raise a flag. Your first question upon receiving such an estimate should be “What am I not getting?” Why would you assume you’re not getting everything you want? Two reasons: first, if your ideal result could be accomplished for well under your proposed budget then the agency (an honest one, at least) would have given you an estimate that came in well under your proposed budget. Second, it is extraordinarily unlikely that your ideal result and your budget are so well in line with each other by chance alone, or by any napkin math done by your own internal teams.

Furthermore, it is quite rare for a client to propose a project with a budget much larger than what’s necessary. There are two reasons for this as well: clients will naturally seek to maximize the number of features in any system, and clients greatly underestimate the effort involved in implementing any given feature.

What does all of this mean? It means that the vast majority of clients will not get everything they want for the resources they have. This wouldn’t be such a problem, except that as we’ve discussed the standard practice in the bidding and agency selection process is for clients to select an agency that not only routinely underestimates what’s required to do the job correctly, but the agency also sets an expectation that the client will get everything they want for that underestimated budget.

And why does the agency do such a silly thing? Because, believe it or not, the client expects them to. This is dysfunction.

Our Dysfunctional Industry, Part 3

In Part 2 of this series we discuss what it means to choose a partner for a creative endeavor, their incentives, and how likely they are to do the job right.

If you’ve read the two previous entries in this series, you’ve surely detected the not-so-subtle parallels between the situations I’ve described concerning someone attempting to choose the right architect and builder to build their dream home and the client-agency relationship in the software industry. We’ve established that the first firm, the firm that endeavors to take the time required to learn as much as reasonably possible prior to beginning the actual building process is more likely to do a good job.

And yet, despite all this, standard operating procedure within the software industry, our industry, is for a client to choose the second architectural firm. That is, it is standard for a client to choose the agency less likely to do a good job. Why?

There are several reasons why clients choose the agency less likely to do the better job. Here are the most common:

1. Clients are unwilling to commit to an agency before they learn what that agency plans to charge them for the entire project.

2. Clients see the work of planning and design as part of the core project itself and don’t want to spend money on that aspect before they know more about what those costs are going to be.

3. Clients want a number at the outset because a number is something they can point to later on to pressure the agency into covering contingency costs.

We will discuss each of these individually.

Clients are unwilling to commit to an agency before they learn what that agency plans to charge them for the entire project.

This is a natural instinct. When you’re talking about hundreds of thousands to millions of dollars, you’re talking about a lot of trust. You’re talking about people’s jobs. You’re talking about millions in future business. The client in this situation is dealing with two opposing forces: the force that impels them to make the right choice in a partner, and the force that impels them to keep costs to a minimum. 

The problem here is that there’s rarely, if ever, enough information to determine which partner will most efficiently maximize available resources. Clients are left to make a decision based on a haphazard bidding process in which a superficial examination of the project’s requirements--typically as written by the client--is performed by agencies who, typically working unbillable hours, are incentivized to gloss over the unknowns and difficult questions and instead focus on setting their estimate not in direct reference to the actual complexity of the project but rather on what estimate will most likely win them the contract. The weaknesses in this approach are obvious.

So what’s the alternative? The alternative is knowledge. Instead of asking for a number of hours or dollars, clients would better serve their own interests by focusing entirely on the solutions proposed by the various agencies. They should ask themselves questions like:

• Does this agency demonstrate clear expertise on the subject matter?

• Does this agency appear to understand the overall vision of the project?

• Does this agency demonstrate an understanding of our challenges?

• Is this agency up front and honest about the unknowns and the gaps in their knowledge regarding the scope and requirements of the project?

• Is the team the agency proposes a good fit for this project and our internal teams they will be working with?

By asking questions like these, the client can begin attempting to establish trust with each agency. Once trust has been established, the client can feel confident that whichever agency they choose will do right by them. In this way, the client is more focused on the how over the how much.

Clients see the work of planning and design as part of the core project itself and don’t want to spend money on that aspect before they know more about what those costs are going to be.

This is purely an expectation that’s been set by years of doing it this way, and quite simply it’s wrong--backwards, in fact. The solution to this problem flows directly from the previous problem. Once a client has established trust with an agency, they should feel more confident in allowing the agency the time and effort necessary to properly scope the project before giving back any estimates on cost or timeline. This exercise may be lengthy and costly, but it is far preferable to finding out months later that the project will actually cost double what was expected. It’s also worth remembering that this is not lost time. Every minute of research and discovery can uncover roadblocks and pitfalls that, if found later, may cost orders of magnitude more time and money to fix as work is unwound and redone.

Clients want a number at the outset because a number is something they can point to later on to pressure the agency into covering contingency costs.

Obviously, clients are self-interested. This isn’t a bad thing. If they weren’t, then agencies would have no accountability and would have no incentive to get anything done in a reasonable timeframe. Thus, it makes sense that clients would want to extract a number (price, timeline, etc) from the agencies they are considering, even if they recognize the number is a fantasy. And make no mistake, many clients do recognize that the numbers they receive are not accurate--in fact, many count on it. This isn’t evil. The fact that agencies are so willing to furnish wildly inaccurate estimates in order to win a bid makes them responsible for it, not the client. And yet, in appealing to their own self-interest in obtaining what they may know or suspect is a nonsense estimate, clients are doing themselves a major disservice. I’ll explain. 

Because the client knows the agency wants to keep them happy (and paying), the client wants to be able to point to the number as a way of incentivizing the agency to do more work for less money as the project progresses. The agency, on the other hand, typically has no choice but to go along with this because they know if they refuse to go along the client can easily go find another agency who will, or even sue them for breach of contract. The problem is that holding agencies to an inaccurate, premature estimate incentivizes them into a race to the bottom where they produce software of the lowest possible quality that will still get them paid. Little or no care or thought is given to long term maintenance, code quality, etc. It also results in a continuing lack of foresight and planning, which results in more crunch, which results in still lower quality results. This is an extraordinarily dangerous feedback loop and can easily sink a project.

All of this results in hugely inflated maintenance costs and an unstable, bug-ridden code base, all of which over time will easily outweigh the benefits of a project ostensibly done “on time and on budget.”

As a client you must ask yourself this question: Is it more important to get it done, or get it done right?

Our Dysfunctional Industry, Part 2

The previous post asks you to imagine a scenario in which you describe your dream home to two different architects in order to solicit an estimate of the total cost to build. The first architect replies that the work required to develop an accurate estimate will cost a significant amount of time and money, leaving you in a situation where you may not want to or even be able to afford the total cost. The second architect tells you he can develop an estimate of how much your house will cost in just a few days. Then we ask the question: at this point, which of the two firms do you think is more likely to do a better job working on your dream home?

(A couple things to note here: We did not ask which firm will be cheaper. This is not because cost is not important. It clearly is very important. We also don’t wish to give the impression that a firm that costs less will do an objectively bad or even worse job, and just because the second firm is willing to produce a quick cost estimate doesn’t mean they won’t be the more expensive of the two.)

What we’re really asking here is this: if the complexity of the dream home you have in mind is the same regardless of which firm you choose, which of the two firms is more likely to do the better job: the one willing to produce an estimate with a minimum of research or the one who will take whatever time they, as experts in the field, deem necessary to develop such an estimate? 

It seems obvious that the firm that takes more time and expends more effort to develop an estimate will be armed with more information from the outset. It’s also likely that this firm, by virtue of their unwillingness to render what they deem a premature assessment, even if it means they lose the business, is more willing to be diligent and conscientious in all their efforts. Therefore, it does indeed seem likely that this firm will do a better job.

If you’re not yet convinced, I would invite you to closely examine the consequences of choosing the second firm. We’ve established that the objective complexity of the house you want to build is the same in either case, so by selecting the second firm on the basis of their estimate you are taking a very real risk that the firm in question is performing only a cursory examination of the actual costs of building your house and instead is more concerned with winning your business and getting you to commit to them before the true costs of building your dream home become clearer.

In other words, the house will cost what it will cost based on its intrinsic complexity and requirements. It won’t magically become cheaper as a result of someone giving you an estimate that you want to hear, and in fact receiving such an estimate should immediately raise a flag. (We’ll discuss that in more detail in Part 4 of this series.)

So if we recognize that the second firm’s incentives in providing you a free, low-effort estimate probably aren’t entirely in line with your own incentives of building your ideal dream home, we’re still left with a dilemma: what happens if I spend a bunch of money to get an estimate to build a house I can’t afford? Two things can happen in this situation:

1. You walk away, a bit poorer but wiser.

2. You work with a now-trusted partner to establish the path forward.

Which would you choose?

Our Dysfunctional Industry, Part 1

In the following series of posts, I attempt to explain the dysfunction at the heart of the client-agency dynamic in the software industry as I see it. My own experience is primarily in web software, but in research and talking to friends and acquaintances I know this dynamic frequently plays out in other software industries as well. My intention is to both illuminate as well as to discuss how best to forge a path toward healthier relationships between clients and agencies and a healthier software industry in general.

To start, I want you to imagine the following scenario, which I call “The Dream Home Architect.”

Imagine that you’ve saved up a bunch of money over many years and you’re ready to build your dream home. You haven’t got every detail worked out, but in your mind you know what you want. Confident and excited, you make an appointment to talk to a special architectural-construction firm about planning and building your home. The day of the appointment arrives and you walk into the architect’s office beaming in anticipation. You sit down and begin to verbally describe your dream home. You describe the location, paint colors, the exterior and interior materials, the size, the kind of decorating you plan to do, the overall style, the various features you want, and so on. All the while the architect dutifully takes notes and asks for occasional clarification.

After maybe 30 minutes of this, you find you’ve exhausted your mental notes. You look expectantly at the architect. She says “All right, I believe I have everything I need to get started here. I will do an initial draft and you’ll be able to review it within a week. It will cost about $100 an hour for us to do necessary research and draw up the initial plans and given what you’ve said I think it’ll take at least 40 hours for the first draft to be finished.”

Puzzled, you ask, “Wait, I thought you were supposed to estimate how much the house will cost to build.” The architect replies, “I have no idea yet how much it will cost to build.”

Now you’re really confused. “But I just told you everything! How can you not know how much it will cost?”

The architect shakes her head. “I simply don’t know enough about the house to give you an estimate. We need to design the house, draw up blueprints, consult with the builders and suppliers, review permit requirements and about a hundred other things before we’ll have any realistic idea of how much the project will cost. This can take weeks, even months.”

“But what if after all that your estimate is more than I can pay?” you ask.

“Unfortunately, that’s the nature of our business. After we’ve drawn up the plans, reviewed them together, done some more research and consulted with some more people, we can give you a rough estimate, but we can’t do better than that right now.”

“So you want me to pay you at least $4000 before I’ll have any idea what you will charge me to actually build the house?” you ask, incredulous.

“Asking me to do otherwise is asking me to work for free. Is it fair for me to work for free?” the architect replies.

You don’t have an answer to that question, but you decide to get a second opinion. You do some digging and find another architect-construction firm that seems to be more willing to do things your way. You repeat the process and after describing your dream home to the architect, he thinks for a moment and then says “Give me three days and I will tell you how much your house will cost to build.”

****

Now, having envisioned this scenario in your mind, ask yourself: which of the two firms is more likely to do a better job working on your dream home?

An entry in this series will be posted each week for the next several weeks. Check back each week!