Product Engineering Delivery Guidance

Process

Focus and drive is the foundation. Consider that the US Army doesn’t teach a single specific martial art (e.g. Krav Maga, Muay Thai, etc) for hand-to-hand combat, because they have found routine practice, cultivating the ability to learn quickly, and instilling a fast-reacting, determined spirit which thrives in uncertainty drives superior outcomes. Training still incorporates aspects of specific martial arts, but with an emphasis on the larger patterns and tactics they embody. These notes are built around the same concept.

Process is necessary, but not sufficient. Scrum Agile, SAFE, kanban, even waterfall – performing all of the steps of a process in the right order on a schedule will not deliver the work in and of itself. Necessary preconditions include:

• A definition of done: you should know what you’re going to have at the end, why you’re doing it, and what success looks like. The precise definition might change along the way, but it should be clear, short, well-understood, and easily discoverable. If you do not know what “done”¹ looks like, you will never get there.
• A shared commitment to delivery as a team: if you and your team are not committed to getting unblocked, helping each other, being flexible, and making progress every day, no amount of process will help you.
• A delivery date: as with the definition of done, it can change along the way, and doesn’t have to be set in stone, but the date will shape your definition of done and sharpen your team’s focus.

Process either helps people get things done or prevents bad, unrecoverable errors.

• Waterfall attempts to do this by requiring exhaustive specification and review steps, but it has the highest failure rate of all processes with regard to on-time delivery of quality software. There is a lesson here: the exhaustive specification, large batch transfer, and gating review steps associated with waterfall are time-consuming and do not deliver on their promises and actively harm delivery even though these are intuitive solutions to preventing re-work and mistakes.
• Scrum Agile processes help people get things done in two ways:
  • Time boxing limits batch transfer sizes, reducing cycle times and
    tightening feedback loops.
  • The regular meeting cadence reduces the cost of schedule coordination on a team.
• Scrum Agile should be lightweight and flexible, so Product Engineering teams generally use Scrum Agile by default.

Remember, Scrum Agile is built to work with these principles:

• Individuals and interactions over processes and tools.
• Working software over comprehensive documentation.
• Customer collaboration over contract negotiation.
• Responding to change over following a plan.

It follows that if your team is spending a significant amount of time on process overhead, making no progress from iteration to iteration, or waiting on “finalized” specifications before engineering starts, even if you have daily standups, retrospectives, and bi-weekly sprint planning meetings, your process is not going to save your project. Declare process bankruptcy and start over from the principles, with time-boxing and a regular meeting cadence. Adding more process overhead with gating steps to speed up delivery of a late project is like pouring water on a grease fire.

The keystone is iterative development. Scrum Agile is iterative and incremental.

• The ideal outcome of an iteration is something a stakeholder can personally test, in order to help refine their own thinking and provide valuable feedback.
• Your cycles should be:
  • Build.
  • Gather feedback on what was built.
  • Adapt your plan to the feedback for the next cycle.
• If you find yourself on an “iterative” project where most of the requirements are written before development begins, or there is an attempt to create thorough and detailed specifications before code is written, it’s a waterfall project.
  • A typical software project experiences a 25% change in requirements, scaling up to 50% for large projects². If your team and process do not plan for some requirements to change, you are wasting up-front planning time and undermining your project.
• Planning and requirements are valuable, but they should start as rough sketches and be refined during the project, rather than finalized and approved before any code is written. The team should be producing code and designs at roughly the same time planning and refinement starts.

The goal is predictable delivery of system updates that help stakeholders. Embracing change, centering people over process, person-to-person stakeholder collaboration, and sharing working code on a daily basis supports that goal.

How to Reduce Unknowns: An Example of Iterative and Evolutionary Analysis and Design

1. At project kickoff, hold a time-boxed requirements workshop which includes the team and outside stakeholders.
   1. Begin with high-level requirements analysis, identifying use cases, features and key non-functional requirements (usability, reliability, performance, supportability).
   2. The technical lead and stakeholder lead select 10% of this list which have the greatest blend of these qualities: architecturally significant, high business value, and high risk.
   3. Do a deeply detailed analysis of the functional and non-functional requirements selected such that 10% of the project now has substantive design artifacts.
2. For your first iteration, create a set of iteration goals based on those top 10% of use cases to design, build, and test.
3. During each iteration:
   1. Engineers start modeling and design work in pairs for the first day or two in collaboration.
   2. Engineers produce code, writing tests, validating assumptions for 1 week, ideally in collaborative pairs.
   3. During a regularly scheduled meeting, such as a standup, about 75% of the way through the iteration, the team takes a realistic look at whether or not they can deliver on the goals, de-scoping where necessary.
   4. Demo the work for stakeholders to get feedback.
4. Prior to the end of the iteration, repeat steps 1 and 2 until about 80-90% of the requirements have been detailed through collaboration, experimentation, and stakeholder review, even though very little (perhaps 10%) of the final code is written. Barring changes to the external environment, the design and analysis elaboration phase should be complete at this point from the perspectives of both the stakeholders and delivery team.
5. At this point, you likely have a well-adapted set of requirements, useful design artifacts, a team with an excellent working model of the problem to be solved, an idea of your final deliverable, and a rough delivery date with a high degree of confidence.

Questions for prompting good requirements

A common problem for software engineering teams is focusing on functional features (i.e. the user story: “As an X, when I Y, I expect Z”) at the exclusion of other requirements.

Good requirements include concern for usability, reliability, performance, supportability, and operations:

• Consider the human factors: what mistakes could someone make, and what kind of feedback can the system provide? When can it fix bad input itself, and when does it have to ask the user? What are the boundary conditions for input?
• How reliable does the system have to be? What should downstream systems experience when it fails? What should it do if upstream systems fail?
• What kind of monitoring do you need to ensure the system is behaving correctly?
• What can happen that should drive an alert which needs a timely (or emergency) response?
• What steps will have to be taken for rollout? Are any of the steps difficult to undo? If the system has to be rolled out in phases, how will that be done? Who
  will do it, and how can the system improve the usability of that rollout?
• What response times, throughput, availability, and resource usage constraints
  are critical?
• Which aspects of the system should be instrumented to facilitate monitoring health and throughput?
• What should be easily configurable by end users, or engineers on the team, and how should it be configured? Are there defaults?
• Who should have access to this? Who shouldn’t? What happens if someone who shouldn’t have access gets it, and vice versa?
• Is the security model of the system changing?
• How could a user abuse this feature? How will we deal with abuse?
• How will this be maintained on a day-to-day basis?
• Consider aspects of the maturity rubric – is this alpha level? Beta? Generally available?

Project Management Tips

Remember to build in room for the unexpected: a post-release support/bug sprint (or 3), work emergencies, team member emergencies, and dealing with technical debt. You and your team are not a machine in a factory doing a set of repetitive tasks to the same inputs over and over, even though processes which help us deliver on a regular cadence may sometimes foster this illusion.

Weeks of writing code can save you hours of planning. Time spent planning your sprint may feel like it takes away from time spent writing code, but making this investment immediately before the team begins to write code for delivery during the sprint will save the team time.

As the team is refining the project, the team lead should be refining the list of tasks and their dependencies based on new information. You should be able to identify the critical path – the longest sequence of finish-start dependencies – by tracking those dependencies, task durations, and resource commitments using your list. Manage the overall project by keeping that information fresh and using it to make decisions:

• Start times of tasks on the critical path are valuable leading indicators which are predictive of late projects, and ensuring start dates are accurate is easier than managing completion dates midway through an iteration.
• Identify critical resources for tasks on the critical path and make sure they are not otherwise overloaded.
• Identify ways to shorten the duration of tasks on the critical path without sacrificing non-negotiable project requirements (including non-functional requirements), or by making a good, data-backed case for the trade-off on negotiable requirements. Pairing engineers with relevant but mostly non-overlapping skill sets can also shorten cycle time on resolving tasks requiring the superset of their skills.
• Identifying tasks off the critical path, as well as their schedule float, helps parallelize tasks as resources become available.

Carryover per iteration is a good leading metric for late delivery, and tickets which aren’t moving or otherwise showing activity (comments, commits, or status changes) should be closely managed.

We have cognitive biases which push us to be overly optimistic in our predictions. We want to help and please our colleagues, or simply fail to consider practical challenges when we imagine the solution to a problem. Applying a coefficient greater than one to estimates you receive for sprint planning purposes, and possibly then a further coefficient to the projected delivery day, is an easy way to help correct for this bias, but it’s imprecise and may occasionally produce wildly unrealistic estimates if multiple teams with interlocking dependencies are applying it in an uncoordinated fashion.

Improving Delivery Estimation

As a product engineering organization, when necessary, we want and need to be able to give our colleagues delivery dates that have a 90% chance of being accurate, but published, peer-reviewed research suggests 90% confidence intervals are only accurate about half of the time.

Why? Overpricision, a.k.a. overconfidence in estimates, is the excessive certainty that one knows the truth. It is the form of overconfidence which is most consistent and extreme.

Of all forms of over-confidence, overprecision is particularly impervious to de-biasing. Empirical evidence has demonstrated that highly trained and experienced domain experts exhibit it as much as laypeople, so it’s reasonable to assume simply training or working harder will not improve your 90% confidence estimates.

There are many reasons a project might be delayed or otherwise miss a deadline, but they aren’t covered in this segment. This lesson is specifically about a robust, experimentally-validated approach to reducing overprecision bias, with the catchy acronym of “SPIES” (“Subjective Probability Interval Estimates”).

Using it doesn’t guarantee you will deliver on time, but it gives you a pretty simple way to get to a 90% confidence interval you can iteratively update as your project proceeds, potentially helping you make the necessary course corrections to deliver, or providing earlier messaging to project stakeholders.

SPIES is pretty easy to implement:

• Determine your maximum and minimum interval values, as well as reasonable interval size. For example, will this project take less than a month, or more than a year? Does a month provide sufficient granularity?
• Assign a probability to each interval, such that the sum of probabilities for all of your estimates is exactly 100%. In other words, you want to cover all possible outcomes.
• Beginning with the interval assigned the highest probability, add pairs of neighboring intervals until you get as close to your target confidence interval (for example, 90%) without going over. Then, add the proportion of the adjacent interval with the next highest probability (or both, if they are assigned equal probabilities) needed to reach 90%.
• This interval gives you a window which is demonstrated to be more accurate than simply estimating a 90% confidence date because you have been forced to consider many more possibilities, summoning more varied data points from memory to inform your mental model. It’s a little “Thinking, Fast and Slow”.
• You can use SPIES as an individual, or as a team, with a variety of approaches. For example, you could determine the intervals and probabilities as a group, weighing different arguments and flushing out uncertainties, or you could have individuals perform their own SPIES estimates, then averaging the results, as well as calculating a standard deviation.

At this point, your favorite coding agent can probably make you your very own custom version in about 4 minutes.

What do you do if your interval is too large for your stakeholders to use?

• Break it down: are there re-orderable or parallelizable sub-projects or deliverables you can estimate separately, using SPIES, and is it possible to front-load work on the biggest one? That might help pull the other tasks forward.
• SORT BY uncertainty DESC: use SPIES in a group setting to discuss estimates for intervals, looking for sources of uncertainty, and then try to prioritize the work which reduces it most efficiently. Then, revisit the project’s SPIES as part of your team cadence: is it narrowing? It’s OK if not; you may be doing very valuable discovery work, just keep stakeholders abreast of your learnings and document them visibly.
• Make sure you’re all on the same page: maybe the variation is caused by people imagining very different projects. Try putting together a rough consensus Gantt chart of tasks and deliverables with dependency links in case anyone is extremely under- or over-weighting necessary (or unnecessary) steps.