It's a story as old as time, boy meets girl, girl falls in love with incomprehensible eldritch horror — oh wait no, sorry wrong story, this one is about TECH DEBT!
In this painfully familiar story, before every other sprint, stakeholders in the business raise some crucial feature that needs to be developed as quickly as possible. The business proclaims "it's an emergency!" (it isn't), and the team is asked to take on more technical debt and told "don't worry, we'll make time to deal with it later". Every day since has been "today" and not one of them seems to be this day called "later". The engineers have stopped wondering if later will ever come...
Over time, the team's productivity asymptotically approaches zero. Product managers are getting frustrated, engineers are frustrated, stakeholders have given up expecting anything from Product (unless they declare it an emergency), and the tech debt seems insurmountable. Developing new features is neither easy nor enjoyable. Everyone has a different solution, from re-writes to an entire quarter without feature work.
In this article, not only do I offer a viable way out of the morass, but a strategy to ensure things don't get this bad in the first place. It's outstandingly simple, and requires only a little focus to execute on. I call it The Three Stream Backlog
The what now?
If this idea catches on I'm going to really regret not coming up with a better name, but The Three Stream Backlog is dividing your backlog up into well... three streams of work.
This stream contains the epics that Product Managers and the team have poured blood, sweat, and tears into developing and preparing for development. The epics in this stream have been through discovery, user research, user testing, prototyping, tech spikes, technical feasibility planning, etc. The team is confident that these epics are a good use of engineering resources.
We're not going to spend too long talking about this stream because the entire Product Management discipline is devoted to doing this well.
Allocate 50 - 70% of the team's engineering resources to this stream.
Business as Usual
This stream contains only tickets for bug fixes, copy improvements, legal document updates, and minor changes. There are no epics in this stream! It's really important that large features do not try to sneak in.
Allocate 20 - 25% of the team's engineering resources to this stream.
This stream contains both tech debt tickets AND epics. However, these epics are created by engineering leadership with the same rigor, research, planning, and confidence in deliverable value as the epics in the feature stream.
This means engineering leadership need to learn some product management skills and maintain their own backlog. The crucial step here is really stepping back and looking at the big picture. Make sure your tech backlog supports and empowers the team to deliver on the feature roadmap.
Does that mean you need to plan an upgrade to Hasura to support web hooks for the upcoming integration feature? Do you need to implement a repeatable solution to building audit tables alongside the mutable tables that Data & Analytics teams are relying on for their reports? Is the developer experience slowing everyone down and causing "It works on my machine!" bugs? Maybe deployments are too difficult and error prone.
The tech stream is partially recursive, because you will find it contains a stream of big features and smaller bug fixes, just like the three streams themselves. You need to put on your Product Management hat and figure out how to efficiently allocate limited resources to seemingly unlimited problems. This leads me to my next point.
Executing on The Three Streams
The critical detail in implementing this effectively is allocating people to particular streams for entire sprints. Context switching kills focus, kills productivity, and saps our team's morale.
So given a team of 4 engineers, you would allocate them to the 3 streams as such:
This focus is the critical detail that lets you plan big, empowering tech epics and deliver serious improvements to the infrastructure, developer experience, and non-functional requirements of the product! You now have dedicated resources each sprint to work on the tech.
What's the alternative? I think we have all tried the "30% of the sprint's story points are for tech debt tickets" approach. Has this ever worked for anyone? The problems with that approach are:
- You can't plan significant tech work
- Tech debt tickets are prioritised last in the sprint, meaning they get done in a rushed and haphazard way, if picked up at all
- You can't predict delivery of tech work
- Often, no one is really looking after these tickets or planning the tech backlog
In the end, the story point approach leaves the tech team feeling even more disempowered, and the technical challenges seemingly even more insurmountable.
Tips and Tricks
Win over Product Managers
Product Managers might push back on the idea of them handing over up to 25% of the available engineering resources every sprint for tech to work on their own backlog. However, remind them that this means they don't have to prioritise tech debt tickets ever again. It means the team will deliver faster and more efficiently. It means the team will be able to estimate work better because the code is easier to work with. It means deployments will become more reliable and less stressful.
Try it for a quarter and ensure you deliver a significant technical challenge that improves your ability to deliver Product features reliably. Then, tell everyone that will listen that this was possible because you were able to plan technical work via the three stream backlog and deliver it with uninterrupted focus. When I first introduced it, my team introduced Continuous Deployment for the first time in a business that many thought would never pull it off.
Avoid Stream Fatigue
It's important to swap people around between the feature, BAU, and tech streams. Everyone needs that shared context, and often the things people learn working in one stream will make them more effective in another.
However, be mindful not to break peoples' flow. Let people stay in the tech stream until their technical epic is complete, and same for the feature stream. Since the BAU stream is made up entirely of small tickets, it's easier to swap people in and out of this stream between sprints than the feature and tech streams.
Report on Outcomes with Discipline
As a new Product Manager of the technical backlog, you need to put in place some systems to measure the success of your tech epics. This means your epics need to not only be well formed coming in, but well analysed on the way out. I do this by creating a report in Confluence for each big feature, and include:
- What went well
- What could be improved next time
- What outcomes occurred that I expected (i.e. reduced lead time, increase release frequency, better team health metrics, etc)
- What outcomes occurred that I did not expect
When people ask you if this approach is working, you will have a repository of delivered work and measured outcomes to direct them towards.
So far I've seen the three streams work best on agile teams running Scrum with sprints. It works decently on Kanban teams, but swapping engineers between streams gets a little messy.
It is not compatible with Shape Up, but then I believe Shape Up is incompatible with life, healthy teams, kittens, and basically anything good and bright in the world.