Put your engineers on support

Let's talk about your bug backlog. You know you have it -- somewhere in your issue tracker, a stack of dozens or maybe even hundreds of bugs awaiting attention. Sometimes new engineers will pull a few off the stack while they're getting up to speed; one or two might get slotted into a release if enough customers write in about it. But ultimately, it's grown into a beast and it's difficult to know where to even start with it. Which bugs are the most important? Which are actively blocking your customers, and which are just minor UI annoyances? How many of them have already been fixed in some other change and you haven't even realized it? The task of simply reviewing the bug backlog is daunting, never mind actually shrinking it.

At Aha!, we pride ourselves on ultra-responsive support. Our customer success team maintains a global average of under two hours for the first response to a ticket. Ultra-responsive support also includes technical resources; it doesn't do much good if you respond to your customer quickly but their issue continues to languish for weeks or months. We know that our product is a central part of our customers' daily workflow. Unfixed bugs can disrupt our customers' productivity, and even worse, undermine their overall trust in the stability of our service.

To combat both of these issues at once, our engineering team maintains an engineering support rotation. Each week, a handful of engineers are assigned to do nothing but monitor the engineering escalations queue and handle the tickets that come in. Some of these tickets are informational in nature, like a technical question about our API. Others are subtle behaviors that require a thoughtful explanation but are ultimately working as intended. But as you might expect, a number of the tickets that we handle each week are true bugs that exist out in the wild. Our engineering support rotation has been crucial to ensuring that we responsively address customer issues and provide users with the stable, quality product that they expect. Here are a few key reasons why.

Staying responsive to customers

When a customer writes in about a bug, it represents both a failure and an opportunity. Obviously, if we had it our way, our application would be totally free of bugs and nobody would ever need to write us about a problem. But given that bugs are an inevitability for any product, the most important issue becomes how you handle them.

If bug reports disappear into a backlog where they are unlikely to be fixed for weeks or even months, it creates an ongoing tax on your customers' mental model about your product. In the short-term, they are likely to be blocked by whatever task they are unable to accomplish, which diminishes the value they receive. But, perhaps even worse, in the long term it undermines their overall trust in the application.

People build deep and lasting associations based on repeated experiences. In the same way that repeating a mundane task like exercise can create a positive habit, repeatedly experiencing bugs that never seem to get fixed creates an overarching impression that the product you're using is intrinsically unreliable. Every subsequent issue you encounter only serves to reinforce that mental model. And while you may continue to use the service (and work around the warts), you certainly won't enjoy using the service. You won't evangelize to your colleagues, and ultimately you won't want to continue spending your budget on something that is unable to reliably deliver the functionality you need.

Conversely, the moment when you receive a bug report is a golden opportunity to create a mental association of a more positive variety. If you turn around an engineering escalation and deliver a fix within a few hours or days, you train your customers to think that they can trust your product. And even more importantly, you train your customers to think that they can trust you. They know that if they do have an issue, they can expect you to take it seriously and quickly devote resources to unblocking their work. And they come to trust your engineering processes as a whole. They may not even realize it, but when you ship an exciting new feature, they will be predisposed to think, "This will be awesome, and if I have a problem, then they'll make it even better." If your current functionality is riddled with unsolved bugs, they are more likely to think, "Oh great, another half-baked addition to an already frustrating product."

Fixing the most urgent bugs first

One of the most difficult challenges of the bug backlog is knowing exactly where to start with it. If you knew how many customers each bug was affecting and the severity of the effect, it would be much easier to understand what to work on and when. But with a backlog that's dozens or hundreds of items deep, it can be difficult to know what is even still reproducible, much less how many customers are running into it.

What if we start at the other end? At the moment when the bug is reported, you understand its severity because the customer is directly telling you, "I am trying to create a report for a big presentation tomorrow, and I am running into this error." We have also found it wise to assume that if one customer is reporting an issue, there are probably dozens more who encountered the issue but silently endured or worked around it. Your error tracker or logs can also add key context. It's easy to assess the ongoing impact by searching data from the past few days, but it’s much harder to find that information for a bug logged months ago. And finally, you never have to question whether the bug still exists or whether you're on a wild goose chase. You can know with certainty that you are expending your engineering bug-fixing effort in the most efficient way possible: directly addressing the contemporaneous issues that were so disruptive that the customer bothered to write in and tell you about it.

Raising team self-awareness

As engineers, it can be difficult to get a complete picture about how customers actually experience the features that we write. You probably hear about some interesting use cases here and there, and you may have instrumentation to tell you how frequently the feature is being used. However, you may have very little insight into how well it works or how satisfied the customer feels after using the new functionality.

Keeping engineers on the front lines of technical support closes the feedback loop and helps to paint a more complete picture. If you ship a feature and a stream of bugs start flowing into support, you're likely to hear about it, even if you aren't directly on the rotation that week. It may open new avenues of thinking about how to properly test your features or make you consider use cases that had never previously crossed your mind. It will certainly encourage you to pay close attention to quality, lest you subject your teammates to a barrage of issues that you created. And it can even improve how you think about what you're building in the first place. You begin to sense trends over time -- like interfaces or patterns that seem to frequently create confusion or be prone to errors. On the whole, you stay much closer to your customers and the ways in which they are actually using your product.

Finally, support is a growth opportunity for engineers. Through direct communication with customers, engineers gain experience in tactfully wording responses and navigating difficult conversations. The lessons learned are broadly cross-applicable within engineering organizations. For example, learning how to say, "No, but here is an alternative that I think addresses your need" is invaluable when interacting with the product team or other stakeholders. Engineers also get to share in the joy from a customer when a bug is promptly fixed, which helps them to understand the real impact that the product has on its users. Nothing is more rewarding than getting a request from a customer, shipping it to production a couple of hours later, and making someone's day.

Implementation

So you're sold on the concept -- how do you actually implement an engineering support rotation? At Aha!, we have found a few key factors that are crucial to the success of our process.

• Everyone participates. Every member of the engineering team at Aha! takes a shift on the support rotation -- including the CTO and the director of engineering. Responsively fixing customer issues is not grunt work to be shunted to junior developers; it is a critical component of our team's responsibilities. By having everyone pitch in, we ensure that support is simply part of the engineering culture at Aha!, rather than something to be avoided or pushed off to less-tenured colleagues.

• Support trumps everything else. When you're on support, you're on support -- your other feature work is completely on hold for the week. We proactively work with our product team to let them know which engineers will be on support in upcoming weeks so they can plan accordingly. We have found that it is simply not possible to balance support with ongoing feature development. You inevitably end up focusing on one to the exclusion of the other, while feeling guilty about the things you're not getting to. By focusing exclusively on support, we empower our engineers to clear their heads and focus deeply on each customer issue they attend to.

• Training is key. Support can be daunting at first, especially for new engineers. Many engineers have never even written an email to a customer, much less had to interact with a customer who is less than thrilled about the issue they're encountering. We ensure that new engineers "ride shotgun" with a more experienced engineer for at least their first week of support, learning how to write to customers and where to start when debugging different kinds of issues. We also work closely with our customer success team. They are experts in customer communication and are always happy to lend a hand in crafting a response or explaining an issue.

See for yourself

Our weekly support rotation is one of my favorite components of the Aha! engineering culture. It helps us stay ultra-responsive, assisting the Customer Success team to ensure that customers receive the prompt and helpful support they desire. It builds customer trust in the quality of our application and in our commitment to quickly addressing issues that arise. It helps us prioritize and ensure that urgent bugs are promptly fixed, rather than disappearing into the backlog for weeks or months. And it closes the feedback loop, helping us to understand how features are being used and improve the way we write code in the first place. If you feel buried under a daunting backlog of bugs, I highly recommend that you assign some engineers to the front lines of support and see where it takes you.