Quality Assistance: Lessons learned

This morning, during my 10 idea routine, I push myself to list techniques and activities that I’ve tried to change the Quality mindset around the company. You already know that we’ve been doing modelling some of the process following the Quality Assistance approach. I know that the outcome of these activities will drastically vary for each team, but hopefully knowing some experiences will help you try something similar. And, as Scott Berkun explained amazingly here, it’s more important to learn from the failures than to just share the final idea. And certainly writing this will help me remember some details. I don’t trust my memory that much…

I’ll try to give you as much background as possible, although reading about how the organisation is working now and how I’m facing it know may help. Now.. Let’s focus on some things I’ve tried!

Three amigos. This is a pretty standard activity, although we needed some twist mainly because of resource issues (just me as QA and one product owner). We started picking some complex tasks per sprint and trying a standard Three Amigos, although at some point we decided that the engineers should do the exercise of listing the scenarios before implementing a solution, leveraging our work to double check and enhance with some “edgy but important” ones.

QA demo. We do it after finishing the development of some sprint stories where me (or Product) showed in 10 minutes at their machine how users are most likely to run through it, as well as trying to break it. There’s no better way to understand how to test than looking someone testing (and trying you communicate the mental process). There are some developers that, after this, embraced some testing activities as they saw me enjoying it, although others are reluctant.

Defining metrics to understand the current status. Things like amount of errors in each version, times, grouping types of errors, clearing non-useful data or even finding ways to generate it (like replicating production load on testing environments). We’re all engineers here, and no matter how much you try: data wins discussions. That’s a fact. I don’t have any data to back it, but you have to believe me.

Testing hour. I can’t think that I have too much time to run many experiments, but one of the activities I love doing is setting voluntary exploratory testing sessions. I’ve done it mainly for big new releases, and I try to guide them to focus on a different aspect each time, although it gets wild most of the times. Unfortunately, we don’t do enough dogfooding, so this activity helps a lot spreading the knowledge about the tool and getting feedback. In my experience, teams that are usually siloed and less involved with the products are the ones enjoying it more, and giving you fresh feedback (R&D, Ops, etc.).

Getting involve with the community. We’re quite small, and I love allocating some time helping out the customer support guys giving not technical answers in the “I think I found a bug” section of our forum. It gives me loads of insight about what they care about, and some valuable examples to back my feedback. And they actually find some really weird bugs!

Proactively spying production. Gathering metrics, we’ve come around with some filters to get weird errors and behaviours, and we can use our tool to understand how the a user can get there. We’ve found edgy bugs because of it. We also use Inspectlet to understand how the users react to new features, helping us understanding further user cases and finding strange behaviours!

Participate in technical design meetings. I’m used to joining product meetings to help define some new features or make the product more resilient, but being involved in the technical discussion gives me more insight into the implementation and the challenges, and allow me to raise issues to ensure a more testable product (backdoors to easily reach some states, ways to inject failures, etc.)

Big picture testing. That every developer should care about testing it own slice is a fact, particularly in an organisation where there is only a QA. But testing integrations and “end to end” is usually something tedious that no one wants to deal with, but having a smoke verification before and after every release brings A LOT of value. That’s why I’ve taken the responsibility of building a “simple scenarios” big picture testing suite for both backend and frontend, serving me as valuable documentation as well. It’s the green light everyone needs before rolling a change.

Reviewing code (particularly tests). I wish I had more time to review more bits of code, but I usually just check some of the new stuff who I haven’t understand yet; and the code for the functional tests, especially the first batch of tests for a new feature. My main concern are readability, scalability and detecting flakiness. Avoiding integrations when possible (using static versions of webs vs live version), some random waits, retries policy that would masquerade problems, modularize for reusability…

Pair whenever possible. Ok, after all of this, I know I don’t have much time left. But, when I have, my first list is pairing as much as possible. It teaches me new technologies and ways to develop, I get some understanding about the solution, I can give early feedback and it bonds me with my colleagues. I’ll make a post about the benefits (and cons) I see in pairing soon!

And this, guys, are some activities I’m trying to infect the team with the love about Quality. Some of them may end as worthless, and many will be evolved and change, but right now I’m learning something new every time I do them. That sounds like enough reason to keep them for me!

May the force be with you,



Quality Assistance: Why we tried it

As I mentioned before, Quality Assistance is a testing methodology that focuses on empowering and support the developer on Testing tasks so the team can deliver at speed. In this post, I’m going to talk about the reasons why we tried this approach with our team, as well as some challenges we faced.

I joined the startup right after our QA lead did. It was in interesting timing, all the decisions regarding Quality were still pending. I was lucky that she counted on me for the change, and one of the approaches we wanted to try was Quality Assistance.

Why? Well, I think that the main three reasons were: resource limitation, we were only two to deal with every time, so traditional methodologies based on manual sign off or us automating every check was out of the table; we sincerely believe this is the step forward the industry needs, especially in Agile environments where speed is a key factor; and personally, I’m always willing to try something that would allow me to spend more time doing the activities I love of my role.

It wasn’t the only option we considered, but the one we wanted to try the most. As any other similar change we had a list of challenges to face in order to success in it, and some of them needed to be addressed before working on it.

As always, the first step was convincing the stakeholders. Luckily for me, most of this work were done by my manager, and they were really keen to try the change because they knew the previous approach wasn’t working and agree that Quality ownership should be shared in the team. But, if we want to really convince stakeholders we needed to start gathering metrics to show the results and start using facts as arguments instead of opinions or beliefs. It makes most conversations easier. We struggled on this and I believe if we had focused more on metrics, embracing Quality Assistance would have been smoother. Although I don’t have facts on that…

Convincing the developers at that stage was easy, but when we started to work together on what were the expectations and how we needed to change our way to work… Well, I can ensure you that it’ll take me a whole post talking about some clashes we had during the process. The lack of documentation made it more difficult. So far, it seems that only Atlassian is trying it, so we didn’t have many documentation or examples to show them, and most of the information was focused on QA practitioners out stakeholders. It is also an approach that requires high doses of customization as it depends on each team. We had to focus on a team by team solution, difficulting us to use one solution to convince other teams.

Personally, I also had the challenge of improving my exploratory testing techniques. Previously in my career, I just focused on automated checkings, tooling and performance testing. And this is a crucial part of quality assistance. It was an enlightening experience, learning a lot in such a small period of time, and helping me with the next challenge: thinking how to teach testing. To empower developers with the ability of testing you need to learn how to express all the tacit knowledge you’ve been building for years. It completely changes the perspective because every time I performed a testing task I was actively thinking what is happening inside my head and trying to translate it into words. Challenging for sure, but fun nevertheless.

These were the main walls we had to face before trying Quality Assistance on your organisation. On following posts, I’ll try to focus on different steps that we tried and what was their impact.

May the force be with you,

P.S. Feel free to use Quality Assistance category to visit the related posts!

Meet Quality Assistance

Quality Assistance is a shift in the Testing industry that focuses on empowering the developer with testing knowledge and guidance so the development process can enough Agile speed. The principal promoter of the change is Atlassian, although some other companies are starting to embrace a similar approach (like Spotify)

In this video, Penny Wyatt (Atlassian) beautifully explains why they decided to try Quality Assistance:

Personally, I’m in love with this way to introduce Quality in software projects. I learnt about this approach from my QA Lead and we tried some of the ideas in our startup, learning in the journey some of the challenges needed to embrace it. We learnt how to deal with different developer archetypes convincing them that our goal was empowering them, not procrastinate because we were sharing Quality’s responsibility.

Particularly, I love that I spent most of my time on activities I love doing with this approach, such as revisiting processes, breaking them down along with the product itself looking for dependencies and bottlenecks to focus on; pairing with the developers learning the technical details of the product while I show them the joy of the puzzle of Testing; spending sometimes automating tools, enhancing frameworks and improving automated tests… it was such a win-win situation for me when we embraced this approach!

Obviously, we also encountered walls and difficulties and realise that there is not a simple way to adapt this methodology without customization, iteration and a fair amount of time. Most of the challenges are related to the need to change how most of the people work, being particularly difficult if you find some reluctant to change developers.

It’s impossible to tackle down the details of this encouraging shift in one post. I’ve included some really interesting documentation that I recommend you to visit, as well as the promise to detail in further entries some of the aspects of the approach, and some experiences dealing with it.

For me, Quality Assistance convinced me that Testing is where I want to be. I spent a big percentage of my career thinking that Development is what I should be doing, but I enjoy way more the tester role from Quality Assistance than programmer’s one.

May the force be with you,