In agile software development, there is a focus on the feature implementation and delivery of value to the customer and, as such, non-functional aspects of a system should also be of attention. Non-functional requirements testing is challenging due its cross-functional aspects and lack of clarity of their needs by business in the most part of projects. But how do agile team members handle non-functional testing in their projects? And which factors influencing the testing of non-functional requirements, specifically performance and security in agile development?
We conducted interviews with twenty IT professionals in a large multinational company. As result we could identify seven main factors influencing non-functional testing and four main practices adopted by them to overcome the challenges faced. Together with two colleagues Cristina Camacho and Sabrina Marcsak, I published a paper on the ARES Conference that I summarize the findings below.
In 2001, agile development has arisen aiming for a way to deliver working software and to add business value earlier in the software development process. Communication, working software, quality, and cross-functional teams are values highly appreciated in agile development as described in the Agile Manifesto. Up to that moment, software testing was highly dependent on requirements and testing specialists dedicated to transform those requirements into test cases within the overall goal of system quality. In the agile context, this scenario is not anymore possible, once that there is not an established phase of requirements elicitation and analysis that produce a set of requirements that can be used as input for the testing process. On the traditional waterfall development, it was already noticed the complexity of defining, describing, and prioritizing non-functional requirements due to issues such as the natural of the non-functional requirement of being general and not exactly "do" something as features and functions do. Non-functional requirements lose priority against functional requirements, and even when some functional requirements are affected when non-functional characteristics are not considered.
Along with that, the testing of non-functional requirements has not been taken seriously and it is very often classified as low-risk due to its characteristics. Non-functional testing requires long time of execution and an open minded approach. The need of an overall approach and the necessity of a long execution time can be also listed as an additional concern since agile development brings a focus on the feature implementation and faster delivery of value to the customer (generally functional requirements), bringing even more difficulty to identify non-functional aspects.
What are the factors influencing testing of non-functional requirements in agile teams? We identified seven main factors that influences the non-funtional testing in agile teams : priority, time pressure, cost, technical issues, awareness, culture and experience.
- Priority: reason for why the non-functional testing would add value in that moment to the business. The testing priority was reported as the main key factor impacting non-functional testing activities. Many different aspects can determine the prioritization of the non-functional testing, such as feature/system characteristics, project type and criticality to business.
- Time pressure: When defining priority of testing, time pressure was also largely cited as an influencing factor. Developers and testers state that functional testing takes the prioritization over non-funtional testing due to development short time for a given sprint or release.
- Cost: The cost can be divided into two different meanings or contexts. In one hand they considered the cost of failure as a factor to determine the execution of non-functional testing. In this context, an analysis over which would cause an impact to business in case of failure of some non-functional aspect.
- Technical Issues: issues related to the system code deployed into production environment as a factor that may influence the testing of non-functional requirements in the next releases or any resource or environment issue that can avoid or impact or invalidate the execution of non-functional testing.
- Awareness: Business, developers and product owner's awareness with regards the importance of at least the analysis of the non-functional aspects as an important factor that influences the non-functional testing.
- Culture: Culture in this context means to have the habit of thinking and considering non-functional testing in the development tasks.
- Experience and skills set: Participants reported that generally the most experienced team members provide more attention to the non-functional aspects and defend the non-functional testing. In their view this happens due to past experiences they had in the past. The emphasize that mainly younger team members concentrate in what needs to be delivered (mostly functional items) and don't think in the overall system or cross-functional items / features.
The main observation was that factors are related and influence each other. Priority can be influenced by team experience and non-functional culture and awareness of its importance. Time pressure and cost can be better analyzed by experienced team members. Issues such as lack of a dedicated environment can get priority and budget once its acquisition importance is very clear for all team members and customers. According to the results, when importance of non-functional testing is not clear to all, factors observed in this paper culminates in no time enough to execute non-functional testing due to the lack of priority and time and, of additional budget (hours) are added, due to the lack of visibility from business and developers that it is important, culminating in possible production issues that can damage customers' brand or productivity of real users.
How do agile teams handle the identified factors?
The key moments where non-functional aspects should be discussed and detailed are the project inception as the moment where business characteristic should be reviewed. This analysis, help team member on identifying that the non-functional testing needs supporting to define priority. The same exercise is done during sprint planning. Once reviewing the user story to be developed, senior team members reinforce the need of a deeper review under non-functional aspects related to that user story. As the whole team participates, this exercise helps on the dissemination of the non-functional awareness and culture.
Another good approach is to work with a "concerned" mindset. When having a risk mindset and an explanation about how impacted business can be, other factors can be minimized such cost and time, because non-functional needs can be better understood. It is important to have representatives from all roles during the agile ceremonials, especially inception and sprint planning. Developers, architects, product owners, business and in case possible, non-functional experts. It is also important the presence of senior members during these meetings.
The conversation about non-functional requirements and testing needs to continue during the development of the user story and, through discussions on the agile team ceremonials such as stand-up-meetings. Non-functional aspects are related to an evangelizing task, which needs to be done always in order to produce a culture inside the team and company.
Communication was largely listed as a channel to handle with the factors impacting non-functional testing. Participants believe that a good communication around non-functional issues with the features teams aligned with DevOps communication or order to have developers aware of the behavior of the application in production can also help on the identification of non-functional testing needs. They believe that sometimes, developers and testers don't have information enough to elaborate a good testing strategy. The communication with other teams and real users can in their view, to support on identifying where the priorization needs to focus on.
Finally, work with the quality perception is crucial. Quality on top of mind during the whole process and post-delivery. Participants also cited code review practice as an instrument that helps in the identification and execution of non-functional testing. All roles working with the quality in mind and awareness of non-functional needs.
We aim to replicate our investigation in a larger scale. Meanwhile, our work provides initial contributions to practitioners and inspires our future research. Please contact us if you would like to discuss more issues !
