IT industry is in the second half of a decade that is witnessing the buzzword Cloud Computing. At BizTalk360, we released our first SaaS-based application BizTalk360 Cloud last year. When I was the assigned resource to test BizTalk360 Cloud, it felt very encouraging because it was really a thought-provoking process to test something new for the first time in my lifetime and of course, the best way to learn cutting edge technologies.
Difference between testing SaaS versus on premise applications
So with this, the first question which came to my mind was How SaaS application testing differs from on premise applications? By going through various forums and talking to some cloud experts, I understood that “SaaS“ is software delivered over the Internet (cloud) as a service. The main key transformations between on premise and cloud are Infrastructure, Scalability, Resource Pooling, Data Retention, High Integration and Security concerns.
With the above answers, I was able to conclude that SaaS application testing involves functional and non-functional testing based on the requirements and also testing needs to cover on cloud infrastructure and the components like App Services, Web Jobs, Elastic Pool, Event Hubs etc.
Functional testing – As in any other application, we need to make sure the functionality works as per the business requirements.
Non-functional testing here includes the following testing types:
- Availability Testing – To ensure the application is up and accessible to users all the time, without any downtime
- Security Testing – To maintain integrity of data within SaaS application, since security is an advantage of on premise application and vice versa for cloud
- Interoperability Testing – Every SaaS application must be able to function seamlessly in all different environments and platforms so that users from all backgrounds can access the application
- Load and Stress Testing – To know the stability of the system and to ensure how our application reacts when it is targeted beyond its operational capacity
I would like to list some of the challenges that we faced while testing the SaaS product:
This is applicable for both the cloud and on premise testing. Since at the initial stage of product development, there were a lot of chances that the functionality or business flow may change based on the requirements and time complexity to deliver a product. So creating a proper document like test plans and defining the scope of each functionality at right time fill a gap between the team members. Else we may struggle on defining what is defect and what is not.
During the initial stages of product development, we happened to frequently deploy the application for various reasons like unstable DB structure, the team members checking in codes at different times, and so on. Therefore, many times the smoke testing steps was a failure.The best practice for this is automating deployment process.
The main impact of this frequent deployment is that all the test data will be removed. So it is difficult to track how the application behaves when it runs continuously for certain days. It is difficult to see where data is growing, how each functionality responds with huge data/number of days.
We need to ensure whether an issue occurs in the application or in the supporting components of the application like Web Jobs, Elastic Pool etc. So proper logging is required for better debugging. Additionally, white box testing, DB, and API level testing is very important.
In an On Premise application, we can change the date and time in our local environment and we can trigger the function. But in the cloud, this is quite different and is not possible. The tester has to wait until the set time to validate the functionality.
For this, we need to identify the configurable components and date/time can be managed in the config files. So for the scenarios like testing the functionality of purging a Web Job which runs once in 1 day, we need not have to wait until the mentioned time. Instead, we can change 1 day as a minute in the config file which makes testing easy. And the other trick for date/time validation is converting to the nearest time zone.
There are ‘N’ number of customers being served from the same infrastructure that can be used by multiple customers over the world. Therefore, some may access from Western Europe, some may from Southeast Asia and few from different time zones. So we need to ensure the response/access time should be seamless across the world. Testing needs to be done with respect to internet connectivity and with low to the high-end configuration as the cloud is a “pay as you go” platform. So all kind of infrastructure setup needs to be covered in the test environment.
Apart from the business requirements, we also need to test more on cloud components that will be utilized in the SaaS application. This will be covered in a series of blogs in the coming weeks.
Despite these challenges, we successfully released our first SaaS product BizTalk360 Cloud last year. This was definitely a learning experience!
You can get started with BizTalk360 Cloud and start monitoring 10 artifacts for free with lifetime free account.