As teams move towards continuous development and delivery model for the software world, they are exploring solutions to increase their pace of delivery. QA teams try to this by reducing the testing time. Usually, the wrong approach taken is to reduce test coverage based on risk analysis by only testing on limited browsers, browser-versions, and OS based on a maximum target audience usage.
However, if teams move from sequential testing to parallel testing, teams can achieve maximum coverage with reduced execution time. This new era of the continuous testing model, along with parallel testing, helps drive this approach. So first, let’s understand parallel testing more.
What is Parallel Testing (also referred to as Parallel Test Execution)?
In parallel testing, we test different modules or applications on multiple browsers in parallel rather than one by one.
The parallel test execution is different from sequential testing, where we test different modules or functionalities one after the other. Even in the case of testing applications on multiple browsers, tests are performed sequentially on various browsers. This approach of testing is very time-consuming.
Parallel testing helps to reduce execution time and efforts and results in a faster time to delivery. It proves to be handy specifically in case of cross browser testing, compatibility testing, localization, and internalization testing. In a scenario where we have two versions of software available, and we need to check its stability and compatibility, we can run the two versions simultaneously and find issues at a much faster rate.
In this post, we explain how to run parallel tests by using Selenium with TestNG.
Parallel Testing using TestNG and Selenium
TestNG is a testing framework for Java that helps to organize tests in a structured way and enhances maintainability and readability to the scripts. TestNG has made it easier for automation testers owing to its large feature set. One of which is parallel testing or parallel execution. TestNG provides an auto-defined XML file, where one can set the parallel attribute to method/tests/classes and by using the concept of multi-threading of Java, one can set the number of threads, one wants to create for parallel execution. Below is the structure for defining this attribute in the TestNG XML:
<suite name="Parallel_Testing" parallel="methods" thread-count="2">
The parallel attribute can be extended for multiple values, as below:
- Methods: Helps run methods in separate threads
- Tests: Help to run all methods belonging to the same tag in the same thread
- Classes: Helps to run all methods belonging to a class in a single thread
- Instances: Helps run all methods in the same instance in the same thread
Along with the parallel attribute, the thread-count attribute helps in defining the number of threads one wishes to create while running the tests in parallel. For example, in case one has three methods, with thread count as two, then during execution, two threads shall start in parallel with the corresponding methods. As the first method execution is completed, and the thread gets free, it takes up the next method in the queue.
Let’s look into the code snippet to understand the parallel execution process. In the example below, I have created two methods. One uses chrome to open the Browserstack homepage where the other uses the firefox browser to sign up on the Browserstack platform. In this code snippet, we are executing both these methods in parallel, by setting up the thread count in the TestNG XML file as 2. Refer to the below snippet and TestNG XML file:
import org.testng.annotations.AfterClass; import org.testng.annotations.Test; import org.testng.annotations.Test; import java.util.concurrent.TimeUnit; import org.openqa.selenium.By; import org.openqa.selenium.WebDriver; import org.openqa.selenium.chrome.ChromeDriver; import org.openqa.selenium.edge.EdgeDriver; import org.openqa.selenium.firefox.FirefoxDriver; import org.openqa.selenium.ie.InternetExplorerDriver; import org.testng.annotations.Test; public class ParallelTestWithMultiThread { WebDriver driver; @Test() public void testOnChromeWithBrowserStackUrl() { System.setProperty("webdriver.chrome.driver", ".\\Driver\\chromedriver.exe"); driver=new ChromeDriver(); driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS); driver.get("https://www.browserstack.com/"); driver.manage().window().maximize(); System.out.println("this is the test related to chrome browserstack homepage"+ " " +Thread.currentThread().getId()); } @Test() public void testOnChromeWithBrowserStackSignUp() { System.setProperty("webdriver.gecko.driver", ".\\Driver\\geckodriver.exe"); driver=new FirefoxDriver(); driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS); driver.get("https://www.browserstack.com/users/sign_up"); driver.manage().window().maximize(); driver.findElement(By.id("user_full_name")).sendKeys("<name>"); driver.findElement(By.id("user_email_login")).sendKeys("<login email id>"); driver.findElement(By.id("user_password")).sendKeys("<password>"); System.out.println("this is the test related to chrome browserstack login"+ " " +Thread.currentThread().getId()); } @AfterClass public void close() { driver.quit(); } } TestNG Xml file:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd"> <suite name="Suite" parallel="methods" thread-count="2"> <test name="Test"> <classes> <class name="ParallelTestWithMultiThread"/> </classes> </test> <!-- Test --> </suite> <!-- Suite -->
TestNG Report:
Based on the requirements, one can set up the thread count and parallel attribute. Now, let’s understand how parallel execution makes testing faster.
The Execution Race – Parallelization vs Serialization
Let us look into an interesting case study, that would give a clear insight into how parallelization makes testing faster and gives a better RoI. Let us consider the same code snippet that contains the two methods, which would run in parallel in two different browsers, one is a test on Chrome, and another is a test on Firefox.
In one instance of execution, we shall run the methods one by one, whereas in the other we shall run both the methods in parallel with 2 threads created simultaneously. The below image indicates the time taken to execute both the methods in serial i.e. one after the other:
The below image indicates the time taken to execute both the methods in parallel:
From the statistics above, we can see that parallel test execution took almost 1.5 times less time than serialization. With this reduced execution time, we can complete the testing cycle faster, resulting in quicker delivery, and also leading to a better ROI.
Try Parallel Testing with Selenium
An interesting fact to note is, increasing the number of threads may not substantially reduce this number to a lot in comparison to serialization, so choosing the threads is also an essential factor in parallelization.
Disadvantages of Parallel Testing
- In the case of parallelization, for different modules to run in parallel, we need to create independent modules. Modules with dependencies cannot be included in the parallel approach.
- For parallelization, one needs to have a detailed understanding of the product and its flow for better results. Even though parallelization can help in cross browser compatibility testing, its coverage of multiple browsers is restricted until and unless it is accompanied by distributed testing where the setup of multiple machines and browsers is provided
The era of Cloud-Based Testing
Owing to the need for having access to multiple platforms and browsers to run tests in parallel, the cost of compatibility testing with parallel testing increases. Also, we may come to a point where access to all browsers and versions may not be possible. BrowserStack provides access to numerous platforms and browsers with their corresponding versions on the real device cloud. One can run automated parallel tests and use multiple browsers and versions.
ncG1vNJzZmivp6x7o77OsKqeqqOprqS3jZympmeXqralsY6pmKuZnKGyrXnTnqqtoZ6ceri106FkrJ2cmruqwcw%3D