Soft Assertions in Test Automation

An assertion is a statement that evaluates a condition and asserts that the condition is true. In test automation, assertions are used to verify that the application under test behaves as expected.

There are two types of assertions: hard assertions and soft assertions.

• Hard assertions fail the test immediately if the condition is not met. This is useful for verifying critical conditions.
• Soft assertions do not fail the test immediately if the condition is not met. Instead, they log the error and continue with the test. This is useful for verifying non-critical conditions.

Over the years, testing frameworks and libraries have started incorporating soft assertion functionality, making it easier for testers to implement this approach in their automated testing practices. Frameworks like TestNG, JUnit, and various JavaScript testing libraries have built-in support for soft assertions. Playwright also provides this functionality.

The importance of soft assertions in test automation is that they allow you to verify multiple conditions in a single test without worrying about the test failing prematurely. This can help identify subtle bugs that might not be caught by hard assertions.

For example, let’s say you are testing a login form. You could use a hard assertion to verify that the username and password fields are both filled in. However, you could also use a soft assertion to verify that the text in the username field is your actual username. If the username field is empty, the hard assertion will fail, but the soft assertion will still log the error and allow you to continue with the test.

A hard assertion immediately throws an exception, stopping the test steps. Consequently, the test report will only capture the first failure message. On the other hand, soft assertions don’t throw an exception right away. They allow all the steps and soft assertions in the automated test to be executed before failing the test. Therefore, the test report will include all the failure messages.

Playwright has built-in soft assertions that mark the test as failed but don’t terminate the test execution when it fails. Combining soft assertions with hard assertions ensures that all failed assertions are captured, and the test execution stops when we decide.

test('test', async ({ page }) => {
    await page.goto('<https://owasp.org/membership/>');
    const emailAddress = page.locator('[aria-label="Email Address"]')
    expect.soft(await emailAddress.getAttribute('placeholder')).toBe('Member Address'); 
    //Error: expect(received).toBe(expected) // Object.is equality
    await emailAddress.fill('email.testing@gmail.com');
    
    const confirmEmailAddress = page.getByPlaceholder('Confirm Member Email Address')
    expect.soft(await confirmEmailAddress.getAttribute('placeholder')).toBe('Confirm Member Address');
    //Error: expect(received).toBe(expected) // Object.is equality
    await confirmEmailAddress.fill('email.testing@gmail.com');
    
    const companyName = page.getByPlaceholder('Company Name')
    expect.soft(await companyName.getAttribute('placeholder')).toBe('Company Name');
    await companyName.fill('Testing Company');
    
    const memberName = page.locator('[aria-label="Name"]')
    expect.soft(await memberName.getAttribute('placeholder')).toBe('Member Name');
    await memberName.fill('New Member');
  
    expect(test.info().errors).toHaveLength(0);
    //Error: expect(received).toHaveLength(expected)
});

My recent publication compiled a comprehensive collection of 100 similar typescript programs. Each program not only elucidates essential Typescript concepts and expounds upon the significance of test automation but also provides practical guidance on its implementation using Playwright. This resource will undoubtedly be valuable if you look deeper into similar topics.

Playwright's Cookbook: Recipes with Typescript