Conditional Testing

Last update on August 19 2022 21:50:57 (UTC/GMT +8 hours)

In this tutorial, you will learn when it is a good choice to use conditional testing for your tests. You will also learn where it is not possible to use conditional testing. Finally, we will show you different strategies that you can use to handle common scenarios of conditional testing.

Definition

When we say conditional testing, we are referring to this common programming pattern:

If X, then Y, else Z

Some sample use cases are as shown below:

How do I do something depending on whether an element does or doesn’t exit?

How do I account for my application that does A/B testing?

How do I recover from a failed Cypress command such as when cy.get() does not find an element?

I want to write dynamic tests that will do something based on the text on the page.

I want to find all <li> elements in a page automatically and then based on the ones that are found, I want to check that each links work.

Although this tests seem simple at first look, writing test in this manner will lead to flaky tests more often than not, it will also lead to random failures and make it very difficult to track edge cases.

Let us look at the problem more critically and then we will look at how we can overcome them.

Statement of Problem

Modern JavaScript applications are highly dynamic and mutable. The state and DOM change continuously over a period of time.

However, the problem with conditional testing is that you can only use it when the state has stabilized. Most times it is impossible to know when a state is stable.

We might not be able to notice something that changes per 10ms or even per 100ms. But a computer will be able to track those changes.

Let us illustrate this by trying to test an unstable state conditionally.

The DOM is unstable

// your app code

'''// random amount of time
const random = Math.random() * 100

// create a <button> element
const btn = document.createElement('button')

// attach it to the body
document.body.appendChild(btn)

setTimeout(() => {
  // add the class active after an indeterminate amount of time
  btn.setAttribute('class', 'active')
}, random)'''
// your cypress test code
'''it('does something different based on the class of the button', () => {
  // RERUN THIS TEST OVER AND OVER AGAIN
  // AND IT WILL SOMETIMES BE TRUE, AND
  // SOMETIMES BE FALSE.

  cy.get('button').then(($btn) => {
    if ($btn.hasClass('active')) {
      // do something if it's active
    } else {
      // do something else
    }
  })
})'''

The problem with this test is that it is non-deterministic. The <button> does not always have the class active. And in most cases, you cannot rely on the state of the DOM to determine what you are to do conditionally.

This leads to flaky tests, and Cypress API is designed to combat flakiness at every step.

The situations

You can only perform conditional testing on the DOM when you are 100% sure that the state has settled and can’t change. Aside that, you will get a flaky test if you rely on the state of the DOM for conditional testing.

Server side rendering

In the case where your application is server side rendered, without JavaScript that asynchronously modifies the DOM- congratulation, you will be able to do conditional testing on the DOM, this is because, if the DOM will not change after the load event occurs, you can then represent a stable state of truth accurately.

Client side rendering

Although it is possible to perform server side rendering, it is not what is obtainable in most modern applications. In these applications, when the load event occurs nothing will be rendered to the screen, the content of the page render asynchronously. In this case it is not possible to use the DOM to perform conditional testing, except when you are sure that all the asynchronous instructions have been rendered and there are no pending network requests, setTimeouts, intervals, postMessage or async/await code.

This is a big problem as it is very difficult (and most times impossible) to capture every async possibility without making changes to your application.

There are workarounds to this problem however- you will have to anchor yourself to the piece of truth that it is not mutable.

The strategies

As we stated in the previous sections, you cannot always guarantee the state of the DOM, if you cannot guarantee that the DOM is stable. There are other ways you can perform conditional testing or perform a work around the problems that are inherent with it.

You could:

Remove the need to do conditional testing
Force your application to behave deterministically
Check for other sources of truth such as your server or database.
Embed data into other places (cookies/ local storage) that you could read off.
Add data to the DOM that you could read off in order to know how to proceed.

Here are some examples of conditional testing that will pass or fail 100% of the time.

A/B campaign

Let us assume a scenario where you visit your application and the content that is displayed depends on which A/B campaign that the server decides to send. This may be as a result of the geo-location, IP address, time of day, locale, or some other factors that you cannot control.

To write tests in this manner, you will have to control which campaign is sent or you have to provide a reliable means to know which one it is:

Use URL query params:

'''// tell your back end server which campaign to send
// so you can deterministically know what it is early
cy.visit('https://app.com?campaign=A')
...
cy.visit('https://app.com?campaign=B')
...
cy.visit('https://app.com?campaign=C')'''

in this case there is no need to write conditional testing since you know ahead of time what campaign will be sent.

Use the server

If the campaign is saved with a session, you can ask your server to tell you the particular campaign that you are on:

Conditional testing 2

'''// this sends us the session cookies
cy.visit('https://app.com')

// assuming this will send us back the 
// campaign information
cy.request('https://app.com/me')
  .its('body.campaign')
  .then((campaign) => {
    // runs different cypress test code
    // based on the type of campaign
    return campaigns.test(campaign)
  })'''

Use session cookies

Alternatively, you can test this if your server sent the campaign in a session cookie that you could read off.

'''cy.visit('https://app.com')
cy.getCookie('campaign')
  .then((campaign) => {
    return campaigns.test(campaign)
  })'''

Embed data in the DOM

Another strategy that you can use it to embed data directly into the DOM- however you should do this in a way where the data is always present and query-able.

'''cy.get('html')
  .should('have.attr', 'data-campaign').then((campaign) => {
    return campaigns.test(campaign)
  })'''

Welcome wizard

Let us assume that you are running a bunch of tests and it should show a “Welcome Wizrard” modal each time you load the application.

You want to close the wizard if it exists and ignore it if it does not exist. However, the wizard most likely renders asynchronously, so you cannot use the DOM to dismiss it conditionally. In this case you will need to another reliable way to achieve this without involving the DOM.

The pattern is the same as we discussed previously:

Use the URL to control it:
Use Cookies to know ahead of time:
Use your server or database:
Embed data in DOM:

Element existence

Whenever you are trying to sue the DOM to do conditional testing, you will be able to use the ability to query an element synchronously in Cypress to create control flow.

Let us imagine a scenario where your application does two separate things that you are unable to control. In such cases you tried each of the strategies above and you were unable to determine ahead of time what your application will do.

However, it is possible to test this in Cypress.

// app code

'''$('button').on('click', (e) => {
  // perform something synchronously randomly
  if (Math.random() < .5) {
    // append an input field
    $('<input />').appendTo($('body'))
  } else {
    // or append a textarea field
    $('<textarea />').appendTo($('body'))
  }
})
// click on the button causing the new
// elements to appear
cy.get('button').click()
cy.get('body').then(($body) => {
  // synchronously query from body to
  // find which element was created
  if ($body.find('input').length) {
    // input field was found, perform another action
    return 'input'
  }

  // else assume that it is a textarea
  return 'textarea'
})
  .then((selector) => {
    // selector is a string that represents the
    // selector we could use to find it
    cy.get(selector).type(`found the element by selector ${selector}`)
  })'''

If the <input> or the <textarea> was rendered asynchronously, you will not be able to use the pattern above. You will have to involve arbitrary delays that will not work in every situation, slow down your tests, and will still make your tests prone to flakiness. The secret to writing good tests in Cypress is to provide Cypress with as much state and facts and to guard it from issuing new commands until your application has reached a desired state that it needs to proceed.

But writing conditional testing adds a huge problem- the test writers are usually unsure what the given state will be. In such cases, the only reliable way is to embed this dynamic state in a reliable and consistent way.

Dynamic text

The pattern of doing something conditionally based on whether a certain text is present or not is similar to the element existence described above.

Conditionally check whether an element has certain text:

'''// this will only work if there's 100% guarantee that the body
// has fully rendered without any pending changes
// to its state
cy.get('body').then(($body) => {
    // synchronously ask for the body of the text
    // and then do something based on whether it includes
    // another string
    if ($body.text().includes('some string')) {
      // yup found it
      cy.get(...).should(...)
    } else {
      // nope not here
      cy.get(...).should(...)
    }
  })'''

Error Recovery

You may be wondering how your tests can recover from failed commands, this is actually the default way to think while programming, especially if you previously programmed in Node.

If you cannot predict accurately the given state of the system, neither can Cypress. Error handling does not offer an additional proof that this can be done deterministically. Failed commands in Cypress is similar to uncaught exceptions in server side code. It is not possible to try to recover in these scenarios as the system has transitioned to an unreliable state. For uncaught exceptions, you will always opt for crash and log. This is essentially what happens when Cypress fails the test. It bails out and skips any commands that is remaining in the test, and then logs out the failure.

Let us consider for a moment that you did have error handling in Cypress. Enabling this would mean that for every single command, it would have to recover from errors, but that is only after each applicable command timeout was reached. Since timeouts start at 4 seconds (and exceed from there), this will mean that it would only fail after a long, long time.