Powerful E2E Tests Combination: Cypress and Git(Hub|Lab) CI/CD
We are working on a rapidly growing project, so the E2E tests and their optimizations are a fundamental part of the development/release process. The project is big with a lot of forms and third-party services like Plaid, Pipedrive, Docusign, etc. — a lot of features offered to the end users — a lot of test cases to be covered. This article serves as a brief summary of our Cypress Reports Generation, its CI/CD integration, and the way we optimised the whole flow without connecting to the CypressIO Dashboard Service (skipping the third-party dependency and preserving all of the data inside our team scope only).
Why Cypress?
A colleague of mine has already written a series of articles on that topic. Be sure to check it out here, here and here. It gives a very detailed comparison between the different E2E testing frameworks. The articles were written around 3 years ago but are still valid today as well.
There are three main reasons why I decided to use Cypress for this project:
- It has enjoyable syntax (the way tests, fixtures, and additional helper functions are injected and used is just awesome)
- It has almost all the major features that other frameworks provide + a lot of plugins that extend the base functionality
- It’s fast to write, fast to integrate, and fast to update if needed (includes all needed Docker images out-of-the-box + well-written documentation)
When I write tests, I follow one simple yet powerful rule — the simpler, the better (from both readability and integration perspectives). It doesn’t matter if they are unit, integration, or E2E, the tests are mandatory from the development point of view, but at the same time, they are something additional to the main codebase — not part of the main functionalities used by the end users. So they should be as descriptive as possible — when something fails, we, as developers, want to just take a look and directly receive information about the feature that has been broken (and the exact parts of it). No need for complex abstract functions or OOP, and no need for reducing the “duplicated” code — each test should be written in a “one look gives you all information” way (without scrolling to different files, logging inputs/outputs, etc. to find the problem). That’s why Cypress completely covers the above requirements.
Cypress Test Cases Optimization
Before we dive deeper into the CI/CD integration-based optimisations, we should ensure that the tests are structured and written in an optimised way as well. If your tests run is slow and heavy by itself you should consider refactoring, not optimisations. A few rules that we can apply:
1. Organise the assertions in an “optimised” way
E2E tests generally don’t follow the “single assertion per test” concept, so feel free to write as many assertions as needed. Yes, it’s nice to have all tests structured in smaller amounts of assertions and more test cases (I would say it’s even mandatory for unit tests), but if it slows the E2E tests down (requiring repetitive actions and page refreshes), I would consider grouping more assertions in one test case.
2. Use Cypress Custom Commands
Cypress offers a very intuitive way of writing custom commands (helper functions). For example, here we have such a command for typing text in text input.
Cypress.Commands.add('fill', (fieldId, value, tagName = 'input') => {
cy.get(`${tagName}#${fieldId}`)
.scrollTo('top', {ensureScrollable: false})
.type(value);
});And then the command can be used anywhere in the tests as:
cy.fill('email', 'input');The command will scroll to the field and then fill the value in (I’ve experienced some issues with not-visible/not-clickable fields, and that completely solved them). It’s not something that would speed up your tests, but it’s something that will speed up the debugging process if something fails (increases the readability). But keep in mind that the commands should not be too complex, otherwise, you would be in the same time-consuming tests debugging position (scrolling through a lot of files, searching different functions).
3. Fill out the forms and test the validation rules iteratively when possible.
If you have large forms with lots of fields, it makes sense to test them iteratively — filling the fields one by one and checking the results without the need for page refreshing.
Cypress Reports Generation
Cypress offers different built-in reporters (from Mocha) as well as a lot of plugins that you can quickly set up and generate some good-looking reports. JSON, HTML, Videos, Images, XML — whatever fits best for your project. For our case, we decided to generate a simple HTML page in a compact inline format — one file that serves all of the information. I’ve tried different libraries during the research and personally liked the Cypress Mockawesome Reporter. Internally it uses Mochawesome JSON structured data to generate the final HTML, which is exactly what Cypress suggests in the documentation as an example for HTML Reports Generation. The setup is very simple, and the best part is that Cypress allows you to combine different reporters using the cypress-multi-reporters. For example, in this snippet of code, you can find the basic Mochawesome JSON reporter combined with the Cypress Mockawesome Reporter. Mochawesome JSON Reporter serves us a complete JSON stats report, and the Cypress Mochawesome Reporters generates the HTML output.
export default defineConfig({
reporter: 'cypress-multi-reporters',
reporterOptions: {
reporterEnabled: 'cypress-mochawesome-reporter, mochawesome',
cypressMochawesomeReporterReporterOptions: {
reportPageTitle: `SF E2E - ${date.toLocaleString("en-US", {timeZone: reportsTimeZone})}`,
charts: true,
overwrite: false,
inlineAssets: true,
embeddedScreenshots: true,
saveAllAttempts: false,
videoOnFailOnly: true,
},
mochawesomeReporterOptions: {
reportDir: 'cypress/reports/json',
overwrite: false,
html: false,
json: true,
},
});You will find more details for the Cypress configuration options and callbacks in the GitHub CI/CD Optimizations sections.
GitLab Cypress Integration
The project was initially placed in GitLab, but we decided to migrate the codebase along with all pipelines to GitHub (as it provides more features like branch-protection rules using required jobs to pass, etc.). However, there are some interesting tricks in GitLab that I want to share with you as that might save you some time in the future (a lot of these facts are hard-to-find in the GitLab documentation or community forum, so I will post them here summarised).
The overall deployment pipeline is:
Four main stages:
1. Build — installing the needed dependencies and building the optimised production code. It’s a NextJS Typescript project, so here we generate both the server code as well as the front-end production builds.
2. Test — starting a temporal server using the already generated code from the Build jobs and running the E2E Cypress tests. Here we use a specific Cypress Docker Image — another big benefit of GitHub over Gitlab — in GitHub you have an already developed official action that does all of the processes automatically (more details in the GitHub Cypress Integration section).
.cypress-e2e-tests:
image: cypress/browsers:node-18.14.1-chrome-110.0.5481.96-1-ff-109.0-edge-110.0.1587.41-1
cache:
- key: $CI_COMMIT_REF_SLUG
paths:
- node_modules
- .next
policy: pull
- key: $CI_COMMIT_REF_SLUG--e2e-report--$CI_JOB_ID
paths:
- cypress/reports
policy: push
when: always
artifacts:
paths:
- cypress/all-videos
- cypress/screenshots
expire_in: 5 days
dependencies: []
before_script:
- echo "$CYPRESS_CONFIG_JSON" > cypress.env.json
- npm start &
script:
- CYPRESS_CACHE_FOLDER=$CYPRESS_CACHE_DIR npm run test:e2e
after_script:
- npm run test:e2e-postprocessAs you can see, we’re using the cypress/browsers:node-18.14.1-chrome-110.0.5481.96-1-ff-109.0-edge-110.0.1587.41-1 Docker image to run the tests in Chrome. There are two caches - the ‘dependencies and build’ one, generated from the Build stage (only for reading), and the Cypress reports one, where we store the generated reports (only for writing). Also, we have one additional artifact for convenience - just in case the developer wants to take a quick look at all of the report resources directly in GitLab. In the main scripts part, we inject the Cypress environment config (echo "$CYPRESS_CONFIG_JSON" > cypress.env.json) and start the NextJS server (npm start &). The NPM test:e2e command starts the Cypress run (executing cypress run --browser chrome). The after_script is used to finish the reports generation process as we use mochawesome-merge to merge JSONs and generate a nice HTML report page.
3. Deploy — the only purpose of this stage is to upload the already built and tested code to the real server.
4. Plaid Check — it’s used to flag if the Plaid integration is still working as expected after the deployment.
We have configured Minio Bucket as the default storage for all GitLab caches. This way, you can easily integrate the report’s data in every project without touching its initial storage structure — without adding and synchronising additional tables in the database and without developing queues/API requests to get the needed information. For example, you can easily create a similar table to summarise the results just by reading the information from the Cache Storage.
GitHub Cypress Integration
There is an already-developed Official Cypress GitHub Action that handles the whole integration (it installs Cypress, builds the project, starts it, and runs the tests). You can still use custom Docker images if you want a specific version of the browser.
You can see the fundamental part of our GitHub Deployment workflow below.
As we’re focusing on the Cypress tests in this article, I will not dive deeper into the other parts of the workflow. But basically, again, we have four main parts:
- Install — installs all dependencies and caches them. Keep in mind that when you cache the Cypress library, you should cache the Cypress Cache as well. For convenience, I would recommend installing it in the ‘node_modules’ folder as well (passing the
CYPRESS_CACHE_FOLDERvariable to the NPM installation command).CYPRESS_CACHE_FOLDER=${{ github.workspace }}/node_modules/.cache/Cypress npm ci - Build — builds the NextJS production code and uploads the production build as an artifact that will be passed to the following jobs.
- Test — runs the Cypress E2E Tests using the Official Cypress GitHub Action. Initially we used a regular job but then switched to a matrix jobs strategy that made the tests drastically faster.
- Deploy/Reports generation — Deploys the code to the server if the tests have passed successfully and generates the HTML reports using the Cypress Mockawesome Reporter.
That’s how the Cypress GitHub Action looks like in our testing job:
- name: Run the E2E Cypress Tests
uses: cypress-io/github-action@v5
with:
install: false
start: npm run start
browser: chrome
spec: cypress/e2e/**/*.spec.ts
env:
CYPRESS_CACHE_FOLDER: ${{ github.workspace}}/node_modules/.cache/CypressInstall is ‘false’ as we’ve already installed Cypress and all of the dependencies in the Install job (they are located in the GitHub Cache now). The server is started using npm run start, and the browser is Chrome. If you want to use the same approach with ‘cached dependencies’, ensure that the CYPRESS_CACHE_FOLDER variable is passed to the environment of the step. We don’t provide a ‘build’ command to the action as we have already built the project in the Build job. All of these “pre-building/pre-installing in previous jobs” are part of the CI/CD optimizations as well - if you build your project once and use the build in a couple of different jobs - it makes sense. If you have a matrix of jobs and every single job needs the project distribution code - it makes sense (you are saving GitHub runtime seconds). But if you just need to run the tests and that’s everything, you can completely leave all install, build, start, and run steps to the Cypress GitHub Action. For example, we have a scheduled daily run of the same tests that notifies the client if something has been broken. There is no need to use artifacts and flood the GitHub Storage with unnecessary data.
That’s the action looks there:
- name: Run the Cypress E2E tests
uses: cypress-io/github-action@v5
with:
install: true
build: npm run build
start: npm run start
browser: chromeCypress CI/CD Optimization via Jobs Parallelization
As you saw in the previous section, instead of one test job, we have a matrix of 12 jobs that run in parallel.
The idea is very simple — divide and conquer. It’s something that Cypress offers out-of-the-box, but they force you to use the CypressIO Dashboard Service. I don’t say it’s something bad, but it creates additional third-party dependency in your flow and exposes some of your test data to their cloud as well. How about if you test some private interfaces?
We decided to separate the tests by ourselves with a simple bash script that runs in every single ‘matrix’ job.
That’s what the overall test job looks like:
test:
name: Test
needs: build
runs-on: ubuntu-22.04
strategy:
matrix:
worker_id: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
env:
WORKER_ID: ${{ matrix.worker_id }}
WORKERS_COUNT: 12
CYPRESS_WORKERS: true
steps:
- name: Checkout the git branch
uses: actions/checkout@v3- uses: actions/cache@v3
with:
path: |
node_modules
key: dependencies-cache-${{ runner.os }}-${{ hashFiles('package-lock.json') }} - name: Download the distribution build zip
uses: actions/download-artifact@v3
with:
name: distribution-build.zip - name: Unzip distribution-build.zip
run: unzip -q distribution-build.zip - name: Setup Node.JS
uses: actions/setup-node@v3
with:
node-version: ${{ env.NODE_VERSION }} - name: Select the worker's tests
run: chmod u+x ./cypress-threads.sh && ./cypress-threads.sh - name: Cypress start and run
uses: cypress-io/github-action@v5
with:
install: false
start: npm run start
browser: chrome
spec: cypress/e2e/**/*.spec.ts
env:
CYPRESS_CACHE_FOLDER: ${{ github.workspace }}/node_modules/.cache/Cypress - name: Zip the Cypress results
if: success() || failure()
run: zip -9qry "worker-results-${{ matrix.worker_id }}.zip" "./" -i "cypress/reports/*" - name: Upload the Cypress results
if: success() || failure()
uses: actions/upload-artifact@v3
with:
name: worker-results-${{ matrix.worker_id }}.zip
path: worker-results-${{ matrix.worker_id }}.zip
retention-days: 1
We keep the current ‘worker_id’ (the number of the job) and the overall workers count in environment variables used by the ‘cypress-threads.sh’ bash script to select the ‘per worker tests’.
#!/bin/bashTESTS_GLOB_EXPRESSION=cypress/e2e/**/*.spec.ts
TESTS_COUNT=$(ls -1bA $TESTS_GLOB_EXPRESSION | wc -l)
TESTS_PER_WORKER_COUNT=$(( ($TESTS_COUNT + ($WORKERS_COUNT / 2)) / $WORKERS_COUNT ))if [ $TESTS_PER_WORKER_COUNT == 0 ]; then
echo "Too many workers! -> WORKERS_COUNT:$((WORKERS_COUNT)) and TESTS_COUNT: $((TESTS_COUNT))"
exit 1
fiWORKER_FIRST_TEST_INDEX=$(( ($WORKER_ID - 1) * $TESTS_PER_WORKER_COUNT ))# The last worker handles the remainder
if [ $WORKER_ID == $WORKERS_COUNT ];then
WORKER_LAST_TEST_INDEX=$(( TESTS_COUNT ))
else
WORKER_LAST_TEST_INDEX=$(( $WORKER_ID * $TESTS_PER_WORKER_COUNT ))
fiINDEX=0
for f in $TESTS_GLOB_EXPRESSION; do
if [[ $INDEX -lt $WORKER_FIRST_TEST_INDEX || $INDEX -ge $WORKER_LAST_TEST_INDEX ]];
then
mv $f "${f}_disabled"
fi ((INDEX++))
done
It just adds the ‘_disabled’ postfix to exclude the rest of the tests (every worker has an equal amount of tests).
After that, Cypress runs the ‘selected’ tests and generates the partial reports. Each report is uploaded as an artifact (the last step of the Test job).
Then, all report artifacts are extracted in the Generate Reports job that waits until the whole matrix of jobs finishes. It sums up all of the reports in one.
It all depends on your Cypress reporters and the type of report data that you want to generate. But everything can be accomplished with just a few minor tweaks in the Cypress ‘before:run’ and ‘after:run’ hooks. If you need just one JSON file containing all of the test information, you can simply merge the extracted reports in that job using CLI.
GitHub & GitLab Pages for Serving Reports
We wanted to generate HTML reports accessible to the client, so we ended up with the question — “How can we serve the HTML?”. Ideally, there should be a way to serve the reports separately — detached from the main product server/resources. Ideally, the test reports serving part should be part of the development process (and the Platform you are using — GitHub, GitLab, Azure DevOps, etc.).
Both GitHub and GitLab provide an easy and flexible way to deploy and serve simple HTML pages — GitLab Pages and GitHub Pages.
So the idea of generating and hosting the reports in the same service seems so close and possible but keep in mind the following facts.
For GitHub Pages:
- You have only one website per repository
- You should have a separate branch to serve the HTML reports
- The testing workflow would become more complicated as you should commit to the branch from it
- The repository might become too “storage killing” if you want to serve the whole history of the runs as the HTML pages contain images and videos (it’s not like storing them in specific storage like S3 or Minio)
For GitLab Pages:
- You have only one website per repository
- You can easily configure the Pages storage. It does not use the repository branches directly (like GitHub)
- Every time you deploy, you lose the previous Pages state (it is still in the storage by default though)
- The only way to deploy is by using artifacts, and again — it might become a “storage killer” if you want to save the whole history of test runs as you will have duplicated reports data in both Artifacts, Cache, and the Pages (there isn’t an easy way to persist the data)
To wrap it up — both Pages functionalities are great if you want to serve only the last run of the tests, but if you want to persist all runs and serve them — it’s better to use something else.
Final Words
As I’ve mentioned before, test performance optimization should be an essential part of the whole development process. It should even be part of the main end-user code development — if your mindset points in the direction of “well, if I develop that function/component in this way, how can I test it?”, you probably have already made the most important step of the optimization journey. That’s why I like the idea of test-driven development. That’s why having the tests always in mind when developing is fundamental — all of these additional speed/efficiency (CI/CD) improvements are something additional.
Hope you enjoyed the information in the article!
Happy Coding!
And…Happy Testing!
Originally published on the MTR Design company website.
MTR Design is a Bulgarian web development company with strong expertise in Python, PHP and Javascript/Typescript, and with more than 15 years of experience working with digital agencies, tech startups and large corporations from all over the world. We are always open for new projects and cooperations, so if there are any projects we can help with, please react out (use the contact form on our website, or email us at office@mtr-design.com) and we will be happy to discuss them.
