Test Automation

1. What is Test Automation?
2. Challenges in automating tests
3. How does Test Automation works with TestBench?
4. Where does the data for the test execution come from?
5. How much effort is involved in Keyword Driven Testing?
6. Is No-Code test automation possible?
7. Can it be tested across all technologies?
8. How can a CI/CD system be connected?
9. Can you evaluate test results without TestBench import?

In test automation, software tests are carried out automatically using special tools – so-called test automats – instead of testing manually. These tools simulate the interaction with the system to be tested by executing test cases implemented in scripts.

Before automated test execution, for example overnight, the tests must first be programmed. The often recommended “capture-and-replay” method, in which tests are executed manually and recorded as program code, usually leads to unreliable code that is difficult to maintain. Programming skills are therefore essential for robust test automation.

In order to make optimum use of test utomation, a test specification based on keywords is required (see Keyword Driven Testing). This method has decisive advantages for automation, such as maintainability, reusability and robustness. In TestBench, every keyword that does not call other keywords requires an implementation in the connected test automation. It is not necessary to create a separate implementation for each test case – one implementation per keyword is sufficient.

Test execution sequence with a wrapper script:

The test sequences composed of keywords determine the order of the test steps. To ensure that the test automat executes this sequence correctly, a wrapper script is used that is developed specifically for the respective test automat.

The wrapper script works as follows:

1. The tests to be executed are loaded.
2. The next test case to be executed is identified.
3. The next keyword in the test case is identified together with its parameter values.
4. The implementation of the keyword is called in the test automat.
5. The test automaton executes the associated program code.
6. The test results and additional information are processed and added to the test result log.
7. The process is repeated until all keywords of a test case and all test cases have been executed.
8. Finally, the test result log is completed and the test execution is ended.

Alternative procedure for special test automats

Not every test automat supports the execution of individual steps or wrapper scripts. In such cases, TestBench generates a specific program code for the test automat that enables test execution. The test results are then imported into TestBench and evaluated.

Integration of common test automats

Open source wrappers already exist for many well-known test automats. These can be easily adapted and extended. TestBench also offers a library that simplifies the interpretation of test instructions. As a result, a wrapper often consists of just a few lines of code.

Interface support

Test results can be imported either via TestBench REST API or manually. This is particularly advantageous for test automats that work in real time or in hardware-in-the-loop (HiL) environments.

(See also integration of the test framework Robot Framework).

TestBench supports a parameter interface via which values are transferred to keywords (see Keyword Driven Testing and Data Driven Testing). This means that no additional effort is required for the assignment of data.

Parameters can be:

• loaded automatically,
• added before the execution of a keyword,
• or transferred to the test result log after execution.

TestBench offers flexible solutions for dynamic data such as session IDs. This makes the management of test data much easier, even in complex or rapidly changing scenarios.

A common argument against Keyword Driven Testing is the initial effort involved in creating and managing keywords. In practice, however, it has been shown that this method pays off in the long term:

Reusability: once implemented, keywords can be used as often as required.

Minimal maintenance: changes only affect the keywords concerned and not entire test cases.

Comparative studies show that the effort required to automate entire test cases with keywords is significantly reduced. This keeps the tests clear and maintainable – even for large test projects.

Quelle: https://www.sigs-datacom.de/uploads/tx_dmjournals/brandes_OS_05_09.pdf

Yes, No-Code test automation is possible with TestBench. As programming is limited to the implementation of individual keywords, people without programming knowledge can also create tests. If already implemented keywords are used, the test cases can be executed automatically immediately.

The more keywords are implemented, the easier it is to create new tests. If new keywords need to be created or existing ones adapted, the workload remains manageable and can be delegated to automation experts.

This enables a clear division of tasks between testers with knowledge of the specialist domain of the test object and programmers without this specialist knowledge but with programming skills.

Domain tester

Know the business logic.
Create the technical test sequences.
Provide the appropriate data.
Have minimal programming knowledge.

Test automation experts

Know the technology of the test object.
Solve technical automation problems.
Have little knowledge of technical contexts.
Have good programming skills.

With Keyword Driven Testing, the test specification is independent of the underlying technology, which means that the tests remain flexible and customizable. The specific technology is only taken into account at the programming level of a keyword.

Example:

Within a test sequence, a change from PlayWright (for web interfaces) to Appium (for mobile apps) can take place. This requires a wrapper that supports several technologies and regulates communication with the corresponding test machines.

As such solutions can become complex, TestBench offers integration with the open source test framework Robot Framework, which supports central components of the test automation architecture. This framework:

• Supports cross-technology testing.
• Offers extensive functions for low-code automation.
• Significantly facilitates the implementation of keywords.

Together with TestBench, this enables efficient and flexible test automation across different technologies – an important component of a powerful test automation architecture.

More information: Integration of the test framework Robot Framework.

Yes, test results can be evaluated independently of a test management system. TestBench offers a wizard for executing manual tests.

This wizard makes it possible to analyze the logs of automated test runs offline. Huge amounts of data can be generated, especially during extensive tests, such as nightly or weekend runs. This data can be processed efficiently with TestBench.

Functions for analysis:

• Quickly jump to specific results, such as the next FAIL.
• Add comments or error descriptions to the test result logs.
• Import the extended logs into TestBench if required.

These functions make work easier even with millions of test points and ensure efficient evaluation – regardless of the TestBench environment.