Test automation is

  • complex and expensive,
  • maintenance-intensive,
  • unreliable and unstable,
  • can only be mastered by specialists in the long term.

Are you firmly convinced of this? Then you can stop reading now!
But you could also find out how simple, fast and manageable it can all be …

In modern software quality assurance, test automation is an essential part of the development process. With the increasing complexity of software applications, the need for efficient test methods is becoming ever more urgent. Test automation has proven to be an effective solution for accelerating test processes and ensuring software quality, particularly in regression testing. Test automation is therefore actually indispensable.

When talking about test automation, one is almost immediately confronted with its technology:

  • What technology does the system to be tested use?
  • How can the objects, events or signals be recognized, manipulated and read out?
  • What problems are there to overcome with one interface or another?
  • Which tools and which programming languages are used?

Basic methods and sensible procedures for setting up test automation and creating the test specification for tests to be automated are only discussed to a lesser extent and are of no interest to most specialists. However, they are the decisive key that helps to overcome the problems listed above. When used correctly, low-code or even no-code test automation is achievable, which makes it possible to specify tests independently of the technology required for this and to answer the questions posed above almost incidentally.

What is low-code test automation?

Low-code test automation refers to the creation and management of automated tests with minimal programming effort. This is achieved through the use of pre-configured modules that allow tests to be created, for example, through drag-and-drop and simple configuration. This makes it possible for people without programming knowledge to create and maintain tests that can be executed automatically. At the same time, collaboration between all those involved in development, testing and specialist areas improves considerably, as everyone can remain in their respective domains.

Keyword-Driven Testing (KDT) is the method that generates and uses the necessary building blocks. Tests are created from predefined “keywords”. Each keyword represents a specific action or a group of actions that are to be executed in a test case. A keyword is stored centrally in a keyword library and can be called from there in many test cases. Analogous to programming, a keyword is very similar to a function or method, as it also represents an encapsulated functionality. When a keyword is changed, all places where this keyword is used are automatically changed.

This method offers several advantages:

Reusability: Keywords can be reused in multiple tests, which makes test development more efficient.

Readability: The use of keywords makes tests more readable and easier to understand, even for people without in-depth programming knowledge.

Maintainability: Changes to a keyword only need to be made once, which makes test maintenance easier.

These advantages are further enhanced if keywords call up other keywords and thus more complex processes or technical requirements can be abstracted.

KDT is therefore used to define comprehensible test cases in a clear and structured manner. This includes the specification of the test conditions, the necessary test data, the expected results and the description of the test steps. These purely technical specifications are technology- and platform-independent.

Where does the implementation come from if no programming is required?

Test automation must be linked to the functional level so that automated execution of the test cases is possible. The prerequisite is that a piece of code exists for each keyword that is no longer composed, which implements the exact functionality of the keyword in the desired technology. How can we avoid the need for intensive programming at this point, even though the goal is low-code or even no-code automation? If we look at an interface, such as a web GUI, it can be operated with a finite number of small actions. Such actions are, for example, clicking on a button or expanding a menu. If you define a keyword for each of these actions, you get a keyword library with all the low-level keywords for operating a web GUI. If these low-level keywords are called up in the correct order, even the most complex processes can be mapped as test cases. If the appropriate code exists for each of these low-level keywords and the code of the individual keywords is called in the sequence defined in the test cases, the test cases can be executed automatically. This is where the Robot Framework test automation framework comes into play, in which KDT is used for specification.

The Robot Framework community provides ready-made libraries of the associated low-level keywords for many technologies. For example, there is the “BrowserLibrary” for the web GUI, which provides such a ready-made implementation based on PlayWright. These low-level keywords can be used and executed via the Robot Framework. However, if you were to specify test cases using only the low-level keywords, this would work, but the result would be long, incomprehensible test sequences that are difficult to change. For this reason, low-level keywords must be combined into more complex, more technical and therefore more comprehensible keywords. These keywords represent larger parts of a workflow, for example logging into an application or creating an order. These combined keywords are used to create the technical specifications described above, making them much easier to read and understand. In the end, there are two to three levels of keywords that map very comprehensible and easily customizable test cases at their top level.

An example to illustrate this: A new customer is to be created in the system.

Three low-level keywords are available in the browser library of the Robot Framework:
Opening a URL in the browser, clicking on an element, entering a text, whose Robot Framework code looks like this:

*** Keywords ***

Open Browser

    New Page  ${URL}

Click Element

    Click  ${locator}

Input Text

    Fill Text  ${locator}  ${text}

From this, two new keywords “Login” and “Create new customer” can now be put together, to which the necessary arguments for use with the low-level keywords are transferred:

*** Keywords ***

Login

    [Arguments]  ${username}  ${password}

    New Page  ${LOGIN_URL}

    Fill Text  id=username  ${username}

    Fill Text  id=password  ${password}

    Click  id=login-button

Lege Neuen Kunden An

    [Arguments]  ${customer_name}  ${contact_info}

    Click  id=new-customer-button

    Fill Text  id=customer-name  ${customer_name}

    Fill Text  id=contact-info  ${contact_info}

    Click  id=save-button

The creation of a new customer can be specified from these two keywords:

*** Test Cases ***

Create new customer

Login admin admin_password

Create New Customer To John Doe john@example.com

A comprehensible and easy-to-read sequence of events has now been created across several levels:

This sequence can be started by both sides at the same time. As a rule, the test sequences “meet” between the second and third step.

In principle, three layers of the test specification are mapped here:

  1. Business layer: Mapping the business processes with business keywords, levels 1 and 2 in the picture above.
  2. Technical layer: Mapping the individual actions of a system under test from technical keywords in level 3
  3. Technological layer: Mapping the technology of a system under test as a keyword library in level 4.

As shown in the example, the test cases of the functional layer can also be specified in the Robot Framework. This works well, but has its limits when a certain number of test cases and their data are involved, or when requirements or errors are to be linked to test cases. It also requires that the test cases are also specified in a development environment, which is usually difficult for people without programming knowledge.

It has proven to be a good idea to switch to a test management environment at the functional level, which also provides the Keyword Driven Test method for specifying test cases and at the same time offers the most important options of a development environment. This also has great advantages in the selection and planning of test cases and enables requirements and defects to be linked to test cases, thus creating traceability from requirements to test specifications and test results to defects without having to manually enter such information into the definition of test cases within a development environment.

The prerequisite for this is that the definition of the technical keywords is imported into the test management specification environment and repeatedly synchronized. The basic principles of a development environment must also be fulfilled, such as ensuring that it is always clear how a keyword is defined or where it is called and that support is available for refactoring.

This is the case for the test management and test design system TestBench. The test sequence of our example then looks as follows:

For the parameters, the arguments of the keywords were imported from the Robot Framework as data types, in which the necessary values are then stored. Instead of the fixed values for the parameters, these data types can also be inserted, which then act like variables:

These variables are supplied with values via a data table:

The result is a data-driven test that can be supplied with several data combinations, for each of which the test sequence is run during test execution. Before the actual execution of the tests from TestBench, the same Robot Framework code is generated as was specified in the Robot Framework at the end of our example.

The architecture of the solution described so far can be represented schematically as two components:

The three layers of our test specification can be clearly seen here.

The execution of the tests

To execute the tests, the low-level keywords must be executed based on the sequence defined at the functional level, which is ensured by the on-the-fly generation of the Robot Framework code. In addition, the test results must be logged at all levels during the test execution, the error cases must be handled and additional information such as log files or screenshots must be generated and attached.

A third component is therefore required to take over these tasks:

These three components form the generic test automation architecture [1] (gTAA), which has proven to be very successful as a best practice:

  1. Specification level: This is where the test specification described above is created using the keywords.
  2. Test execution level: The keywords are executed in the order and with the data from the specification level. Test results are determined and logged.
  3. Technical adaptation: For each keyword that is no longer combined, there is an implementation suitable for the required technology, such as a web GUI.

These three levels ensure that each keyword is used in its correct functional context in the test specification and in its correct technical context in the test execution.

This also provides a clear distribution of tasks and roles in the creation and use of test automation:

  • Test specifiers deal exclusively with the technical content of the tests, the test steps and the test data, but do not need any knowledge of the implementation code.
  • Test automation specialists are responsible for the implementation of the individual keywords, but do not need to have any knowledge of the functional relationships and can concentrate entirely on solving the technical problems.

This means that there are clear points for both roles where changes must be made without necessarily affecting the other role and, if they do, there is a clear interface. The test execution level connects the other two levels at test execution runtime and passes the test results upwards.

A result is determined for each keyword that is no longer combined. These results are cumulated via the composite keywords to form the overall result of a test case or even an entire test suite.

The combination of TestBench and Robot Framework has proven to be a very good implementation of the gTAA. The three levels are divided as follows:

Both TestBench and Robot Framework are capable of Keyword Driven and Data Driven Testing, so there is no method break between the systems. Both ideally divide up the levels of the generic test automation architecture so that there is very little functional overlap. The several hundred low-level libraries of the Robot Framework in particular ensure that new technologies can be implemented quickly. For example, extending the web GUI tests to include tests for a mobile app is not a major problem, as several low-level keyword libraries can be used simultaneously. This means that comprehensive end-to-end tests, e.g. from SAP via browsers and apps to REST APIs and database content, are no longer a problem.

The outlined solution is also able to map non-automated, manual tests only at the functional specification level and only decide later whether to automate them.

Conclusion

Low-code test automation with Keyword Driven Testing and a generic test automation architecture offers numerous advantages for software quality assurance. By reducing programming effort, improving maintainability and promoting collaboration, the efficiency and quality of the test process can be significantly increased. Tools such as TestBench in combination with the Robot Framework support these methods and offer comprehensive functions for planning, implementing, executing and analyzing tests. The easy integration of test automation into development landscapes with requirements and error management and often necessary reporting to fulfill various verification obligations is another important advantage of this combination.

By using these modern testing methods and tools, software teams can ensure that their products meet the highest quality standards and that they can react quickly to changes while minimizing testing costs and effort.

[1] Certified Tester Advanced Level Syllabus – Test Automation Engineer: The Generic Test Automation Architecture und ISO/IEC/IEEE 29119-5, Figure 8 — Components of a Keyword Driven Testing framework for automated test execution