Automated Payment Terminal Testing: Achieving continuous integration for systems that seemed impossible to virtualize.
Article two of two exploring our path to automated testing success.
In the first article of this series, we already looked at the background and the objective of developing test automation for payment terminals. In the following, we will take you on our journey there.
Method one: Virtualization
In order to automate the testing process we first considered virtualizing the payment terminal. An eftpos card reader is formed by a series of hardware components which work together to allow payments.
A simplified depiction of the structure inside a payment terminal can be seen in Figure 1 below. The important components affecting testing is the Pin Entry Device (PED) and the Secure Model, which controls the PED.
In order to automate testing, all input into the terminal has to be automated and output has to be verified. The goal was to develop a virtual set up of the payment application and run a simulation behind the hardware abstraction layer.
While this was the goal in theory, practically the challenges were substantial:
- Card readers were hard to simulate: due to the logic in their chips and the security measures implemented by the card issuers. These would be difficult to duplicate in the virtual set-up.
- The Secure Model was hard to simulate: As the black box which handles all security, this would be a challenge to implement.
- No real end-to-end testing is achieved: When testing a hardware product, not everything can be virtualized. Memory management, file locking and related functionality are some examples.
Method two: Robotics
Given the limitations from pursuing virtualization, we then explored robotics. Our journey took us through various types and stages:
- A lower cost single-arm robot. While, with this type it was easy to automate a simple contactless transaction, doing the same for a contact-based transaction proved more difficult. We did manage to automate inserting physical cards, however, finding a method to press the buttons was not possible.
- Other high-quality robots: Some were quite costly and there was no way of knowing how many we’d need to try till we’d find one which could carry out the tasks we needed.
- Creating our own robot: We wanted something less versatile and more reliable. Our research brought us to 3D printing. With this technology we were able to create finger pressing buttons and a more flexible card arm. This worked great, except for one limitation - only one card could be tested by the robot at one time.
Testing with multiple cards
Cards from different brands behave differently due to the applications and profiles contained on their chips. This meant that testing various cards was important in order to allow customers to purchase with their preferred card.
Since the robot could not hold multiple cards we created another hardware (see figure 2.) which would allow us to input multiple cards and output a single card. We created a Card Multiplexer that can hold up to 16 cards that can be transferred onto a card probe. The multiplexer is stackable to 16 multiplexers, meaning it can hold up to 256 cards.
We created the same device for contactless cards as well.
Allowing for calibration
The next challenge was to allow for the different physical forms of the terminals themselves. Different shapes and button placement meant a method for automation was needed here as well.
We solved this by mapping a layout for each terminal model within the test script.
From simple to sophisticated
We now had a fully automated solution with which we could test simple cases. The next step was adding more complexity to increase testing coverage. We added:
● Drivers simulating POS systems,
● A device to simulate magstripe cards,
● Terminal display validation.
Our software also needed further development so it could be easily implemented with more complex testing. We therefore redesigned our architecture. Together with a partner already using our robots and hardware to test their terminals for large financial entities, we added a simple API to connect our software with third-party systems.
We now had it ready to implement with client systems.
Allowing for future extensions
Looking towards the future, an important element would be the ability to extend its testing capability to cover new options such as POS simulators or hardware. We therefore engineered it in a way where it would be more generic while simultaneously adding command validation to ensure security at every step.
The result was a solution that was extendable and easy to use from the tester’s perspective. At this stage we started to convert most of our manual test cases to automated cases. The first thing we did was to run 5000 test transactions in a row on each terminal. What we discovered was surprising, none of our terminals passed the test and on average we found over 70 bugs on each model. No complex test script was required, nor did we think about test coverage at all. Just a simple test case allowed us to discover many low-level issues (memory leaks, hardware driver crashes, file locking issues etc.) that we had not been able to spot and solve before. Solving these issues led to a much more stable application. This showed us an important aspect of automation: Besides the well-established benefits of automation such as speed, accuracy, repeatability, and scalability, the ability to stress a terminal over a long time without interruption proved very valuable.
The journey continues
With a stable testing system running automatically, the next challenge was releasing it to payment terminals. With high protection and barriers to installing software updates, our system awaited the next stage.
We started the process of continuous integration with high goals and are continually improving our solution. The benefits from our system clearly outweigh the effort and costs that went into its development. Automation is certainly a necessity.
Today, we test more and faster than ever and are able to improve product quality. This keeps our clients happy and confident with the solutions we provide.
- automated testing
- machine automation
- payment methods
- payment terminal
- payment testing
- test automation