Vaccine production had to be ramped up quickly for a globally active company. Goldfuß engineering GmbH and SIMON IBV GmbH developed a robot-based solution for the process step involving loading and unloading trolleys with vials containing the vaccine. The process is fully automated thanks to the robots’ ability to work autonomously through the use of machine vision.

Accelerated vaccine production thanks to automation

Vaccine production had to be ramped up quickly for a globally active company. Goldfuß engineering GmbH and SIMON IBV GmbH developed a robot-based solution for the process step involving loading and unloading trolleys with vials containing the vaccine. The process is fully automated thanks to the robots' ability to work autonomously through the use of machine vision.

Although it is less in the public eye, the establishment of efficient production and logistics is very important in vaccine production. Very large production capacities had to be set up in a very short period of time to be able to quickly produce large volumes of vaccine against a globally circulating virus. This meant that automation was essential.

One stage of the process involves the use of a robot to place vials filled with vaccine, also known as injectable ampoules, from a conveyor belt into drawers and to remove them again later. Goldfuß engineering GmbH developed the overall concept, which consists of several trolleys with drawers, robot cells, and conveyor systems. SIMON IBV GmbH was responsible for the development of the optical 3D system, which enables the robot to grasp the vials independently and without damage. The MVTec HALCON machine vision software is used for this purpose.

The trolleys, which hold around 10,000 vials, serve as a buffer and transfer system between filling, quality control, and packaging. Goldfuß engineering GmbH, based in Balingen, Baden-Württemberg, specializes in the development and production of robotic loading systems for packaging machines and custom solutions for laboratory automation. Siegfried Hameln, a global contract and development manufacturer of active pharmaceutical ingredients and finished products, commissioned the company to develop an automated process for buffering vials. The company Siegfried Hameln, in turn, was commissioned by the vaccine developer to produce the active ingredient. SIMON IBV GmbH was brought on board to provide the vision component of the robot cell. Based in Bayreuth in northern Bavaria, the company has cross-industry expertise in 2D and 3D optical testing and measuring systems.

Robot loads and unloads trolleys with vaccine
Goldfuß engineering developed a new robot cell for the step of loading and unloading the trolleys. Stephan Trunk from Goldfuß engineering explains: "This kind of automation solution has never been available on the market before. In comparable applications, the vials are manually loaded into trays or boxes after filling, manually stacked on pallets or in Kanban trolleys, and subsequently depalletized again. This means that several employees are usually tied up with non-value-added activities. The new, automated process speeds things up, in part because employees can focus on more demanding tasks."

The trolleys used feature nine drawers, with each drawer capable of holding 24 rows of 46 vials each. In total, one trolley can therefore hold almost 10,000 vials. By buffering such a large quantity, it is possible to flexibly control the capacity utilization of the packaging system. The trolleys offer the additional advantage that random samples can be easily removed from them and fed back into the packaging process after quality assurance approval. Since the trolleys are mobile, they can be parked in cooling chambers so that the cold chain is not interrupted. During implementation, one particular challenge involved the deviations and different positions in which the trolleys are positioned, for instance, while the robots still have to work autonomously. Employees manually position the trolleys on the system, which means that the exact position always varies. In addition, the weight of the trolleys and therefore the position of the drawers is constantly changing during loading and unloading. In order to maintain the required positional accuracy so that the vials are always grasped correctly, it is necessary to constantly recalculate the exact position for each row.

In an almost fully automated production process, this task can only be accomplished by a 3D vision system based on machine vision. Other technologies would not have been able to meet the requirements. The use of sensors requires several scanning positions, which would extend the robot cycle considerably, making it impossible to achieve the desired number of cycles. The use of a 2D camera on the gripper would also be problematic because in addition to the extra weight, it would also be necessary to lay sensitive cables in such a way that they can withstand the robot's movements over the long term.

"The 3D cameras with a working distance of approximately 120 cm to the top drawer and 180 cm to the bottom drawer are extremely precise and still offer the robot a sufficient working range for collision-free handling. In combination with powerful machine vision software, however, it has been possible to meet all the requirements for this application," explains Daniel Simon, authorized representative and responsible for technical sales at SIMON IBV.

3D vision system facilitates a fully automated process
In addition to the robot, the hardware components of the robot cell include high-resolution 3D cameras using the stereometric method with pattern projection and industrial computer technology with fast processors for PC-based evaluation. MVTec HALCON was selected for the machine vision system. HALCON is the comprehensive standard machine vision software. On the one hand, the software offers a wide range of powerful machine vision methods. On the other hand, because the software is hardware-independent, it can be used flexibly with different cameras, for example. SIMAVIS® from SIMON IBV is utilized as the user interface for visualization and for the simple operation of the system. Here is how the process works: First, an employee pushes a trolley with the vials into one of two possible positions. A 3D camera within the robot cell pinpoints the trolley and checks the status of the drawers, i.e. whether they are open or closed. The programmable logic controller (PLC) contains information on whether the trolley currently being processed is to be loaded or unloaded, which drawer is to be opened, and how many vials are already there and where.

To begin with, the 3D camera always takes a picture of the drawer to be opened. The MVTec HALCON machine vision software uses this image to create a coordinate system and shares it with the robot. This enables the robot to open the drawer. In the next step, the 3D camera takes a picture of the contents of the drawer. This is used to determine how many vials are stored there and where exactly they are located. In addition to the number, the system also checks for errors, such as whether individual ampoules are upright or have tipped over and therefore cannot be picked up by the robot. The robot's gripper is equipped to handle 46 vials at a time. Once a drawer has been completely unloaded, the 3D camera determines the position of the drawer handle so that the robot can close it again. This process is then repeated with the remaining eight drawers until the trolley has been completely unloaded.

Machine vision software handles different tasks
Implementing this type of solution, in which the robot is largely autonomous, presents a number of challenges. Stephan Trunk from Goldfuß engineering describes it this way: "The development of the robot cell took place under tremendous time pressure. The demand for the vaccine had to be met as quickly as possible, which also served as an intrinsic motivation for our employees."

The main challenge was to achieve high precision. The position of the vials must be detected with an accuracy of 0.1 millimeters. And with a working area of 800 x 600 mm and a depth of 600 mm, this is no easy task. The extremely valuable contents of the vials make this requirement even more critical. The vials must not be damaged under any circumstances. However, loading and unloading must be done quickly, in part to facilitate the rapid vaccine delivery. "The vision system also faced two special challenges. On the one hand, it was necessary to work with different materials. The glass of the vials and the metal, such as that of the trolleys, have transparent or reflective surfaces and are therefore difficult to detect. On the other hand, the system will only work if the robots can operate autonomously in three-dimensional space. To do this, however, the machine vision software must also include powerful 3D vision technologies. There are not many machine vision software products that achieve the necessary performance and robustness in this regard," says Daniel Simon. The companies decided to go with the MVTec HALCON machine vision software. "Based on our many years of experience with HALCON, we knew that the software has an extensive library with many extremely powerful methods," explains Daniel Simon.

Various machine vision technologies are used in the robot cell that was developed. One of the technological requirements in the system is the so-called hand-eye calibration. This technology is essential for any application in which cameras work in conjunction with robots. During this process, the robot's coordinate system and the camera's coordinate system are synchronized. This makes it possible to match the movements of the robot to the images from the camera with the utmost precision. HALCON's hand-eye calibration delivers extremely high accuracy in determining the relative position between the camera and the robot. This makes it possible to determine the exact positions and orientations of the vials in relation to the robot. This calibration is the basis for all further machine vision applications with HALCON that are necessary for this application. At the same time, the system additionally utilizes the stereo vision 3D vision technology, which is also included in HALCON. This technology is designed for 3D reconstruction and is particularly useful for large or medium-sized structured objects. In addition, this technology facilitates quality control or position detection of three-dimensional objects. The technology also calculates 3D coordinates on object surfaces. This can be done with one or with multiple cameras. Stereo vision is particularly well suited for the precise measurement of elevations. One feature within the technology is multigrid stereo, an advanced method of interpolating the 3D data in homogeneous parts of the image. This method yields greater accuracy for small objects.

New robot cell in operation since July 2021
"We have managed to develop a completely new robot cell and get it up and running with process stability in just six months. We are proud to have contributed to accelerating the production of a vaccine against a dangerous virus," says Stephan Trunk of Goldfuß engineering. The system was put into operation in July 2021. The strict requirements—in terms of speed and precision, for example—were also met. "This project illustrates all of the potential and possibilities offered by machine vision. Thanks to the successful implementation, we are very motivated to automate other challenging tasks. There are a number of promising projects already underway with Goldfuß engineering," says Daniel Simon.


About MVTec Software GmbH
MVTec is a leading manufacturer of standard software for machine vision. MVTec products are used in a wide range of industries, such as semiconductor and electronics manufacturing, battery production, agriculture and food, as well as logistics. They enable applications like surface inspection, optical quality control, robot guidance, identification, measurement, classification, and more. By providing modern technologies such as 3D vision, deep learning, and embedded vision, software by MVTec also enables new automation solutions for the Industrial Internet of Things aka Industry 4.0. With locations in Germany, the USA, France, Benelux, China, and Taiwan, as well as an established network of international distributors, MVTec is represented in more than 35 countries worldwide. www.mvtec.com