This work
continues and further develops our research [1-5] on the development of
software and laser technological equipment for the practical implementation of
the Selective Laser Melting (SLM) technology in domestic enterprises. The SLM
technology is based on the process of manufacturing products by selectively
melting metallic powder layers using laser radiation. In the first stage, a
layer of metallic powder of a specified size is evenly applied to the working
surface, and in the second stage, the laser beam locally heats and melts the
material according to predefined data from the 3D model layer. The previously
developed software complex [1-5], which includes the LAMachineVision software
platform created for visualizing the SLM process [2], has successfully solved
the multi-parameter task of selecting equipment operation parameters for
selective laser melting of metallic powders [5]. In the practical industrial
realization of the SLM technology, we encountered a problem related to the
duration of the operation for selecting technological parameters. At each
stage, due to incorrect selection of technological parameters, defects may
arise. After the powder is applied, longitudinal stripes orthogonal to one of
the axes of the working surface can appear during leveling, depending on the
leveling direction, resulting in damage to the leveling device.
To reduce the duration of selecting
technological parameters while considering these factors, a defect recognition
software module named FlexADD (Additive Defect Detector) was developed to
visualize, detect leveling defects, and perform layer-by-layer photo-documentation
of the process. The program code was written in C++ 17 using the QT 5.15
software platform. The MvCameraSDK library was used to work with the industrial
Hikrobot camera, and the OpenCV library was utilized to simplify the
implementation of recognition algorithms and frame processing. The built-in QT
mechanisms for working with the OpenGL scene were used for visualizing the
video stream from the camera and leveling defects. The FlexADD defect
recognition software module supports simultaneous operation with multiple video
cameras and controls physical lighting modules located in the working camera.
Lighting control is achieved through communication with a specifically
developed controller via the HID standard. Due to the cross-platform architecture
of the technologies used, the program can run on both Windows and Linux
operating systems. The created FlexADD defect recognition software module
utilizes the StateMachine architectural concept (Fig. 1), which allows for the
division of states and transitions between them. This approach uses to manage
complex processes or algorithms. StateMachine tracks the current state of the
FlexADD program and defines possible next states and transitions between them.
This approach makes the code more understandable, modular, and easier to
support and further develop, as each state and its behavior are described
separately from others. Additionally, using StateMachine helps avoid errors in
the program related to incorrect execution order of operations or improper state
transitions. Text logging of program actions plays an important role in
industrial development, facilitated by the nature of the StateMachine concept.
Fig 1. Visualization of
the StateMachine concept using the implemented SLM process sequence as an
example.
For stable defect
recognition, it is not always enough to rely solely on software and
mathematical tools; sometimes it is necessary to influence the intensity,
color, and direction of illumination in the area of the working surface. To
address this issue in the configuration for which the development was
conducted, there are several lighting modules that can be controlled from the
FlexADD defect recognition program. In addition to controlling the state of a
specific module, brightness adjustment is also available (Fig. 2). The lighting
configuration is linked to the currently selected camera, so when switching
cameras, the lighting parameters also change. In the working camera, in
addition to the additional lighting modules, there is also main light. Through
experiments, it was found that the main light has a negative impact on the
quality of recognition by causing overexposure of the working area. To eliminate
the negative effects of the main light and enhance the contrast of these
defects, a software control of its state was added.
Fig. 2. Backlight control
interface.
This
solution provides the ability to physically influence the contrast of defects
in the video stream by adjusting the angle at which light falls on the working
surface, thereby increasing the effectiveness of software automation methods.
The design of the
laser setup being developed does not allow for the camera to be positioned
directly above the working surface, however, such positioning is necessary for
accurate determination of the coordinates of detected defects. To compensate
for the non-axis positioning of the camera with the working surface, the
software includes a perspective compensation function that allows the surface
to be viewed from the desired angle (Fig. 3). During the calibration stage of
the software, the operator must specify the height and width of the area marked
in Fig. 3 in millimeters. By having the dimensions of the area in pixels and
the user-entered values of the area dimensions, we determine a conversion
coefficient that is then used to transform coordinates. This approach involves
cropping unnecessary areas for processing, thereby further reducing the size of
the analyzed image and increasing the speed of the algorithms.
Fig. 3. Software
perspective correction.
Defect recognition
occurs at a specific moment in the process, specifically after leveling when
the powder should evenly cover the working surface. The sequence of recognition
(Fig. 4) is based on the StateMachine concept. Initially, the system is in a
waiting state for a request to search for defects (stWait), and the connection
via TCP protocol with the control program of the 3D printing setup has been
established to provide information about the current printed layer. Upon
receiving a command to search for defects, the defect recognition program
FlexADD sequentially captures N images with N cameras (stGetCameraFrames), with
consecutive processing required to switch lighting configurations. The command
for defect detection also contains an array of detector names, the results of
which should be provided as a response. After capturing images, the specified
filters and recognition algorithms from the protocol (stAnalyzeFrames) are
applied, and upon completion, the detected defects are sent to the control
program (stSendResult).
Fig. 4. Visualization of
the sequence of states of the recognition state machine.
Therefore, a
methodology has been developed for successful defect detection, in which the
operator needs to programmatically adjust the perspective, configure the
lighting, and determine the video stream processing pipeline.
To simplify the
analysis of the causes of defects in the FlexADD defect recognition software
block, a mechanism for layered photo fixation was introduced. When a command is
received to search for defects, this mechanism is activated. At the
stGetCameraFrames stage (Fig. 4), in addition to obtaining images for further defect
detection, these images are saved in memory. In total, with each of the N
cameras, two or three images of the following types will be saved:
•
Original frame obtained from the camera.
•
Frame with corrected perspective.
•
Frame with corrected perspective and
defects visualized on it. (This option is not used if there are no defects in
the frame).
Creating a detail
using SLM technology can contain a large number of layers, and consequently a
large number of their saved images. To systematize the photos, a file system
has been created. Thus, when the SLM process is launched in the FlexADD
program, a special command is received to create a directory containing the
project name from the 3D printing process management program. In addition to
the project name, the date and time of the process launch are added to
distinguish the images made during the production of one project at different
times. When the first images are saved in the previously created directory, N
folders are created (based on the number of cameras used in the system), with
names corresponding to the camera name in the system, set by the user during
the initial personalization of the program. The images saved also contain the
time of their acquisition (as it may differ from the actual time of saving),
the type of image being saved, and the layer number received with the defect
recognition command. After applying a structural approach to layered photo
fixation, the process of analyzing the causes of defects has been significantly
simplified. Progress is especially noticeable compared to the previously used
video fixation, where the technician had to search for the moment of defect
formation in a hours-long video. Moreover, even after identifying the necessary
frame, they did not have information on which specific layer of the 3D model
the defect occurred.
The diversity of possible powder
materials used in selective laser melting technology dictates the need to
provide the operator with the ability to adjust not only the parameters of the
detector itself, but also the types and order of image filters used in
processing the input image. In order to provide the user with this capability,
a function for customizable video stream processing conveyor was introduced
(Fig. 5), following the methodology proposed in [6].
Fig. 5.
Video
stream processing pipeline.
This mechanism
operates with two types of primitives. The first type is detectors,
implementing algorithms for defect detection on the incoming frame. The second
type is filters, performing preprocessing of the frame before sending it to the
detector. The FlexADD defect recognition program implements flexible
customization of filters and detectors, allowing interactive visualization of
the impact of parameter changes on the frame state (Fig. 6). The arrangement
order of filters in the conveyor plays an important role in tuning, as each
filter receives an image processed by the previous filter (see Fig. 5).
Fig. 6. An example of
setting a threshold filter.
Through such
processing of the input frame, it is possible to neutralize the difference in
physical properties of the powder affecting its display on the camera.
As already
mentioned, stripes can appear as a result of a damaged leveling tool passing
over the working surface. To detect the lines left by the leveling tool, a
popular method based on Hough transformations [6–8] was applied, let's delve a
bit deeper into the principles of its operation with primitives like
"straight lines".
A straight line in
the image space can be expressed with two variables. For example, in the Cartesian
coordinate system parameters: (m, b), or in the polar coordinate system
parameters: (r,
θ).
For Hough
transformations, the linear equation of a line in polar coordinates can be
written as:
For each point
(x0, y0), we can determine a family of lines passing through this point as
follows:
This means that
each pair (r
θ,
θ)
represents each line passing through (x0, y0). If we build a family of lines
passing through a given point (x0, y0), we obtain a sinusoid. If we perform the
same operation for all image points and find that curves from two different
points intersect in the
(θ,
r)
plane (Fig. 7), it means that both points belong to the same line.
Fig. 7. Three points
belonging to the same line.
After traversing
the image with this algorithm, the output is the equation of the line in polar
coordinates, thanks to which the visualization of the defect on the camera
frame is implemented (Fig. 8).
Fig. 8. Detected defects.
Additive SLM
technology is primarily used to create products with complex geometry, which
are either much more difficult or impossible to produce using traditional
machining methods. To achieve the required mechanical parameters of the final
product, it is necessary to fine-tune the 3D printing process by adjusting the
parameters of the technological process [5], which requires significant time
investment.
The introduction
of our developed defect recognition software module, FlexADD, into the existing
structure of the software complex for implementing SLM technology has
significantly reduced the time required for adjusting equipment parameters
during the 3D printing process. This was achieved by identifying defects at any
stage of 3D printing, allowing the immediate termination of the lengthy
technological process of creating a defective product. Early defect detection
and process stoppage enabled savings on metal powder, thus reducing the
production costs of parts.
The application of
the FlexADD defect recognition software module in practice resulted in a more
complex equipment setup for 3D printing, as the operator of the laser
technological system needs to adjust a number of parameters: filters,
detectors, as well as lighting configuration and perspective. However, it is
worth noting that the configuration of our software module is not tailored to
specific products but to the metal powder used in printing, which is convenient
for serial production.
The implementation
of the FlexADD defect recognition software module has led to a reduction in the
overall time spent on adjusting technological parameters of SLM technology and
a reduction in the production costs of this process through metal powder
savings by halting the process in the event of defects. The analysis of the reasons
for defects has been simplified by integrating layer-by-layer photofixation
into the software complex, indicating the time and layer number of the saved
image.
The authors
express their gratitude to the management of the "Lasers and Equipment
TM" group of companies for their assistance in providing material and
technical support for conducting experimental research and modeling the
considered process.
1. Molotkov A.A., Tretiyakova O.N. On possible approaches to visualizing the process of selective laser melting // Scientific Visualization. 2019. Vol 11. ¹ 4. PP. 1-12.
2. Molotkov A.A., Tretiyakova O.N., Tuzhilin D. N. About development and application of a software platform for machine vision for various laser technologies // Scientific Visualization 2022. ¹5. Ð.P.108-118.
3. Molotkov A.A., Tretiyakova O.N. Application of machine vision and mathematical modeling methods for the development of electronic device creation technologies // Instruments. 2022. ¹ 4. P.55-58.
4. Molotkov A. A., Saprykin D.L., Tretiyakova O. N., Tuzhilin D. N. Development of a complex of programs for the creation of industrial laser technological equipment // Instruments. 2022. ¹ 5. P.15-22.
5. Molotkov A.A., Tretiyakova O.N. Work out of technological modes in the creation of SLM technology. // Instruments. 2023. ¹ 8. P. 44-47.
6. Kudrina M.A. The use of Hough transformation for detecting straight lines and circles in an image // Proceedings of the Samara Scientific Center of the Russian Academy of Sciences, 2014.
7. Kudrina M.A., Kiryakov S.A., Kudrin K.A., Degtyareva O.A. Vectorization of raster images using Hough transformation // "Reliability and Quality". 2013. ¹1. P. 337-340.
8. I. A. Matveev, V. I. Tsurkov, N. N. Chinaev, “Forming the pupil circle using the Hough transformation for the boundaries of connected components”, Autom. Remote Control. 2015, ¹ 11, 104–117; Autom. Remote Control, 76:11 (2015), 1988–1999.
