Enhancing Pharmaceutical Inspection through Poisson Image Editing Techniques Christian O. Pérez Pérez Electrical Engineering Viktor Zaharov – adviser Electrical Engineering Department Polytechnic University of Puerto Rico Abstract  This research explores the application The recent years have witnessed a of Poisson image editing techniques for enhancing transformative shift in these industries, driven by the the dataset quality in the detection of defects within convergence of technology and healthcare. This pharmaceutical products. The focus is on addressing evolution has seen the emergence of image common defect types that compromise product processing models and deep learning algorithms as integrity and safety. By employing Poisson image potent tools for automating a variety of tasks that editing, the researcher aim to improve the accuracy glass was traditionally performed manually [2]. and efficiency of machine learning models used in Specifically, in the context of glass containers, these pharmaceutical quality control. The outcomes technological advancements have been leveraged to indicated substantial enhancements in detecting and develop classification models capable of analyzing classifying defects, thereby promising to elevate the visual data to determine the usability of these standards of pharmaceutical safety. This study not containers. By identifying defects that render glass only underscores the value of advanced image containers unfit for use, these models play a crucial editing in quality assurance processes, but also role in upholding the standards of patient care and encourages further exploration into its potential safety. Deep learning applications for across various aspects of pharmaceutical pharmaceutical packaging inspection have manufacturing and inspection. underscored the potential of these technologies to Key Terms  Data Augmentation, Image enhance quality assurance processes significantly [3] Blending, Pharmaceutical Vials, Quality Assurance. [4] [18]. Despite the promise and potential of these INTRODUCTION technologies, the journey towards developing accurate and reliable classification models is fraught In the realm of global healthcare, the with challenges. Central among these is the reliance pharmaceutical and medical device industries are on extensive datasets comprising both defect-free foundational pillars, tasked with the critical mission (good-quality) and defective samples—a of delivering safe and effective treatments to patients requirement that is both resource-intensive and worldwide. Central to this mission are glass contains costly [5]. Traditionally, these datasets are created in like vials and syringes, the primary vessels through laboratory settings, where defects are artificially which medical interventions—ranging from routine induced under controlled conditions. This process, vaccinations to complex treatments—are while necessary, is time-consuming and financially administered. The integrity, quality, and reliability burdensome, posing significant logistical challenges of these containers are not merely operational that can impede the pace of model development and concerns, but are of paramount importance for innovation. ensuring patient safety, preventing medication Considering the challenges mentioned above, errors, and avoiding contamination that could lead to there is a need to explore alternative methodologies adverse health outcomes. Studies have highlighted that can circumvent the limitations posed by the the crucial role of packaging integrity in traditional approach to dataset creation. Among the pharmaceuticals, emphasizing the need for stringent most promising of these alternatives is the quality control measures to prevent contamination exploration of data augmentation techniques, and ensure patient safety [1]. particularly those that can synthetically enhance the necessity for robust image classification models that size and variability of datasets [6]. Such techniques can effectively handle such variability and hold the potential to not only mitigate the reliance on imbalance. costly physical samples but also to accelerate the Robustness and Generalization development of robust classification models capable of meeting the stringent demands of the Is crucial to achieving robust performance pharmaceutical and medical device industries. across different types of glass containers and This research project seeks to contribute to this ensuring that these methods generalize well to evolving landscape by exploring the application of unseen data. The real-world application of these Poisson image editing—a technique that operates by models demands a high degree of accuracy and seamlessly integrating image gradients—to enhance reliability, especially when dealing with a variety of the identification and classification of defective and defects that can compromise the safety and efficacy non-defective glass containers [7]. By delving into of pharmaceutical products. For this reason, the USP the mathematical foundations of Poisson image Chapter <1790>, requires a 100% inspection of all editing, its practical implementation, and the injectables [1] [10]. subsequent evaluation of its efficacy, this project Deep Learning Approaches and Ensemble aims to illuminate a path forward for leveraging Methods image augmentation techniques in the pursuit of ensuring the highest standards of safety and The advent of Convolutional Neural Networks reliability in healthcare delivery. (CNNs) has revolutionized the field of image classification, offering profound improvements over LITERATURE REVIEW traditional methods [11]. Techniques such as transfer learning, where pre-trained models on large datasets The exploration of image augmentation are fine-tuned for specific tasks and using attention techniques, particularly Poisson image editing, mechanisms to improve focus on relevant image offers a promising avenue for addressing the features, represent promising avenues for enhancing challenges inherent in the classification of defective classification performance [12]. Similarly, ensemble and non-defective pharmaceutical glass containers. methods that combine predictions from multiple This section delves into the key areas of challenge models can offer increased robustness and accuracy, and opportunity within this domain, supported by a presenting another viable strategy for tackling the review of existing methodologies and the challenges in this domain [13]. introduction of Poisson image editing as a solution. Domain Adaptation Image Variability and Data Imbalance The concept of domain adaptation is particularly A significant challenge in automating the relevant when considering the application of models inspection of glass containers arises from the trained on synthetic or augmented data to real-world inherent variability in their shape, size, and scenarios. Techniques that enable models to adapt appearance [8]. This variability, compounded by from synthetic to real-world data are crucial in diverse image conditions such as lighting variations ensuring that the improvements seen during training and occlusions, often stymies existing classification translate into effective performance in practical methods. Moreover, the imbalance between defect- applications [6] [14]. free (good-quality) and defective images during model training exacerbates the difficulty, as Poisson Image Editing: Bridging the Gap collecting a well-balanced dataset is a critical yet One of the most promising techniques emerging challenging task [9]. These issues underscore the in this context is Poisson image editing [7] [15]. Originally proposed by Pérez et al. [3], this paramount for ensuring the overall quality and safety technique operates by seamlessly blending images of pharmaceutical products. through the integration of gradient information This investigation zeroes in on cosmetic defects rather than direct pixel values. This approach offers located specifically in the neck region of these vials, several advantages for image augmentation: recognizing this zone as crucial for maintaining the • Mathematical Foundations: At its core, Poisson sterility and efficacy of the pharmaceutical contents. image editing leverages the Laplacian operator The defect types scrutinized in this study include to capture the essence of how a scalar field, such Twist, Lap, Check, Crack, Spitticule, and as pixel intensity, varies across an image. The Ondulation. These defects, ranging from minor subsequent formulation of the Poisson equation visual flaws to more significant structural facilitates the construction of new images that weaknesses, can compromise the vial's integrity and, approximate a desired gradient field, enabling by extension, patient safety. the seamless integration of synthetic defects into Additionally, to complement investigation defect-free images. rigorous quantitative analysis, the researcher • Applications and Benefits: This technique finds integrated a qualitative testing phase into the utility in a range of applications, from seamless methodology. It evaluates the performance of the cloning, which allows for the clean integration Poisson method in different backgrounds. of new elements into an image, to texture Poisson Equation for Image Editing flattening and high-dynamic-range imaging. Its ability to generate realistic, seamlessly blended At the core of Poisson image editing lies the images makes it particularly valuable for Poisson equation, which is utilized to blend images augmenting datasets with synthetic defects, seamlessly. Mathematically, the problem is set up as thereby addressing the challenges of data follows: imbalance and the need for robust, generalized • Guided Interpolation: This method involves models. solving a Poisson equation for each color The exploration of Poisson image editing in the component of the image, treating the image as a context of pharmaceutical glass containers scalar function. The aim is to find an unknown classification represents a great approach to function 𝑓𝑓 that, within a domain 𝛺𝛺, best matches overcoming the limitations of current the gradient of a guidance vector field 𝑣𝑣, under methodologies. By enhancing the quality and certain boundary conditions defined by the diversity of training datasets through synthetic target image. augmentation, this technique holds the promise of • Mathematical Formulation: Given a guidance accelerating the development of highly accurate and field 𝑣𝑣, the goal is to minimize the difference reliable classification models, marking a significant between the gradient of the unknown function 𝑓𝑓 step forward in the pursuit of ensuring the safety and and 𝑣𝑣, formulated as a variational problem: efficacy of pharmaceutical products. 𝑚𝑚𝑚𝑚𝑚𝑚 ∫⬚𝑓𝑓 𝛺𝛺 ‖∇𝑓𝑓 − 𝑣𝑣‖ 2𝑑𝑑𝑑𝑑, (1) subject to boundary conditions on 𝑓𝑓 that ensure METHODOLOGY it matches the target image at the domain's This section delves into the methodological boundary. framework utilized to exploit Poisson image editing • Poisson Equation with Dirichlet Boundary techniques, aimed at augmenting the accuracy of Conditions: The solution to the minimization defect classification in pharmaceutical glass problem leads to the Poisson equation: containers, with a particular focus on the neck area ∆𝑓𝑓 = 𝑑𝑑𝑚𝑚𝑣𝑣(𝑣𝑣) 𝑚𝑚𝑚𝑚 𝛺𝛺 (2) of vials. The integrity and cleanliness of this area are with 𝑓𝑓 = 𝑓𝑓∗ on 𝜕𝜕𝛺𝛺, where 𝑓𝑓∗ represents the technical description of Poisson Image Editing but known values on the boundary of 𝛺𝛺, ∆ denotes also underscores its versatility and power as a tool the Laplacian operator, and 𝑑𝑑𝑚𝑚𝑣𝑣 denotes the for creative and corrective image manipulation. divergence operator. Practical Implementation • Numerical Solution: In practice, the Poisson equation is solved numerically since digital The practical implementation of Poisson image images are discrete. This involves discretizing editing was achieved using Python and the OpenCV the continuous domain into pixels and solving a library, known for its comprehensive set of functions sparse linear system that approximates the catering to computer vision tasks, including image Poisson equation [7]. blending [16]. • Applications in Image Editing: This • Python and OpenCV: The implementation mathematical framework enables a range of involved leveraging OpenCV's image editing tasks, such as seamless cloning, “seamlessClone” function, which is designed object removal, and texture blending. By for tasks like object insertion and image choosing appropriate guidance fields, one can blending, based on the Poisson equation. This manipulate images in ways that preserve natural function simplifies the blending process, gradients and textures, achieving effects that are allowing for the automatic adjustment of colors difficult or impossible with traditional image and illumination to match the source image with editing tools [7]. the target background. • Custom User Interface (UI): A custom UI was Enhanced Clarity on Discrete Poisson Solver developed to facilitate interaction with the The discrete version of this problem considers Poisson blending process. This UI allows users the pixelated nature of digital images. Here, the to: image domain is represented as a grid of pixels, and o Interactively extract defects and create the Poisson equation is solved for each pixel. The masks (source of the defect image), process involves setting up and solving a system of providing precise control over areas linear equations that represent the discretized for blending. version of the Poisson equation, with the solution providing the new pixel values for the edited image. This approach offers unprecedented control over how image features are blended, allowing for edits that maintain the coherence of lighting, texture, and color, even when introducing elements from another image or altering existing components of the picture. By grounding the editing process in the mathematical principles of the Poisson equation, Poisson image editing leverages the natural properties of images (such as gradients and boundary Figure 1 UI for Area Selection of the Area with a Defect to be conditions) to produce results that are visually Extracted harmonious and free of artifacts commonly associated with image manipulation, such as abrupt transitions or halo effects. This in-depth understanding of the mathematical foundations not only enriches the model's classification capabilities post- augmentation [17]. These metrics are fundamental in assessing the effectiveness of classification models. Below are the definitions and formulas for each metric: Figure 2 UI for Mask Creation and Poisson Image Editing Progress • TP (True Positives): The number of correct positive predictions. o Visualize the blending process, • TN (True Negatives): The number of correct including the selection of defect, mask negative predictions. application, and the final Poisson- • FP (False Positives): The number of negative blended image, providing immediate cases incorrectly categorized as positive. feedback on the blending outcome in • Accuracy: This measures the proportion of true the destination image D. results among the total number of cases examined. It is calculated as the sum of true positives and true negatives divided by the total number of cases. 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 = 𝑇𝑇𝑇𝑇+𝑇𝑇𝑇𝑇 (3) 𝑇𝑇𝑇𝑇+𝑇𝑇𝑇𝑇+𝐹𝐹𝑇𝑇+𝐹𝐹𝑇𝑇 Figure 3 • Precision: This measures the proportion of true UI for Image Blending Using the Poisson. To the Left, the Destination Image is Without Defect. To the Right, the positive predictions in the total positive Destination Image After the Blending predictions. It is also known as the positive Evaluation predictive value. 𝑃𝑃𝐴𝐴𝑃𝑃𝐴𝐴𝑚𝑚𝑃𝑃𝑚𝑚𝑃𝑃𝑚𝑚 = 𝑇𝑇𝑇𝑇 (4) The evaluation of the enhanced classification 𝑇𝑇𝑇𝑇+𝐹𝐹𝑇𝑇 models, incorporating images augmented through • Recall (or Sensitivity): This measures the Poisson blending, involved several key steps: proportion of actual positives that were • Neural Network Training: A Convolutional identified correctly. It is calculated as the Neural Network (CNN) was trained to number of true positives divided by the sum of distinguish between good (defect-free) and bad true positives and false negatives. 𝑇𝑇𝑇𝑇 (defective) images. The training set was 𝑅𝑅𝑃𝑃𝐴𝐴𝐴𝐴𝑅𝑅𝑅𝑅 = (5) 𝑇𝑇𝑇𝑇+𝐹𝐹𝑇𝑇 augmented using Poisson blending to artificially • F1 Score: This is the harmonic mean of introduce defects into good images, creating a precision and recall and is a measure of the complete set of images with artificially blended model's accuracy. An F1 score reaches its best defects. value at 1 (perfect precision and recall) and • Testing: The trained model was then evaluated worst at 0. on unseen data, including both real-world • 𝐹𝐹1 𝑆𝑆𝐴𝐴𝑃𝑃𝐴𝐴𝑃𝑃 = 2 × 𝑇𝑇𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 × 𝑅𝑅𝑃𝑃𝑃𝑃𝑅𝑅𝑅𝑅𝑅𝑅 (6) defective images and additional good images 𝑇𝑇𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃+ 𝑅𝑅𝑃𝑃𝑃𝑃𝑅𝑅𝑅𝑅𝑅𝑅 not used during training. This step aimed to Through this methodology, the project aims to assess the model's ability to generalize and demonstrate the effectiveness of Poisson image perform accurately on new, unencountered editing in enhancing the performance of samples. classification models for defective and non-defective • Performance Metrics: The evaluation focused pharmaceutical glass containers, addressing key on standard performance metrics such as challenges such as data imbalance and the need for accuracy, precision, recall, and F1 score, robust generalization. It’s important to note that providing a comprehensive assessment of the while the testing was performed using only vials, the same principles will apply to all types of glass model's operational efficacy. Nevertheless, the containers. observed reduction in precision, albeit slight, warrants a closer examination to mitigate the DISCUSSION incidence of false positives in practical deployment Quantitative Evaluation of Model Performance scenarios. The following tables encapsulate the Qualitative Assessment of Augmented Image performance metrics derived from the empirical Fidelity evaluation of the classification model. These metrics Complementing the quantitative analysis, a furnish a quantitative appraisal of the model's qualitative examination of the augmented images proficiency in discerning between non-defective was conducted. This assessment entailed ('good') and defective ('bad') pharmaceutical vials comparative visual scrutiny between images across the training and independent testing datasets. generated via the Poisson blending technique and specifically, the defects are localized to the neck of those produced through conventional copy-paste the vials. augmentation methods. Refer to Figures 4 and 5. Table 1 Neural Network Results Outcome Training Dataset Testing Dataset True Positives 1254 647 True Negatives 1255 939 False Positives 1 66 False Negatives 0 0 Based on these outcomes, the performance metrics were calculated as follows: Table 2 Neural Network Metric Results Figure 4 Crack Defect Insertion Using Copy and Paste Metric Training Dataset Testing Dataset Accuracy 99.96% 97.66% Precision 99.92% 94.45% Recall 100.00% 100.00% F1 Score 99.96% 97.15% The analysis reveals exemplary model performance in the training phase, with nearly all metrics approaching the upper theoretical limits. Such findings suggest that the Poisson image editing technique, employed in the augmentation of the training set, has substantially contributed to the Figure 5 model's discriminative capabilities. Crack Defect Insertion Using the Poisson Image Editing Technique Conversely, the testing phase metrics, while exhibiting a marginal decrement, maintain a high The images subjected to Poisson blending level of accuracy and an unblemished recall rate. The exhibited a pronounced enhancement in realism, preservation of a maximal recall rate in the testing with artificial defects seamlessly integrated into the phase is critical, as it ensures that all defective items vial imagery. This seamless integration is pivotal in are invariably identified, thereby affirming the ensuring the authenticity of the training data. However, certain images, particularly those with disadvantages-and-inspector-qualifications. [Accessed: intricate background gradients, displayed minor February 17, 2024]. [2] shadowing phenomena post-blending. These visual A. Flaquiere, J. Malthête and G. Vaccines, "Parental anomalies, although not significantly deterring the Drug Associaiton," Parental Drug Associaiton, 19 March 2020. [Online]. Available: quantitative metrics, highlight the necessity for https://www.pda.org/pda-letter-portal/home/full- article/qualification-of-manual-visual-inspection-still- algorithmic refinement to optimize the visual critical. [Accessed: February 17, 2024]. congruity of the training images. (Refer to Figure 6). [3] N. Nagi and d. Dechow, "Quality Magazine," 1 March 2022. [Online]. Available: https://www.qualitymag.com/articles/96902-advanced- imaging-and-deep-learning-technologies-execute- critical-packaging-inspection. [Accessed: February 17, 2024]. [4] L. K. Vora, A. D. Gholap, K. Jetha, R. R. S. Thakur, H. K. Solanki and V. P. Chavda, "Artificial Intelligence in Pharmaceutical Technology and Drug Delivery Design," Pharmaceutics, vol. 15, no. 1716, pp. 1-46, 2023. [5] A. Aldoseri, K. N. Al-Khalifa and A. M. Hamouda, "Re- Thinking Data Strategy and Integration for Artificial Intelligence: Concepts, Opportunities, and Challenges," Applied Sciences, vol. 13, no. 7082, pp. 1-33, 2023. [6] V. Dugan, "Digital Poirots," 6 June 2023. [Online]. Available: https://digitalpoirots.com/synthetic-data-generation-and-why-is-it-becoming-popular/. [Accessed: Figure 6: February 17, 2024]. Blending Difficulties When an Uneven Background is Present [7] P. Perez, M. Gangnet and A. Blake, "Poisson Image Editing," ACM Transactions on Graphics, vol. 22, no. 3, pp. 313-318, 203. CONCLUSION [8] F. Renard, S. Guedria, N. D. Palma and N. Vuillerme, "Variability and Reproducibility in Deep Learning for The research successfully demonstrated the Medical Image Segmentation," Scientific Reports, vol. efficacy of Poisson image editing in creating 10, no. 13724, pp. 1-16, 2020. [9] J. M. Johnson and T. M. Khoshgoftaar, "Survey on deep augmented datasets for the classification of cosmetic learning with class imbalance," Journal of Big Data, defects in pharmaceutical vials. The quantitative vol. 6, no. 27, pp. 1-54, 2019. evaluation revealed that the CNN model achieved [10] ECA Fundation, "ECA Fundation," ECA Foudnation, 2 April 2015. [Online]. Available: https://www.gmp- high accuracy and recall rates, with precision compliance.org/gmp-news/usp-chapter-1790-visual- inspection-of-injections-published. [Accessed: February indicating a need for further model refinement. 17, 2024]. Qualitatively, the blended images showed enhanced [11] A. Krizhevsky, I. Sutskever and G. E. Hinton, realism, although some introduced shadows in "ImageNet Classification with Deep Convolutional Neural Networks," in Proceedings of the 25th gradient-varied backgrounds suggest areas for future International Conference on Neural Information improvement. This study paves the way for further Processing Systems, Lake Tahoe, Nevada, 2012. [12] M.-H. G. Guo and T.-X. Xu, "Attention Mechanisms in research into advanced data augmentation Computer Vision: A Survey," Computational Visual techniques and their application in ensuring the Media, vol. 9, no. 3, pp. 331-368, 2022. [13] M. Ganaie, M. Hu, A. Malik, M. Tanveer and P. safety and integrity of pharmaceutical glass Suganthan, "Ensemble deep learning: A review," containers. Engineering Applications of Artificial Intelligence, vol. 115, 2022. [14] C. Gao, B. D. Killeen, Y. Hu, R. B. Grupp, ,. R. H. REFERENCES Taylor, M. Armand and M. Unberath, "Synthetic data accelerates the development of generalizable learning- [Online] based algorithms for X-ray image analysis," Nature Machine Intelligence, vol. 5, pp. 294-308, 2023. [1] Restore Health Consulting, "Restore Health Consulting," [15] W.-H. Shen and M.-L. Li, "Arxiv," 28 August 2023. 8 April 2022. [Online]. Available: [Online]. Available: https://arxiv.org/abs/2308.14772. https://www.restorehealthconsulting.com/news/understa [Accessed: February 17, 2023]. nding-regulatory-expectations-for-manual-visible- particle-detection-human-advantages-and- [16] S. Mallick, "learnopencv," learnopencv, 2 March 2015. [Online]. Available: https://learnopencv.com/tag/poisson-image-editing/. [Accessed: September 9, 2023]. [17] M. Vakili, M. Ghamsari and M. Rezaei, "Performance Analysis and Comparison of Machine and Deep Learning Algorithms for IoT Data Classification," Global Transitions Proceedings, no. 2, pp. 212-219, 2021.