Diseño e Implementación de una arquitectura orientada a eventos en Kubernetes para la plataforma cognitiva MIRIAM

Abstract

RESUMEN : La arquitectura basada en eventos es ampliamente popular debido a su impacto en el desarrollo de software, debido a que los procesos dentro de esta arquitectura son asíncronos y distribuidos y la hace ideal para desarrollar sistemas altamente escalables, esto se acomoda perfectamente a procesos de ETL (extracción, transformación y carga) porque se puede implementar un punto de entrada que reciba los datos de un publicador y delegar los pasos siguientes a los módulos suscriptores correspondientes, así desentendiéndose del resultado del siguiente paso porque de eso se encargará otro suscriptor y cumpliendo únicamente su labor cuando le llegan tareas. Este documento aborda la implementación de un proceso ETL para contratos logísticos a través de la plataforma de gestión cognitiva MIRIAM.

ABSTRACT : The event-driven architecture is widely popular due to its impact on software development, because the processes within this architecture are asynchronous and distributed and makes it ideal for developing highly scalable systems, this is perfectly suited to ETL processes (extraction, transformation and load) because you can implement an entry point that receives data from a publisher and delegate the following steps to the corresponding subscriber modules, thus disregarding the outcome of the next step because that will take care of another subscriber and fulfilling only their work when tasks arrive. This paper addresses the implementation of an ETL process for logistics contracts through the MIRIAM cognitive management platform. As a final result of this project, a new version for the MIRIAM platform was developed, implementing an event-based architecture with docker components and capable of running within a Kubernetes environment, the process starts with a producer service that takes from a topic the information of the logistic contracts to be processed, it is responsible for sending tasks to a messaging queue to which Celery workers are subscribed and which trigger all the processing tasks of the document to be presented to the users in the graphical interface at the end. This application is deployed in a Kubernetes cluster through cloud services offered by Microsoft Azure and is about to be used in a fully production environment.