A Practical Approach of Different Programming Techniques to Implement a Real-time Application using Django

@article{Stigler2020,
    Author = "Stigler, Sebastian and Burdack, Marina",
    Title = "{A Practical Approach of Different Programming Techniques to Implement a Real-time Application using Django}",
    Journal = "ATHENS JOURNAL OF SCIENCES",
    Year = "2020",
    Month = "March",
    Number = "1",
    Pages = "43-66",
    Volume = "7",
    Abstract = "Real-time web-applications surround our daily life. The most popular are shopping platforms like Amazon or Zalando. Behind these websites huge and complex shop-systems are typically used. These systems manage and handle thousands of parallel synchronous and asynchronous tasks like user interactions, user authentications, communication with an ERP system (enterprise resource planning), or the handling of the website visualization in the background. Through the independence of the computer system and the portability on other devices, like touch panels or mobile phones the web-technology is also used for industry 4.0 applications. The applications can be e. g. predictive maintenance dashboards or monitoring dashboards for the production line. The key task for these dashboards are the visualization and analyzing of a huge amount of data in real-time. For both scenarios, the data have to be preprocessed continuously in different steps. In summary, the industry 4.0 application has to manage a different kind of tasks compared to a typical webshop. These tasks contain for example big data preprocessing tasks, which are mostly CPU bound, in contrast to the typically I/O bound task of a webshop (e.g. database access). The main problem with web application frameworks is, that they are not designed to execute a lot of CPU bound tasks and handle webrequests at the same time. The purpose of this paper is to compare two different programming techniques for the work with CPU bound tasks in a real-time web application for the use in industry 4.0 without the use of third-part software like Spark. The research employed two approaches: on the one hand multiprocessing and on the other hand task queues. In this work, two kinds of web-applications on the base of python’s web framework Django are implemented. The first application uses multiprocessing and the second application uses the Celery task queue to organize different tasks. The general building blocks are the sensor data streaming via MQTT and the data transformation in various steps. The result of this transformation can be used to visualize the data in real-time charts for example. The implemented web-applications are evaluated by the following criteria: Performance while working with a huge amount of streaming data. Possibilities to optimize the load distribution and avoid overloading. Possibilities to scale up and down hardware.",
    Doi = "10.30958/ajs.7-1-4",
    Owner = "sebastian",
    Timestamp = "2019.11.27",
    customlinkslides = "/papers/Stigler2020.slides.pdf"
}