From Devices to Insights: Transforming Business Needs with MQTT and Node.js in an IoT Data Pipeline

“From Devices to Insights: Transforming Business Needs with MQTT and Node.js in an IoT Data Pipeline” written from the visionary perspective of our Head of Web, Geo Apopei and his stellar experts: Cosmin Stoian and Victor Badulescu .

Data Pipeline with MQTT, Node.js, and Pub/Sub

The Internet of Things (IoT) has emerged as a pivotal technology, facilitating the seamless exchange of data between devices and systems. With its applications it looks like IoT is bringing the possibilities of tomorrow closer to the user. With such a momentum on development and increasing appetite from the market, challenges and breakthroughs go hand in hand. One such challenge lies in data generation, how can developers create systems that manage and process data generated by IoT devices efficiently?

In this article, we explore how at Salt and Pepper we used a combination of MQTT, Node.js and Pub/Sub architecture to form the backbone of a robust data pipeline, enabling the continuous flow of information from IoT devices to data storage, data warehousing systems & 3rd parties.

MQTT: Lightweight Communication

The main glue of many IoT communication protocols is MQTT (Message Queuing Telemetry Transport). MQTT is renowned for its lightweight nature and efficient pub/sub messaging pattern, making it ideal for IoT applications where resources are often limited, as per our case. In our scenario, thousands of IoT devices leverage MQTT to publish data to a broker, which acts as a central hub for message distribution.

Data size and data publish frequency were important factors while delivery guarantee for a specific message was less of an issue, as the message could be recreated from scattered events.

Node.js: The Listener

Node.js, with its event-driven architecture and non-blocking I/O capabilities, serves as a perfect choice for building the listener component of our data pipeline. In our setup, a dockerised Node.js application acts as a subscriber to the MQTT broker, continuously listening for incoming messages from the IoT devices.

As messages arrive, Node.js decodes them, extracting the relevant information for further processing. Each message is an array of ones or zeros, not the best for DX or readability (takes me back to college Computer Science), that’s why we opted for typescript, an essential extra-step for demystifying the input data structure.

Pub/Sub Architecture: Orchestrating Data Flow

To ensure scalability and flexibility in our data pipeline, we employ a Pub/Sub (Publish/Subscribe) architecture. Pub/Sub enables decoupling between message producers and consumers, allowing for consistent integration of additional systems. In our case, Node.js acts as both a subscriber and a publisher. Once data is decoded, Node.js publishes it to a Pub/Sub system, where it is subsequently consumed by downstream systems.

Integration with 3rd parties

The most important piece of the architecture of the data pipeline involves integrating with data warehousing and NoSQL databases. By leveraging Pub/Sub as the intermediary layer, we enable a decoupled communication between our Node.js application and these data storage systems.

The data consumed from Pub/Sub can be stored in a data warehouse for analytical purposes, providing valuable insights into IoT device performance, usage and error patterns. Simultaneously, data can be persisted in the NoSQL database for real-time querying and processing, ensuring the user’s rapid access to device information and also could be stored into object storage systems to be able to replay flows or export data but the latter wasn’t the case, at least not for this phase of the project.

Conclusion

In conclusion, the melange of MQTT, Node.js, and Pub/Sub architecture forms a robust foundation for building scalable and efficient IoT data pipelines. By integrating communication, data processing, and storage components, organizations can harness the full potential of their IoT ecosystems. Whether it’s monitoring sensor data, tracking device health, or optimizing operational efficiency, the power of MQTT, Node.js, and Pub/Sub unlocks endless possibilities in the realm of IoT data management.

For this project we used: MQTT (emqx), Node.js, Docker, Bitbucket Pipeline and GCP services. We will go more in depth into the implemented solution in the next articles.