Get in touch →

Architecture Overview for Budget Friendly IoT Communication and Data Processing

by | Jun 18, 2024 | Blog

 

In this article, we’ll explore the architecture designed for an IoT project focused on efficient data communication, processing, and storage for around 2000 units.

The project involved receiving data via MQTT, decoding it, and publishing it to different services for consumption and storage. We’ll also discuss the flow designed to keep the EMQX serverless account alive in development and staging environments.

The tools utilized include EMQX, GCP Compute Engine, GCP BigQuery, Firestore, GCP Cloud Functions, and GCP Pub/Sub.
 

Architecture Components:

 

  • 1. EMQX: An MQTT broker for publishing and subscribing to messages from IoT devices.
  • 2. GCP Compute Engine: Used to host the Node.js listener that decodes MQTT messages.
  • 3. GCP Pub/Sub: A messaging service to facilitate asynchronous communication between the Node.js listener and other systems.
  • 4. GCP BigQuery: A data warehouse for storing raw event data.
  • 5. GCP Cloud Functions: Serverless functions triggered by Firestore to process and aggregate data.
  • 6. Firestore: A NoSQL database for storing events and user usage data.

 

Data Flow

 

Insert Event & User Usage

 

  1. Data Ingestion:
    • IoT devices send data via MQTT to the EMQX broker.
    • The EMQX broker routes these messages to a Node.js listener hosted on GCP Compute Engine.
    • The Publisher & Subscriber clients were separate and had limited responsibilities and intuitive naming.
  2. Message Processing:
    • The Node.js listener decodes the received MQTT messages.
    • Decoded messages are published to GCP Pub/Sub.
  3. Data Storage and Triggering Events:
    • GCP Pub/Sub publishes messages to both GCP BigQuery and Firestore.
    • In GCP BigQuery, each event is stored “as is” for historical and analytical purposes.
    • In Firestore, events are saved in an “Events Ledger” collection.
  4. Triggering Cloud Functions:
    • Saving a document in Firestore triggers a GCP Cloud Function.
    • This function aggregates the user data based on the event data and updates another collection in Firestore named “User Usages”.
  5. Data Consumption:
    • The aggregated data from “User Usages” in Firestore is consumed by a mobile application, providing users with the necessary insights.

 

Keep-alive Flow

 

  1. Scheduled Keep-alive:
    • A GCP Pub/Sub topic is scheduled to trigger a GCP Cloud Function weekly.
    • The Cloud Function publishes a keep-alive message to the EMQX broker.
    • This ensures the EMQX serverless account remains active in both development and staging environments.

 

Detailed Diagram

 

The architecture diagram can be broken down into two main flows: Insert Event & User Usage, and Keep-alive.
We chose these services based on budget, infrastructure constraints and scalability. The architecture is easily scalable or replaceable due to separation of concerns and decoupling of services.

 

Conclusion

 

The architecture is designed to efficiently handle real-time data from IoT devices using MQTT, ensuring scalability, reliability, and flexibility. Each component plays a specific role, from initial message brokering with EMQX to final data consumption by a mobile application.
The use of GCP services like Compute Engine, Pub/Sub, BigQuery, Firestore, and Cloud Functions ensures that the system can process large volumes of data, trigger automated workflows, and provide real-time insights to users. Additionally, the keep-alive flow maintains system activity in non-production environments, ensuring smooth development and testing phases.