[Avg. reading time: 12 minutes]

Protocol Buffers

What are Protocol Buffers

• A method to serialize structured data into binary format
• Created by Google
• Its like JSON, but smaller and faster.
• Protocol Buffers are more commonly used in industrial IoT scenarios.

Why Protobuf is great for IoT

• Smaller size: Uses binary format instead of text, saving bandwidth
• Faster processing: Binary format means less CPU usage on IoT devices
• Strict schema: Helps catch errors early
• Language neutral: Works across different programming languages
• Great for limited devices: Uses less memory and battery power
• Extensibility: Add new fields to your message definitions without breaking existing code.

Industrial Use Cases

• Bridge structural sensors (vibration, stress)
• Factory equipment monitors
• Power grid sensors
• Oil/gas pipeline monitors
• Wind turbine telemetry
• Industrial HVAC systems

Why Industries prefer Protobuf:

• High data volume (thousands of readings per second)
• Need for efficient bandwidth usage
• Complex data structures
• Multiple systems need to understand the data
• Long-term storage requirements
• Cross-platform compatibility needs

graph LR
    subgraph Bridge["Bridge Infrastructure"]
        S1[Vibration Sensor] --> GW
        S2[Strain Gauge] --> GW
        S3[Temperature Sensor] --> GW
        subgraph Gateway["Linux Gateway (Solar)"]
            GW[Edge Gateway]
            DB[(Local Storage)]
            GW --> DB
        end
    end
    
    subgraph Communication["Communication Methods"]
        GW --> |4G/LTE| Cloud
        GW --> |LoRaWAN| Cloud
        GW --> |Satellite| Cloud
    end
    
    Cloud[Cloud Server] --> DA[Data Analysis]
    
    style Bridge fill:#87CEEB,stroke:#333,stroke-width:2px,color:black
    style Gateway fill:#90EE90,stroke:#333,stroke-width:2px,color:red
    style Communication fill:#FFA500,stroke:#333,stroke-width:2px,color:black
    style Cloud fill:#4169E1,stroke:#333,stroke-width:2px,color:white
    style DA fill:#4169E1,stroke:#333,stroke-width:2px,color:white
    style GW fill:#000000,stroke:#333,stroke-width:2px,color:white
    style DB fill:#800020,stroke:#333,stroke-width:2px,color:white
    
    classDef sensor fill:#00CED1,stroke:#333,stroke-width:1px,color:black
    class S1,S2,S3 sensor

Consumer IoT devices (in general)

• Use simpler formats (JSON, proprietary)
• Have lower data volumes
• Work within closed ecosystems (Google Home, Apple HomeKit)
• Don’t need the optimization Protobuf provides

Data Types in Protobufs

Scalar Types:

int32, int64, uint32, uint64, sint32, sint64, fixed32, fixed64, sfixed32, sfixed64 float, double, bool, string, bytes

Composite Types:

• message: Defines a structured collection of other fields.
• enum: Defines a set of named integer constants.

Collections:

• repeated: Allows you to define a list of values of the same type. Like Array.

Steps involved in creating a Proto Buf data file.

Step 1: Define the Data Structure of your data file as .proto text file. Ex: my_data.proto

syntax = "proto3";

message MyData {
  int32 id = 1;
  string name = 2;
  float value = 3;
}

Step 2: Compile the .proto file to Python Class (.pb) or Java Class (.java) using protoc library.

protoc --python_out=. my_data.proto

Generates my_data_pb2.py

Install Protoc

Step 3: Use the Generated Python Class file and use it to store data.

Note: Remember protoc –version should be same or closer as protobuf minor version number from pypi library.

In my setup protoc –version = 29.3, pypi protobuf = 5.29.2 Minor version of protobuf is 29.2 which is closer to 29.3

See example.

Demo Script

git clone https://github.com/gchandra10/python_protobuf_demo

flowchart LR
    subgraph Sensor["Temperature/Humidity Sensor"]
        S1[DHT22/BME280]
    end
    
    subgraph MCU["Microcontroller"]
        M1[ESP32/Arduino]
    end
    
    subgraph Gateway["Gateway/Edge Device"]
        G1[Raspberry Pi/\nIntel NUC]
    end
    
    subgraph Cloud["Cloud Server"]
        C1[AWS/Azure/GCP]
    end
    
    S1 -->|Raw Data 23.5°C, 45%| M1
    M1 -->|"JSON over MQTT {temp: 23.5,humidity: 45}"| G1
    G1 -->|Protocol Buffers\nover HTTPS| C1

#protobuf #googleVer 6.0.5