SmartSpectra Protocol Buffer Reference
Overview
This page contains complete reference for all Protocol Buffer messages used in every SmartSpectra SDK and the Physiology gRPC service, which is only available via the SmartSpectra OnPrem SDK to select customers and partners. If you'd like to use the OnPrem SDK, please reach out to us via the contact form.
Integration Guides
For details on how to access message fields in different languages & SmartSpectra SDKs, please consult their respective integration guides linked in the table below.
| SDK (Language) | Related Integration Guide Documentation |
|---|---|
| Android (Kotlin) | SmartSpectra Android SDK Integration Guide |
| Swift | SmartSpectraSwiftSDK Data Access |
| C++ | Using Metrics with SmartSpectra C++ SDK |
Table of Contents
physiology_service.proto
TopFrameWithTimestamp
Container for video frames with precise timing information.
Associates image data with capture timestamps for temporal analysis.
| Field | Type | Label | Description |
| timestamp | int64 | Timestamp when the frame was captured (microseconds since capture start) |
|
| frame | Mat | Image data in OpenCV Mat format |
LogEntry
Log entry for streaming diagnostic and debug information.
Provides timestamped log messages from the processing pipeline.
| Field | Type | Label | Description |
| message | string | Log message content |
|
| timestamp | int64 | Timestamp when the log entry was generated (microseconds since epoch) |
Mat
Represents an OpenCV Mat structure for image data transmission.
Encapsulates image dimensions, data type, and raw pixel data for processing.
| Field | Type | Label | Description |
| rows | int32 | Number of rows (height) in the image matrix (pixels) |
|
| cols | int32 | Number of columns (width) in the image matrix (pixels) |
|
| element_type | int32 | OpenCV element type identifier (e.g., CV_8UC3 for 8-bit 3-channel), see GitHub gist by Changjiang Yang at https://gist.github.com/yangcha/38f2fa630e223a8546f9b48ebbb3e61a for details |
|
| element_size | int32 | Size in bytes of each matrix element, i.e. [ channel count ] * [ depth in bytes ] |
|
| data | bytes | Raw image data as byte array. Note on Python usage: no need to use cv.encode when packaging a NumPy array into this field, just use the array's .tobytes() function. (OpenCV frames/Mats are NumPy matrices in Python.) |
Physiology
The physiology service provides comprehensive physiological analysis capabilities based on supplied video frames.
This service manages the complete pipeline from video frame input to physiological metrics output,
including heart rate, respiratory rate, blood pressure, and other vital sign analysis.
| Method Name | Request Type | Response Type | Description |
| StartCore | Empty | Empty | Starts the Core processing engine for physiological analysis. Initializes the backend processing components required for metrics calculation. |
| StartEdgeGraph | Empty | Empty | Starts the Edge processing graph for real-time frame analysis. Initializes the edge-side preprocessing pipeline for incoming video frames. |
| StartPreprocessing | Empty | Empty | Starts the Edge processing graph for real-time frame analysis (deprecated). @deprecated Use StartEdgeGraph instead. |
| WaitUntilGraphIsIdle | Empty | Empty | Waits until the Edge processing graph becomes idle. Blocks until all pending frame processing operations are complete. |
| GetStatusCode | Empty | StatusValue stream | Streams real-time status codes indicating processing state. Provides continuous updates about face detection, image quality, and processing issues. |
| GetStatusDescription | StatusValue | StringValue | Gets human-readable description for a given status code. Converts status codes into descriptive text for user interfaces. |
| GetStatusHint | StatusValue | StringValue | Gets actionable hints for resolving issues indicated by status codes. Provides user guidance for addressing detected problems. |
| SetRecording | BoolValue | Empty | Controls recording state of the physiological analysis. Starts or stops the active processing and metrics generation. |
| AddFrameWithTimestamp | FrameWithTimestamp stream | Empty | Accepts streaming video frames with timestamps for processing. The input video is expected to be captured at approximately 25-30 frames per second for optimal results, and timestamps are expected to be set accordingly, preferably using the camera shutter timing (if available). Primary input method for real-time physiological analysis from video data. |
| GetMetrics | Empty | MetricsBuffer stream | Streams comprehensive physiological metrics from the Core processing engine (deprecated). @deprecated Use GetCoreMetrics instead (alternatively, use GetEdgeMetrics for on-edge metrics). |
| GetCoreMetrics | Empty | MetricsBuffer stream | Streams comprehensive physiological metrics from the Core processing engine. Provides complete metrics buffers with metadata from cloud/server-side analysis. |
| GetEdgeMetrics | Empty | Metrics stream | Streams real-time physiological metrics from Edge processing. Provides lightweight metrics from device-side analysis for immediate feedback. |
| StopEdgeGraph | Empty | Empty | Stops the Edge processing graph. Cleanly shuts down edge-side preprocessing and analysis components. |
| StopPreprocessing | Empty | Empty | Stops the Edge processing graph (deprecated). @deprecated Use StopEdgeGraph instead for edge-side processing shutdown. |
| StopCore | Empty | Empty | Stops the Core processing engine. Cleanly shuts down backend processing components and releases resources. |
| ResetProcessing | Empty | Empty | Convenience procedure that resets all processing state and clears accumulated data. Useful for starting fresh analysis (e.g. new session or subject) or recovering from error states. Replaces a StopCore+StopEdgeGraph combination, followed by a StartCore+StartEdgeGraph combination. May be used without Core or Edge running to launch the first analysis session. |
| CheckHealth | Empty | StringValue | Performs health check on the service. Returns service status. |
| IsEdgeGraphRunning | Empty | BoolValue | Checks if the Edge processing graph is currently running. Returns the operational status of edge-side processing components. |
| IsPreprocessingRunning | Empty | BoolValue | Checks if preprocessing is currently running (deprecated). @deprecated Use IsEdgeGraphRunning instead for edge processing status. |
| IsPhysiologyCoreUp | Empty | BoolValue | Checks if the Physiology Core backend is operational. Returns the connectivity and operational status of core processing services. |
| StreamLogs | Empty | LogEntry stream | Streams real-time log entries from the processing pipeline. Provides detailed logging information for debugging and monitoring. |
Methods with deprecated option
| Method Name | Option |
| StartPreprocessing | true |
| GetMetrics | true |
| StopPreprocessing | true |
| IsPreprocessingRunning | true |
metrics.proto
TopBloodPressure
Blood pressure measurements and related cardiovascular metrics.
Currently focuses on phasic blood pressure variations.
| Field | Type | Label | Description |
| phasic | MeasurementWithConfidence | repeated | Phasic blood pressure measurements with confidence scores |
Breathing
Comprehensive breathing/respiratory measurements and derived metrics.
Contains respiratory rate, traces, and various breathing pattern indicators.
| Field | Type | Label | Description |
| rate | MeasurementWithConfidence | repeated | Respiratory rate measurements with confidence scores |
| upper_trace | Measurement | repeated | Chest breathing movement trace measurements |
| lower_trace | Measurement | repeated | Abdominal breathing movement trace measurements |
| amplitude | Measurement | repeated | Breathing amplitude measurements |
| apnea | DetectionStatus | repeated | Apnea (breathing cessation) detection status |
| respiratory_line_length | Measurement | repeated | Respiratory line length measurements for breathing pattern analysis |
| baseline | Measurement | repeated | Baseline breathing measurements |
| inhale_exhale_ratio | Measurement | repeated | Inhale to exhale duration ratio measurements |
| strict | Strict | Strict/high-precision breathing measurements |
DetectionStatus
Represents detection status with timestamp information.
Used to track whether a particular physiological feature or state is detected.
| Field | Type | Label | Description |
| time | float | Deprecated. Relative time of the detection status in seconds since start of recording (deprecated) |
|
| detected | bool | Whether the feature/state was detected |
|
| stable | bool | Whether the detection is considered stable/reliable |
|
| timestamp | int64 | Absolute timestamp at which the detection status was updated, in microseconds, since Linux epoch |
Fields with deprecated option
| Name | Option |
| time | true |
Eda
Electrodermal Activity (EDA) measurements.
Tracks skin conductance changes related to autonomic nervous system activity.
| Field | Type | Label | Description |
| trace | Measurement | repeated | EDA trace measurements over time |
Face
Comprehensive facial analysis measurements and detections.
Contains blinking, talking detection, landmarks, and micro-expressions.
| Field | Type | Label | Description |
| blinking | DetectionStatus | repeated | Blinking detection status over time |
| talking | DetectionStatus | repeated | Talking/speech detection status over time |
| landmarks | Landmarks | repeated | Facial landmark coordinates over time |
| micro_expression | MicroExpression | repeated | Detected micro-expressions over time |
Landmarks
Facial landmark coordinates with temporal and stability information.
Used for face tracking and facial feature analysis.
| Field | Type | Label | Description |
| time | float | Deprecated. Relative time of landmark detection in seconds since start of recording (deprecated) |
|
| value | Point2dFloat | repeated | Array of 2D coordinate points representing facial landmarks |
| stable | bool | Whether the landmark detection is considered stable/reliable |
|
| reset | bool | Indicates whether the landmark set was reset (cannot be directly associated with previous set) |
|
| timestamp | int64 | Absolute timestamp at which the landmarks were detected, in microseconds, since Linux epoch |
Fields with deprecated option
| Name | Option |
| time | true |
Measurement
Represents a single measurement with timestamp and stability information.
Used for various physiological measurements throughout the system.
| Field | Type | Label | Description |
| time | float | Deprecated. Relative time of the measurement in seconds since start of recording (deprecated) |
|
| value | float | The measured or estimated value |
|
| stable | bool | Whether the measurement is considered stable/reliable |
|
| timestamp | int64 | Absolute timestamp at which the measurement was taken, in microseconds, since Linux epoch |
Fields with deprecated option
| Name | Option |
| time | true |
MeasurementWithConfidence
Represents a measurement with an associated confidence score.
Extends basic measurement with confidence information for quality assessment.
| Field | Type | Label | Description |
| time | float | Deprecated. Relative time of the measurement in seconds since start of recording (deprecated) |
|
| value | float | The measured value |
|
| stable | bool | Whether the measurement is considered stable/reliable |
|
| confidence | float | Confidence score for the measurement (0.0 to 1.0) |
|
| timestamp | int64 | Absolute timestamp at which the measurement was taken, in microseconds, since Linux epoch |
Fields with deprecated option
| Name | Option |
| time | true |
Metadata
Metadata information for metrics data packets.
Contains identification, timing, and frame information for data correlation.
| Field | Type | Label | Description |
| id | string | Unique identifier for the metrics data |
|
| upload_timestamp | string | Timestamp when the data was uploaded/sent, as a full ISO-8601 string |
|
| api_version | string | API version used to generate the metrics |
|
| sent_at_s | double | Seconds since Linux epoch when the data was sent from Physiology Core (cloud API or OnPrem Core Server) |
|
| frame_timestamp | int64 | Microseconds since start of capture, associated with PreprocessedDataBuffer that triggered this MetricsBuffer |
|
| frame_count | int32 | Number of actual image frames used to generate the associated PreprocessedDataBuffer, excluding frames partially used for previous buffers |
Metrics
Comprehensive physiological metrics container.
Contains all available physiological measurements and analysis results.
| Field | Type | Label | Description |
| breathing | Breathing | Breathing and respiratory analysis results |
|
| micromotion | MicroMotion | Micro-motion analysis results |
|
| eda | Eda | Electrodermal activity measurements. Note: processing needs to run for over 35 seconds to generate the first EDA result. |
|
| face | Face | Facial analysis results |
MetricsBuffer
Container for a complete set of physiological metrics from a processing buffer.
Represents the output of processing a batch of preprocessed data.
| Field | Type | Label | Description |
| pulse | Pulse | Pulse and heart rate related measurements |
|
| breathing | Breathing | Breathing and respiratory measurements |
|
| blood_pressure | BloodPressure | Blood pressure measurements |
|
| face | Face | Facial analysis measurements |
|
| metadata | Metadata | Metadata for this metrics buffer |
MicroExpression
Represents a detected micro-expression with metadata.
Used for facial expression analysis and emotion detection.
| Field | Type | Label | Description |
| time | float | Deprecated. Relative time of the micro-expression detection in seconds since start of recording (deprecated) |
|
| expression | string | String identifier of the detected expression |
|
| stable | bool | Whether the detection is considered stable/reliable |
|
| confidence | float | Confidence score for the detection (0.0 to 1.0) |
|
| timestamp | int64 | Absolute timestamp at which the micro-expression was detected, in microseconds, since Linux epoch |
Fields with deprecated option
| Name | Option |
| time | true |
MicroMotion
Micro-motion measurements for body movement analysis.
Tracks small movements in specific body regions.
| Field | Type | Label | Description |
| glutes | Measurement | repeated | Micro-motion measurements in the glutes region |
| knees | Measurement | repeated | Micro-motion measurements in the knees region |
Pulse
Comprehensive pulse-related measurements and derived metrics.
Contains heart rate, pulse trace, and respiratory coupling information.
| Field | Type | Label | Description |
| rate | MeasurementWithConfidence | repeated | Heart rate measurements with confidence scores |
| trace | Measurement | repeated | Raw pulse trace measurements |
| pulse_respiration_quotient | Measurement | repeated | Pulse-respiration quotient measurements indicating cardio-respiratory coupling |
| strict | Strict | Strict/high-precision pulse measurements |
Strict
Container for strict/exact values that require high precision.
Used when measurements need to be treated with special precision requirements.
| Field | Type | Label | Description |
| value | float | The strict value requiring high precision |
Trace
Generic trace data container for time-series measurements.
Used for various physiological signals that require temporal sampling.
| Field | Type | Label | Description |
| sample | Measurement | repeated | Array of time-stamped measurement samples |
point_types.proto
TopPoint2dFloat
Represents a 2D point with floating-point coordinates.
Used for precise positioning, normalized coordinates, and sub-pixel accuracy.
| Field | Type | Label | Description |
| x | float | X coordinate as floating-point value |
|
| y | float | Y coordinate as floating-point value |
Point2dInt32
Represents a 2D point with integer coordinates.
Used for pixel-based coordinate systems and discrete positioning.
| Field | Type | Label | Description |
| x | int32 | X coordinate as 32-bit signed integer |
|
| y | int32 | Y coordinate as 32-bit signed integer |
Point3dFloat
Represents a 3D point with floating-point coordinates.
Used for spatial positioning, depth information, and 3D landmark representation.
| Field | Type | Label | Description |
| x | float | X coordinate as floating-point value |
|
| y | float | Y coordinate as floating-point value |
|
| z | float | Z coordinate (depth) as floating-point value |
status.proto
TopStatusValue
Container for status code values.
Wraps status codes for transmission and storage.
| Field | Type | Label | Description |
| value | StatusCode | The status code indicating current processing state |
|
| timestamp | int64 | A frame timestamp, in absolute microseconds since the Unix epoch, associated with the status code |
StatusCode
Status codes for physiological video processing operations.
Indicates any issues encountered during preliminary visual frame analysis or processing.
| Name | Number | Description |
| OK | 0 | Processing completed successfully without issues |
| NO_FACES_FOUND | 1 | No faces were detected in the input data |
| MORE_THAN_ONE_FACE_FOUND | 2 | Multiple faces detected when only one is expected |
| FACE_NOT_CENTERED | 3 | The detected face is not properly centered in the frame |
| FACE_TOO_BIG_OR_TOO_SMALL | 4 | The face size in the frame is outside acceptable range (too big or too small) |
| IMAGE_TOO_DARK | 5 | The input image lighting is too dark for reliable processing |
| IMAGE_TOO_BRIGHT | 6 | The input image lighting is too bright for reliable processing |
| CHEST_TOO_FAR_OR_NOT_ENOUGH_SHOWING | 7 | The chest is too far away or insufficient chest area is visible |
| PROCESSING_NOT_STARTED | 8 | Processing has not been started yet |
| UNAUTHORIZED | 9 | User is not authorized to perform this operation |
| CAMERA_TUNING_IN_PROGRESS | 10 | Camera is currently tuning; wait before recording |
Scalar Value Types
| .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby |
| double | double | double | float | float64 | double | float | Float | |
| float | float | float | float | float32 | float | float | Float | |
| int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
| int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum |
| uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) |
| uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) |
| sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
| sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum |
| fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) |
| fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum |
| sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) |
| sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum |
| bool | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass | |
| string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) |
| bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) |