m
The SmartSpectra OnPrem SDK provides a local gRPC service for measuring vital signs, such as heart rate and breathing rate, which can be used in custom applications written in any language that gRPC supports. We currently offer tutorials and compiled Protocol Buffer libraries for Python, but others are available on demand. If you'd like to see higher support levels for specific languages, please reach out to <support@presagetech.com>.
Important: The SmartSpectra OnPrem SDK is not part of the SmartSpectra community-edition SDKs but is available to select customers and partners. If you would like to use the OnPrem SDK, please reach out to us via the contact form.
ATTENTION: This is external-facing documentation for SmartSpectra OnPrem. For internal usage in the context of the Physiology monorepo, please refer to the SmartSpectra OnPrem Developer Documentation.
Getting Started
This section, when followed linearly, will guide you through the process of setting up a basic SmartSpectra OnPrem client in Python for the very first time. If you would like an abbreviated version of this document check out the Quick Start Guide.
Installation Prerequisites
Install Python 3.11 System-wide Dependency
1) Install PPA for Python 3.11:
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
2) Install Python 3.11, pip, & development headers:
sudo apt install python3.11 python3.11-dev python3.11-venv
3) You should now be able to run Python 3.11 from any terminal by invoking python3.11. Verify Python 3.11 installation:
This should print: Python 3.11.xx, where xx is the latest micro/bugfix version available.
4) Update pip on Python 3.11:
python3.11 -m pip install -U pip
Installation
- Before you begin, if you haven't already, you'll need to obtain the SmartSpectra OnPrem package. It is not available with the rest of the public-facing SDK sources or packages; contact us to obtain it through contractual agreements with Presage Technologies. You will be provided with a link to the package download.
Once you receive the download link, download the entire package and extract its contents to a directory of your choice. You should see files and directories with the following names, sans the version number. Adjust the version number as needed to match yours in the subsequent commands.
- docs/ (HTML documentation - open index.html in browser)
- docs_markdown/ (Raw markdown documentation for LLM-assisted development)
- samples/ (Python example clients)
- typescript/ (TypeScript SDKs and sample applications)
- QUICKSTART.md (Quick start guide)
- README.md (This file)
- grpc_core_server.py
- libopencv-dev_4.10.0-jammy-presage-2_amd64.deb
- libphysiologyedge-dev-1.7.0-amd64.deb
- libsmartspectra-dev-1.2.0-amd64.deb
- physiology_core-1.7.0-py3-none-any.whl
- physiology_edge_protobuf-1.7.0-py3-none-any.whl
- smartspectra_protobuf-1.7.0-py3-none-any.whl
- smartspectra_ipc-1.7.0-py3-none-any.whl
📚 Documentation Guide (Important for LLM-Assisted Development):
The package includes documentation in two formats for different use cases:
- HTML Documentation (docs/index.html):
- Professional web-based documentation with navigation, search, and styling
- Open in browser for reading and reference
- Best for: Human browsing, comprehensive navigation
- Raw Markdown Documentation (docs_markdown/*.md):
- Plain text markdown files optimized for LLM context and terminal viewing
- Use with AI assistants (Claude, ChatGPT, Copilot, etc.)
- grep-friendly and searchable with standard command-line tools
- Works with terminal markdown viewers (bat, glow, mdless, etc.)
- Best for: LLM-assisted development, command-line workflows, offline work
Available Markdown Documentation Files:
docs_markdown/
├── architecture.md # System architecture overview
├── developer.md # Build and development guide
├── errors_and_logs.md # Error handling and logging patterns
├── examples.md # Python example clients guide
├── file_ipc.md # File IPC backend configuration
├── redis_ipc.md # Redis IPC backend configuration (recommended)
├── smartspectra_ipc.md # SmartSpectra-IPC Python package API
├── typescript.md # TypeScript SDK and web applications guide
└── health_readiness_tutorial.md # Health check endpoints
For Intermittent Developers (returning after weeks/months):
These raw markdown files are designed for developers who work on-and-off with the SDK. When returning to the codebase:
- Feed to your LLM: Copy entire files to Claude/ChatGPT for instant context
- Terminal-friendly: cat docs_markdown/typescript.md | less for quick reference
- Searchable: grep -r "MetricsCollector" docs_markdown/ to find APIs
- Self-contained: Each doc includes complete examples and troubleshooting
Quick LLM Workflow Example:
# need to build Redis dashboard
cat docs_markdown/typescript.md # Send to Claude/ChatGPT
# Ask: "How do I set up the React dashboard with Redis backend?"
# LLM now has complete context from comprehensive markdown
Set up a Python 3.11 virtual environment in a directory from which you intend to run the server ("server startup location").
python3.11 -m venv .venv
source .venv/bin/activate
- Install deb packages strictly in this order via sudo apt install ./PACKAGE_FILENAME.
- libopencv-dev_4.10.0-jammy-presage-<build_number>_amd64.deb
- libphysiologyedge-dev-<build_number>amd64.deb
- libsmartspectra-dev-<build_number>-amd64.deb
- Install wheel packages in the active virtual environment via pip install ./<package-filename>
- physiology_edge_protobuf-<build_number>-py3-none-any.whl
- smartspectra_protobuf-<build_number>-py3-none-any.whl
- physiology_core-<build_number>-py3-none-any.whl
- smartspectra_ipc-<build_number>-py3-none-any.whl
- Copy the grpc_core_server.py file from the package into the server startup location.
Running the Physiology Server
Once you have completed the installation steps, you can start the Physiology Server to begin processing video data for vital sign measurements.
Activate your virtual environment (if you created one during installation):
source .venv/bin/activate
Optionally, set environment variable for BP calibration (if using blood pressure features):
Start the Physiology Server:
physiology_server --also_log_to_stderr --port 50051 --venv_directory .venv --buffer_duration 0.2 --enable_phasic_bp=true
Command line options explained:
- --also_log_to_stderr: Display log messages in the terminal
- --port 50051: Set the server port (default: 50051)
- --venv_directory .venv: Specify the virtual environment directory (use the path where you created your venv)
- --buffer_duration 0.2: Set data buffer duration in seconds (affects Core metrics output frequency)
- --enable_phasic_bp=true: Enable blood pressure inference (optional)
- The server will start and display initialization messages. You should see output indicating the server is ready to accept connections.
Important Notes:
Running the Hello World Example Client
The Hello World example demonstrates basic usage of the SmartSpectra OnPrem SDK by connecting to the Physiology Server, processing camera input, and displaying vital sign metrics.
Prerequisites
- Physiology Server must be running (see Running the Physiology Server)
- A camera connected to your system (webcam, USB camera, etc.)
- Python example files in your current working directory
Running the Example
- Ensure the Physiology Server is running on port 50051 (default).
- Note: With the Hello world example the physiology_server can't have the –use_camera flag enabled
Run the Hello World client:
python samples/hello_world_on_prem_client.py
- Using the client:
- The application will open a camera feed window
- Press ‘'s’ to start recording and begin vital sign processing
- Press 'q'` to quit the application
- Vital sign metrics (heart rate, breathing rate, etc.) will be displayed in the terminal
- Expected behavior:
- The client connects to the Physiology Server
- Camera feed appears in a window
- After pressing 's' to start recording, you'll see metrics output in the terminal
- Processing continues until you press 'q' to quit
Troubleshooting
- Camera not found: Ensure your camera is connected and not being used by another application
- Connection failed: Verify the Physiology Server is running and accessible on port 50051
- No metrics output: Make sure you pressed 's' to start recording, and that your face is visible in the camera feed
Other Examples
The OnPrem package includes additional example clients demonstrating advanced features:
- full_example_on_prem_client.py: Comprehensive example showing complete gRPC API usage including frame upload, metrics streaming, and control endpoints
- alt_ipc_example_on_prem_client.py: Alternative IPC backend example supporting both File and Redis backends for systems that cannot use gRPC
- redis_ipc_metrics_saving_client.py: Redis metrics collection using the SmartSpectra-IPC package, demonstrating session management and data aggregation
These examples demonstrate alternative deployment patterns, IPC backend usage, camera rotation for micromotion tracking, and additional physiological metrics (EDA, phasic BP).
For detailed information on running these examples with appropriate physiology_server configurations, see: OnPrem Examples Guide OnPrem Python Examples Guide
Minimal Test Script
If you want to verify everything works without the examples:
import grpc
import smartspectra.on_prem.physiology_service_pb2_grpc as ps_grpc
import smartspectra.on_prem.physiology_service_pb2 as ps
channel = grpc.insecure_channel('localhost:50051')
client = ps_grpc.PhysiologyServiceStub(channel)
response = client.GetVersion(ps.VersionRequest())
print(f"Server version: {response.version}")
print("✅ Connection successful!")
Save as test_connection.py and run:
python test_connection.py
Advanced Features
Direct Camera Input
The physiology_server executable supports direct camera input, allowing it to process video frames without requiring a separate client application to upload frames via gRPC.
Command Line Options:
- --use_camera: Enable direct camera input (required for alternative IPC backends)
- --camera_device_index=N: Select camera device by index (default: 0)
Example Usage:
physiology_server --use_camera --camera_device_index=0 --also_log_to_stderr
This mode is particularly useful when:
- Using alternative IPC backends (File or Redis) for data streaming
- Running standalone processing without a client application
- Integrating with systems that need direct server-side video capture
Note: When using the standard gRPC backend, clients typically upload frames via the AddFrameWithTimestamp RPC endpoint. Direct camera input bypasses this and captures frames directly on the server side.
Face Perception Models
The flags --use_full_range_face_detection and --use_full_pose_landmarks enable more powerful face perception models. These can deal with the face occupying a smaller pixel area because the camera is rotated and/or farther away. They are enabled by default with --enable_micromotion but this flag requires the knees and glutes to be in view of the camera. This usually requires the camera to be rotated and greater camera-to-face distance.
Alternative IPC Backends
In addition to the standard gRPC streaming interface, physiology_server supports alternative IPC (Inter-Process Communication) backends for data output:
- Redis IPC (Recommended): Publishes metrics and frames to Redis pub/sub channels - ideal for distributed systems, multiple concurrent clients, and web applications
- File IPC (Alternative option): Writes metrics and frames to local filesystem (JSONL format) - available for systems where Redis is not an option
- Use options to turn on Redis–streaming_backend=redis --use_camera
These backends enable integration with systems that cannot use gRPC or prefer different data transport mechanisms.
Key Features:
- Real-time metrics streaming to files or Redis
- JPEG frame compression for efficient storage/transmission
- Configurable output formats and destinations
- Support for multiple concurrent consumers
For detailed information on configuring and using these backends, see:
Video and Timestamp Recording
The physiology_server supports recording raw video frames to MKV files with companion timestamp files. This feature is useful for:
- Offline analysis and debugging
- Creating datasets for research
- Audit trails and compliance requirements
Command Line Option:
physiology_server --video_output_dir=/path/to/output --use_camera
note –use_camera isn't strictly necessary for video recording, i.e. you can use the AddFrameWithTimestamp gRPC endpoint to upload frames to the server and physiology_server record them to a file.
Output Files:
- video_YYYYMMDD_HHMMSS.mkv: Video file with MJPEG codec
- video_YYYYMMDD_HHMMSS.txt: Timestamp file (microseconds, one per frame)
Input recording for metrics processing is controlled via the SetRecording gRPC endpoint.
Quick replay workflow:
- Start any Redis subscribers you care about (metrics collector, React dashboard, etc.).
- Launch physiology_server with --input_video_path (and optional --input_video_time_path) instead of --use_camera.
- The server auto-enables recording for the replay and will publish a stop when the file ends.
When feeding a pre-recorded video the server also relaxes the preprocessing inter-frame delay to 33 ms unless you override --interframe_delay_ms, which helps prevent dropped frames. Tune the flag if your captures require a different cadence.
SmartSpectra-IPC Python Package
The smartspectra-ipc Python package provides utilities for working with alternative IPC backends. This package is distributed as a wheel file (smartspectra_ipc-*.whl) with the OnPrem package and can be installed via pip.
Installation:
pip install smartspectra_ipc-1.7.0-py3-none-any.whl
Current Features:
- MetricsCollector: High-level class for collecting and aggregating metrics from Redis pub/sub channels
- Automatic conversion from array-of-structs (streaming) to struct-of-arrays (analysis)
- Session management with metadata tracking
- Dual output: continuous JSONL file + final JSON summary
Key Capabilities:
The MetricsCollector class simplifies Redis metrics collection by:
- Subscribing to core and edge metrics channels automatically
- Aggregating time-series data with proper timestamp handling
- Detecting and handling timestamp resets/wraps
- Writing both streaming JSONL and final aggregated JSON output
- Providing real-time snapshots of collected data
Example Usage:
from pathlib import Path
config = MetricsCollectorConfig(
redis_host="localhost",
redis_port=6379,
redis_key_prefix="physiology",
output_jsonl_path=Path("./metrics.jsonl"),
output_json_path=Path("./metrics_summary.json")
)
collector = MetricsCollector(config)
collector.start()
collector.set_session_id("experiment_001")
snapshot = collector.get_current_snapshot()
print(f"Collected {snapshot['meta']['total_samples']} samples")
collector.stop()
For detailed information on the MetricsCollector API, configuration options, output JSON format, and usage patterns, see: SmartSpectra-IPC Package Documentation SmartSpectra-IPC Python Package
Future Extensions:
The smartspectra-ipc package is designed for expansion. Future versions may include additional utilities for File IPC processing, data transformation pipelines, and integration with popular data analysis frameworks.
Metrics Overview
The SmartSpectra SDK provides two types of metrics:
- Core Metrics (MetricsBuffer): High-level physiological measurements computed via the Physiology REST API, including refined pulse rates, breathing rates, and face detection data.
- Edge Metrics (Metrics): Real-time measurements computed locally on the device, providing breathing traces, micromotion data, and face landmarks at frame rate.
NOTE: some metrics are not publicly available yet, please contact Presage Support for any inquiries.
Both MetricsBuffer and Metrics classes are generated from the Protobuf messages. See the SmartSpectra Protocol Buffer Reference for comprehensive descriptions of all their available fields.
***Important***
- The frequency of Core Mectrics (MetricsBuffer) correspond to the buffer_duration argument on the physiology_server.
- The breathing & micromotion trace signals for Edge Metrics (Metrics) appear at the target framerate (Currently fixed at 30 Hz) and the EDA signal for Edge Metrics (Metrics) comes in periodically at approximately every 0.2 seconds
TypeScript SDKs and Web Applications
The OnPrem package includes TypeScript client libraries and sample web applications for building browser-based interfaces to SmartSpectra physiology metrics. These packages enable modern web development workflows with full TypeScript type safety.
Included Packages
@smartspectra/physiology-client
TypeScript client for accessing SmartSpectra physiology metrics from Redis. Provides:
- Real-time subscriptions to core and edge metrics via Redis pub/sub
- WebSocket gateway (MetricsGateway) for browser clients
- HTTP endpoints for MJPEG HUD streaming and health checks
- Full TypeScript type definitions for all metrics payloads
- Recording state management and control commands
@smartspectra/video-stream
MJPEG video stream client with automatic reconnection and health monitoring. Provides:
- Automatic reconnection on network errors
- Support for multiple synchronized video views
- Frame health monitoring and status display
- Placeholder management for loading states
- Zero external dependencies
Sample Applications
React Dashboard
Modern React + TypeScript dashboard demonstrating:
- WebSocket-based real-time metrics visualization
- Responsive charts with 30-second rolling windows (Recharts)
- MJPEG HUD video stream integration
- Recording control UI with shadcn/ui components
- Minimal Node.js gateway for browser-to-Redis bridge
Quick Start:
cd typescript/samples/react-dashboard
./run_dashboard.sh --server-port 8090 --port 5173
# Opens http://localhost:5173
JavaScript Frontend (Alternative option - React dashboard recommended)
Vanilla JavaScript dashboard (available as a simpler alternative):
- Direct MetricsGateway integration (no framework required)
- Real-time plotting with custom charting
- MJPEG HUD stream display
- Minimal dependencies and simple architecture
Note: For production deployments, we recommend using the React dashboard above for better performance and maintainability.
Installation and Usage
The TypeScript packages are pre-built in the OnPrem distribution and ready to use:
Install from local path:
cd your-project/
npm install ../onprem-package/typescript/packages/physiology-client
npm install ../onprem-package/typescript/packages/video-stream
Copy packages to your workspace:
cp -r typescript/packages/physiology-client your-workspace/
cp -r typescript/packages/video-stream your-workspace/
cd your-workspace/physiology-client
npm install
Basic MetricsGateway Example
import { createMetricsGateway } from '@smartspectra/physiology-client';
// Create WebSocket + HTTP gateway
const gateway = createMetricsGateway({
port: 8080,
metricsClient: {
host: '127.0.0.1',
port: 6379,
prefix: 'physiology',
},
});
await gateway.start();
// WebSocket: ws://localhost:8080/ws
// HUD Stream: http://localhost:8080/hud.mjpg
// Health: http://localhost:8080/api/health
Then connect from your browser:
const ws = new WebSocket('ws://localhost:8080/ws');
ws.onmessage = (event) => {
const message = JSON.parse(event.data);
if (message.type === 'plot_update') {
// Update charts with metrics.traces
}
if (message.type === 'rate_update') {
// Update rate displays with metrics.values
}
if (message.type === 'recording_state') {
// Update recording UI
}
};
// Start recording
ws.send(JSON.stringify({ type: 'set_recording', value: true }));
Integration with physiology_server
The TypeScript SDKs work with Redis IPC backend:
# Terminal 1: Start physiology_server with Redis backend
physiology_server \
--streaming_backend=redis \
--use_camera \
--redis_host=localhost \
--redis_port=6379 \
--redis_key_prefix=physiology \
--also_log_to_stderr
# Terminal 2: Run sample dashboard
cd typescript/samples/react-dashboard
./run_dashboard.sh
Use Cases
- Web-based monitoring dashboards: Real-time vitals display in browser
- Remote monitoring: Multiple concurrent browser clients via WebSocket
- Custom UIs: Build tailored interfaces with React, Vue, or vanilla JS
- Integration with existing web apps: Add physiology metrics to your stack
- Mobile-responsive interfaces: Modern responsive design with TypeScript safety
Documentation
For a comprehensive guide to TypeScript packages, sample applications, and integration patterns, see: TypeScript SDK and Web Applications Guide TypeScript SDK and Web Applications Guide
Additional resources:
- Package READMEs: typescript/packages/*/README.md
- Sample documentation: typescript/samples/*/README.md
- Main TypeScript overview: typescript/README.md
- Redis IPC backend configuration: Redis IPC Backend Redis IPC Backend
📚 Documentation Formats:
The package includes dual documentation formats:
- HTML (docs/index.html): Web documentation with navigation and search
- Markdown (docs_markdown/*.md): Raw files for LLM assistants (Claude, ChatGPT, etc.) and terminal viewing
For LLM-Assisted Development:
If you're using AI coding assistants or work intermittently, the markdown files in docs_markdown/ are designed to be fed directly to LLMs for instant context. Example:
# need to set up Redis metrics collection
cat docs_markdown/smartspectra_ipc.md # Copy to Claude/ChatGPT
# Now your LLM has complete MetricsCollector API context
Common Troubleshooting
Server Won't Start
Problem: physiology_server: command not found
Cause: .deb packages not installed or not in PATH
Solution:
# Verify installation
which physiology_server
# Should return: /usr/local/bin/physiology_server
# If not, reinstall packages:
sudo apt install --reinstall ./libsmartspectra-dev-*.deb
Problem: Failed to start Core server
Cause: grpc_core_server.py not found or venv not activated
Solution:
# Ensure grpc_core_server.py is in current directory
ls grpc_core_server.py
# Verify virtual environment is activated
echo $VIRTUAL_ENV # Should show path to .venv
# If not activated:
source .venv/bin/activate
Python Import Errors
Problem: ‘ImportError: No module named 'smartspectra’`
Cause: Protobuf wheels not installed
Solution:
source .venv/bin/activate
pip install smartspectra_protobuf-*.whl
pip install physiology_edge_protobuf-*.whl
pip install physiology_core-*.whl
pip install smartspectra_ipc-*.whl # For Redis examples
Problem: ‘ModuleNotFoundError: No module named 'grpcio’`
Cause: Sample dependencies not installed
Solution:
cd samples/
pip install -r requirements.txt
Camera Issues
Problem: Camera not detected or Failed to open video device
Cause: Camera in use, wrong device index, or permission issues
Solution:
# Check available cameras
ls -l /dev/video*
# Test camera access
python3 -c "import cv2; cap = cv2.VideoCapture(0); print('OK' if cap.isOpened() else 'FAIL')"
# Try different device index
python samples/full_example_on_prem_client.py --video-device 1
# Check permissions (add user to video group if needed)
sudo usermod -a -G video $USER
# Then log out and back in
Connection Errors
Problem: grpc.RpcError: failed to connect to all addresses
Cause: physiology_server not running
Solution:
# Start server in separate terminal first
physiology_server --also_log_to_stderr --port 50051 --venv_directory .venv
# Verify server is listening
netstat -tuln | grep 50051
Problem: redis.exceptions.ConnectionError: Error connecting to Redis
Cause: Redis server not running
Solution:
# Start Redis server
redis-server
# Verify Redis is running
redis-cli ping # Should return "PONG"
Metrics Not Appearing
Problem: No metrics output after starting client
Solution:
- Press s to start recording after launching the client
- Metrics only flow when recording is active
- Ensure your face is visible in the camera frame
Problem: Metrics are delayed or choppy
Solution:
# Reduce buffer duration for faster updates
physiology_server --buffer_duration 0.2 ...
# Check system load
top # Look for high CPU usage
Installation Issues
Problem: dpkg: dependency problems when installing .deb packages
Cause: .deb packages installed out of order
Solution:
# Uninstall in reverse order
sudo apt remove libsmartspectra-dev libphysiologyedge-dev libopencv-dev
# Reinstall in correct order
sudo apt install ./libopencv-dev_*.deb
sudo apt install ./libphysiologyedge-dev-*.deb
sudo apt install ./libsmartspectra-dev-*.deb
Problem: python3.11: command not found
Solution:
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.11 python3.11-dev python3.11-venv
Getting More Help
If issues persist:
- Check logs: Run server with --also_log_to_stderr flag
- Verify setup: See QUICKSTART.md for step-by-step verification
- Review examples: Check samples/README.md for detailed usage
- Contact support: Email support@presagetech.com
Further Reference
Architecture overview
┌─────────────────────┐
│ Client Application │ ← Your code (Python, TypeScript, etc.)
│ (hello_world.py) │
└──────────┬──────────┘
│ gRPC
▼
┌─────────────────────┐
│ physiology_server │ ← C++ wrapper (port 50051)
│ (SmartSpectra SDK) │
└──────────┬──────────┘
│
┌──────┴──────┐
│ │
▼ ▼
┌────────┐ ┌──────────┐
│ Edge │ │ Core │ ← Python gRPC server (port 50052)
│ (C++) │ │ (Python) │
└────────┘ └──────────┘
SmartSpectra OnPrem consists of three cooperating components:
- Core: remote analysis component that provides pulse and additional signal analysis. In OnPrem, it is exposed via a Python gRPC server (grpc_core_server.py).
- Edge: on-device preprocessing and on-edge metrics ("Edge Metrics"). It prepares inputs for Core and computes some metrics on-device.
- Physiology Server: a C++ gRPC server (physiology_server) that wraps and orchestrates Edge and Core. It starts the Core gRPC server process on demand and controls Edge via the SmartSpectra C++ SDK; Edge communicates with Core over a dedicated gRPC link.
For a deeper dive, see: SmartSpectra OnPrem Architecture SmartSpectra OnPrem Architecture
Handling errors and logs
Client applications can collect logs either by mirroring server stderr to a file (e.g., piping through tee) or programmatically via the StreamLogs gRPC endpoint. For error handling, prefer wrapping unary RPCs (e.g., StartCore, StartPreprocessing, ResetProcessing) in try/except blocks and surfacing meaningful messages; streaming RPC errors (e.g., frame upload) will surface directly in the awaiting coroutine and should trigger graceful shutdown. The docs include code snippets showing recommended patterns with asyncio.gather and exception propagation.
For a deeper dive with examples, see: Error Handling & Logging in Python SmartSpectra OnPrem Error Handling & Logging
Protobuf Reference
This reference covers the OnPrem Physiology RPC service and its endpoints, all protobuf Messages that make up the SmartSpectra OnPrem gRPC interface. It includes the service definitions exposed by the Physiology Server as well as the message types exchanged between the client and server.
For the full, generated reference, see SmartSpectraProtobuf Reference
