File IPC Backend

The File IPC backend writes metrics and video frames to the local filesystem in JSONL (JSON Lines) format. This enables integration with batch processing systems, offline analysis tools, and applications that prefer file-based data exchange.

Overview

When using the File IPC backend, physiology_server writes four types of output files:

• Core Metrics (core_metrics.jsonl): Pulse, breathing, and other vitals from Physiology Core
• Edge Metrics (edge_metrics.jsonl): On-device metrics computed by Edge
• Status (status.jsonl): System status updates and state changes
• Frames (latest.jpg or numbered frames): JPEG-compressed video frames

All metrics are written as JSON envelopes with timestamps, enabling precise temporal correlation.

Configuration

Command Line Usage

Enable File IPC backend with direct webcam / camera device input:

physiology_server \
  --streaming_backend=file \
  --use_camera \
  --filesystem_ipc_config_path=/path/to/config.json \
  --also_log_to_stderr

Required Flags:

• --streaming_backend=file: Select File IPC backend
• --use_camera: Enable direct camera input (required for alternative IPC backends)

Optional Flags:

• --filesystem_ipc_config_path: Path to JSON configuration file
• Individual configuration flags (see below)

Configuration File

Generate a default configuration file:

physiology_server --save_default_file_ipc_config

This creates default_file_ipc_config.json with all available options:

{
  "base_directory": "./file_ipc_output",
  "create_if_missing": true,
  "truncate_on_init": true,
  "max_frame_count": 0,
  "use_file_locking": true,
  "frame_extension": "jpg",
  "compress_frames": true,
  "io_buffer_size": 8192,
  "enable_debug_logging": false
}

Configuration Options

Option	Type	Default	Description
base_directory	string	"./file_ipc_output"	Root directory for all output files
create_if_missing	bool	true	Create base directory if it doesn't exist
truncate_on_init	bool	true	Clear existing files on startup
max_frame_count	int	0	Maximum frames to keep (0 = unlimited)
use_file_locking	bool	true	Enable file locking for concurrent access
frame_extension	string	"jpg"	Frame file extension
compress_frames	bool	true	Enable JPEG compression for frames
io_buffer_size	int	8192	Buffer size for file I/O (bytes)
enable_debug_logging	bool	false	Enable verbose logging

Command Line Overrides

Individual configuration options can be overridden via command line flags:

physiology_server \
  --streaming_backend=file \
  --use_camera \
  --file_ipc_base_directory=/custom/path \
  --file_ipc_compress_frames=true \
  --file_ipc_max_frame_count=1000

Output Format

JSON Envelope Structure

All metric outputs use a consistent JSON envelope format:

{
  "type": "core_metrics",
  "timestamp": 1234567890123456,
  "payload": { ... }
}

• type: Message type identifier, one of {"core_metrics","edge_metrics", "status", "frame"}
• timestamp: Microseconds since epoch
• payload: Type-specific data

Core Metrics Example

{
  "type": "core_metrics",
  "timestamp": 1234567890123456,
  "payload": {
    "pulse": {
      "strict": {"value": 72.5, "confidence": 0.95}
    },
    "breathing": {
      "strict": {"value": 16.2, "confidence": 0.89}
    }
  }
}

Edge Metrics Example

{
  "type": "edge_metrics",
  "timestamp": 1234567890123457,
  "payload": {
    "eda": 0.42,
    "upper_breathing_trace": 1.23
  }
}

Status Example

{
  "type": "status",
  "timestamp": 1234567890123458,
  "payload": {
    "code": 0,
    "message": "OK"
  }
}

Frame Example

{
  "type": "frame",
  "timestamp": 1234567890123459,
  "payload": {
    "data": "/9j/4AAQSkZJRgABAQEAYABgAAD...",
    "width": 640,
    "height": 480,
    "format": "jpeg"
  }
}

Frame data is base64-encoded JPEG image data.

Directory Structure

The File IPC backend creates the following directory structure:

base_directory/
├── core_metrics.jsonl      # Core vitals (pulse, breathing, etc.)
├── edge_metrics.jsonl      # Edge metrics (EDA, breathing traces, etc.)
├── status.jsonl            # System status updates
└── latest.jpg              # Most recent frame (or numbered frames)

Python Client Example

Reading metrics from File IPC output:

import json
from pathlib import Path
 
def read_file_ipc_metrics(base_dir):
    """Read metrics from File IPC output directory."""
    base_path = Path(base_dir)
 
    # Read core metrics
    core_metrics = []
    with open(base_path / "core_metrics.jsonl") as f:
        for line in f:
            envelope = json.loads(line)
            core_metrics.append(envelope)
 
    # Read edge metrics
    edge_metrics = []
    with open(base_path / "edge_metrics.jsonl") as f:
        for line in f:
            envelope = json.loads(line)
            edge_metrics.append(envelope)
 
    return core_metrics, edge_metrics
 
# Usage
core, edge = read_file_ipc_metrics("./file_ipc_output")
print(f"Collected {len(core)} core metric samples")
print(f"Collected {len(edge)} edge metric samples")

A complete example client is available in the samples directory:

• samples/alt_ipc_example_on_prem_client.py: Reads File IPC output and displays frames

Performance Considerations

File I/O

• Buffering: Adjust io_buffer_size based on your storage device (default: 8192 bytes)
• SSD Recommended: For high frame rates, use SSD storage to minimize I/O latency
• File Locking: Disable use_file_locking if single-consumer or performance-critical

Frame Output

• Compression: JPEG compression significantly reduces disk usage (recommended: enabled)
• Frame Limiting: Set max_frame_count to prevent unbounded storage growth
• Frame Format: Only JPEG is currently supported

Concurrent Access

• File Locking: Enabled by default for safe concurrent reads
• Multiple Readers: JSONL format supports streaming reads (tail -f style)
• Write Ordering: Metrics are written in timestamp order

Troubleshooting

Permission Errors

Symptom: Failed to create directory or Permission denied errors

Solution:

# Ensure directory exists and has correct permissions
mkdir -p /path/to/output
chmod 755 /path/to/output

Disk Space Issues

Symptom: System runs out of disk space

Solutions:

• Enable truncate_on_init to clear old data on restart
• Set max_frame_count to limit frame accumulation
• Implement external log rotation for JSONL files
• Monitor disk usage with system tools

File Locking Conflicts

Symptom: Resource temporarily unavailable errors

Solutions:

• Ensure no zombie processes are holding file locks
• Disable use_file_locking if not needed for your use case
• Use lsof to identify processes with open files:

  lsof | grep file_ipc_output

Missing Output Files

Symptom: Expected files are not created

Solutions:

• Verify --use_camera flag is set (required for alternative IPC)
• Check base_directory path is correct
• Enable enable_debug_logging for detailed diagnostics
• Ensure create_if_missing is enabled

Use Cases

Offline Analysis

Record a session for later analysis:

# Start recording
physiology_server \
  --streaming_backend=file \
  --use_camera \
  --file_ipc_base_directory=./session_$(date +%Y%m%d_%H%M%S)
 
# Analyze later
python analyze_session.py ./session_20250103_143022/

Batch Processing

Integrate with data pipelines:

# Write to staging directory
physiology_server \
  --streaming_backend=file \
  --file_ipc_base_directory=/data/staging/physiology
 
# Process with external tools
./process_metrics.sh /data/staging/physiology

Debugging and Development

Capture ground truth data:

# Record with verbose logging
physiology_server \
  --streaming_backend=file \
  --file_ipc_enable_debug_logging=true \
  --also_log_to_stderr \
  --file_ipc_base_directory=./debug_session

See Also

• Redis IPC Backend - Redis IPC backend for pub/sub streaming
• SmartSpectra OnPrem Architecture - Overall system architecture
• samples/alt_ipc_example_on_prem_client.py - Example Python client