================================================================================
EQ-MEMOPS CRATE: COMPREHENSIVE EXPLORATION SUMMARY
================================================================================

LOCATION: /mnt/code/dev/Rust/energy-quotient/pq-synapse/server/crates/eq-memops

================================================================================
1. SHARED MEMORY ABSTRACTIONS
================================================================================

1.1 Core Abstraction Layer
----------------------------
- SharedMemoryAccess<T> trait: Generic interface for any memory backend
  - Implemented by: SensorSharedMemory, ArrowSharedMemory
  - Key methods: write_position(), read_position(), prepare_data_at()
  - Type parameters for flexibility: Error type, Control structure, Data type

- SharedMemBufferState enum: Health tracking
  - Normal: Healthy operation
  - Empty: No data
  - OverrunRiskWarning: >50% full (circular buffer at risk)
  - Overrun: Writer wrapped, may overwrite unread data
  - Error: Unrecoverable state

1.2 Buffer Structure
-------------------
- Circular ring buffers for deterministic memory usage
- No dynamic allocation after initialization
- Fixed capacity prevents memory fragmentation
- Wrap-around arithmetic for efficient circular access


================================================================================
2. DATA STRUCTURES FOR SHARED MEMORY
================================================================================

2.1 Raw Sensor Data (SensorDataRow)
------------------------------------
7 channels × 24-bit signed integers per sample:
  - ia, ib, ic: Phase A/B/C currents
  - va, vb, vc: Phase A/B/C voltages
  - i_n: Neutral current

2.2 Batch Processing (DataBuffer)
----------------------------------
- Converts 24-bit samples to 32-bit integers
  ├─ i24→i32 conversion: sign-extension for negatives, zero-pad positives
  ├─ Circular buffer: write_pos tracks insertion point
  ├─ Capacity-based size tracking: len tracks current data
  └─ Per-row: 7 channels × 4 bytes = 28 bytes/row

- Batch Views provide two access patterns:
  ├─ Raw View: All channels, unscaled i32 values
  └─ Scaled View: Selected channels, f32 values with per-channel scaling

2.3 Arrow Data Structures (ArrowSharedMemory)
----------------------------------------------
- ArrowSharedControl: Lock-free write coordination
  ├─ write_pos: Current offset in circular buffer
  ├─ generation: Batch counter (incremented atomically)
  └─ batch_infos[1024]: Metadata for up to 1024 batches
     └─ Each entry: offset, size, timestamp (all Atomic)

- ReaderControl: Per-reader position tracking
  ├─ Up to 16 concurrent readers (0-15)
  ├─ positions[16]: Where each reader is in data
  └─ active[16]: Active status for reader slots

- Memory Layout (4096-byte header):
  Offset 0-64:      ArrowSharedControl (64-byte aligned)
  Offset 64-65536:  batch_infos[1..1023] continuation
  Offset 65536+:    ReaderControl
  Offset 4096+:     Circular data buffer (serialized Arrow IPC)


================================================================================
3. READER/WRITER PATTERNS
================================================================================

3.1 Generic SharedMemoryReader<M, S, T>
----------------------------------------
Three type parameters:
  M: Shared memory backend (SensorSharedMemory, ArrowSharedMemory)
  S: Data sink (channel, buffer, file writer)
  T: Data type (WaveBlockData, RecordBatch)

Builder Pattern:
  SharedMemoryReaderBuilder provides:
  - with_shmem(Arc<M>): Set memory backend
  - with_sink(S): Set data destination
  - with_core_id(CoreId): Set CPU affinity
  - with_metrics(bool): Enable telemetry

Specialized default for CPOW:
  SharedMemoryReaderBuilder::default(sink)
  - Auto-loads SynapseConfig
  - Defaults to core 3 for affinity
  - Configured for WaveBlockData type

3.2 Main Run Loop Algorithm
-----------------------------
Loop flow (while !terminate_flag):

1. Check write_pos from shared memory (Acquire ordering)
2. Calculate available data:
   - if write_pos > read_pos: Normal progression
   - if write_pos < read_pos: Wrapped circular buffer
   - if equal: Empty, sleep with exponential backoff (1-1000µs)
3. Detect overrun: prev_write_pos > write_pos indicates wrap-around
4. Read N_ROWS_PER_BLOCK (256 rows = 8ms @ 32kHz):
   - prepare_data_at(read_pos, 256)
   - sink.try_consume(data)
   - Advance: read_pos = (read_pos + 256) % num_rows
5. Update metrics and buffer state
6. Handle failures: Sleep 10ms, retry

3.3 Multi-Reader Support
------------------------
- Up to 16 concurrent readers (slots 0-15)
- Thread-local caching: Each thread remembers its reader ID
  └─ HashMap<(instance_id, _), reader_id>
  └─ Reduces registration overhead
- Cache-line aligned reader states (64-byte alignment)
  └─ Prevents false sharing between CPUs
- Registration via compare-and-swap:
  - Find inactive slot
  - CAS loop until successful
  - Initialize position = write_pos


================================================================================
4. ARROW SHARED MEMORY SUPPORT
================================================================================

4.1 ArrowIpc Writer
-------------------
Generic writer: ArrowIpc<T, S, W, D>
  T: Data type (WaveBlockData)
  S: Scaling strategy (iscale, vscale)
  W: Backend writer (FileWriter for Parquet)
  D: Destination (file path)

4.2 Batch-to-Arrow Conversion
------------------------------
convert_batch_to_arrays(batch, timestamp):
1. Create timestamp array: Int64Array([ts, ts, ...])
2. Get per-channel scale factors
3. Parallel conversion (rayon) per channel:
   for each channel in parallel:
     - Read i32 values
     - Multiply by scale factor
     - Create Float32Array
4. Combine into Vec<Arc<dyn Array>>
5. RecordBatch::try_new(schema, arrays)

Result: Arrow IPC serialized format
  - Schema: {timestamp: i64, ia: f32, va: f32, ...}
  - Rows: Typically 256 (one block)
  - Size: 1-10KB after serialization

4.3 Write to Shared Memory
---------------------------
write(batch, timestamp):
1. Serialize RecordBatch to Arrow IPC format
2. Check free space accounting for all active readers
3. Handle wrap-around: if wp + size >= total_size, wrap to data_offset
4. Copy serialized data to shmem[new_wp]
5. Update batch_info[generation % 1024]:
   - offset = new_wp
   - size = serialized_size
   - timestamp = ts
   (All Release ordered)
6. Increment generation counter
7. Update write_pos
(All atomic operations for lock-free coordination)

4.4 Read from Shared Memory
----------------------------
read(reader_id):
1. Verify reader_id is registered and active
2. Load generation (Acquire ordered)
3. Calculate batch_idx = generation % 1024
4. Get batch metadata:
   - offset = batch_infos[idx].offset
   - size = batch_infos[idx].size
   - timestamp = batch_infos[idx].timestamp
5. Check for overrun: reader's position relative to offset/write_pos
6. Copy data from shmem to temporary buffer
7. Deserialize using Arrow StreamReader
8. Update reader's position
9. Return (RecordBatch, timestamp)


================================================================================
5. MEMORY LAYOUT AND SYNCHRONIZATION
================================================================================

5.1 Memory Alignment Strategy
------------------------------
Cache-line alignment (64 bytes) prevents false sharing:
  - ArrowSharedControl: 64-byte aligned
  - ReaderControl: 64-byte aligned
  - ReaderState: 64-byte aligned with exact padding calculation

Layout ensures:
  - Each atomic update by one thread doesn't invalidate another's cache line
  - Lock-free performance without mutex contention

5.2 Atomic Memory Ordering
---------------------------
Release/Acquire pattern for cross-process synchronization:

Writer (Release ordered):
  1. Copy data to shared memory
  2. Update batch_info fields (Release)
  3. Update write_pos (Release)
  4. Increment generation (Release)
  → All reader sees complete batch before position update

Reader (Acquire ordered):
  1. Load generation (Acquire)
  2. Load batch_info fields (Acquire)
  3. Can safely read from shmem[offset...size]
  → Sees all of writer's changes before accessing data

5.3 Circular Buffer Arithmetic
-------------------------------
Write side:
  new_pos = if (pos + size >= total_size) { data_offset } else { pos + size }

Read side:
  read_pos = (read_pos + N_ROWS) % num_rows

Available data calculation:
  if write_pos > read_pos:
    available = write_pos - read_pos
  else if write_pos < read_pos:
    available = total_size - read_pos + write_pos  (wrapped)
  else:
    available = 0  (empty)


================================================================================
6. ERROR HANDLING HIERARCHY
================================================================================

MemopsError
  ├─ Ipc(IpcError)
  ├─ Buffer(String)
  ├─ Io(io::Error)
  └─ Other(String)

IpcError
  ├─ ArrowIpc(ArrowIPCError)
  ├─ Sensor(SensorError)
  ├─ Channel(ChannelError)
  ├─ Reader(SharedMemoryReaderError)
  └─ Other(String)

ArrowIPCError
  ├─ Io(io::Error)
  ├─ Arrow(ArrowError)
  ├─ Parquet(ParquetError)
  ├─ Writer(WriterError)
  ├─ Schema(String)
  ├─ Batch(BatchError)
  ├─ Config(String)
  ├─ Destination(String)
  └─ Other(String)

ArrowSharedMemoryError
  ├─ SharedMemory(String)
  ├─ Arrow(ArrowError)
  ├─ MemoryFull
  ├─ NoData
  ├─ InvalidReader(usize)
  └─ BufferOverrun

Context propagation pattern:
  error.with_context("where this happened")
  - String errors embed context in message
  - Enum errors log context via tracing::debug
  - Preserves error type while enriching diagnostics


================================================================================
7. METRICS AND TRACING
================================================================================

ReaderMetrics struct (atomic fields):
  - blocks_processed: Total blocks successfully read
  - last_read_time: Timestamp of last successful read
  - shared_mem_buffer_state: Current buffer health
  - sink_buffer_state: Downstream channel status
  - overflow_count: Times sink rejected data
  - read_errors: Recoverable errors encountered
  - processing_rate: Blocks per unit time (reserved)

Buffer states tracked:
  - SharedMemBufferState: Normal, Empty, OverrunRiskWarning, Overrun, Error
  - ChannelBufferState: Normal, Buffering, WritingToDisk

Integration:
  - Global READER_METRICS (lazy static)
  - Implements RawMetrics trait for eq-trace
  - Per-reader state tracking via ReaderTrace trait
  - Centralized telemetry collection


================================================================================
8. NEON OPTIMIZATION (ARM)
================================================================================

neon_ops/conversion.rs provides:
  - convert_i32_to_f32_neon(): Safe wrapper for NEON acceleration
  - convert_i32_to_f32_neon_raw(): Unsafe in-place conversion
  - i32_to_f32_scaled(): Conditional dispatcher

Process:
  1. Pads i32 buffer to multiple of 4
  2. Calls C function: process_frames_neon()
  3. In-place memory reinterpretation as f32
  4. Scaling applied during SIMD operations
  5. Falls back to scalar on non-ARM architectures

Performance benefit:
  - Vectorized conversion (4 values per instruction)
  - Scaling included in same pass
  - No intermediate buffer allocation
  - ~3-4x speedup vs scalar on ARM64


================================================================================
9. KEY PERFORMANCE CHARACTERISTICS
================================================================================

Throughput:
  - 32kHz × 7 channels = 224,000 samples/second
  - Raw data: 896 KB/s (i32 format)
  - With Arrow overhead: ~1.8 MB/s
  - Parquet batches: 320,000 rows (10s) = ~9-12 MB/batch

Latency:
  - Acquisition to processing: ~8ms per block
  - Processing to Arrow IPC: ~1-2ms
  - Arrow IPC to reader: <1ms
  - End-to-end: ~10-15ms (negligible for 10s+ windows)

Memory:
  - Header: 4096 bytes
  - Per-reader overhead: 64 bytes (cache-line aligned)
  - 16 readers: ~1KB total
  - Circular buffer: Configurable, typically 1-5MB

Lock-free properties:
  - No mutexes in data path
  - Atomic operations with Release/Acquire ordering
  - Cache-line alignment prevents false sharing
  - Minimal copying (2 copies: sensor→processing→Arrow IPC)


================================================================================
10. KNOWN LIMITATIONS AND TODOS
================================================================================

Incomplete implementations:
  ✗ ArrowSharedMemory::new() (line 364): Blocked on requirements
  ✗ MultiReaderState::new() (line 607): Incomplete state management
  ✗ cpow_raw.rs: SharedMemoryReaderRunner implementation stub

Design TODOs:
  - File rotation timestamp handling (arrow.rs:70-72)
  - Batch view purpose unclear (batch.rs, may be removed)
  - context method pattern (error handling could be optimized)
  - Multi-reader state implementation details


================================================================================
11. INTEGRATION POINTS
================================================================================

Consumer crates:
  - synapse-cpow/acquire: Writes raw sensor data
  - synapse-cpow/process: Reads, processes, writes Arrow
  - eq-processing: Signal processing pipelines
  - eq-io: File I/O operations
  - synapse-web services: Real-time dashboard APIs

Dependency graph:
  eq-memops depends on:
  ├─ eq-cpow-sensor: SensorSharedMemory interface
  ├─ eq-cpow-types: WaveBlockData, SensorDataRow
  ├─ eq-common: SharedMemoryAccess trait, DataSink trait
  ├─ eq-core: Zenoh, VOLTAGE_SCALE, CURRENT_SCALE
  ├─ eq-trace: Telemetry integration
  ├─ synapse-config: Configuration loading
  └─ External: arrow, shared_memory, flume, rayon, etc.


================================================================================
12. ARCHITECTURE SUMMARY
================================================================================

Three-layer shared memory architecture:

Layer 1: Raw Sensor Data
  └─ SensorSharedMemory: 24-bit samples from hardware
     └─ 7 channels × 32kHz = 224k samples/sec

Layer 2: Processing
  └─ DataBuffer: Converts i24→i32, provides batch views
  └─ SharedMemoryReader: Core-pinned polling loop
  └─ Batch processing: Scaling, conversion to Arrow

Layer 3: Arrow IPC
  └─ ArrowSharedMemory: Lock-free multi-reader coordination
  └─ Up to 16 concurrent readers across processes
  └─ Generation-based batch indexing
  └─ Atomic position tracking

Layer 4: Persistence
  └─ Parquet files: Time-based rotation
  └─ Web APIs: Real-time dashboards via Arrow

This enables:
  ✓ Deterministic 32kHz acquisition (core-pinned reader)
  ✓ Efficient batch processing (column-major Arrow)
  ✓ Lock-free cross-process communication
  ✓ Real-time analytics dashboards
  ✓ Historical data persistence

================================================================================
END OF SUMMARY
================================================================================