diff --git a/docs/design/database_architecture.md b/docs/design/database_architecture.md index a40ad68..6e148b7 100644 --- a/docs/design/database_architecture.md +++ b/docs/design/database_architecture.md @@ -1,12 +1,14 @@ -# POWR Database Architecture Diagrams +# POWR Database Architecture ## 1. Entity Relationship Diagram -This diagram shows the core database structure and relationships between tables. The design supports both raw Nostr event storage and processed data for efficient querying. +This diagram shows the core database structure and relationships between tables. The design supports both local-first operations with performance optimizations and Nostr protocol integration. Key Features: - Raw Nostr event storage in `nostr_events` - Processed exercise data in `exercise_definitions` +- Media content storage in `exercise_media` +- Cache management in `cache_metadata` - Dependency tracking in `incomplete_templates` - Efficient tag indexing in `event_tags` @@ -15,6 +17,9 @@ erDiagram nostr_events ||--o{ event_tags : contains nostr_events ||--o| exercise_definitions : processes nostr_events ||--o| incomplete_templates : tracks + exercise_definitions ||--o{ exercise_media : stores + exercise_definitions ||--o{ cache_metadata : tracks + nostr_events { string id PK string pubkey @@ -23,12 +28,14 @@ erDiagram number created_at number received_at } + event_tags { string event_id FK string name string value number index } + exercise_definitions { string event_id FK string title @@ -36,6 +43,23 @@ erDiagram string format string format_units } + + exercise_media { + string exercise_id FK + string media_type + blob content + blob thumbnail + number created_at + } + + cache_metadata { + string content_id PK + string content_type + number last_accessed + number access_count + number priority + } + incomplete_templates { string template_id FK number missing_exercise_count @@ -45,180 +69,206 @@ erDiagram ## 2. Event Processing Flow -This diagram illustrates how different types of Nostr events (Exercise Definitions, Workout Templates, and Workout Records) are processed, validated, and stored. +This diagram illustrates how both local and Nostr events are processed, validated, and stored. The system handles Exercise Definitions (33401), Workout Templates (33402), and Workout Records (33403). Key Features: -- Event type differentiation -- Validation process +- Support for both local and Nostr events +- Unified validation process +- Media content handling +- Cache management - Dependency checking -- Storage paths for complete/incomplete data +- Storage optimization ```mermaid flowchart TB subgraph Input - A[New Nostr Event] --> B{Event Type} + A[New Event] --> B{Source} + B -->|Local| C[Local Creation] + B -->|Nostr| D[Nostr Event] + C --> E{Event Type} + D --> E end - B -->|kind 33401| C[Exercise Definition] - B -->|kind 33402| D[Workout Template] - B -->|kind 33403| E[Workout Record] + E -->|kind 33401| F[Exercise Definition] + E -->|kind 33402| G[Workout Template] + E -->|kind 33403| H[Workout Record] subgraph Processing - C --> F[Validate Event] - D --> F - E --> F + F --> I[Validate Event] + G --> I + H --> I - F --> G{Valid?} - G -->|No| H[Reject Event] - G -->|Yes| I[Store Raw Event] + I --> J{Valid?} + J -->|No| K[Reject Event] + J -->|Yes| L[Store Raw Event] - I --> J[Process Event] - J --> K{Dependencies?} + L --> M[Process Event] + M --> N{Has Media?} - K -->|Missing| L[Store as Incomplete] - K -->|Complete| M[Store Processed Data] + N -->|Yes| O[Process Media] + N -->|No| P{Dependencies?} + O --> P - L --> N[Queue Missing Events] + P -->|Missing| Q[Store as Incomplete] + P -->|Complete| R[Store Processed Data] + + Q --> S[Queue Missing Events] + R --> T[Update Cache] end subgraph Storage - M --> O[(NostrEvents)] - M --> P[(ProcessedData)] - L --> Q[(IncompleteQueue)] + R --> U[(NostrEvents)] + R --> V[(ProcessedData)] + O --> W[(MediaStore)] + T --> X[(Cache)] + Q --> Y[(IncompleteQueue)] end ``` ## 3. Query and Cache Flow -This sequence diagram shows how data is retrieved, utilizing the LRU cache for performance and handling template dependencies. +This sequence diagram shows how data is retrieved, using a performance-optimized approach with LRU caching, efficient media handling, and template dependency resolution. Key Features: -- Cache hit/miss handling +- Smart cache management +- Media streaming - Template dependency resolution -- Efficient data retrieval paths -- Incomplete template handling -- Solid line with arrow (->>) means "makes a request to" -- Dashed line (-->) means "returns data to" +- Query optimization +- Priority-based caching ```mermaid sequenceDiagram - participant C as Client + participant UI as UI Layer participant Cache as LRU Cache + participant Media as Media Store participant DB as SQLite - participant Q as Query Builder + participant Query as Query Builder - C->>Q: Client asks Query Builder for templates - Q->>Cache: Query Builder first checks if data is in cache + UI->>Query: Request Content + Query->>Cache: Check Cache alt Cache Hit - Cache-->>C: Quickly Returns Data found in Cache - else Cache Miss: Data not in cache, must query DB - Q->>DB: Query Database - DB-->>Q: Raw Results - Q->>Q: Process Raw Data - Q->>Cache: Store in Cache - Q-->>C: Return Results + Cache-->>UI: Return Cached Data + + opt Has Media References + UI->>Media: Request Media + Media-->>UI: Stream Media + end + + else Cache Miss + Query->>DB: Query Database + DB-->>Query: Raw Results + + opt Has Media + Query->>Media: Load Media + Media-->>Query: Media Content + end + + Query->>Query: Process Results + Query->>Cache: Update Cache + Query-->>UI: Return Results end - Note over C,DB: Template Dependency Resolution + Note over Query,DB: Template Resolution - C->>Q: Template Request - Q->>DB: Fetch Template - DB-->>Q: Template Data - Q->>DB: Check Dependencies - - alt Missing Dependencies - DB-->>Q: Missing Exercises - Q->>C: Return Incomplete - else Complete - DB-->>Q: All Dependencies - Q->>C: Return Complete Template + opt Template Dependencies + Query->>DB: Check Dependencies + alt Missing Dependencies + DB-->>Query: Missing References + Query-->>UI: Return Incomplete + else Complete + DB-->>Query: All Dependencies + Query-->>UI: Return Complete + end end ``` -The second part shows template dependency resolution: -- Template request process -- Dependency checking -- Different responses based on whether all exercises exist -The key learning points: -- Cache is checked first to improve performance -- Database is only queried if necessary -- Results are cached for future use -- Dependencies are verified before returning complete templates ## 4. Component Architecture -This diagram shows the overall application architecture, including service layers and future Nostr integration points. +This diagram shows the application architecture, focusing on the interaction between local-first operations and Nostr integration. Key Features: -- Clear layer separation -- Service interactions -- Cache management -- Future Nostr integration points +- Local-first prioritization +- Efficient service layers +- Clear data boundaries +- Performance optimization +- NDK integration points ```mermaid graph TB subgraph UI Layer - A[Library Screen] - B[Exercise Form] - C[Template Form] + A[Views] + B[Forms] + C[Media Display] end subgraph Service Layer D[Library Service] E[Event Processor] F[Cache Manager] + G[Media Service] end - subgraph Data Layer - G[(SQLite)] - H[Query Builder] - I[Event Validators] + subgraph Storage Layer + H[(SQLite)] + I[Media Store] + J[Event Store] + K[Query Builder] end - subgraph Future Nostr - J[Relay Manager] - K[Event Publisher] - L[Sync Manager] + subgraph NDK Layer + L[Relay Manager] + M[Event Publisher] + N[Sync Manager] end A --> D B --> D - C --> D + C --> G D --> E D --> F - E --> I - E --> G - - F --> G + E --> H + E --> J F --> H + G --> I - H --> G - - D -.-> J - D -.-> K D -.-> L + D -.-> M + + H --> K + K --> F ``` ## Implementation Notes -These diagrams represent the core architecture of POWR's database implementation. Key considerations: +These diagrams represent POWR's database implementation with a focus on local-first performance while maintaining Nostr compatibility. -1. **Data Flow** - - All Nostr events are stored in raw form - - Processed data is stored separately for efficiency - - Cache layer improves read performance - - Dependency tracking ensures data integrity +1. **Local-First Design** + - SQLite as primary storage + - Efficient caching layer + - Optimized media handling + - Smart query patterns + - Background processing -2. **Scalability** - - Modular design allows for future expansion - - Clear separation of concerns - - Efficient query patterns - - Prepared for Nostr integration +2. **Nostr Integration** + - Raw event preservation + - NDK compatibility + - Event validation + - Dependency tracking + - Sync management -3. **Performance** - - LRU caching for frequent queries - - Optimized indexes for common operations - - Efficient dependency resolution - - Batch processing capability \ No newline at end of file +3. **Performance Features** + - LRU caching with priorities + - Media optimization + - Query optimization + - Batch processing + - Background sync + +4. **Data Integrity** + - Transaction management + - Dependency tracking + - Event validation + - Error handling + - Recovery procedures \ No newline at end of file diff --git a/docs/design/database_implementation.md b/docs/design/database_implementation.md index 543fdde..08c805f 100644 --- a/docs/design/database_implementation.md +++ b/docs/design/database_implementation.md @@ -1,216 +1,237 @@ -# POWR Database Implementation Design Document +# POWR Database Implementation PRD ## Problem Statement -Implement a SQLite database that supports local-first fitness tracking while enabling seamless Nostr protocol integration. The system must handle exercise definitions (NIP-33401), workout templates (NIP-33402), and workout records (NIP-33403) while managing event dependencies, caching, and efficient querying. +Implement a local-first SQLite database that efficiently handles single-user fitness tracking while enabling seamless Nostr protocol integration through NDK. The system must handle exercise definitions (NIP-33401), workout templates (NIP-33402), and workout records (NIP-33403) while optimizing local performance and maintaining decentralized sync capabilities. -## Requirements +## Core Requirements -### Functional Requirements -- Store and process Nostr events (33401, 33402, 33403) -- Handle incomplete workout templates with missing exercise references -- Support both local and Nostr-sourced content -- Enable efficient content querying and filtering -- Track template completeness and dependencies -- Manage event replacements and updates +### 1. Local-First Database +- SQLite database optimized for single-user access +- Efficient schema for workout tracking +- Media content storage for exercise demos +- Strong data consistency +- Performant query patterns -### Non-Functional Requirements -- Query response time < 100ms -- Support offline-first operations -- Handle concurrent write operations safely -- Efficient storage for device constraints -- Maintain data integrity with event dependencies +### 2. Nostr Integration +- NDK-compatible event handling +- Raw event storage with validation +- Template dependency management +- Tag-based indexing +- Event replacements and updates -## Design Decisions - -### 1. Event Storage Strategy -Store both raw Nostr events and processed data: -- Raw events for perfect relay replication -- Processed data for efficient querying -- Separate incomplete template tracking -- Tag indexing for fast lookups - -Rationale: -- Maintains Nostr protocol compliance -- Enables efficient local operations -- Supports dependency tracking -- Facilitates sync operations - -### 2. Dependency Management -Track missing exercise references: -- Store incomplete templates -- Track missing dependencies -- Enable background fetching -- Allow filtering by completeness - -Rationale: -- Better user experience -- Data integrity -- Eventual consistency -- Clear status tracking +### 3. Performance Features +- LRU cache for frequent content +- Blob storage for media +- Query optimization +- Background sync +- Efficient indexing ## Technical Design ### Core Schema +```sql +-- Version tracking - keeps track of the database versioin +CREATE TABLE schema_version ( + version INTEGER PRIMARY KEY, + updated_at INTEGER NOT NULL +); + +-- Raw Nostr Events (NDK Compatible) +CREATE TABLE nostr_events ( + id TEXT PRIMARY KEY, -- 32-bytes hex + pubkey TEXT NOT NULL, -- 32-bytes hex + kind INTEGER NOT NULL, -- 33401 | 33402 | 33403 + created_at INTEGER NOT NULL, + content TEXT NOT NULL, + sig TEXT, -- 64-bytes hex + raw_event TEXT NOT NULL, -- Full JSON + received_at INTEGER NOT NULL +); + +-- Tag Indexing +CREATE TABLE event_tags ( + event_id TEXT NOT NULL, + name TEXT NOT NULL, + value TEXT NOT NULL, + index_num INTEGER NOT NULL, + FOREIGN KEY(event_id) REFERENCES nostr_events(id) +); + +-- Processed Exercise Data +CREATE TABLE exercises ( + id TEXT PRIMARY KEY, + title TEXT NOT NULL, + type TEXT NOT NULL CHECK(type IN ('strength', 'cardio', 'bodyweight')), + category TEXT NOT NULL, + equipment TEXT, + description TEXT, + created_at INTEGER NOT NULL, + source TEXT NOT NULL DEFAULT 'local' +); + +-- Exercise Media +CREATE TABLE exercise_media ( + exercise_id TEXT NOT NULL, + media_type TEXT NOT NULL, + content BLOB NOT NULL, + thumbnail BLOB, + created_at INTEGER NOT NULL, + FOREIGN KEY(exercise_id) REFERENCES exercises(id) +); + +-- Template Tracking +CREATE TABLE templates ( + id TEXT PRIMARY KEY, + nostr_event_id TEXT, + title TEXT NOT NULL, + type TEXT NOT NULL, + category TEXT NOT NULL, + description TEXT, + created_at INTEGER NOT NULL, + is_complete BOOLEAN NOT NULL DEFAULT 0, + FOREIGN KEY(nostr_event_id) REFERENCES nostr_events(id) +); + +-- Cache Management +CREATE TABLE cache_metadata ( + content_id TEXT PRIMARY KEY, + content_type TEXT NOT NULL, + last_accessed INTEGER NOT NULL, + access_count INTEGER NOT NULL, + cache_priority INTEGER NOT NULL +); +``` + +### Event Processing +This section shows how the app handles data coming from the Nostr network. The EventProcessor class handles: +- Validating incoming events +- Storing the raw data +- Processing different types of events (exercises, templates, workouts) +- Updating search indexes ```typescript -interface DatabaseSchema { - // Raw Nostr event storage - nostr_events: { - id: string; // 32-bytes hex - pubkey: string; // 32-bytes hex - kind: number; // 33401 | 33402 | 33403 - raw_event: string; // Full JSON event - created_at: number; // Unix timestamp - received_at: number; // Local timestamp - }; +interface NostrEvent { + id: string; // 32-bytes hex + pubkey: string; // 32-bytes hex + created_at: number; + kind: number; // 33401 | 33402 | 33403 + tags: string[][]; + content: string; + sig?: string; // 64-bytes hex +} - // Processed exercise definitions - exercise_definitions: { - event_id: string; // Reference to nostr_events - title: string; - equipment: string; - format: string; // JSON stringified - format_units: string; // JSON stringified - }; +class EventProcessor { + async processIncomingEvent(event: NostrEvent) { + await this.validateEvent(event); + await this.storeRawEvent(event); + await this.processEventByKind(event); + await this.updateIndices(event); + } - // Template dependency tracking - incomplete_templates: { - template_id: string; // Reference to nostr_events - missing_exercise_count: number; - missing_exercises: string; // JSON stringified - }; - - // Tag indexing - event_tags: { - event_id: string; // Reference to nostr_events - name: string; // Tag name - value: string; // Tag value - index: number; // Position in tag array - }; + private async processEventByKind(event: NostrEvent) { + switch(event.kind) { + case 33401: return this.processExerciseTemplate(event); + case 33402: return this.processWorkoutTemplate(event); + case 33403: return this.processWorkoutRecord(event); + } + } } ``` -### Event Validators +### Cache Implementation ```typescript -interface EventValidator { - validateEvent(event: NostrEvent): ValidationResult; - processEvent(event: NostrEvent): ProcessedEvent; +interface CacheConfig { + maxSize: number; + exerciseLimit: number; + templateLimit: number; + mediaLimit: number; + pruneThreshold: number; } -class ExerciseDefinitionValidator implements EventValidator { - // Implementation for kind 33401 -} +class CacheManager { + private cache: LRUCache; + + async get(key: string): Promise { + const cached = this.cache.get(key); + if (cached) { + await this.updateAccessMetrics(key); + return cached.data as T; + } + return null; + } -class WorkoutTemplateValidator implements EventValidator { - // Implementation for kind 33402 + async set(key: string, value: any): Promise { + await this.ensureCacheSpace(); + this.cache.set(key, { + data: value, + lastAccessed: Date.now(), + accessCount: 0 + }); + } } ``` ## Implementation Plan -### Phase 1: Core Event Handling +### Phase 1: Core Infrastructure (Week 1-2) 1. Basic schema implementation -2. Event validation and processing -3. Tag indexing system -4. Basic CRUD operations +2. NDK event processor setup +3. CRUD operations +4. Media storage system -### Phase 2: Template Dependencies -1. Incomplete template tracking -2. Missing exercise handling -3. Background fetching system -4. Template status management +### Phase 2: Nostr Integration (Week 2-3) +1. Raw event storage +2. Event validation +3. Tag indexing +4. Template dependencies -### Phase 3: Query Optimization -1. Implement efficient indexes -2. Query caching system -3. Common query patterns -4. Performance monitoring +### Phase 3: Performance Layer (Week 3-4) +1. Cache implementation +2. Query optimization +3. Index tuning +4. Media optimization + +### Phase 4: Sync & Polish (Week 4-5) +1. NDK sync integration +2. Background operations +3. Performance tuning +4. Error handling ## Testing Strategy ### Unit Tests - Event validation - Data processing -- CRUD operations -- Tag indexing +- Cache operations +- Media handling ### Integration Tests -- Template dependency handling -- Event synchronization -- Cache operations +- NDK compatibility +- Sync operations +- Template dependencies - Query performance ### Performance Tests - Query response times -- Write operation latency -- Cache efficiency -- Storage utilization - -## Security Considerations -- Event signature validation -- Input sanitization -- Access control -- Data integrity +- Cache effectiveness +- Media load times +- Storage efficiency ## Future Considerations ### Potential Enhancements -- Advanced caching strategies -- Relay management -- Batch operations +- Advanced sync strategies +- Predictive caching +- Enhanced search - Compression options ### Known Limitations -- SQLite concurrent access -- Dependency completeness -- Cache memory constraints +- SQLite constraints +- Local storage limits - Initial sync performance - -## Dependencies - -### Runtime Dependencies -- expo-sqlite -- @react-native-async-storage/async-storage - -### Development Dependencies -- Jest for testing -- SQLite tooling -- TypeScript - -## Query Examples - -### Template Queries -```typescript -// Get templates by author -async getTemplatesByPubkey( - pubkey: string, - options: { - includeIncomplete?: boolean; - limit?: number; - since?: number; - } -): Promise - -// Get template with exercises -async getTemplateWithExercises( - templateId: string -): Promise -``` - -### Exercise Queries -```typescript -// Search exercises -async searchExercises( - params: SearchParams -): Promise - -// Get exercise history -async getExerciseHistory( - exerciseId: string -): Promise -``` +- Cache memory bounds ## References - NDK SQLite Implementation -- Nostr NIP-01 Specification -- POWR Exercise NIP Draft -- POWR Library Tab PRD \ No newline at end of file +- Nostr NIPs (01, 33401-33403) +- SQLite Performance Guide +- LRU Cache Patterns \ No newline at end of file