saulteafarmer/POWR
1
0
Fork 0
mirror of https://github.com/DocNR/POWR.git synced 2025-06-06 18:31:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

db doc update

Browse Source
This commit is contained in:
DocNR 2025-02-11 16:23:23 -05:00
parent c771af1b08
commit 96925999df
2 changed files with 339 additions and 268 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

256
docs/design/database_architecture.md
View File

@ -1,12 +1,14 @@
# POWR Database Architecture Diagrams # POWR Database Architecture
## 1. Entity Relationship Diagram ## 1. Entity Relationship Diagram
This diagram shows the core database structure and relationships between tables. The design supports both raw Nostr event storage and processed data for efficient querying. This diagram shows the core database structure and relationships between tables. The design supports both local-first operations with performance optimizations and Nostr protocol integration.
Key Features: Key Features:
- Raw Nostr event storage in `nostr_events` - Raw Nostr event storage in `nostr_events`
- Processed exercise data in `exercise_definitions` - Processed exercise data in `exercise_definitions`
- Media content storage in `exercise_media`
- Cache management in `cache_metadata`
- Dependency tracking in `incomplete_templates` - Dependency tracking in `incomplete_templates`
- Efficient tag indexing in `event_tags` - Efficient tag indexing in `event_tags`
@ -15,6 +17,9 @@ erDiagram
nostr_events ||--o{ event_tags : contains nostr_events ||--o{ event_tags : contains
nostr_events ||--o| exercise_definitions : processes nostr_events ||--o| exercise_definitions : processes
nostr_events ||--o| incomplete_templates : tracks nostr_events ||--o| incomplete_templates : tracks
exercise_definitions ||--o{ exercise_media : stores
exercise_definitions ||--o{ cache_metadata : tracks
nostr_events { nostr_events {
string id PK string id PK
string pubkey string pubkey
@ -23,12 +28,14 @@ erDiagram
number created_at number created_at
number received_at number received_at
} }
event_tags { event_tags {
string event_id FK string event_id FK
string name string name
string value string value
number index number index
} }
exercise_definitions { exercise_definitions {
string event_id FK string event_id FK
string title string title
@ -36,6 +43,23 @@ erDiagram
string format string format
string format_units string format_units
} }
exercise_media {
string exercise_id FK
string media_type
blob content
blob thumbnail
number created_at
}
cache_metadata {
string content_id PK
string content_type
number last_accessed
number access_count
number priority
}
incomplete_templates { incomplete_templates {
string template_id FK string template_id FK
number missing_exercise_count number missing_exercise_count
@ -45,180 +69,206 @@ erDiagram
## 2. Event Processing Flow ## 2. Event Processing Flow
This diagram illustrates how different types of Nostr events (Exercise Definitions, Workout Templates, and Workout Records) are processed, validated, and stored. This diagram illustrates how both local and Nostr events are processed, validated, and stored. The system handles Exercise Definitions (33401), Workout Templates (33402), and Workout Records (33403).
Key Features: Key Features:
- Event type differentiation - Support for both local and Nostr events
- Validation process - Unified validation process
- Media content handling
- Cache management
- Dependency checking - Dependency checking
- Storage paths for complete/incomplete data - Storage optimization
```mermaid ```mermaid
flowchart TB flowchart TB
subgraph Input subgraph Input
A[New Nostr Event] --> B{Event Type} A[New Event] --> B{Source}
B -->|Local| C[Local Creation]
B -->|Nostr| D[Nostr Event]
C --> E{Event Type}
D --> E
end end
B -->|kind 33401| C[Exercise Definition] E -->|kind 33401| F[Exercise Definition]
B -->|kind 33402| D[Workout Template] E -->|kind 33402| G[Workout Template]
B -->|kind 33403| E[Workout Record] E -->|kind 33403| H[Workout Record]
subgraph Processing subgraph Processing
C --> F[Validate Event] F --> I[Validate Event]
D --> F G --> I
E --> F H --> I
F --> G{Valid?} I --> J{Valid?}
G -->|No| H[Reject Event] J -->|No| K[Reject Event]
G -->|Yes| I[Store Raw Event] J -->|Yes| L[Store Raw Event]
I --> J[Process Event] L --> M[Process Event]
J --> K{Dependencies?} M --> N{Has Media?}
K -->|Missing| L[Store as Incomplete] N -->|Yes| O[Process Media]
K -->|Complete| M[Store Processed Data] N -->|No| P{Dependencies?}
O --> P
L --> N[Queue Missing Events] P -->|Missing| Q[Store as Incomplete]
P -->|Complete| R[Store Processed Data]
Q --> S[Queue Missing Events]
R --> T[Update Cache]
end end
subgraph Storage subgraph Storage
M --> O[(NostrEvents)] R --> U[(NostrEvents)]
M --> P[(ProcessedData)] R --> V[(ProcessedData)]
L --> Q[(IncompleteQueue)] O --> W[(MediaStore)]
T --> X[(Cache)]
Q --> Y[(IncompleteQueue)]
end end
``` ```
## 3. Query and Cache Flow ## 3. Query and Cache Flow
This sequence diagram shows how data is retrieved, utilizing the LRU cache for performance and handling template dependencies. This sequence diagram shows how data is retrieved, using a performance-optimized approach with LRU caching, efficient media handling, and template dependency resolution.
Key Features: Key Features:
- Cache hit/miss handling - Smart cache management
- Media streaming
- Template dependency resolution - Template dependency resolution
- Efficient data retrieval paths - Query optimization
- Incomplete template handling - Priority-based caching
- Solid line with arrow (->>) means "makes a request to"
- Dashed line (-->) means "returns data to"
```mermaid ```mermaid
sequenceDiagram sequenceDiagram
participant C as Client participant UI as UI Layer
participant Cache as LRU Cache participant Cache as LRU Cache
participant Media as Media Store
participant DB as SQLite participant DB as SQLite
participant Q as Query Builder participant Query as Query Builder
C->>Q: Client asks Query Builder for templates UI->>Query: Request Content
Q->>Cache: Query Builder first checks if data is in cache Query->>Cache: Check Cache
alt Cache Hit alt Cache Hit
Cache-->>C: Quickly Returns Data found in Cache Cache-->>UI: Return Cached Data
else Cache Miss: Data not in cache, must query DB
Q->>DB: Query Database opt Has Media References
DB-->>Q: Raw Results UI->>Media: Request Media
Q->>Q: Process Raw Data Media-->>UI: Stream Media
Q->>Cache: Store in Cache end
Q-->>C: Return Results
else Cache Miss
Query->>DB: Query Database
DB-->>Query: Raw Results
opt Has Media
Query->>Media: Load Media
Media-->>Query: Media Content
end
Query->>Query: Process Results
Query->>Cache: Update Cache
Query-->>UI: Return Results
end end
Note over C,DB: Template Dependency Resolution Note over Query,DB: Template Resolution
C->>Q: Template Request opt Template Dependencies
Q->>DB: Fetch Template Query->>DB: Check Dependencies
DB-->>Q: Template Data alt Missing Dependencies
Q->>DB: Check Dependencies DB-->>Query: Missing References
Query-->>UI: Return Incomplete
alt Missing Dependencies else Complete
DB-->>Q: Missing Exercises DB-->>Query: All Dependencies
Q->>C: Return Incomplete Query-->>UI: Return Complete
else Complete end
DB-->>Q: All Dependencies
Q->>C: Return Complete Template
end end
``` ```
The second part shows template dependency resolution:
- Template request process
- Dependency checking
- Different responses based on whether all exercises exist
The key learning points:
- Cache is checked first to improve performance
- Database is only queried if necessary
- Results are cached for future use
- Dependencies are verified before returning complete templates
## 4. Component Architecture ## 4. Component Architecture
This diagram shows the overall application architecture, including service layers and future Nostr integration points. This diagram shows the application architecture, focusing on the interaction between local-first operations and Nostr integration.
Key Features: Key Features:
- Clear layer separation - Local-first prioritization
- Service interactions - Efficient service layers
- Cache management - Clear data boundaries
- Future Nostr integration points - Performance optimization
- NDK integration points
```mermaid ```mermaid
graph TB graph TB
subgraph UI Layer subgraph UI Layer
A[Library Screen] A[Views]
B[Exercise Form] B[Forms]
C[Template Form] C[Media Display]
end end
subgraph Service Layer subgraph Service Layer
D[Library Service] D[Library Service]
E[Event Processor] E[Event Processor]
F[Cache Manager] F[Cache Manager]
G[Media Service]
end end
subgraph Data Layer subgraph Storage Layer
G[(SQLite)] H[(SQLite)]
H[Query Builder] I[Media Store]
I[Event Validators] J[Event Store]
K[Query Builder]
end end
subgraph Future Nostr subgraph NDK Layer
J[Relay Manager] L[Relay Manager]
K[Event Publisher] M[Event Publisher]
L[Sync Manager] N[Sync Manager]
end end
A --> D A --> D
B --> D B --> D
C --> D C --> G
D --> E D --> E
D --> F D --> F
E --> I E --> H
E --> G E --> J
F --> G
F --> H F --> H
G --> I
H --> G
D -.-> J
D -.-> K
D -.-> L D -.-> L
D -.-> M
H --> K
K --> F
``` ```
## Implementation Notes ## Implementation Notes
These diagrams represent the core architecture of POWR's database implementation. Key considerations: These diagrams represent POWR's database implementation with a focus on local-first performance while maintaining Nostr compatibility.
1. **Data Flow** 1. **Local-First Design**
- All Nostr events are stored in raw form - SQLite as primary storage
- Processed data is stored separately for efficiency - Efficient caching layer
- Cache layer improves read performance - Optimized media handling
- Dependency tracking ensures data integrity - Smart query patterns
- Background processing
2. **Scalability** 2. **Nostr Integration**
- Modular design allows for future expansion - Raw event preservation
- Clear separation of concerns - NDK compatibility
- Efficient query patterns - Event validation
- Prepared for Nostr integration - Dependency tracking
- Sync management
3. **Performance** 3. **Performance Features**
- LRU caching for frequent queries - LRU caching with priorities
- Optimized indexes for common operations - Media optimization
- Efficient dependency resolution - Query optimization
- Batch processing capability - Batch processing
- Background sync
4. **Data Integrity**
- Transaction management
- Dependency tracking
- Event validation
- Error handling
- Recovery procedures

351
docs/design/database_implementation.md
View File

@ -1,216 +1,237 @@
# POWR Database Implementation Design Document # POWR Database Implementation PRD
## Problem Statement ## Problem Statement
Implement a SQLite database that supports local-first fitness tracking while enabling seamless Nostr protocol integration. The system must handle exercise definitions (NIP-33401), workout templates (NIP-33402), and workout records (NIP-33403) while managing event dependencies, caching, and efficient querying. Implement a local-first SQLite database that efficiently handles single-user fitness tracking while enabling seamless Nostr protocol integration through NDK. The system must handle exercise definitions (NIP-33401), workout templates (NIP-33402), and workout records (NIP-33403) while optimizing local performance and maintaining decentralized sync capabilities.
## Requirements ## Core Requirements
### Functional Requirements ### 1. Local-First Database
- Store and process Nostr events (33401, 33402, 33403) - SQLite database optimized for single-user access
- Handle incomplete workout templates with missing exercise references - Efficient schema for workout tracking
- Support both local and Nostr-sourced content - Media content storage for exercise demos
- Enable efficient content querying and filtering - Strong data consistency
- Track template completeness and dependencies - Performant query patterns
- Manage event replacements and updates
### Non-Functional Requirements ### 2. Nostr Integration
- Query response time < 100ms - NDK-compatible event handling
- Support offline-first operations - Raw event storage with validation
- Handle concurrent write operations safely - Template dependency management
- Efficient storage for device constraints - Tag-based indexing
- Maintain data integrity with event dependencies - Event replacements and updates
## Design Decisions ### 3. Performance Features
- LRU cache for frequent content
### 1. Event Storage Strategy - Blob storage for media
Store both raw Nostr events and processed data: - Query optimization
- Raw events for perfect relay replication - Background sync
- Processed data for efficient querying - Efficient indexing
- Separate incomplete template tracking
- Tag indexing for fast lookups
Rationale:
- Maintains Nostr protocol compliance
- Enables efficient local operations
- Supports dependency tracking
- Facilitates sync operations
### 2. Dependency Management
Track missing exercise references:
- Store incomplete templates
- Track missing dependencies
- Enable background fetching
- Allow filtering by completeness
Rationale:
- Better user experience
- Data integrity
- Eventual consistency
- Clear status tracking
## Technical Design ## Technical Design
### Core Schema ### Core Schema
```sql
-- Version tracking - keeps track of the database versioin
CREATE TABLE schema_version (
version INTEGER PRIMARY KEY,
updated_at INTEGER NOT NULL
);
-- Raw Nostr Events (NDK Compatible)
CREATE TABLE nostr_events (
id TEXT PRIMARY KEY, -- 32-bytes hex
pubkey TEXT NOT NULL, -- 32-bytes hex
kind INTEGER NOT NULL, -- 33401 | 33402 | 33403
created_at INTEGER NOT NULL,
content TEXT NOT NULL,
sig TEXT, -- 64-bytes hex
raw_event TEXT NOT NULL, -- Full JSON
received_at INTEGER NOT NULL
);
-- Tag Indexing
CREATE TABLE event_tags (
event_id TEXT NOT NULL,
name TEXT NOT NULL,
value TEXT NOT NULL,
index_num INTEGER NOT NULL,
FOREIGN KEY(event_id) REFERENCES nostr_events(id)
);
-- Processed Exercise Data
CREATE TABLE exercises (
id TEXT PRIMARY KEY,
title TEXT NOT NULL,
type TEXT NOT NULL CHECK(type IN ('strength', 'cardio', 'bodyweight')),
category TEXT NOT NULL,
equipment TEXT,
description TEXT,
created_at INTEGER NOT NULL,
source TEXT NOT NULL DEFAULT 'local'
);
-- Exercise Media
CREATE TABLE exercise_media (
exercise_id TEXT NOT NULL,
media_type TEXT NOT NULL,
content BLOB NOT NULL,
thumbnail BLOB,
created_at INTEGER NOT NULL,
FOREIGN KEY(exercise_id) REFERENCES exercises(id)
);
-- Template Tracking
CREATE TABLE templates (
id TEXT PRIMARY KEY,
nostr_event_id TEXT,
title TEXT NOT NULL,
type TEXT NOT NULL,
category TEXT NOT NULL,
description TEXT,
created_at INTEGER NOT NULL,
is_complete BOOLEAN NOT NULL DEFAULT 0,
FOREIGN KEY(nostr_event_id) REFERENCES nostr_events(id)
);
-- Cache Management
CREATE TABLE cache_metadata (
content_id TEXT PRIMARY KEY,
content_type TEXT NOT NULL,
last_accessed INTEGER NOT NULL,
access_count INTEGER NOT NULL,
cache_priority INTEGER NOT NULL
);
```
### Event Processing
This section shows how the app handles data coming from the Nostr network. The EventProcessor class handles:
- Validating incoming events
- Storing the raw data
- Processing different types of events (exercises, templates, workouts)
- Updating search indexes
```typescript ```typescript
interface DatabaseSchema { interface NostrEvent {
// Raw Nostr event storage id: string; // 32-bytes hex
nostr_events: { pubkey: string; // 32-bytes hex
id: string; // 32-bytes hex created_at: number;
pubkey: string; // 32-bytes hex kind: number; // 33401 | 33402 | 33403
kind: number; // 33401 | 33402 | 33403 tags: string[][];
raw_event: string; // Full JSON event content: string;
created_at: number; // Unix timestamp sig?: string; // 64-bytes hex
received_at: number; // Local timestamp }
};
// Processed exercise definitions class EventProcessor {
exercise_definitions: { async processIncomingEvent(event: NostrEvent) {
event_id: string; // Reference to nostr_events await this.validateEvent(event);
title: string; await this.storeRawEvent(event);
equipment: string; await this.processEventByKind(event);
format: string; // JSON stringified await this.updateIndices(event);
format_units: string; // JSON stringified }
};
// Template dependency tracking private async processEventByKind(event: NostrEvent) {
incomplete_templates: { switch(event.kind) {
template_id: string; // Reference to nostr_events case 33401: return this.processExerciseTemplate(event);
missing_exercise_count: number; case 33402: return this.processWorkoutTemplate(event);
missing_exercises: string; // JSON stringified case 33403: return this.processWorkoutRecord(event);
}; }
}
// Tag indexing
event_tags: {
event_id: string; // Reference to nostr_events
name: string; // Tag name
value: string; // Tag value
index: number; // Position in tag array
};
} }
``` ```
### Event Validators ### Cache Implementation
```typescript ```typescript
interface EventValidator { interface CacheConfig {
validateEvent(event: NostrEvent): ValidationResult; maxSize: number;
processEvent(event: NostrEvent): ProcessedEvent; exerciseLimit: number;
templateLimit: number;
mediaLimit: number;
pruneThreshold: number;
} }
class ExerciseDefinitionValidator implements EventValidator { class CacheManager {
// Implementation for kind 33401 private cache: LRUCache<string, CacheEntry>;
}
async get<T>(key: string): Promise<T | null> {
const cached = this.cache.get(key);
if (cached) {
await this.updateAccessMetrics(key);
return cached.data as T;
}
return null;
}
class WorkoutTemplateValidator implements EventValidator { async set(key: string, value: any): Promise<void> {
// Implementation for kind 33402 await this.ensureCacheSpace();
this.cache.set(key, {
data: value,
lastAccessed: Date.now(),
accessCount: 0
});
}
} }
``` ```
## Implementation Plan ## Implementation Plan
### Phase 1: Core Event Handling ### Phase 1: Core Infrastructure (Week 1-2)
1. Basic schema implementation 1. Basic schema implementation
2. Event validation and processing 2. NDK event processor setup
3. Tag indexing system 3. CRUD operations
4. Basic CRUD operations 4. Media storage system
### Phase 2: Template Dependencies ### Phase 2: Nostr Integration (Week 2-3)
1. Incomplete template tracking 1. Raw event storage
2. Missing exercise handling 2. Event validation
3. Background fetching system 3. Tag indexing
4. Template status management 4. Template dependencies
### Phase 3: Query Optimization ### Phase 3: Performance Layer (Week 3-4)
1. Implement efficient indexes 1. Cache implementation
2. Query caching system 2. Query optimization
3. Common query patterns 3. Index tuning
4. Performance monitoring 4. Media optimization
### Phase 4: Sync & Polish (Week 4-5)
1. NDK sync integration
2. Background operations
3. Performance tuning
4. Error handling
## Testing Strategy ## Testing Strategy
### Unit Tests ### Unit Tests
- Event validation - Event validation
- Data processing - Data processing
- CRUD operations - Cache operations
- Tag indexing - Media handling
### Integration Tests ### Integration Tests
- Template dependency handling - NDK compatibility
- Event synchronization - Sync operations
- Cache operations - Template dependencies
- Query performance - Query performance
### Performance Tests ### Performance Tests
- Query response times - Query response times
- Write operation latency - Cache effectiveness
- Cache efficiency - Media load times
- Storage utilization - Storage efficiency
## Security Considerations
- Event signature validation
- Input sanitization
- Access control
- Data integrity
## Future Considerations ## Future Considerations
### Potential Enhancements ### Potential Enhancements
- Advanced caching strategies - Advanced sync strategies
- Relay management - Predictive caching
- Batch operations - Enhanced search
- Compression options - Compression options
### Known Limitations ### Known Limitations
- SQLite concurrent access - SQLite constraints
- Dependency completeness - Local storage limits
- Cache memory constraints
- Initial sync performance - Initial sync performance
- Cache memory bounds
## Dependencies
### Runtime Dependencies
- expo-sqlite
- @react-native-async-storage/async-storage
### Development Dependencies
- Jest for testing
- SQLite tooling
- TypeScript
## Query Examples
### Template Queries
```typescript
// Get templates by author
async getTemplatesByPubkey(
pubkey: string,
options: {
includeIncomplete?: boolean;
limit?: number;
since?: number;
}
): Promise<ProcessedTemplate[]>
// Get template with exercises
async getTemplateWithExercises(
templateId: string
): Promise<CompleteTemplate>
```
### Exercise Queries
```typescript
// Search exercises
async searchExercises(
params: SearchParams
): Promise<ProcessedExerciseDefinition[]>
// Get exercise history
async getExerciseHistory(
exerciseId: string
): Promise<ExerciseHistory[]>
```
## References ## References
- NDK SQLite Implementation - NDK SQLite Implementation
- Nostr NIP-01 Specification - Nostr NIPs (01, 33401-33403)
- POWR Exercise NIP Draft - SQLite Performance Guide
- POWR Library Tab PRD - LRU Cache Patterns