saulteafarmer/POWR
1
0
Fork 0
mirror of https://github.com/DocNR/POWR.git synced 2025-04-19 10:51:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

db doc update

Browse Source
This commit is contained in:
DocNR 2025-02-11 16:23:23 -05:00
parent c771af1b08
commit 96925999df
2 changed files with 339 additions and 268 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

256
docs/design/database_architecture.md
View File

@ -1,12 +1,14 @@
# POWR Database Architecture Diagrams
# POWR Database Architecture
## 1. Entity Relationship Diagram
This diagram shows the core database structure and relationships between tables. The design supports both raw Nostr event storage and processed data for efficient querying.
This diagram shows the core database structure and relationships between tables. The design supports both local-first operations with performance optimizations and Nostr protocol integration.
Key Features:
- Raw Nostr event storage in `nostr_events`
- Processed exercise data in `exercise_definitions`
- Media content storage in `exercise_media`
- Cache management in `cache_metadata`
- Dependency tracking in `incomplete_templates`
- Efficient tag indexing in `event_tags`
@ -15,6 +17,9 @@ erDiagram
nostr_events ||--o{ event_tags : contains
nostr_events ||--o| exercise_definitions : processes
nostr_events ||--o| incomplete_templates : tracks
exercise_definitions ||--o{ exercise_media : stores
exercise_definitions ||--o{ cache_metadata : tracks
nostr_events {
string id PK
string pubkey
@ -23,12 +28,14 @@ erDiagram
number created_at
number received_at
}
event_tags {
string event_id FK
string name
string value
number index
}
exercise_definitions {
string event_id FK
string title
@ -36,6 +43,23 @@ erDiagram
string format
string format_units
}
exercise_media {
string exercise_id FK
string media_type
blob content
blob thumbnail
number created_at
}
cache_metadata {
string content_id PK
string content_type
number last_accessed
number access_count
number priority
}
incomplete_templates {
string template_id FK
number missing_exercise_count
@ -45,180 +69,206 @@ erDiagram
## 2. Event Processing Flow
This diagram illustrates how different types of Nostr events (Exercise Definitions, Workout Templates, and Workout Records) are processed, validated, and stored.
This diagram illustrates how both local and Nostr events are processed, validated, and stored. The system handles Exercise Definitions (33401), Workout Templates (33402), and Workout Records (33403).
Key Features:
- Event type differentiation
- Validation process
- Support for both local and Nostr events
- Unified validation process
- Media content handling
- Cache management
- Dependency checking
- Storage paths for complete/incomplete data
- Storage optimization
```mermaid
flowchart TB
subgraph Input
A[New Nostr Event] --> B{Event Type}
A[New Event] --> B{Source}
B -->|Local| C[Local Creation]
B -->|Nostr| D[Nostr Event]
C --> E{Event Type}
D --> E
end
B -->|kind 33401| C[Exercise Definition]
B -->|kind 33402| D[Workout Template]
B -->|kind 33403| E[Workout Record]
E -->|kind 33401| F[Exercise Definition]
E -->|kind 33402| G[Workout Template]
E -->|kind 33403| H[Workout Record]
subgraph Processing
C --> F[Validate Event]
D --> F
E --> F
F --> I[Validate Event]
G --> I
H --> I
F --> G{Valid?}
G -->|No| H[Reject Event]
G -->|Yes| I[Store Raw Event]
I --> J{Valid?}
J -->|No| K[Reject Event]
J -->|Yes| L[Store Raw Event]
I --> J[Process Event]
J --> K{Dependencies?}
L --> M[Process Event]
M --> N{Has Media?}
K -->|Missing| L[Store as Incomplete]
K -->|Complete| M[Store Processed Data]
N -->|Yes| O[Process Media]
N -->|No| P{Dependencies?}
O --> P
L --> N[Queue Missing Events]
P -->|Missing| Q[Store as Incomplete]
P -->|Complete| R[Store Processed Data]
Q --> S[Queue Missing Events]
R --> T[Update Cache]
end
subgraph Storage
M --> O[(NostrEvents)]
M --> P[(ProcessedData)]
L --> Q[(IncompleteQueue)]
R --> U[(NostrEvents)]
R --> V[(ProcessedData)]
O --> W[(MediaStore)]
T --> X[(Cache)]
Q --> Y[(IncompleteQueue)]
end
```
## 3. Query and Cache Flow
This sequence diagram shows how data is retrieved, utilizing the LRU cache for performance and handling template dependencies.
This sequence diagram shows how data is retrieved, using a performance-optimized approach with LRU caching, efficient media handling, and template dependency resolution.
Key Features:
- Cache hit/miss handling
- Smart cache management
- Media streaming
- Template dependency resolution
- Efficient data retrieval paths
- Incomplete template handling
- Solid line with arrow (->>) means "makes a request to"
- Dashed line (-->) means "returns data to"
- Query optimization
- Priority-based caching
```mermaid
sequenceDiagram
participant C as Client
participant UI as UI Layer
participant Cache as LRU Cache
participant Media as Media Store
participant DB as SQLite
participant Q as Query Builder
participant Query as Query Builder
C->>Q: Client asks Query Builder for templates
Q->>Cache: Query Builder first checks if data is in cache
UI->>Query: Request Content
Query->>Cache: Check Cache
alt Cache Hit
Cache-->>C: Quickly Returns Data found in Cache
else Cache Miss: Data not in cache, must query DB
Q->>DB: Query Database
DB-->>Q: Raw Results
Q->>Q: Process Raw Data
Q->>Cache: Store in Cache
Q-->>C: Return Results
Cache-->>UI: Return Cached Data
opt Has Media References
UI->>Media: Request Media
Media-->>UI: Stream Media
end
else Cache Miss
Query->>DB: Query Database
DB-->>Query: Raw Results
opt Has Media
Query->>Media: Load Media
Media-->>Query: Media Content
end
Query->>Query: Process Results
Query->>Cache: Update Cache
Query-->>UI: Return Results
end
Note over C,DB: Template Dependency Resolution
Note over Query,DB: Template Resolution
C->>Q: Template Request
Q->>DB: Fetch Template
DB-->>Q: Template Data
Q->>DB: Check Dependencies
alt Missing Dependencies
DB-->>Q: Missing Exercises
Q->>C: Return Incomplete
else Complete
DB-->>Q: All Dependencies
Q->>C: Return Complete Template
opt Template Dependencies
Query->>DB: Check Dependencies
alt Missing Dependencies
DB-->>Query: Missing References
Query-->>UI: Return Incomplete
else Complete
DB-->>Query: All Dependencies
Query-->>UI: Return Complete
end
end
```
The second part shows template dependency resolution:
- Template request process
- Dependency checking
- Different responses based on whether all exercises exist
The key learning points:
- Cache is checked first to improve performance
- Database is only queried if necessary
- Results are cached for future use
- Dependencies are verified before returning complete templates
## 4. Component Architecture
This diagram shows the overall application architecture, including service layers and future Nostr integration points.
This diagram shows the application architecture, focusing on the interaction between local-first operations and Nostr integration.
Key Features:
- Clear layer separation
- Service interactions
- Cache management
- Future Nostr integration points
- Local-first prioritization
- Efficient service layers
- Clear data boundaries
- Performance optimization
- NDK integration points
```mermaid
graph TB
subgraph UI Layer
A[Library Screen]
B[Exercise Form]
C[Template Form]
A[Views]
B[Forms]
C[Media Display]
end
subgraph Service Layer
D[Library Service]
E[Event Processor]
F[Cache Manager]
G[Media Service]
end
subgraph Data Layer
G[(SQLite)]
H[Query Builder]
I[Event Validators]
subgraph Storage Layer
H[(SQLite)]
I[Media Store]
J[Event Store]
K[Query Builder]
end
subgraph Future Nostr
J[Relay Manager]
K[Event Publisher]
L[Sync Manager]
subgraph NDK Layer
L[Relay Manager]
M[Event Publisher]
N[Sync Manager]
end
A --> D
B --> D
C --> D
C --> G
D --> E
D --> F
E --> I
E --> G
F --> G
E --> H
E --> J
F --> H
G --> I
H --> G
D -.-> J
D -.-> K
D -.-> L
D -.-> M
H --> K
K --> F
```
## Implementation Notes
These diagrams represent the core architecture of POWR's database implementation. Key considerations:
These diagrams represent POWR's database implementation with a focus on local-first performance while maintaining Nostr compatibility.
1. **Data Flow**
- All Nostr events are stored in raw form
- Processed data is stored separately for efficiency
- Cache layer improves read performance
- Dependency tracking ensures data integrity
1. **Local-First Design**
- SQLite as primary storage
- Efficient caching layer
- Optimized media handling
- Smart query patterns
- Background processing
2. **Scalability**
- Modular design allows for future expansion
- Clear separation of concerns
- Efficient query patterns
- Prepared for Nostr integration
2. **Nostr Integration**
- Raw event preservation
- NDK compatibility
- Event validation
- Dependency tracking
- Sync management
3. **Performance**
- LRU caching for frequent queries
- Optimized indexes for common operations
- Efficient dependency resolution
- Batch processing capability
3. **Performance Features**
- LRU caching with priorities
- Media optimization
- Query optimization
- Batch processing
- Background sync
4. **Data Integrity**
- Transaction management
- Dependency tracking
- Event validation
- Error handling
- Recovery procedures

351
docs/design/database_implementation.md
View File

@ -1,216 +1,237 @@
# POWR Database Implementation Design Document
# POWR Database Implementation PRD
## Problem Statement
Implement a SQLite database that supports local-first fitness tracking while enabling seamless Nostr protocol integration. The system must handle exercise definitions (NIP-33401), workout templates (NIP-33402), and workout records (NIP-33403) while managing event dependencies, caching, and efficient querying.
Implement a local-first SQLite database that efficiently handles single-user fitness tracking while enabling seamless Nostr protocol integration through NDK. The system must handle exercise definitions (NIP-33401), workout templates (NIP-33402), and workout records (NIP-33403) while optimizing local performance and maintaining decentralized sync capabilities.
## Requirements
## Core Requirements
### Functional Requirements
- Store and process Nostr events (33401, 33402, 33403)
- Handle incomplete workout templates with missing exercise references
- Support both local and Nostr-sourced content
- Enable efficient content querying and filtering
- Track template completeness and dependencies
- Manage event replacements and updates
### 1. Local-First Database
- SQLite database optimized for single-user access
- Efficient schema for workout tracking
- Media content storage for exercise demos
- Strong data consistency
- Performant query patterns
### Non-Functional Requirements
- Query response time < 100ms
- Support offline-first operations
- Handle concurrent write operations safely
- Efficient storage for device constraints
- Maintain data integrity with event dependencies
### 2. Nostr Integration
- NDK-compatible event handling
- Raw event storage with validation
- Template dependency management
- Tag-based indexing
- Event replacements and updates
## Design Decisions
### 1. Event Storage Strategy
Store both raw Nostr events and processed data:
- Raw events for perfect relay replication
- Processed data for efficient querying
- Separate incomplete template tracking
- Tag indexing for fast lookups
Rationale:
- Maintains Nostr protocol compliance
- Enables efficient local operations
- Supports dependency tracking
- Facilitates sync operations
### 2. Dependency Management
Track missing exercise references:
- Store incomplete templates
- Track missing dependencies
- Enable background fetching
- Allow filtering by completeness
Rationale:
- Better user experience
- Data integrity
- Eventual consistency
- Clear status tracking
### 3. Performance Features
- LRU cache for frequent content
- Blob storage for media
- Query optimization
- Background sync
- Efficient indexing
## Technical Design
### Core Schema
```sql
-- Version tracking - keeps track of the database versioin
CREATE TABLE schema_version (
version INTEGER PRIMARY KEY,
updated_at INTEGER NOT NULL
);
-- Raw Nostr Events (NDK Compatible)
CREATE TABLE nostr_events (
id TEXT PRIMARY KEY, -- 32-bytes hex
pubkey TEXT NOT NULL, -- 32-bytes hex
kind INTEGER NOT NULL, -- 33401 | 33402 | 33403
created_at INTEGER NOT NULL,
content TEXT NOT NULL,
sig TEXT, -- 64-bytes hex
raw_event TEXT NOT NULL, -- Full JSON
received_at INTEGER NOT NULL
);
-- Tag Indexing
CREATE TABLE event_tags (
event_id TEXT NOT NULL,
name TEXT NOT NULL,
value TEXT NOT NULL,
index_num INTEGER NOT NULL,
FOREIGN KEY(event_id) REFERENCES nostr_events(id)
);
-- Processed Exercise Data
CREATE TABLE exercises (
id TEXT PRIMARY KEY,
title TEXT NOT NULL,
type TEXT NOT NULL CHECK(type IN ('strength', 'cardio', 'bodyweight')),
category TEXT NOT NULL,
equipment TEXT,
description TEXT,
created_at INTEGER NOT NULL,
source TEXT NOT NULL DEFAULT 'local'
);
-- Exercise Media
CREATE TABLE exercise_media (
exercise_id TEXT NOT NULL,
media_type TEXT NOT NULL,
content BLOB NOT NULL,
thumbnail BLOB,
created_at INTEGER NOT NULL,
FOREIGN KEY(exercise_id) REFERENCES exercises(id)
);
-- Template Tracking
CREATE TABLE templates (
id TEXT PRIMARY KEY,
nostr_event_id TEXT,
title TEXT NOT NULL,
type TEXT NOT NULL,
category TEXT NOT NULL,
description TEXT,
created_at INTEGER NOT NULL,
is_complete BOOLEAN NOT NULL DEFAULT 0,
FOREIGN KEY(nostr_event_id) REFERENCES nostr_events(id)
);
-- Cache Management
CREATE TABLE cache_metadata (
content_id TEXT PRIMARY KEY,
content_type TEXT NOT NULL,
last_accessed INTEGER NOT NULL,
access_count INTEGER NOT NULL,
cache_priority INTEGER NOT NULL
);
```
### Event Processing
This section shows how the app handles data coming from the Nostr network. The EventProcessor class handles:
- Validating incoming events
- Storing the raw data
- Processing different types of events (exercises, templates, workouts)
- Updating search indexes
```typescript
interface DatabaseSchema {
// Raw Nostr event storage
nostr_events: {
id: string; // 32-bytes hex
pubkey: string; // 32-bytes hex
kind: number; // 33401 | 33402 | 33403
raw_event: string; // Full JSON event
created_at: number; // Unix timestamp
received_at: number; // Local timestamp
};
interface NostrEvent {
id: string; // 32-bytes hex
pubkey: string; // 32-bytes hex
created_at: number;
kind: number; // 33401 | 33402 | 33403
tags: string[][];
content: string;
sig?: string; // 64-bytes hex
}
// Processed exercise definitions
exercise_definitions: {
event_id: string; // Reference to nostr_events
title: string;
equipment: string;
format: string; // JSON stringified
format_units: string; // JSON stringified
};
class EventProcessor {
async processIncomingEvent(event: NostrEvent) {
await this.validateEvent(event);
await this.storeRawEvent(event);
await this.processEventByKind(event);
await this.updateIndices(event);
}
// Template dependency tracking
incomplete_templates: {
template_id: string; // Reference to nostr_events
missing_exercise_count: number;
missing_exercises: string; // JSON stringified
};
// Tag indexing
event_tags: {
event_id: string; // Reference to nostr_events
name: string; // Tag name
value: string; // Tag value
index: number; // Position in tag array
};
private async processEventByKind(event: NostrEvent) {
switch(event.kind) {
case 33401: return this.processExerciseTemplate(event);
case 33402: return this.processWorkoutTemplate(event);
case 33403: return this.processWorkoutRecord(event);
}
}
}
```
### Event Validators
### Cache Implementation
```typescript
interface EventValidator {
validateEvent(event: NostrEvent): ValidationResult;
processEvent(event: NostrEvent): ProcessedEvent;
interface CacheConfig {
maxSize: number;
exerciseLimit: number;
templateLimit: number;
mediaLimit: number;
pruneThreshold: number;
}
class ExerciseDefinitionValidator implements EventValidator {
// Implementation for kind 33401
}
class CacheManager {
private cache: LRUCache<string, CacheEntry>;
async get<T>(key: string): Promise<T | null> {
const cached = this.cache.get(key);
if (cached) {
await this.updateAccessMetrics(key);
return cached.data as T;
}
return null;
}
class WorkoutTemplateValidator implements EventValidator {
// Implementation for kind 33402
async set(key: string, value: any): Promise<void> {
await this.ensureCacheSpace();
this.cache.set(key, {
data: value,
lastAccessed: Date.now(),
accessCount: 0
});
}
}
```
## Implementation Plan
### Phase 1: Core Event Handling
### Phase 1: Core Infrastructure (Week 1-2)
1. Basic schema implementation
2. Event validation and processing
3. Tag indexing system
4. Basic CRUD operations
2. NDK event processor setup
3. CRUD operations
4. Media storage system
### Phase 2: Template Dependencies
1. Incomplete template tracking
2. Missing exercise handling
3. Background fetching system
4. Template status management
### Phase 2: Nostr Integration (Week 2-3)
1. Raw event storage
2. Event validation
3. Tag indexing
4. Template dependencies
### Phase 3: Query Optimization
1. Implement efficient indexes
2. Query caching system
3. Common query patterns
4. Performance monitoring
### Phase 3: Performance Layer (Week 3-4)
1. Cache implementation
2. Query optimization
3. Index tuning
4. Media optimization
### Phase 4: Sync & Polish (Week 4-5)
1. NDK sync integration
2. Background operations
3. Performance tuning
4. Error handling
## Testing Strategy
### Unit Tests
- Event validation
- Data processing
- CRUD operations
- Tag indexing
- Cache operations
- Media handling
### Integration Tests
- Template dependency handling
- Event synchronization
- Cache operations
- NDK compatibility
- Sync operations
- Template dependencies
- Query performance
### Performance Tests
- Query response times
- Write operation latency
- Cache efficiency
- Storage utilization
## Security Considerations
- Event signature validation
- Input sanitization
- Access control
- Data integrity
- Cache effectiveness
- Media load times
- Storage efficiency
## Future Considerations
### Potential Enhancements
- Advanced caching strategies
- Relay management
- Batch operations
- Advanced sync strategies
- Predictive caching
- Enhanced search
- Compression options
### Known Limitations
- SQLite concurrent access
- Dependency completeness
- Cache memory constraints
- SQLite constraints
- Local storage limits
- Initial sync performance
## Dependencies
### Runtime Dependencies
- expo-sqlite
- @react-native-async-storage/async-storage
### Development Dependencies
- Jest for testing
- SQLite tooling
- TypeScript
## Query Examples
### Template Queries
```typescript
// Get templates by author
async getTemplatesByPubkey(
pubkey: string,
options: {
includeIncomplete?: boolean;
limit?: number;
since?: number;
}
): Promise<ProcessedTemplate[]>
// Get template with exercises
async getTemplateWithExercises(
templateId: string
): Promise<CompleteTemplate>
```
### Exercise Queries
```typescript
// Search exercises
async searchExercises(
params: SearchParams
): Promise<ProcessedExerciseDefinition[]>
// Get exercise history
async getExerciseHistory(
exerciseId: string
): Promise<ExerciseHistory[]>
```
- Cache memory bounds
## References
- NDK SQLite Implementation
- Nostr NIP-01 Specification
- POWR Exercise NIP Draft
- POWR Library Tab PRD
- Nostr NIPs (01, 33401-33403)
- SQLite Performance Guide
- LRU Cache Patterns