NeahNew/PROJECT_DEEP_ANALYSIS.md

# Neah Project - Deep Technical Analysis

## Executive Summary

This document provides a comprehensive analysis of the Neah project architecture, focusing on:
- Update Services & Refresh Management
- Widgets Architecture
- Notifications System
- Authentication & Token Refresh
- Performance & Memory Management
- API Routes Tracing

---

## 1. Update Services & Refresh Management

### 1.1 Unified Refresh Manager (`lib/services/refresh-manager.ts`)

**Architecture:**
- **Singleton Pattern**: Single instance manages all refresh operations
- **Resource-Based**: Each refreshable resource has its own configuration
- **Deduplication**: Prevents duplicate refresh requests
- **Interval Management**: Centralized interval control

**Refreshable Resources:**
```typescript
type RefreshableResource = 
  | 'notifications'
  | 'notifications-count'
  | 'calendar'
  | 'news'
  | 'email'
  | 'parole'
  | 'duties'
  | 'navbar-time';
```

**Key Features:**

1. **Request Deduplication**
   - Minimum 1 second between refreshes for same resource
   - Tracks pending requests to prevent duplicates
   - Uses `pendingRequests` Map with promise tracking

2. **Interval Management**
   - Each resource can have different refresh intervals
   - Automatic cleanup on unregister
   - Pause/Resume functionality for all resources

3. **Error Handling**
   - Errors don't update `lastRefresh` timestamp (allows retry)
   - Comprehensive logging for debugging
   - Graceful degradation on failures

**Memory Impact:**
- **Low**: Uses Maps for efficient lookups
- **Cleanup**: Proper cleanup on component unmount
- **Potential Issue**: If components don't unregister, intervals persist

**Performance Considerations:**
- ✅ Deduplication prevents unnecessary API calls
- ✅ Minimum 1s throttle prevents excessive refreshes
- ⚠️ Multiple resources = multiple intervals (but necessary)
- ⚠️ No priority-based scheduling (all resources treated equally)

### 1.2 Unified Refresh Hook (`hooks/use-unified-refresh.ts`)

**Purpose:** React hook wrapper for RefreshManager

**Key Features:**
- Automatic registration/unregistration on mount/unmount
- Session-aware (only active when authenticated)
- Callback ref pattern to avoid stale closures
- Manual refresh trigger with force option

**Usage Pattern:**
```typescript
const { refresh, isActive } = useUnifiedRefresh({
  resource: 'calendar',
  interval: 300000, // 5 minutes
  enabled: status === 'authenticated',
  onRefresh: fetchEvents,
  priority: 'low',
});
```

**Memory Leak Prevention:**
- ✅ Cleanup in useEffect return
- ✅ isMountedRef prevents state updates after unmount
- ✅ Automatic unregister on unmount

---

## 2. Widgets Architecture

### 2.1 Widget Components Overview

**Main Dashboard Widgets** (`app/page.tsx`):
1. **QuoteCard** - Inspirational quotes
2. **Calendar** - Upcoming events (7 events)
3. **News** - News articles (100 limit)
4. **Duties** - Leantime tasks (7 tasks)
5. **Email** - Email preview (5 emails)
6. **Parole** - RocketChat messages

### 2.2 Widget Refresh Patterns

**Current Implementation Issues:**

1. **Calendar Widget** (`components/calendar.tsx`)
   - ❌ No unified refresh integration
   - ❌ Manual refresh only via button
   - ❌ Fetches on mount only
   - ⚠️ Uses `?refresh=true` parameter (bypasses cache)

2. **News Widget** (`components/news.tsx`)
   - ❌ No unified refresh integration
   - ✅ Manual refresh button
   - ✅ Fetches on authentication
   - ⚠️ Uses `?refresh=true` parameter

3. **Email Widget** (`components/email.tsx`)
   - ❌ No unified refresh integration
   - ✅ Manual refresh button
   - ⚠️ Fetches on mount only
   - ⚠️ Uses `?refresh=true` parameter

4. **Parole Widget** (`components/parole.tsx`)
   - ❌ No unified refresh integration
   - ⚠️ **Custom polling**: `setInterval(() => fetchMessages(), 30000)` (30s)
   - ⚠️ **Memory Leak Risk**: Interval not cleared if component unmounts during fetch
   - ✅ Manual refresh button

5. **Duties Widget** (`components/flow.tsx`)
   - ❌ No unified refresh integration
   - ❌ Fetches on mount only
   - ⚠️ Uses `?refresh=true` parameter

### 2.3 Widget Memory & Performance Issues

**Critical Issues:**

1. **Multiple Polling Mechanisms**
   - Parole widget uses `setInterval` (30s)
   - No coordination with RefreshManager
   - Risk of memory leaks if cleanup fails

2. **Cache Bypassing**
   - Most widgets use `?refresh=true`
   - Bypasses Redis cache
   - Increases server load

3. **No Unified Refresh**
   - Widgets don't use `useUnifiedRefresh` hook
   - Inconsistent refresh patterns
   - Hard to manage globally

4. **State Management**
   - Each widget manages its own state
   - No shared state/cache
   - Potential duplicate API calls

**Recommendations:**
- ✅ Migrate all widgets to use `useUnifiedRefresh`
- ✅ Remove custom `setInterval` implementations
- ✅ Use cache-first strategy (remove `?refresh=true` by default)
- ✅ Implement widget-level error boundaries

---

## 3. Notifications System

### 3.1 Architecture Overview

**Service Pattern:** Singleton with adapter pattern

**Location:** `lib/services/notifications/notification-service.ts`

**Adapters:**
- `LeantimeAdapter` (implemented)
- NextcloudAdapter (planned)
- GiteaAdapter (planned)
- DolibarrAdapter (planned)
- MoodleAdapter (planned)

### 3.2 Caching Strategy

**Redis Cache Keys:**
```typescript
NOTIFICATION_COUNT_CACHE_KEY = `notifications:count:${userId}`
NOTIFICATIONS_LIST_CACHE_KEY = `notifications:list:${userId}:${page}:${limit}`
```

**Cache TTL:**
- Count cache: 30 seconds
- List cache: 30 seconds
- Refresh lock: 30 seconds

**Cache Invalidation:**
- On `markAsRead`: Invalidates all user caches
- Uses Redis SCAN for pattern matching
- Prevents blocking operations

### 3.3 Refresh Management

**Integration with RefreshManager:**
- ✅ Uses unified refresh system
- ✅ Registered as 'notifications' and 'notifications-count'
- ✅ 30-second refresh interval (aligned with cache TTL)

**Hook Usage** (`hooks/use-notifications.ts`):
- Request deduplication (2-second window)
- Automatic refresh on mount
- Manual refresh capability
- Error handling with retry

### 3.4 Performance Characteristics

**Strengths:**
- ✅ Redis caching reduces database load
- ✅ Adapter pattern allows easy extension
- ✅ Parallel fetching from multiple adapters
- ✅ Request deduplication prevents duplicate calls

**Potential Issues:**
- ⚠️ SCAN operations can be slow with many keys
- ⚠️ No pagination limits on adapter results
- ⚠️ All adapters fetched in parallel (could be optimized)

**Memory Impact:**
- **Low**: Cached data in Redis, not memory
- **Medium**: Notification objects in React state
- **Low**: Adapter instances are singletons

---

## 4. Authentication & Token Refresh

### 4.1 Keycloak Integration

**Provider:** NextAuth with KeycloakProvider

**Location:** `app/api/auth/options.ts`

### 4.2 Token Refresh Flow

**JWT Callback Logic:**

1. **Initial Sign-In:**
   - Stores access token, refresh token, ID token
   - Extracts roles from access token
   - Sets expiration timestamp

2. **Subsequent Requests:**
   - Checks if token is expired
   - If expired, calls `refreshAccessToken()`
   - Updates token with new values

**Refresh Function** (`refreshAccessToken`):

```typescript
async function refreshAccessToken(token: ExtendedJWT) {
  // Calls Keycloak token endpoint
  // Handles various error scenarios:
  // - SessionNotActive (user logged out)
  // - RefreshTokenExpired (inactivity)
  // - InvalidGrant (session invalidated)
}
```

**Error Handling:**
- ✅ Detects session invalidation
- ✅ Handles refresh token expiration
- ✅ Clears tokens on critical errors
- ✅ Returns null session to trigger re-auth

### 4.3 Session Management

**Session Configuration:**
- Strategy: JWT (stateless)
- Max Age: 4 hours (14,400 seconds)
- Automatic refresh on activity

**Cookie Configuration:**
- HttpOnly: true
- SameSite: 'lax'
- Secure: Based on NEXTAUTH_URL

### 4.4 Email OAuth Token Refresh

**Service:** `lib/services/token-refresh.ts`

**Purpose:** Refresh Microsoft OAuth tokens for email access

**Flow:**
1. Check Redis cache for credentials
2. If cache miss, check Prisma database
3. Validate token expiration (5-minute buffer)
4. Refresh if needed via Microsoft OAuth
5. Update both Redis and Prisma

**Dual Storage:**
- **Redis**: Fast access, 24-hour TTL
- **Prisma**: Persistent storage, survives Redis restarts

**Memory Impact:**
- **Low**: Credentials stored in Redis/DB, not memory
- **Medium**: Token refresh operations are async
- **Low**: No memory leaks (proper cleanup)

### 4.5 Performance Considerations

**Token Refresh Frequency:**
- Keycloak: On every request if expired
- Email OAuth: Only when expired (5-min buffer)

**Optimization Opportunities:**
- ⚠️ Token refresh happens synchronously in JWT callback
- ⚠️ Could implement background refresh
- ✅ Caching reduces refresh frequency

---

## 5. Performance & Memory Management

### 5.1 Next.js Configuration

**Build Configuration** (`next.config.mjs`):
```javascript
experimental: {
  webpackBuildWorker: true,
  parallelServerBuildTraces: true,
  parallelServerCompiles: true,
}
```

**Memory Impact:**
- ✅ Parallel builds reduce build time
- ⚠️ Multiple workers increase memory during build
- ✅ Production builds are optimized

### 5.2 Redis Connection Management

**Singleton Pattern** (`lib/redis.ts`):
- Single Redis client instance
- Connection pooling
- Automatic reconnection with retry strategy

**Memory Impact:**
- **Low**: Single connection per process
- **Medium**: Connection pool (if configured)
- **Low**: Proper cleanup on disconnect

**Connection Strategy:**
- Max reconnect attempts: 5
- Exponential backoff
- Connection timeout: 10 seconds
- Keep-alive: 10 seconds

### 5.3 Caching Strategy

**Redis Cache TTLs:**
```typescript
CREDENTIALS: 24 hours
SESSION: 4 hours
EMAIL_LIST: 5 minutes
EMAIL_CONTENT: 15 minutes
CALENDAR: 10 minutes
NEWS: 15 minutes
TASKS: 10 minutes
MESSAGES: 2 minutes
NOTIFICATIONS: 30 seconds
```

**Memory Impact:**
- **Low**: Data in Redis, not application memory
- **Medium**: Large cache can consume Redis memory
- **Low**: TTL ensures automatic cleanup

### 5.4 Component Memory Management

**Potential Memory Leaks:**

1. **Parole Widget** (`components/parole.tsx`):
   ```typescript
   // ⚠️ RISK: Interval might not clear if component unmounts during fetch
   useEffect(() => {
     if (status === 'authenticated') {
       fetchMessages();
       const interval = setInterval(() => fetchMessages(), 30000);
       return () => clearInterval(interval); // ✅ Good, but...
     }
   }, [status]);
   ```
   **Issue**: If `fetchMessages()` is async and component unmounts, state updates may occur

2. **Widget State:**
   - Each widget maintains its own state
   - No cleanup on unmount for pending requests
   - Potential memory leaks with large data arrays

3. **Event Listeners:**
   - No evidence of unregistered event listeners
   - ✅ React handles most cleanup automatically

### 5.5 API Route Performance

**Common Patterns:**

1. **Session Validation:**
   ```typescript
   const session = await getServerSession(authOptions);
   ```
   - Called on every request
   - JWT validation overhead
   - Could be optimized with middleware

2. **Database Queries:**
   - Prisma ORM adds overhead
   - No query optimization visible
   - Connection pooling handled by Prisma

3. **Redis Operations:**
   - Most routes check cache first
   - SCAN operations for pattern matching
   - Could be optimized with better key patterns

### 5.6 Memory Optimization Recommendations

**High Priority:**
1. ✅ Fix Parole widget interval cleanup
2. ✅ Migrate widgets to unified refresh
3. ✅ Implement request cancellation for unmounted components
4. ✅ Add error boundaries to prevent memory leaks

**Medium Priority:**
1. ⚠️ Implement API route middleware for auth
2. ⚠️ Optimize Redis SCAN operations
3. ⚠️ Add request timeout handling
4. ⚠️ Implement connection pooling for external APIs

**Low Priority:**
1. ⚠️ Consider React Query for state management
2. ⚠️ Implement virtual scrolling for large lists
3. ⚠️ Add memory profiling tools

---

## 6. API Routes Tracing

### 6.1 Logging Infrastructure

**Logger** (`lib/logger.ts`):
- Environment-aware (silent in production for debug/info)
- Always logs errors
- Simple console-based logging

**Limitations:**
- ❌ No structured logging (JSON)
- ❌ No log levels in production
- ❌ No centralized log aggregation
- ❌ No request tracing IDs

### 6.2 Current Logging Patterns

**API Routes:**
- 343 `console.log/error/warn` calls across 68 files
- Inconsistent logging patterns
- Some routes have detailed logging, others minimal

**Examples:**

1. **Good Logging** (`app/api/missions/mission-created/route.ts`):
   ```typescript
   logger.debug('Mission Created Webhook Received');
   logger.debug('Received mission-created data', { ... });
   ```

2. **Inconsistent Logging** (`app/api/courrier/route.ts`):
   ```typescript
   console.log(`[API] Received request with: ...`);
   // Mix of console.log and logger
   ```

### 6.3 API Route Categories

**Authentication Routes:**
- `/api/auth/[...nextauth]` - NextAuth handler
- `/api/auth/refresh-keycloak-session` - Session refresh
- `/api/auth/debug-keycloak` - Debug endpoint

**Email Routes (Courrier):**
- `/api/courrier` - Email list
- `/api/courrier/emails` - Email list (alternative)
- `/api/courrier/[id]` - Single email
- `/api/courrier/refresh` - Token refresh
- `/api/courrier/session` - IMAP session
- `/api/courrier/account` - Account management

**Calendar Routes:**
- `/api/calendars` - Calendar list
- `/api/calendars/[id]` - Single calendar
- `/api/calendars/[id]/events` - Calendar events
- `/api/events` - Event CRUD

**Notification Routes:**
- `/api/notifications` - Notification list
- `/api/notifications/count` - Notification count
- `/api/notifications/[id]/read` - Mark as read
- `/api/notifications/read-all` - Mark all as read

**Mission Routes:**
- `/api/missions` - Mission list
- `/api/missions/[missionId]` - Single mission
- `/api/missions/upload` - File upload
- `/api/missions/mission-created` - Webhook handler

### 6.4 Tracing Recommendations

**Immediate Improvements:**

1. **Request ID Tracking:**
   ```typescript
   // Add to middleware or API route wrapper
   const requestId = crypto.randomUUID();
   logger.info('Request started', { requestId, path, method });
   ```

2. **Structured Logging:**
   ```typescript
   logger.info('API Request', {
     requestId,
     method,
     path,
     userId,
     duration: Date.now() - startTime,
   });
   ```

3. **Error Tracking:**
   ```typescript
   logger.error('API Error', {
     requestId,
     error: error.message,
     stack: error.stack,
     path,
     userId,
   });
   ```

4. **Performance Monitoring:**
   ```typescript
   const startTime = Date.now();
   // ... route logic
   logger.debug('API Response', {
     requestId,
     duration: Date.now() - startTime,
     statusCode,
   });
   ```

**Advanced Tracing:**

1. **OpenTelemetry Integration:**
   - Distributed tracing
   - Performance metrics
   - Error tracking

2. **APM Tools:**
   - New Relic
   - Datadog
   - Sentry

3. **Custom Middleware:**
   ```typescript
   // app/api/middleware.ts
   export function withTracing(handler: Function) {
     return async (req: Request, res: Response) => {
       const requestId = crypto.randomUUID();
       const startTime = Date.now();
       
       try {
         const result = await handler(req, res);
         logger.info('Request completed', {
           requestId,
           duration: Date.now() - startTime,
         });
         return result;
       } catch (error) {
         logger.error('Request failed', {
           requestId,
           error,
           duration: Date.now() - startTime,
         });
         throw error;
       }
     };
   }
   ```

### 6.5 API Route Performance Metrics

**Current State:**
- ❌ No performance metrics collected
- ❌ No request duration tracking
- ❌ No error rate monitoring
- ❌ No cache hit/miss tracking

**Recommended Metrics:**
1. Request duration (p50, p95, p99)
2. Error rate by route
3. Cache hit/miss ratio
4. Database query count
5. Redis operation count
6. External API call duration

---

## 7. Critical Issues & Recommendations

### 7.1 Critical Issues

1. **Memory Leak Risk - Parole Widget**
   - Custom `setInterval` without proper cleanup
   - **Fix**: Migrate to `useUnifiedRefresh`

2. **Inconsistent Refresh Patterns**
   - Widgets don't use unified refresh system
   - **Fix**: Migrate all widgets to `useUnifiedRefresh`

3. **Cache Bypassing**
   - Widgets use `?refresh=true` by default
   - **Fix**: Use cache-first strategy

4. **No Request Tracing**
   - Difficult to debug production issues
   - **Fix**: Implement request ID tracking

5. **No Performance Monitoring**
   - No visibility into slow routes
   - **Fix**: Add performance metrics

### 7.2 High Priority Recommendations

1. ✅ Migrate all widgets to unified refresh system
2. ✅ Fix Parole widget interval cleanup
3. ✅ Implement request ID tracking
4. ✅ Add performance metrics
5. ✅ Standardize logging patterns

### 7.3 Medium Priority Recommendations

1. ⚠️ Implement API route middleware
2. ⚠️ Optimize Redis SCAN operations
3. ⚠️ Add error boundaries
4. ⚠️ Implement request cancellation
5. ⚠️ Add structured logging

### 7.4 Low Priority Recommendations

1. ⚠️ Consider React Query
2. ⚠️ Implement virtual scrolling
3. ⚠️ Add memory profiling
4. ⚠️ Consider OpenTelemetry
5. ⚠️ Add APM tooling

---

## 8. Architecture Strengths

### 8.1 Well-Designed Components

1. **Unified Refresh Manager**
   - Excellent abstraction
   - Proper deduplication
   - Clean API

2. **Notification Service**
   - Adapter pattern allows extension
   - Good caching strategy
   - Proper error handling

3. **Redis Integration**
   - Comprehensive caching
   - Proper TTL management
   - Good key naming conventions

4. **Token Refresh**
   - Dual storage (Redis + Prisma)
   - Proper error handling
   - Automatic refresh

### 8.2 Code Quality

- ✅ TypeScript throughout
- ✅ Consistent component structure
- ✅ Proper error handling in most places
- ✅ Good separation of concerns

---

## 9. Conclusion

The Neah project demonstrates a well-architected Next.js application with several sophisticated systems:

**Strengths:**
- Unified refresh management system
- Comprehensive caching strategy
- Robust authentication flow
- Extensible notification system

**Areas for Improvement:**
- Widget refresh consistency
- Memory leak prevention
- API route tracing
- Performance monitoring

**Overall Assessment:**
The codebase is production-ready but would benefit from the recommended improvements, particularly around widget refresh management and observability.

---

## Appendix: File Reference Map

### Core Services
- `lib/services/refresh-manager.ts` - Unified refresh management
- `lib/services/notifications/notification-service.ts` - Notification system
- `lib/services/token-refresh.ts` - Email OAuth token refresh
- `lib/redis.ts` - Redis caching utilities
- `lib/logger.ts` - Logging utility

### Hooks
- `hooks/use-unified-refresh.ts` - Unified refresh hook
- `hooks/use-notifications.ts` - Notification hook

### Widgets
- `components/calendar.tsx` - Calendar widget
- `components/news.tsx` - News widget
- `components/email.tsx` - Email widget
- `components/parole.tsx` - Messages widget
- `components/flow.tsx` - Tasks widget

### API Routes
- `app/api/auth/options.ts` - NextAuth configuration
- `app/api/notifications/` - Notification endpoints
- `app/api/courrier/` - Email endpoints
- `app/api/calendars/` - Calendar endpoints

---

*Document generated: 2024*
*Last updated: Analysis session*