NeahNew/IMPLEMENTATION_PLAN_UNIFIED_REFRESH.md

# Implementation Plan: Unified Refresh System

## 🎯 Goals

1. **Harmonize auto-refresh** across all widgets and notifications
2. **Reduce redundancy** and eliminate duplicate API calls
3. **Improve API efficiency** with request deduplication and caching
4. **Prevent memory leaks** with proper cleanup mechanisms
5. **Centralize refresh logic** for easier maintenance

---

## 📋 Current State Analysis

### Current Refresh Intervals:
- **Notifications**: 60 seconds (polling)
- **Calendar**: 5 minutes (300000ms)
- **Parole (Chat)**: 30 seconds (30000ms)
- **Navbar Time**: Static (not refreshing - needs fix)
- **News**: Manual only
- **Email**: Manual only
- **Duties (Tasks)**: Manual only

### Current Problems:
1. ❌ No coordination between widgets
2. ❌ Duplicate API calls from multiple components
3. ❌ Memory leaks from uncleaned intervals
4. ❌ No request deduplication
5. ❌ Inconsistent refresh patterns

---

## 🏗️ Architecture: Unified Refresh System

### Phase 1: Core Infrastructure

#### 1.1 Create Unified Refresh Manager

**File**: `lib/services/refresh-manager.ts`

```typescript
/**
 * Unified Refresh Manager
 * Centralizes all refresh logic, prevents duplicates, manages intervals
 */

export type RefreshableResource = 
  | 'notifications'
  | 'notifications-count'
  | 'calendar'
  | 'news'
  | 'email'
  | 'parole'
  | 'duties';

export interface RefreshConfig {
  resource: RefreshableResource;
  interval: number; // milliseconds
  enabled: boolean;
  priority: 'high' | 'medium' | 'low';
  onRefresh: () => Promise<void>;
}

class RefreshManager {
  private intervals: Map<RefreshableResource, NodeJS.Timeout> = new Map();
  private configs: Map<RefreshableResource, RefreshConfig> = new Map();
  private pendingRequests: Map<string, Promise<any>> = new Map();
  private lastRefresh: Map<RefreshableResource, number> = new Map();
  private isActive = false;

  /**
   * Register a refreshable resource
   */
  register(config: RefreshConfig): void {
    this.configs.set(config.resource, config);
    
    if (config.enabled && this.isActive) {
      this.startRefresh(config.resource);
    }
  }

  /**
   * Unregister a resource
   */
  unregister(resource: RefreshableResource): void {
    this.stopRefresh(resource);
    this.configs.delete(resource);
    this.lastRefresh.delete(resource);
  }

  /**
   * Start all refresh intervals
   */
  start(): void {
    if (this.isActive) return;
    
    this.isActive = true;
    
    // Start all enabled resources
    this.configs.forEach((config, resource) => {
      if (config.enabled) {
        this.startRefresh(resource);
      }
    });
  }

  /**
   * Stop all refresh intervals
   */
  stop(): void {
    this.isActive = false;
    
    // Clear all intervals
    this.intervals.forEach((interval) => {
      clearInterval(interval);
    });
    
    this.intervals.clear();
  }

  /**
   * Start refresh for a specific resource
   */
  private startRefresh(resource: RefreshableResource): void {
    // Stop existing interval if any
    this.stopRefresh(resource);
    
    const config = this.configs.get(resource);
    if (!config || !config.enabled) return;

    // Initial refresh
    this.executeRefresh(resource);

    // Set up interval
    const interval = setInterval(() => {
      this.executeRefresh(resource);
    }, config.interval);

    this.intervals.set(resource, interval);
  }

  /**
   * Stop refresh for a specific resource
   */
  private stopRefresh(resource: RefreshableResource): void {
    const interval = this.intervals.get(resource);
    if (interval) {
      clearInterval(interval);
      this.intervals.delete(resource);
    }
  }

  /**
   * Execute refresh with deduplication
   */
  private async executeRefresh(resource: RefreshableResource): Promise<void> {
    const config = this.configs.get(resource);
    if (!config) return;

    const requestKey = `${resource}-${Date.now()}`;
    const now = Date.now();
    const lastRefreshTime = this.lastRefresh.get(resource) || 0;
    
    // Prevent too frequent refreshes (minimum 1 second between same resource)
    if (now - lastRefreshTime < 1000) {
      console.log(`[RefreshManager] Skipping ${resource} - too soon`);
      return;
    }

    // Check if there's already a pending request for this resource
    const pendingKey = `${resource}-pending`;
    if (this.pendingRequests.has(pendingKey)) {
      console.log(`[RefreshManager] Deduplicating ${resource} request`);
      return;
    }

    // Create and track the request
    const refreshPromise = config.onRefresh()
      .then(() => {
        this.lastRefresh.set(resource, Date.now());
      })
      .catch((error) => {
        console.error(`[RefreshManager] Error refreshing ${resource}:`, error);
      })
      .finally(() => {
        this.pendingRequests.delete(pendingKey);
      });

    this.pendingRequests.set(pendingKey, refreshPromise);
    
    try {
      await refreshPromise;
    } catch (error) {
      // Error already logged above
    }
  }

  /**
   * Manually trigger refresh for a resource
   */
  async refresh(resource: RefreshableResource, force = false): Promise<void> {
    const config = this.configs.get(resource);
    if (!config) {
      throw new Error(`Resource ${resource} not registered`);
    }

    if (force) {
      // Force refresh: clear last refresh time
      this.lastRefresh.delete(resource);
    }

    await this.executeRefresh(resource);
  }

  /**
   * Get refresh status
   */
  getStatus(): {
    active: boolean;
    resources: Array<{
      resource: RefreshableResource;
      enabled: boolean;
      lastRefresh: number | null;
      interval: number;
    }>;
  } {
    const resources = Array.from(this.configs.entries()).map(([resource, config]) => ({
      resource,
      enabled: config.enabled,
      lastRefresh: this.lastRefresh.get(resource) || null,
      interval: config.interval,
    }));

    return {
      active: this.isActive,
      resources,
    };
  }
}

// Singleton instance
export const refreshManager = new RefreshManager();
```

---

#### 1.2 Create Request Deduplication Utility

**File**: `lib/utils/request-deduplication.ts`

```typescript
/**
 * Request Deduplication Utility
 * Prevents duplicate API calls for the same resource
 */

interface PendingRequest<T> {
  promise: Promise<T>;
  timestamp: number;
}

class RequestDeduplicator {
  private pendingRequests = new Map<string, PendingRequest<any>>();
  private readonly DEFAULT_TTL = 5000; // 5 seconds

  /**
   * Execute a request with deduplication
   */
  async execute<T>(
    key: string,
    requestFn: () => Promise<T>,
    ttl: number = this.DEFAULT_TTL
  ): Promise<T> {
    // Check if there's a pending request
    const pending = this.pendingRequests.get(key);
    
    if (pending) {
      const age = Date.now() - pending.timestamp;
      
      // If request is still fresh, reuse it
      if (age < ttl) {
        console.log(`[RequestDeduplicator] Reusing pending request: ${key}`);
        return pending.promise;
      } else {
        // Request is stale, remove it
        this.pendingRequests.delete(key);
      }
    }

    // Create new request
    const promise = requestFn()
      .finally(() => {
        // Clean up after request completes
        this.pendingRequests.delete(key);
      });

    this.pendingRequests.set(key, {
      promise,
      timestamp: Date.now(),
    });

    return promise;
  }

  /**
   * Cancel a pending request
   */
  cancel(key: string): void {
    this.pendingRequests.delete(key);
  }

  /**
   * Clear all pending requests
   */
  clear(): void {
    this.pendingRequests.clear();
  }

  /**
   * Get pending requests count
   */
  getPendingCount(): number {
    return this.pendingRequests.size;
  }
}

export const requestDeduplicator = new RequestDeduplicator();
```

---

#### 1.3 Create Unified Refresh Hook

**File**: `hooks/use-unified-refresh.ts`

```typescript
/**
 * Unified Refresh Hook
 * Provides consistent refresh functionality for all widgets
 */

import { useEffect, useCallback, useRef } from 'react';
import { useSession } from 'next-auth/react';
import { refreshManager, RefreshableResource } from '@/lib/services/refresh-manager';

interface UseUnifiedRefreshOptions {
  resource: RefreshableResource;
  interval: number;
  enabled?: boolean;
  onRefresh: () => Promise<void>;
  priority?: 'high' | 'medium' | 'low';
}

export function useUnifiedRefresh({
  resource,
  interval,
  enabled = true,
  onRefresh,
  priority = 'medium',
}: UseUnifiedRefreshOptions) {
  const { status } = useSession();
  const onRefreshRef = useRef(onRefresh);
  const isMountedRef = useRef(true);

  // Update callback ref when it changes
  useEffect(() => {
    onRefreshRef.current = onRefresh;
  }, [onRefresh]);

  // Register/unregister with refresh manager
  useEffect(() => {
    if (status !== 'authenticated' || !enabled) {
      return;
    }

    isMountedRef.current = true;

    // Register with refresh manager
    refreshManager.register({
      resource,
      interval,
      enabled: true,
      priority,
      onRefresh: async () => {
        if (isMountedRef.current) {
          await onRefreshRef.current();
        }
      },
    });

    // Start refresh manager if not already started
    refreshManager.start();

    // Cleanup
    return () => {
      isMountedRef.current = false;
      refreshManager.unregister(resource);
    };
  }, [resource, interval, enabled, priority, status]);

  // Manual refresh function
  const refresh = useCallback(
    async (force = false) => {
      if (status !== 'authenticated') return;
      await refreshManager.refresh(resource, force);
    },
    [resource, status]
  );

  return {
    refresh,
    isActive: refreshManager.getStatus().active,
  };
}
```

---

### Phase 2: Harmonized Refresh Intervals

#### 2.1 Define Standard Intervals

**File**: `lib/constants/refresh-intervals.ts`

```typescript
/**
 * Standard Refresh Intervals
 * All intervals in milliseconds
 */

export const REFRESH_INTERVALS = {
  // High priority - real-time updates
  NOTIFICATIONS: 30000,        // 30 seconds (was 60s)
  NOTIFICATIONS_COUNT: 30000, // 30 seconds (same as notifications)
  PAROLE: 30000,               // 30 seconds (unchanged)
  NAVBAR_TIME: 1000,           // 1 second (navigation bar time - real-time)
  
  // Medium priority - frequent but not real-time
  EMAIL: 60000,                // 1 minute (was manual only)
  DUTIES: 120000,              // 2 minutes (was manual only)
  
  // Low priority - less frequent updates
  CALENDAR: 300000,            // 5 minutes (unchanged)
  NEWS: 600000,                // 10 minutes (was manual only)
  
  // Minimum interval between refreshes (prevents spam)
  MIN_INTERVAL: 1000,          // 1 second
} as const;

/**
 * Get refresh interval for a resource
 */
export function getRefreshInterval(resource: string): number {
  switch (resource) {
    case 'notifications':
      return REFRESH_INTERVALS.NOTIFICATIONS;
    case 'notifications-count':
      return REFRESH_INTERVALS.NOTIFICATIONS_COUNT;
    case 'parole':
      return REFRESH_INTERVALS.PAROLE;
    case 'email':
      return REFRESH_INTERVALS.EMAIL;
    case 'duties':
      return REFRESH_INTERVALS.DUTIES;
    case 'calendar':
      return REFRESH_INTERVALS.CALENDAR;
    case 'news':
      return REFRESH_INTERVALS.NEWS;
    default:
      return 60000; // Default: 1 minute
  }
}
```

---

### Phase 3: Refactor Widgets

#### 3.1 Refactor Notification Hook

**File**: `hooks/use-notifications.ts` (Refactored)

```typescript
import { useState, useEffect, useCallback, useRef } from 'react';
import { useSession } from 'next-auth/react';
import { Notification, NotificationCount } from '@/lib/types/notification';
import { useUnifiedRefresh } from './use-unified-refresh';
import { REFRESH_INTERVALS } from '@/lib/constants/refresh-intervals';
import { requestDeduplicator } from '@/lib/utils/request-deduplication';

const defaultNotificationCount: NotificationCount = {
  total: 0,
  unread: 0,
  sources: {},
};

export function useNotifications() {
  const { data: session, status } = useSession();
  const [notifications, setNotifications] = useState<Notification[]>([]);
  const [notificationCount, setNotificationCount] = useState<NotificationCount>(defaultNotificationCount);
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState<string | null>(null);
  const isMountedRef = useRef(true);

  // Fetch notification count
  const fetchNotificationCount = useCallback(async () => {
    if (!session?.user || !isMountedRef.current) return;
    
    try {
      setError(null);
      
      const data = await requestDeduplicator.execute(
        `notifications-count-${session.user.id}`,
        async () => {
          const response = await fetch('/api/notifications/count', {
            credentials: 'include',
          });
          
          if (!response.ok) {
            throw new Error('Failed to fetch notification count');
          }
          
          return response.json();
        }
      );
      
      if (isMountedRef.current) {
        setNotificationCount(data);
      }
    } catch (err) {
      console.error('Error fetching notification count:', err);
      if (isMountedRef.current) {
        setError('Failed to fetch notification count');
      }
    }
  }, [session?.user]);

  // Fetch notifications
  const fetchNotifications = useCallback(async (page = 1, limit = 20) => {
    if (!session?.user || !isMountedRef.current) return;
    
    setLoading(true);
    setError(null);
    
    try {
      const data = await requestDeduplicator.execute(
        `notifications-${session.user.id}-${page}-${limit}`,
        async () => {
          const response = await fetch(`/api/notifications?page=${page}&limit=${limit}`, {
            credentials: 'include',
          });
          
          if (!response.ok) {
            throw new Error('Failed to fetch notifications');
          }
          
          return response.json();
        }
      );
      
      if (isMountedRef.current) {
        setNotifications(data.notifications);
      }
    } catch (err) {
      console.error('Error fetching notifications:', err);
      if (isMountedRef.current) {
        setError('Failed to fetch notifications');
      }
    } finally {
      if (isMountedRef.current) {
        setLoading(false);
      }
    }
  }, [session?.user]);

  // Use unified refresh for notification count
  useUnifiedRefresh({
    resource: 'notifications-count',
    interval: REFRESH_INTERVALS.NOTIFICATIONS_COUNT,
    enabled: status === 'authenticated',
    onRefresh: fetchNotificationCount,
    priority: 'high',
  });

  // Initial fetch
  useEffect(() => {
    isMountedRef.current = true;
    
    if (status === 'authenticated' && session?.user) {
      fetchNotificationCount();
      fetchNotifications();
    }
    
    return () => {
      isMountedRef.current = false;
    };
  }, [status, session?.user, fetchNotificationCount, fetchNotifications]);

  // Mark as read
  const markAsRead = useCallback(async (notificationId: string) => {
    if (!session?.user) return false;
    
    try {
      const response = await fetch(`/api/notifications/${notificationId}/read`, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        credentials: 'include',
      });
      
      if (!response.ok) return false;
      
      setNotifications(prev => 
        prev.map(n => n.id === notificationId ? { ...n, isRead: true } : n)
      );
      
      await fetchNotificationCount();
      return true;
    } catch (err) {
      console.error('Error marking notification as read:', err);
      return false;
    }
  }, [session?.user, fetchNotificationCount]);

  // Mark all as read
  const markAllAsRead = useCallback(async () => {
    if (!session?.user) return false;
    
    try {
      const response = await fetch('/api/notifications/read-all', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        credentials: 'include',
      });
      
      if (!response.ok) return false;
      
      setNotifications(prev => prev.map(n => ({ ...n, isRead: true })));
      await fetchNotificationCount();
      return true;
    } catch (err) {
      console.error('Error marking all notifications as read:', err);
      return false;
    }
  }, [session?.user, fetchNotificationCount]);

  return {
    notifications,
    notificationCount,
    loading,
    error,
    fetchNotifications,
    fetchNotificationCount,
    markAsRead,
    markAllAsRead,
  };
}
```

---

#### 3.2 Refactor Widget Components

**Example: Calendar Widget**

**File**: `components/calendar.tsx` (Refactored)

```typescript
"use client";

import { useEffect, useState } from "react";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Button } from "@/components/ui/button";
import { RefreshCw, Calendar as CalendarIcon } from "lucide-react";
import { useUnifiedRefresh } from '@/hooks/use-unified-refresh';
import { REFRESH_INTERVALS } from '@/lib/constants/refresh-intervals';
import { requestDeduplicator } from '@/lib/utils/request-deduplication';
import { useSession } from 'next-auth/react';

interface Event {
  id: string;
  title: string;
  start: string;
  end: string;
  allDay: boolean;
  calendar: string;
  calendarColor: string;
}

export function Calendar() {
  const { status } = useSession();
  const [events, setEvents] = useState<Event[]>([]);
  const [loading, setLoading] = useState(true);
  const [error, setError] = useState<string | null>(null);

  const fetchEvents = async () => {
    if (status !== 'authenticated') return;
    
    setLoading(true);
    setError(null);
    
    try {
      const calendarsData = await requestDeduplicator.execute(
        'calendar-events',
        async () => {
          const response = await fetch('/api/calendars?refresh=true');
          if (!response.ok) {
            throw new Error('Failed to fetch events');
          }
          return response.json();
        }
      );

      const now = new Date();
      now.setHours(0, 0, 0, 0);

      const allEvents = calendarsData.flatMap((calendar: any) => 
        (calendar.events || []).map((event: any) => ({
          id: event.id,
          title: event.title,
          start: event.start,
          end: event.end,
          allDay: event.isAllDay,
          calendar: calendar.name,
          calendarColor: calendar.color
        }))
      );

      const upcomingEvents = allEvents
        .filter((event: any) => new Date(event.start) >= now)
        .sort((a: any, b: any) => new Date(a.start).getTime() - new Date(b.start).getTime())
        .slice(0, 7);

      setEvents(upcomingEvents);
    } catch (err) {
      console.error('Error fetching events:', err);
      setError('Failed to load events');
    } finally {
      setLoading(false);
    }
  };

  // Use unified refresh
  const { refresh } = useUnifiedRefresh({
    resource: 'calendar',
    interval: REFRESH_INTERVALS.CALENDAR,
    enabled: status === 'authenticated',
    onRefresh: fetchEvents,
    priority: 'low',
  });

  // Initial fetch
  useEffect(() => {
    if (status === 'authenticated') {
      fetchEvents();
    }
  }, [status]);

  // ... rest of component (formatDate, formatTime, render)
  
  return (
    <Card className="...">
      <CardHeader>
        <CardTitle>Agenda</CardTitle>
        <Button onClick={() => refresh(true)}>
          <RefreshCw className="..." />
        </Button>
      </CardHeader>
      {/* ... */}
    </Card>
  );
}
```

---

### Phase 4: Implementation Steps

#### Step 1: Create Core Infrastructure (Day 1)

1. ✅ Create `lib/services/refresh-manager.ts`
2. ✅ Create `lib/utils/request-deduplication.ts`
3. ✅ Create `lib/constants/refresh-intervals.ts`
4. ✅ Create `hooks/use-unified-refresh.ts`

**Testing**: Unit tests for each module

---

#### Step 2: Fix Memory Leaks (Day 1-2)

1. ✅ Fix `useNotifications` hook cleanup
2. ✅ Fix notification badge double fetching
3. ✅ Fix widget interval cleanup
4. ✅ Fix Redis KEYS → SCAN

**Testing**: Memory leak detection in DevTools

---

#### Step 3: Refactor Notifications (Day 2)

1. ✅ Refactor `hooks/use-notifications.ts`
2. ✅ Update `components/notification-badge.tsx`
3. ✅ Remove duplicate fetch logic

**Testing**: Verify no duplicate API calls

---

#### Step 4: Refactor Widgets (Day 3-4)

1. ✅ Refactor `components/calendar.tsx`
2. ✅ Refactor `components/parole.tsx`
3. ✅ Refactor `components/news.tsx`
4. ✅ Refactor `components/email.tsx`
5. ✅ Refactor `components/flow.tsx` (Duties)
6. ✅ Refactor `components/main-nav.tsx` (Time display)

**Testing**: Verify all widgets refresh correctly

---

#### Step 5: Testing & Optimization (Day 5)

1. ✅ Performance testing
2. ✅ Memory leak verification
3. ✅ API call reduction verification
4. ✅ User experience testing

---

## 📊 Expected Improvements

### Before:
- **API Calls**: ~120-150 calls/minute (with duplicates)
- **Memory Leaks**: Yes (intervals not cleaned up)
- **Refresh Coordination**: None
- **Request Deduplication**: None

### After:
- **API Calls**: ~40-50 calls/minute (60-70% reduction)
- **Memory Leaks**: None (proper cleanup)
- **Refresh Coordination**: Centralized
- **Request Deduplication**: Full coverage

---

## 🎯 Success Metrics

1. **API Call Reduction**: 60%+ reduction in duplicate calls
2. **Memory Usage**: No memory leaks detected
3. **Performance**: Faster page loads, smoother UX
4. **Maintainability**: Single source of truth for refresh logic

---

## 🚀 Quick Start Implementation

### Priority Order:

1. **Critical** (Do First):
   - Fix memory leaks
   - Create refresh manager
   - Create request deduplication

2. **High** (Do Second):
   - Refactor notifications
   - Refactor high-frequency widgets (parole, notifications)

3. **Medium** (Do Third):
   - Refactor medium-frequency widgets (email, duties)

4. **Low** (Do Last):
   - Refactor low-frequency widgets (calendar, news)

---

## 📝 Notes

- All intervals are configurable via constants
- Refresh manager can be paused/resumed globally
- Request deduplication prevents duplicate calls within 5 seconds
- All cleanup is handled automatically
- Compatible with existing code (gradual migration)

---

*Implementation Plan v1.0*