    54→    public function ask(string $prompt, array $modelOverrides = [], array $providerSelection = [], string $system = '', array $history = []): array
    55→    {
    56→        $responses = [];
    57→        $modelsUsed = [];
    58→        $totalCost = 0.0;
    59→
    60→        foreach ($this->providers as $code => $provider) {
    61→            if (!empty($providerSelection) && !in_array($code, $providerSelection, true)) {
    62→                continue;
    63→            }

Prompt:
Design an implementation plan for adding a "context builder + AI chat" feature to the Email Intelligence web app. Here's all the context:

## What the user wants:
The user wants to replicate the Chrome extension's "Ask AI" functionality in the web app, with ENHANCED capabilities:
1. Search for contacts by email/name
2. Select a single thread, multiple threads, or ALL threads for a contact as AI context
3. Choose an AI model from a dropdown (multi-provider: Claude, GPT, Gemini, Grok, DeepSeek, etc.)
4. Ask questions about the selected context - e.g. "how much should I quote?", "what's the deception level?"

Example use cases:
- "Grab all emails for stephanie.k***@asendia.com and check how much we quoted, her profile, and ask how much to quote for task X"
- "Grab multiple threads of david.zolanz@digitalj2.com and ask the level of deception from david"

## Current web app AI chat (what already exists):
- File: `/var/www/mssql.danmarcrm.com/dev1/extension2/public/assets/app.js` (lines 2672-2882)
- Has modes: general, thread (single), contact, quote, reply
- Sends to `api.php?action=ai_chat` which builds context from ONE thread_id or ONE contact_id
- Has model dropdown but only basic models
- Draggable panel with chat history, quick prompts

## Current backend:
- `api.php?action=ai_chat` (lines 2310-2491) - builds context from single thread or contact
- `api.php?action=threads` - lists threads, filterable by contact_id and search
- `QueryAdapter::getThreads($contactId, $search, $limit, $offset)` - fetches threads
- MongoDB + SQL dual support throughout
- Thread emails stored with thread_id grouping
- Contact profiles with trust scores, communication style

## Chrome extension (what to draw inspiration from):
- Has provider dropdown + model dropdown with ALL providers (Claude, OpenAI, Gemini, Grok, DeepSeek, Qwen, Mistral, Cohere)
- Model list defined in JS (lines 18-56 of contentScript.js)
- `fetchEmailContext()` can fetch by thread ID or by email (returns contact + recent threads)
- Conversation history (last 10 exchanges)
- Quick question buttons

## The enhancement needed:
The current web chat can only load ONE thread or ONE contact as context. We need:

1. A **contact/thread search** UI in the AI chat panel (or a new dedicated panel)
2. Ability to **add multiple threads to context** - like a "shopping cart" for context
3. Ability to **"Add all threads for contact X"** with one click
4. The **multi-provider model dropdown** from the chrome extension
5. A **visible context summary** showing what's loaded (e.g. "3 threads, 15 emails from stephanie@...")
6. Backend API changes to accept multiple thread_ids and build combined context

## Key files to modify:
- `/var/www/mssql.danmarcrm.com/dev1/extension2/public/assets/app.js` - Frontend JS
- `/var/www/mssql.danmarcrm.com/dev1/extension2/public/api.php` - Backend API
- `/var/www/mssql.danmarcrm.com/dev1/extension2/public/index.php` - HTML for new UI elements

## Constraints:
- Keep using the existing draggable AI chat panel (enhance it, don't replace it)
- Must work with both MongoDB and SQL backends
- Keep the existing modes (thread, contact, quote, reply, general) but enhance them
- Reuse existing QueryAdapter and MongoDatabase classes
- Use the same AI proxy at `aiapi/api_codex51/api/index.php`

Design a detailed, step-by-step implementation plan.


TodoWrite

Explore existing QueryAdapter methods for threads

Exploring existing QueryAdapter methods for threads

Explore Chrome extension's fetchEmailContext and Ask AI flow

Exploring Chrome extension's Ask AI flow

Explore the AI proxy API endpoint structure

Explore existing search/contacts API actions and frontend patterns

Exploring existing search/contacts API actions

Design the complete implementation plan

Designing the complete implementation plan

Prompt:
Design an implementation plan for adding a "context builder + AI chat" feature to the Email Intelligence web app. Here's all the context:

## What the user wants:
The user wants to replicate the Chrome extension's "Ask AI" functionality in the web app, with ENHANCED capabilities:
1. Search for contacts by email/name
2. Select a single thread, multiple threads, or ALL threads for a contact as AI context
3. Choose an AI model from a dropdown (multi-provider: Claude, GPT, Gemini, Grok, DeepSeek, etc.)
4. Ask questions about the selected context - e.g. "how much should I quote?", "what's the deception level?"

Example use cases:
- "Grab all emails for stephanie.k***@asendia.com and check how much we quoted, her profile, and ask how much to quote for task X"
- "Grab multiple threads of david.zolanz@digitalj2.com and ask the level of deception from david"

## Current web app AI chat (what already exists):
- File: `/var/www/mssql.danmarcrm.com/dev1/extension2/public/assets/app.js` (lines 2672-2882)
- Has modes: general, thread (single), contact, quote, reply
- Sends to `api.php?action=ai_chat` which builds context from ONE thread_id or ONE contact_id
- Has model dropdown but only basic models
- Draggable panel with chat history, quick prompts

## Current backend:
- `api.php?action=ai_chat` (lines 2310-2491) - builds context from single thread or contact
- `api.php?action=threads` - lists threads, filterable by contact_id and search
- `QueryAdapter::getThreads($contactId, $search, $limit, $offset)` - fetches threads
- MongoDB + SQL dual support throughout
- Thread emails stored with thread_id grouping
- Contact profiles with trust scores, communication style

## Chrome extension (what to draw inspiration from):
- Has provider dropdown + model dropdown with ALL providers (Claude, OpenAI, Gemini, Grok, DeepSeek, Qwen, Mistral, Cohere)
- Model list defined in JS (lines 18-56 of contentScript.js)
- `fetchEmailContext()` can fetch by thread ID or by email (returns contact + recent threads)
- Conversation history (last 10 exchanges)
- Quick question buttons

## The enhancement needed:
The current web chat can only load ONE thread or ONE contact as context. We need:

1. A **contact/thread search** UI in the AI chat panel (or a new dedicated panel)
2. Ability to **add multiple threads to context** - like a "shopping cart" for context
3. Ability to **"Add all threads for contact X"** with one click
4. The **multi-provider model dropdown** from the chrome extension
5. A **visible context summary** showing what's loaded (e.g. "3 threads, 15 emails from stephanie@...")
6. Backend API changes to accept multiple thread_ids and build combined context

## Key files to modify:
- `/var/www/mssql.danmarcrm.com/dev1/extension2/public/assets/app.js` - Frontend JS
- `/var/www/mssql.danmarcrm.com/dev1/extension2/public/api.php` - Backend API
- `/var/www/mssql.danmarcrm.com/dev1/extension2/public/index.php` - HTML for new UI elements

## Constraints:
- Keep using the existing draggable AI chat panel (enhance it, don't replace it)
- Must work with both MongoDB and SQL backends
- Keep the existing modes (thread, contact, quote, reply, general) but enhance them
- Reuse existing QueryAdapter and MongoDatabase classes
- Use the same AI proxy at `aiapi/api_codex51/api/index.php`

Design a detailed, step-by-step implementation plan.


Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable

Prompt:
Design an implementation plan for adding a "context builder + AI chat" feature to the Email Intelligence web app. Here's all the context:

## What the user wants:
The user wants to replicate the Chrome extension's "Ask AI" functionality in the web app, with ENHANCED capabilities:
1. Search for contacts by email/name
2. Select a single thread, multiple threads, or ALL threads for a contact as AI context
3. Choose an AI model from a dropdown (multi-provider: Claude, GPT, Gemini, Grok, DeepSeek, etc.)
4. Ask questions about the selected context - e.g. "how much should I quote?", "what's the deception level?"

Example use cases:
- "Grab all emails for stephanie.k***@asendia.com and check how much we quoted, her profile, and ask how much to quote for task X"
- "Grab multiple threads of david.zolanz@digitalj2.com and ask the level of deception from david"

## Current web app AI chat (what already exists):
- File: `/var/www/mssql.danmarcrm.com/dev1/extension2/public/assets/app.js` (lines 2672-2882)
- Has modes: general, thread (single), contact, quote, reply
- Sends to `api.php?action=ai_chat` which builds context from ONE thread_id or ONE contact_id
- Has model dropdown but only basic models
- Draggable panel with chat history, quick prompts

## Current backend:
- `api.php?action=ai_chat` (lines 2310-2491) - builds context from single thread or contact
- `api.php?action=threads` - lists threads, filterable by contact_id and search
- `QueryAdapter::getThreads($contactId, $search, $limit, $offset)` - fetches threads
- MongoDB + SQL dual support throughout
- Thread emails stored with thread_id grouping
- Contact profiles with trust scores, communication style

## Chrome extension (what to draw inspiration from):
- Has provider dropdown + model dropdown with ALL providers (Claude, OpenAI, Gemini, Grok, DeepSeek, Qwen, Mistral, Cohere)
- Model list defined in JS (lines 18-56 of contentScript.js)
- `fetchEmailContext()` can fetch by thread ID or by email (returns contact + recent threads)
- Conversation history (last 10 exchanges)
- Quick question buttons

## The enhancement needed:
The current web chat can only load ONE thread or ONE contact as context. We need:

1. A **contact/thread search** UI in the AI chat panel (or a new dedicated panel)
2. Ability to **add multiple threads to context** - like a "shopping cart" for context
3. Ability to **"Add all threads for contact X"** with one click
4. The **multi-provider model dropdown** from the chrome extension
5. A **visible context summary** showing what's loaded (e.g. "3 threads, 15 emails from stephanie@...")
6. Backend API changes to accept multiple thread_ids and build combined context

## Key files to modify:
- `/var/www/mssql.danmarcrm.com/dev1/extension2/public/assets/app.js` - Frontend JS
- `/var/www/mssql.danmarcrm.com/dev1/extension2/public/api.php` - Backend API
- `/var/www/mssql.danmarcrm.com/dev1/extension2/public/index.php` - HTML for new UI elements

## Constraints:
- Keep using the existing draggable AI chat panel (enhance it, don't replace it)
- Must work with both MongoDB and SQL backends
- Keep the existing modes (thread, contact, quote, reply, general) but enhance them
- Reuse existing QueryAdapter and MongoDatabase classes
- Use the same AI proxy at `aiapi/api_codex51/api/index.php`

Design a detailed, step-by-step implementation plan.


- we should also make sure not sending  over max capacity of tokens
- also make sure not to send all over again the previous message replies, that way would send a much repetitive number of emails for longer email threads
- replies from AI should be saved so we can re-load them and also should be used for further conversation as in a chat


Good points. Let me investigate how the current system handles these concerns.


  2310→        case 'ai_chat':
  2311→            if ($_SERVER['REQUEST_METHOD'] !== 'POST') jsonError('POST required', 405);
  2312→
  2313→            $jsonData = json_decode(file_get_contents('php://input'), true) ?? [];
  2314→            $chatMessage  = trim($jsonData['message'] ?? '');
  2315→            $chatModel    = $jsonData['model'] ?? 'gpt-4o';
  2316→            $chatMode     = $jsonData['mode'] ?? 'general';   // thread|contact|quote|reply|general
  2317→            $chatThreadId = $jsonData['thread_id'] ?? null;
  2318→            $chatContactId= (int)($jsonData['contact_id'] ?? 0);
  2319→            $chatHistory  = $jsonData['history'] ?? [];       // [{role,content}]
  2320→            $chatEmailId  = (int)($jsonData['email_id'] ?? 0);
  2321→
  2322→            if (!$chatMessage) jsonError('message required');
  2323→            if ($chatThreadId) $chatThreadId = resolveThreadId($chatThreadId);
  2324→
  2325→            // --- Build context block ---
  2326→            $ctxParts = [];
  2327→            $ctxAdapter = QueryAdapter::getInstance();
  2328→            $ctxMongo = \EmailIntelligence\MongoDatabase::getInstance();
  2329→            $useMongoDB = $ctxAdapter->getActiveDatabase() === 'mongodb';
  2330→
  2331→            // Thread context - use MongoDB-aware queries
  2332→            if ($chatThreadId) {
  2333→                if ($useMongoDB) {
  2334→                    $ctxEmails = $ctxMongo->findAll('emails', ['thread_id' => $chatThreadId], [
  2335→                        'sort' => ['date' => 1]
  2336→                    ]);
  2337→                } else {
  2338→                    $ctxEmails = $db->fetchAll(
  2339→                        "SELECT direction, date, from_email, to_email, subject, body_text FROM emails WHERE thread_id = ? ORDER BY date ASC",
  2340→                        [$chatThreadId]
  2341→                    );
  2342→                }
  2343→                if ($ctxEmails) {
  2344→                    $ctxParts[] = "=== EMAIL THREAD (" . count($ctxEmails) . " emails) ===";
  2345→                    foreach ($ctxEmails as $i => $em) {
  2346→                        $dir = ($em['direction'] ?? '') === 'inbound' ? 'FROM THEM' : 'FROM US';
  2347→                        $body = substr($em['body_text'] ?? '', 0, 800);
  2348→                        $ctxParts[] = "Email #".($i+1)." {$dir} ({$em['date']})\nSubject: {$em['subject']}\n{$body}";
  2349→                    }
  2350→                    // Include existing analysis if available
  2351→                    if ($useMongoDB) {
  2352→                        $ctxAnalysis = $ctxMongo->findOne('thread_analysis', ['thread_id' => $chatThreadId, 'is_analyzed' => 1]);
  2353→                        if (!$ctxAnalysis) {
  2354→                            // Try without is_analyzed filter (MongoDB may store as bool or int)
  2355→                            $ctxAnalysis = $ctxMongo->findOne('thread_analysis', ['thread_id' => $chatThreadId]);
  2356→                        }
  2357→                    } else {
  2358→                        $ctxAnalysis = $db->fetchOne("SELECT summary, raw_analysis FROM thread_analysis WHERE thread_id = ? AND is_analyzed=1", [$chatThreadId]);
  2359→                    }

        case 'ai_chat':
            if ($_SERVER['REQUEST_METHOD'] !== 'POST') jsonError('POST required', 405);

            $jsonData = json_decode(file_get_contents('php://input'), true) ?? [];
            $chatMessage  = trim($jsonData['message'] ?? '');
            $chatModel    = $jsonData['model'] ?? 'gpt-4o';
            $chatMode     = $jsonData['mode'] ?? 'general';   // thread|contact|quote|reply|general
            $chatThreadId = $jsonData['thread_id'] ?? null;
            $chatContactId= (int)($jsonData['contact_id'] ?? 0);
            $chatHistory  = $jsonData['history'] ?? [];       // [{role,content}]
            $chatEmailId  = (int)($jsonData['email_id'] ?? 0);

            if (!$chatMessage) jsonError('message required');
            if ($chatThreadId) $chatThreadId = resolveThreadId($chatThreadId);

            // --- Build context block ---
            $ctxParts = [];
            $ctxAdapter = QueryAdapter::getInstance();
            $ctxMongo = \EmailIntelligence\MongoDatabase::getInstance();
            $useMongoDB = $ctxAdapter->getActiveDatabase() === 'mongodb';

            // Thread context - use MongoDB-aware queries
            if ($chatThreadId) {
                if ($useMongoDB) {
                    $ctxEmails = $ctxMongo->findAll('emails', ['thread_id' => $chatThreadId], [
                        'sort' => ['date' => 1]
                    ]);
                } else {
                    $ctxEmails = $db->fetchAll(
                        "SELECT direction, date, from_email, to_email, subject, body_text FROM emails WHERE thread_id = ? ORDER BY date ASC",
                        [$chatThreadId]
                    );
                }
                if ($ctxEmails) {
                    $ctxParts[] = "=== EMAIL THREAD (" . count($ctxEmails) . " emails) ===";
                    foreach ($ctxEmails as $i => $em) {
                        $dir = ($em['direction'] ?? '') === 'inbound' ? 'FROM THEM' : 'FROM US';
                        $body = substr($em['body_text'] ?? '', 0, 800);
                        $ctxParts[] = "Email #".($i+1)." {$dir} ({$em['date']})\nSubject: {$em['subject']}\n{$body}";
                    }
                    // Include existing analysis if available
                    if ($useMongoDB) {
                        $ctxAnalysis = $ctxMongo->findOne('thread_analysis', ['thread_id' => $chatThreadId, 'is_analyzed' => 1]);
                        if (!$ctxAnalysis) {
                            // Try without is_analyzed filter (MongoDB may store as bool or int)
                            $ctxAnalysis = $ctxMongo->findOne('thread_analysis', ['thread_id' => $chatThreadId]);
                        }
                    } else {
                        $ctxAnalysis = $db->fetchOne("SELECT summary, raw_analysis FROM thread_analysis WHERE thread_id = ? AND is_analyzed=1", [$chatThreadId]);
                    }


   340→  async function loadSavedAIContext() {
   341→    try {
   342→      const storage = await chrome.storage.local.get('savedAIContext');
   343→      if (storage.savedAIContext && Array.isArray(storage.savedAIContext)) {
   344→        savedAIContext = storage.savedAIContext;
   345→        console.log("[Email Intelligence] Loaded", savedAIContext.length, "saved context items");
   346→      }
   347→    } catch (e) {
   348→      console.log("[Email Intelligence] Could not load saved context:", e);
   349→    }
   350→  }
   351→
   352→  /**
   353→   * Save context to Chrome storage
   354→   */
   355→  async function saveSavedAIContext() {
   356→    try {
   357→      await chrome.storage.local.set({ savedAIContext });
   358→      console.log("[Email Intelligence] Saved context:", savedAIContext.length, "items");
   359→    } catch (e) {
   360→      console.log("[Email Intelligence] Could not save context:", e);
   361→    }
   362→  }
   363→
   364→  /**
   365→   * Add item to saved context
   366→   * @param {string} type - 'question' | 'answer' | 'qa_pair' | 'selection'
   367→   * @param {string} content - The content to save
   368→   * @param {string} description - Optional short description
   369→   * @param {string} question - The question (for qa_pair type)
   370→   */
   371→  function addToSavedContext(type, content, description = '', question = '') {
   372→    const item = {
   373→      id: Date.now().toString(),
   374→      type: type,
   375→      content: content,
   376→      description: description || content.substring(0, 50),
   377→      question: question,
   378→      created_at: new Date().toISOString()
   379→    };
   380→
   381→    savedAIContext.push(item);
   382→    saveSavedAIContext();
   383→    updateSavedContextDisplay();
   384→
   385→    // Show confirmation
   386→    showToast('Context saved! It will be included in future AI requests.');
   387→  }
   388→
   389→  /**
   390→   * Remove item from saved context
   391→   */
   392→  function removeFromSavedContext(id) {
   393→    savedAIContext = savedAIContext.filter(item => item.id !== id);
   394→    saveSavedAIContext();
   395→    updateSavedContextDisplay();
   396→  }
   397→
   398→  /**
   399→   * Clear all saved context
   400→   */
   401→  function clearSavedAIContext() {
   402→    savedAIContext = [];
   403→    saveSavedAIContext();
   404→    updateSavedContextDisplay();
   405→  }
   406→
   407→  /**
   408→   * Build saved context string for AI request
   409→   */
   410→  function buildSavedContextPrompt() {
   411→    if (savedAIContext.length === 0) return '';
   412→
   413→    let context = '\n\n=== USER-SAVED CONTEXT (IMPORTANT - Always Consider) ===\n';
   414→    context += 'The user has saved the following important information that should inform all responses:\n\n';
   415→
   416→    savedAIContext.forEach((item, index) => {
   417→      const typeLabel = {
   418→        'question': '📝 Saved Question',
   419→        'answer': '💡 Saved Answer',
   420→        'qa_pair': '🔗 Saved Q&A',
   421→        'selection': '📌 Saved Note'
   422→      }[item.type] || '📌 Context';
   423→
   424→      context += `[${typeLabel} ${index + 1}]\n`;
   425→      if (item.type === 'qa_pair' && item.question) {
   426→        context += `Q: ${item.question}\n`;
   427→        context += `A: ${item.content}\n`;
   428→      } else {
   429→        context += `${item.content}\n`;
   430→      }
   431→      context += '\n';
   432→    });
   433→
   434→    context += '=== END SAVED CONTEXT ===\n\n';
   435→    return context;
   436→  }
   437→
   438→  /**
   439→   * Update saved context display in sidebar

  async function loadSavedAIContext() {
    try {
      const storage = await chrome.storage.local.get('savedAIContext');
      if (storage.savedAIContext && Array.isArray(storage.savedAIContext)) {
        savedAIContext = storage.savedAIContext;
        console.log("[Email Intelligence] Loaded", savedAIContext.length, "saved context items");
      }
    } catch (e) {
      console.log("[Email Intelligence] Could not load saved context:", e);
    }
  }

  /**
   * Save context to Chrome storage
   */
  async function saveSavedAIContext() {
    try {
      await chrome.storage.local.set({ savedAIContext });
      console.log("[Email Intelligence] Saved context:", savedAIContext.length, "items");
    } catch (e) {
      console.log("[Email Intelligence] Could not save context:", e);
    }
  }

  /**
   * Add item to saved context
   * @param {string} type - 'question' | 'answer' | 'qa_pair' | 'selection'
   * @param {string} content - The content to save
   * @param {string} description - Optional short description
   * @param {string} question - The question (for qa_pair type)
   */
  function addToSavedContext(type, content, description = '', question = '') {
    const item = {
      id: Date.now().toString(),
      type: type,
      content: content,
      description: description || content.substring(0, 50),
      question: question,
      created_at: new Date().toISOString()
    };

    savedAIContext.push(item);
    saveSavedAIContext();
    updateSavedContextDisplay();

    // Show confirmation
    showToast('Context saved! It will be included in future AI requests.');
  }

  /**
   * Remove item from saved context
   */
  function removeFromSavedContext(id) {
    savedAIContext = savedAIContext.filter(item => item.id !== id);
    saveSavedAIContext();
    updateSavedContextDisplay();
  }

  /**
   * Clear all saved context
   */
  function clearSavedAIContext() {
    savedAIContext = [];
    saveSavedAIContext();
    updateSavedContextDisplay();
  }

  /**
   * Build saved context string for AI request
   */
  function buildSavedContextPrompt() {
    if (savedAIContext.length === 0) return '';

    let context = '\n\n=== USER-SAVED CONTEXT (IMPORTANT - Always Consider) ===\n';
    context += 'The user has saved the following important information that should inform all responses:\n\n';

    savedAIContext.forEach((item, index) => {
      const typeLabel = {
        'question': '📝 Saved Question',
        'answer': '💡 Saved Answer',
        'qa_pair': '🔗 Saved Q&A',
        'selection': '📌 Saved Note'
      }[item.type] || '📌 Context';

      context += `[${typeLabel} ${index + 1}]\n`;
      if (item.type === 'qa_pair' && item.question) {
        context += `Q: ${item.question}\n`;
        context += `A: ${item.content}\n`;
      } else {
        context += `${item.content}\n`;
      }
      context += '\n';
    });

    context += '=== END SAVED CONTEXT ===\n\n';
    return context;
  }

  /**
   * Update saved context display in sidebar


2310:        case 'ai_chat':

[Showing results with pagination = limit: 20, offset: 0]

2310:        case 'ai_chat':


src/ThreadAnalyzer.php:912:        "problem": "<data issue they have - scattered, duplicates, manual entry, etc>",
src/ThreadAnalyzer.php:1031:            // Generate unique request ID to track duplicate calls
src/ThreadAnalyzer.php:1464:                // Ignore duplicate errors
src/ThreadAnalyzer.php:1614:     * - "From: ... Sent: ... To: ... Subject: ..." forwarded headers
src/ThreadAnalyzer.php:1626:            '/From:\s+.+\nSent:\s+.+\nTo:\s+.+\nSubject:\s+.+/i',  // Outlook forwarded header
src/PatternDetector.php:180:        // Sort by similarity and remove duplicates
src/PatternDetector.php:183:        // Deduplicate by email_id
src/Database.php:336:     * @param array $updateColumns Columns to update on duplicate (empty = all)
src/WebScraper.php:196:        // Remove duplicate contact info
src/WebScraper.php:338:        // Clean and deduplicate
src/AIAnalyzer.php:545:            // The memory service may handle duplicates or we can add deduplication later
src/QuoteExtractor.php:597:        // Check for duplicate (same email, amount, type)

[Showing results with pagination = limit: 20, offset: 0]


backup_20251217_005857/src/ContactAnalyzer.php:37:            'max_tokens' => defined('CLAUDE_MAX_TOKENS') ? CLAUDE_MAX_TOKENS : 8000,
backup_20251217_005857/src/ContactAnalyzer.php:448:                'max_tokens' => $this->config['max_tokens'],
backup_20251217_005857/src/AIAnalyzer.php:295:            $bodyText = substr($bodyText, 0, 8000) . "\n\n[... email truncated for analysis ...]";
backup_20251217_005857/src/AIAnalyzer.php:381:                'max_tokens' => $this->maxTokens,
backup_20251217_005857/src/ContactProfiler.php:75:            'max_tokens' => defined('CLAUDE_MAX_TOKENS') ? CLAUDE_MAX_TOKENS : 4096,
backup_20251217_005857/src/ContactProfiler.php:440:                'max_tokens' => $this->config['max_tokens'],
backup_20251217_005857/src/ThreadAnalyzer.php:60:                'max_tokens' => defined('OPENAI_MAX_TOKENS') ? OPENAI_MAX_TOKENS : 16000,
backup_20251217_005857/src/ThreadAnalyzer.php:81:                'max_tokens' => defined('CLAUDE_MAX_TOKENS') ? CLAUDE_MAX_TOKENS : 16000,
backup_20251217_005857/src/ThreadAnalyzer.php:393:                $body = substr($body, 0, 2000) . "\n[... email truncated ...]";
backup_20251217_005857/src/ThreadAnalyzer.php:1025:                'max_tokens' => $this->config['max_tokens'],
backup_20251217_005857/src/ThreadAnalyzer.php:1035:                    'max_tokens' => $this->config['max_tokens'],
backup_20251217_005857/src/ThreadAnalyzer.php:1044:                    'max_tokens' => $this->config['max_tokens'],
backup_20251217_005857/src/VectorMemory.php:320:            // Clean and truncate body
backup_20251217_005857/src/VectorMemory.php:485:                return '[truncated - max depth reached]';
backup_20251217_005857/src/VectorMemory.php:493:                $data = substr($data, 0, 100000) . '... [truncated]';
cli/ai_analyze.php:311:                $body = substr($body, 0, 3000) . "\n[... truncated ...]";
cli/ai_analyze.php:492:            $body = substr($body, 0, 4000) . "\n[... truncated ...]";
cli/ai_analyze.php:838:            'max_tokens' => 4000,
cli/ai_analyze.php:883:            'max_tokens' => 4000,
cli/reset_data.php:90:            echo "  ⚠ Warning: Could not truncate {$table}: " . $e->getMessage() . "\n";

[Showing results with pagination = limit: 20, offset: 0]


  1600→            $summaryLine = date('Y-m-d H:i:s') . " | Thread: {$threadId} | Provider: {$provider} | Time: {$elapsedTime}s | Tokens: " . (($usage['input_tokens'] ?? 0) + ($usage['output_tokens'] ?? 0)) . " | File: {$filename}\n";
  1601→            file_put_contents($summaryFile, $summaryLine, FILE_APPEND);
  1602→
  1603→        } catch (Exception $e) {
  1604→            // Don't fail the analysis if logging fails
  1605→            error_log("Failed to save analysis log: " . $e->getMessage());
  1606→        }
  1607→    }
  1608→
  1609→    /**
  1610→     * Strip quoted content from email body to avoid duplication
  1611→     * Removes:
  1612→     * - Lines starting with > (quoted replies)
  1613→     * - "On X date, Y wrote:" patterns and everything after
  1614→     * - "From: ... Sent: ... To: ... Subject: ..." forwarded headers
  1615→     */
  1616→    private function stripQuotedContent(string $body): string
  1617→    {
  1618→        // Remove lines starting with > (quoted text)
  1619→        $body = preg_replace('/^>+.*$/m', '', $body);
  1620→
  1621→        // Remove "On [date], [person] wrote:" and everything after
  1622→        $patterns = [
  1623→            '/On\s+.{10,60}\s+wrote:\s*$/mi',  // "On Monday, Jan 1, John wrote:"
  1624→            '/On\s+\d{1,2}[\/-]\d{1,2}[\/-]\d{2,4}.+wrote:\s*$/mi',  // "On 01/02/2023, John wrote:"
  1625→            '/-{2,}.*Original Message.*-{2,}/i',  // "-----Original Message-----"
  1626→            '/From:\s+.+\nSent:\s+.+\nTo:\s+.+\nSubject:\s+.+/i',  // Outlook forwarded header
  1627→            '/_{5,}.*$/m',  // _____ separator lines
  1628→        ];
  1629→
  1630→        foreach ($patterns as $pattern) {
  1631→            if (preg_match($pattern, $body, $matches, PREG_OFFSET_CAPTURE)) {
  1632→                // Cut everything from the match onwards
  1633→                $body = substr($body, 0, $matches[0][1]);
  1634→            }
  1635→        }
  1636→
  1637→        // Clean up multiple consecutive newlines
  1638→        $body = preg_replace('/\n{3,}/', "\n\n", $body);
  1639→
  1640→        return trim($body);
  1641→    }
  1642→}
  1643→

            $summaryLine = date('Y-m-d H:i:s') . " | Thread: {$threadId} | Provider: {$provider} | Time: {$elapsedTime}s | Tokens: " . (($usage['input_tokens'] ?? 0) + ($usage['output_tokens'] ?? 0)) . " | File: {$filename}\n";
            file_put_contents($summaryFile, $summaryLine, FILE_APPEND);

        } catch (Exception $e) {
            // Don't fail the analysis if logging fails
            error_log("Failed to save analysis log: " . $e->getMessage());
        }
    }

    /**
     * Strip quoted content from email body to avoid duplication
     * Removes:
     * - Lines starting with > (quoted replies)
     * - "On X date, Y wrote:" patterns and everything after
     * - "From: ... Sent: ... To: ... Subject: ..." forwarded headers
     */
    private function stripQuotedContent(string $body): string
    {
        // Remove lines starting with > (quoted text)
        $body = preg_replace('/^>+.*$/m', '', $body);

        // Remove "On [date], [person] wrote:" and everything after
        $patterns = [
            '/On\s+.{10,60}\s+wrote:\s*$/mi',  // "On Monday, Jan 1, John wrote:"
            '/On\s+\d{1,2}[\/-]\d{1,2}[\/-]\d{2,4}.+wrote:\s*$/mi',  // "On 01/02/2023, John wrote:"
            '/-{2,}.*Original Message.*-{2,}/i',  // "-----Original Message-----"
            '/From:\s+.+\nSent:\s+.+\nTo:\s+.+\nSubject:\s+.+/i',  // Outlook forwarded header
            '/_{5,}.*$/m',  // _____ separator lines
        ];

        foreach ($patterns as $pattern) {
            if (preg_match($pattern, $body, $matches, PREG_OFFSET_CAPTURE)) {
                // Cut everything from the match onwards
                $body = substr($body, 0, $matches[0][1]);
            }
        }

        // Clean up multiple consecutive newlines
        $body = preg_replace('/\n{3,}/', "\n\n", $body);

        return trim($body);
    }
}


No files found


Good findings. Let me summarize what I found:

1. **Token/context limits**: Currently there's NO token budget management in the `ai_chat` endpoint. Email bodies are truncated to 800 chars each but no overall limit. The ThreadAnalyzer has `stripQuotedContent()` (line 1616) that strips quoted replies, but this is NOT used in the `ai_chat` endpoint - it's only used in the analysis pipeline.

2. **Quoted content stripping**: `ThreadAnalyzer::stripQuotedContent()` exists (lines 1616-1641) and does exactly what we need - strips `> quoted text`, "On X wrote:" patterns, "Original Message" separators, etc. But it's a private method on ThreadAnalyzer, not used in the ai_chat API endpoint.

3. **Chat history persistence**: There's no server-side storage for conversations - everything lives in the frontend's in-memory history array, and the Chrome extension relies on local browser storage. I need to add proper session persistence to a database.

So my plan is to implement token budget management with smart truncation, leverage the existing quoted content stripping logic to reduce duplication, and create server-side chat session storage.


Important findings:

1. **No token budget** in `ai_chat` - emails are truncated to 800 chars each but no overall limit
2. **`stripQuotedContent()` exists** in [ThreadAnalyzer.php:1616](src/ThreadAnalyzer.php#L1616) but is `private` and NOT used in the `ai_chat` endpoint - it strips `> quoted` text, "On X wrote:" patterns, etc.
3. **No chat persistence** - conversation history only lives in frontend JS memory (`aiChat.history[]`), nothing saved server-side

Let me update the plan with these three concerns addressed.


     1→# Plan: AI Context Builder with Multi-Thread Support
     2→
     3→## Context
     4→
     5→The web app's AI chat panel currently only supports asking AI about **one thread** or **one contact** at a time. The Chrome extension already has a richer "Ask AI" with multi-provider model selection and optional context input. The user wants the web app to support:
     6→
     7→- **Searching** for contacts by email/name
     8→- **Selecting multiple threads** (or all threads for a contact) as AI context
     9→- **Choosing AI provider + model** from a dropdown (synced with Chrome extension's list)
    10→- **Asking questions** about the combined context (e.g. "how much to quote?", "deception level?")
    11→
    12→## Files to Modify
    13→
    14→| File | Changes |
    15→|------|---------|
    16→| [public/index.php](public/index.php) | Replace static model dropdown with provider+model selectors; add context builder HTML section |
    17→| [public/assets/app.js](public/assets/app.js) | Extend `aiChat` state; add `PROVIDER_MODELS`; add context builder functions; modify `sendAiChat()` for multi-thread; update `renderQuickPrompts()` |
    18→| [public/api.php](public/api.php) | Accept `thread_ids[]` + `provider`; build multi-thread context; fix model override format for proxy |
    19→
    20→## Implementation Steps
    21→
    22→### Step 1: Backend – api.php `ai_chat` action (lines 2310-2491)
    23→
    24→**1a.** Parse new fields from JSON input:
    25→- `thread_ids` (array of thread IDs) alongside existing `thread_id`
    26→- `provider` (explicit provider code like "claude", "openai")
    27→
    28→**1b.** Replace single-thread context build (lines 2331-2363) with a loop over all `thread_ids`:
    29→- If `thread_ids` is provided, loop and fetch emails for each thread
    30→- Label each as "THREAD 1 OF N", "THREAD 2 OF N" etc.
    31→- Dynamically reduce per-email body length based on total email count: `min(800, max(200, 60000 / totalEmailCount))`
    32→- Prepend summary line: "MULTI-THREAD CONTEXT: X total emails across Y threads"
    33→- Include per-thread analysis summaries if available
    34→- If only `thread_id` (single) is sent, keep existing behavior
    35→
    36→**1c.** Fix provider detection (lines 2419-2424):
    37→- Use explicit `provider` field from frontend when available
    38→- Add `deepseek` detection for model name fallback
    39→
    40→**1d.** Fix proxy payload model format (line 2431):
    41→- Change `'models' => [$chatModel]` to `'models' => [$chatProvider => $chatModel]`
    42→- This fixes a bug where the AI proxy ignores the selected model
    43→
    44→### Step 2: Frontend JS – app.js
    45→
    46→**2a.** Extend `aiChat` state (line 2672):
    47→- Add `contextThreads: []` – array of `{thread_id, subject, email_count, contact_name}`
    48→- Add `contextContact: null` – `{id, name, email}`
    49→
    50→**2b.** Add `PROVIDER_MODELS` and `PROVIDER_NAMES` constants (after line 2684), synced with Chrome extension's list from `contentScript.js` lines 18-67:
    51→- 8 providers: Claude, OpenAI, Gemini, Grok, DeepSeek, Qwen, Mistral, Cohere
    52→- Same model IDs as the Chrome extension
    53→
    54→**2c.** Add provider dropdown functions:
    55→- `initAiProviderDropdown()` – populates provider select, calls `updateAiModelDropdown()`
    56→- `updateAiModelDropdown()` – populates model select based on selected provider
    57→- Call `initAiProviderDropdown()` from DOMContentLoaded
    58→
    59→**2d.** Add context builder functions:
    60→- `toggleContextBuilder()` – show/hide the search area
    61→- `searchContextContacts()` – calls `apiCall('contacts', {search, limit:5})`, renders clickable results
    62→- `selectContextContact(id, name, email)` – calls `apiCall('threads', {contact:id, limit:50})`, shows thread picker with "+ Add" buttons and an "Add All" button
    63→- `addThreadToContext(threadId, subject, emailCount, contactName)` – pushes to `contextThreads[]`
    64→- `removeThreadFromContext(threadId)` – removes from `contextThreads[]`
    65→- `addAllContactThreads()` – adds all available threads at once
    66→- `clearContextCart()` – clears all context
    67→- `updateContextCart()` – renders the "cart" showing selected threads count and list with remove buttons
    68→
    69→**2e.** Update `updateAiChatContextLabel()` (line 2728):
    70→- When `contextThreads.length > 0`, show green text: "3 threads, ~15 emails from Stephanie"
    71→- Otherwise keep existing mode-based label
    72→
    73→**2f.** Add `multithread` quick prompts to `AI_QUICK_PROMPTS` (line 2686):
    74→- "Summarize all conversations", "How much have we quoted?", "Deception level?", "Build a complete profile", "What patterns emerge?"
    75→- Update `renderQuickPrompts()` to use `multithread` prompts when context builder has threads
    76→
    77→**2g.** Modify `sendAiChat()` (line 2756):
    78→- Read provider from `#ai-chat-provider` select
    79→- If `contextThreads.length > 0`, send `thread_ids: [...]` array
    80→- Otherwise send single `thread_id` as before (backward compat)
    81→- Add `provider` to payload
    82→
    83→### Step 3: HTML – index.php
    84→
    85→**3a.** Replace static model dropdown (lines 538-561) with two selects:
    86→- Provider dropdown (`#ai-chat-provider`) with `onchange="updateAiModelDropdown()"`
    87→- Model dropdown (`#ai-chat-model`) – populated dynamically by JS
    88→
    89→**3b.** Insert context builder section between mode tabs (line 570) and messages div (line 574):
    90→- Context summary bar (always visible, clickable to expand) showing "No context loaded" or "3 threads, ~15 emails"
    91→- Collapsible search area with:
    92→  - Contact/email search input + search button
    93→  - Search results container
    94→  - Thread picker container (shows after contact selected)
    95→  - Context cart (selected threads with remove buttons and "Clear All")
    96→
    97→## Verification
    98→
    99→1. Open https://mssql.danmarcrm.com/dev1/extension2/public/
   100→2. Click the AI chat FAB button (bottom-right)
   101→3. Verify provider dropdown shows all 8 providers, model updates on provider change
   102→4. Click the context summary bar → search area expands
   103→5. Search for a contact email (e.g. "stephanie" or "david.zolanz")
   104→6. Click a contact → threads load with "+ Add" and "Add All" buttons
   105→7. Add individual threads or click "Add All"
   106→8. Verify context label turns green showing thread/email count
   107→9. Verify quick prompts switch to multi-thread prompts
   108→10. Ask a question → verify the AI response has context from ALL selected threads
   109→11. Test backward compat: open a single thread view, click "Ask AI About This Thread" → still works with single thread mode
   110→12. Test different providers/models work correctly
   111→

# Plan: AI Context Builder with Multi-Thread Support

## Context

The web app's AI chat panel currently only supports asking AI about **one thread** or **one contact** at a time. The Chrome extension already has a richer "Ask AI" with multi-provider model selection and optional context input. The user wants the web app to support:

- **Searching** for contacts by email/name
- **Selecting multiple threads** (or all threads for a contact) as AI context
- **Choosing AI provider + model** from a dropdown (synced with Chrome extension's list)
- **Asking questions** about the combined context (e.g. "how much to quote?", "deception level?")

## Files to Modify

| File | Changes |
|------|---------|
| [public/index.php](public/index.php) | Replace static model dropdown with provider+model selectors; add context builder HTML section |
| [public/assets/app.js](public/assets/app.js) | Extend `aiChat` state; add `PROVIDER_MODELS`; add context builder functions; modify `sendAiChat()` for multi-thread; update `renderQuickPrompts()` |
| [public/api.php](public/api.php) | Accept `thread_ids[]` + `provider`; build multi-thread context; fix model override format for proxy |

## Implementation Steps

### Step 1: Backend – api.php `ai_chat` action (lines 2310-2491)

**1a.** Parse new fields from JSON input:
- `thread_ids` (array of thread IDs) alongside existing `thread_id`
- `provider` (explicit provider code like "claude", "openai")

**1b.** Replace single-thread context build (lines 2331-2363) with a loop over all `thread_ids`:
- If `thread_ids` is provided, loop and fetch emails for each thread
- Label each as "THREAD 1 OF N", "THREAD 2 OF N" etc.
- Dynamically reduce per-email body length based on total email count: `min(800, max(200, 60000 / totalEmailCount))`
- Prepend summary line: "MULTI-THREAD CONTEXT: X total emails across Y threads"
- Include per-thread analysis summaries if available
- If only `thread_id` (single) is sent, keep existing behavior

**1c.** Fix provider detection (lines 2419-2424):
- Use explicit `provider` field from frontend when available
- Add `deepseek` detection for model name fallback

**1d.** Fix proxy payload model format (line 2431):
- Change `'models' => [$chatModel]` to `'models' => [$chatProvider => $chatModel]`
- This fixes a bug where the AI proxy ignores the selected model

### Step 2: Frontend JS – app.js

**2a.** Extend `aiChat` state (line 2672):
- Add `contextThreads: []` – array of `{thread_id, subject, email_count, contact_name}`
- Add `contextContact: null` – `{id, name, email}`

**2b.** Add `PROVIDER_MODELS` and `PROVIDER_NAMES` constants (after line 2684), synced with Chrome extension's list from `contentScript.js` lines 18-67:
- 8 providers: Claude, OpenAI, Gemini, Grok, DeepSeek, Qwen, Mistral, Cohere
- Same model IDs as the Chrome extension

**2c.** Add provider dropdown functions:
- `initAiProviderDropdown()` – populates provider select, calls `updateAiModelDropdown()`
- `updateAiModelDropdown()` – populates model select based on selected provider
- Call `initAiProviderDropdown()` from DOMContentLoaded

**2d.** Add context builder functions:
- `toggleContextBuilder()` – show/hide the search area
- `searchContextContacts()` – calls `apiCall('contacts', {search, limit:5})`, renders clickable results
- `selectContextContact(id, name, email)` – calls `apiCall('threads', {contact:id, limit:50})`, shows thread picker with "+ Add" buttons and an "Add All" button
- `addThreadToContext(threadId, subject, emailCount, contactName)` – pushes to `contextThreads[]`
- `removeThreadFromContext(threadId)` – removes from `contextThreads[]`
- `addAllContactThreads()` – adds all available threads at once
- `clearContextCart()` – clears all context
- `updateContextCart()` – renders the "cart" showing selected threads count and list with remove buttons

**2e.** Update `updateAiChatContextLabel()` (line 2728):
- When `contextThreads.length > 0`, show green text: "3 threads, ~15 emails from Stephanie"
- Otherwise keep existing mode-based label

**2f.** Add `multithread` quick prompts to `AI_QUICK_PROMPTS` (line 2686):
- "Summarize all conversations", "How much have we quoted?", "Deception level?", "Build a complete profile", "What patterns emerge?"
- Update `renderQuickPrompts()` to use `multithread` prompts when context builder has threads

**2g.** Modify `sendAiChat()` (line 2756):
- Read provider from `#ai-chat-provider` select
- If `contextThreads.length > 0`, send `thread_ids: [...]` array
- Otherwise send single `thread_id` as before (backward compat)
- Add `provider` to payload

### Step 3: HTML – index.php

**3a.** Replace static model dropdown (lines 538-561) with two selects:
- Provider dropdown (`#ai-chat-provider`) with `onchange="updateAiModelDropdown()"`
- Model dropdown (`#ai-chat-model`) – populated dynamically by JS

**3b.** Insert context builder section between mode tabs (line 570) and messages div (line 574):
- Context summary bar (always visible, clickable to expand) showing "No context loaded" or "3 threads, ~15 emails"
- Collapsible search area with:
  - Contact/email search input + search button
  - Search results container
  - Thread picker container (shows after contact selected)
  - Context cart (selected threads with remove buttons and "Clear All")

## Verification

1. Open https://mssql.danmarcrm.com/dev1/extension2/public/
2. Click the AI chat FAB button (bottom-right)
3. Verify provider dropdown shows all 8 providers, model updates on provider change
4. Click the context summary bar → search area expands
5. Search for a contact email (e.g. "stephanie" or "david.zolanz")
6. Click a contact → threads load with "+ Add" and "Add All" buttons
7. Add individual threads or click "Add All"
8. Verify context label turns green showing thread/email count
9. Verify quick prompts switch to multi-thread prompts
10. Ask a question → verify the AI response has context from ALL selected threads
11. Test backward compat: open a single thread view, click "Ask AI About This Thread" → still works with single thread mode
12. Test different providers/models work correctly


Write

# Plan: AI Context Builder with Multi-Thread Support

## Context

The web app's AI chat panel currently only supports asking AI about **one thread** or **one contact** at a time. The Chrome extension already has a richer "Ask AI" with multi-provider model selection and optional context input. The user wants the web app to support:

- **Searching** for contacts by email/name
- **Selecting multiple threads** (or all threads for a contact) as AI context
- **Choosing AI provider + model** from a dropdown (synced with Chrome extension's list)
- **Asking questions** about the combined context (e.g. "how much to quote?", "deception level?")

Additionally:
- **Token budget management** – don't exceed model context limits
- **Strip quoted/duplicate content** from emails (reuse existing `stripQuotedContent()`)
- **Persist chat sessions** server-side so conversations can be reloaded and continued

## Files to Modify

| File | Changes |
|------|---------|
| [public/api.php](public/api.php) | Accept `thread_ids[]` + `provider` + `session_id`; strip quoted content; token budget; save/load chat sessions |
| [public/assets/app.js](public/assets/app.js) | Extend `aiChat` state; add `PROVIDER_MODELS`; context builder functions; modify `sendAiChat()`; session persistence |
| [public/index.php](public/index.php) | Replace static model dropdown with provider+model selectors; add context builder HTML; add session list UI |
| [src/ThreadAnalyzer.php](src/ThreadAnalyzer.php) | Make `stripQuotedContent()` public static so it can be reused |

## Implementation Steps

### Step 1: Backend – api.php `ai_chat` action (lines 2310-2491)

**1a.** Parse new fields from JSON input:
- `thread_ids` (array of thread IDs) alongside existing `thread_id`
- `provider` (explicit provider code like "claude", "openai")
- `session_id` (optional – to continue an existing chat session)

**1b.** Strip quoted content from email bodies using `ThreadAnalyzer::stripQuotedContent()`:
- Currently the `ai_chat` sends raw `body_text` truncated to 800 chars (line 2347)
- Each email reply typically contains the entire previous conversation quoted below
- A 5-email thread sends ~5x duplicate content because email #5 quotes emails #1-4
- **Fix**: call `stripQuotedContent($body)` before truncating, to keep only the new content in each email
- This alone can reduce context size by 60-80% for long threads

**1c.** Replace single-thread context build (lines 2331-2363) with a multi-thread loop:
- If `thread_ids` array provided, loop and fetch emails for each thread
- Label each as "THREAD 1 OF N", "THREAD 2 OF N"
- Include per-thread analysis summaries if available
- If only `thread_id` (single) is sent, keep existing behavior (backward compat)

**1d.** Add token budget management after building context:
- Define model context limits map: `{'gpt-4o': 128000, 'claude-sonnet-4-5': 200000, ...}`
- Estimate tokens: `strlen($contextBlock) / 3.5` (rough chars-to-tokens ratio)
- Reserve ~30% for AI response + conversation history
- If context exceeds budget: truncate oldest emails first (keep newest), add "[X earlier emails omitted]" note
- Return `context_stats` in response: `{threads, emails, estimated_tokens, truncated}`

**1e.** Fix provider detection (lines 2419-2424):
- Use explicit `provider` field from frontend when available
- Add `deepseek` detection for model name fallback

**1f.** Fix proxy payload model format (line 2431):
- Change `'models' => [$chatModel]` to `'models' => [$chatProvider => $chatModel]`
- Fixes bug where the AI proxy ignores the selected model

### Step 2: Chat Session Persistence (new)

**2a.** Create `ai_chat_sessions` collection/table:
```
{
  session_id: string (uuid),
  created_at: datetime,
  updated_at: datetime,
  title: string (auto-generated from first question),
  mode: string,
  provider: string,
  model: string,
  context_thread_ids: array,
  contact_id: int|null,
  contact_name: string|null,
  messages: [
    { role: 'user'|'assistant', content: string, timestamp: datetime, model: string|null }
  ]
}
```

**2b.** New API actions in `api.php`:
- `ai_chat_save` – saves/updates a session (called after each AI response)
- `ai_chat_sessions` – lists recent sessions (for the session list UI)
- `ai_chat_session` – loads a specific session by `session_id`
- `ai_chat_session_delete` – deletes a session

**2c.** Modify existing `ai_chat` action:
- Accept `session_id` in payload
- After successful AI response, auto-save the message pair to the session
- Return `session_id` in response (create new if not provided)

### Step 3: Frontend JS – app.js

**3a.** Extend `aiChat` state (line 2672):
- Add `contextThreads: []` – array of `{thread_id, subject, email_count, contact_name}`
- Add `contextContact: null` – `{id, name, email}`
- Add `sessionId: null` – current chat session ID
- Add `sessions: []` – list of saved sessions

**3b.** Add `PROVIDER_MODELS` and `PROVIDER_NAMES` constants (after line 2684), synced with Chrome extension (`contentScript.js` lines 18-67):
- 8 providers: Claude, OpenAI, Gemini, Grok, DeepSeek, Qwen, Mistral, Cohere
- Same model IDs as the Chrome extension

**3c.** Add provider dropdown functions:
- `initAiProviderDropdown()` – populates provider select, calls `updateAiModelDropdown()`
- `updateAiModelDropdown()` – populates model select based on selected provider
- Call `initAiProviderDropdown()` from DOMContentLoaded

**3d.** Add context builder functions:
- `toggleContextBuilder()` – show/hide the search area
- `searchContextContacts()` – calls `apiCall('contacts', {search, limit:5})`, renders clickable results
- `selectContextContact(id, name, email)` – calls `apiCall('threads', {contact:id, limit:50})`, shows thread picker with "+ Add" buttons and an "Add All" button
- `addThreadToContext(threadId, subject, emailCount, contactName)` – pushes to `contextThreads[]`
- `removeThreadFromContext(threadId)` – removes from `contextThreads[]`
- `addAllContactThreads()` – adds all available threads at once
- `clearContextCart()` – clears all context
- `updateContextCart()` – renders the "cart" showing selected threads count and list with remove buttons

**3e.** Update `updateAiChatContextLabel()` (line 2728):
- When `contextThreads.length > 0`, show green text: "3 threads, ~15 emails from Stephanie"
- Otherwise keep existing mode-based label

**3f.** Add `multithread` quick prompts to `AI_QUICK_PROMPTS` (line 2686):
- "Summarize all conversations", "How much have we quoted?", "Deception level?", "Build a complete profile", "What patterns emerge?"
- Update `renderQuickPrompts()` to use `multithread` prompts when context builder has threads

**3g.** Modify `sendAiChat()` (line 2756):
- Read provider from `#ai-chat-provider` select
- If `contextThreads.length > 0`, send `thread_ids: [...]` array
- Otherwise send single `thread_id` as before (backward compat)
- Add `provider` and `session_id` to payload
- On response: store returned `session_id`, display `context_stats` if truncation happened

**3h.** Add session management functions:
- `loadChatSessions()` – fetches recent sessions from API, renders in a dropdown/list
- `loadChatSession(sessionId)` – loads a session, restores messages + context + model selection
- `startNewSession()` – clears chat and starts fresh (generates new session_id on first message)
- `deleteChatSession(sessionId)` – deletes from server

### Step 4: HTML – index.php

**4a.** Replace static model dropdown (lines 538-561) with two selects:
- Provider dropdown (`#ai-chat-provider`) with `onchange="updateAiModelDropdown()"`
- Model dropdown (`#ai-chat-model`) – populated dynamically by JS

**4b.** Add session controls to the chat header (line 531 area):
- "New Chat" button
- "History" button that toggles a session list overlay

**4c.** Insert context builder section between mode tabs (line 570) and messages div (line 574):
- Context summary bar (always visible, clickable to expand) showing "No context loaded" or "3 threads, ~15 emails"
- Collapsible search area with:
  - Contact/email search input + search button
  - Search results container
  - Thread picker container (shows after contact selected)
  - Context cart (selected threads with remove buttons and "Clear All")

### Step 5: Make `stripQuotedContent()` reusable – ThreadAnalyzer.php

**5a.** Change `private function stripQuotedContent(string $body): string` (line 1616) to `public static function stripQuotedContent(string $body): string`
- No other changes needed – the method is pure/stateless, no `$this` references

## Key Design Decisions

### Email deduplication strategy
Each email in a thread typically quotes all previous emails in the body. By calling `stripQuotedContent()` we keep only the **new content** per email. Since we send emails in chronological order, AI sees the full conversation without duplication. For a 10-email thread this can reduce context from ~40KB to ~8KB.

### Token budget approach
Rather than hard-failing, we use **smart truncation**:
1. Strip quoted content first (biggest win)
2. Calculate estimated tokens
3. If over budget: reduce per-email body length proportionally
4. If still over: omit oldest emails per thread, noting "[X earlier emails omitted for context budget]"
5. Never truncate the user's question or conversation history

### Chat session persistence
- Sessions are saved server-side (MongoDB `ai_chat_sessions` collection)
- Auto-saved after each AI response (not manually)
- Session title auto-generated from first user message
- Frontend sends `session_id` with each request; server appends to message history
- Loading a session restores: messages, context thread IDs, provider/model, mode
- This replaces the Chrome extension's `localStorage` approach with a proper server-side solution

## Verification

1. Open https://mssql.danmarcrm.com/dev1/extension2/public/
2. Click the AI chat FAB button (bottom-right)
3. Verify provider dropdown shows all 8 providers, model updates on provider change
4. Click the context summary bar → search area expands
5. Search for a contact email (e.g. "stephanie" or "david.zolanz")
6. Click a contact → threads load with "+ Add" and "Add All" buttons
7. Add individual threads or click "Add All"
8. Verify context label turns green showing thread/email count
9. Verify quick prompts switch to multi-thread prompts
10. Ask a question → verify response has context from ALL selected threads
11. Check response includes `context_stats` showing token count and if truncation happened
12. Ask a follow-up question → verify conversation continues with history
13. Refresh the page → click "History" → verify the session appears in the list
14. Click to reload the session → verify messages, context, and model selection are restored
15. Test backward compat: open a single thread view, click "Ask AI About This Thread" → still works
16. Test with a contact that has many threads (e.g. 30+) → verify token budget truncation kicks in and AI still responds coherently


The file /root/.claude/plans/scalable-doodling-peach.md has been updated successfully.

# Plan: AI Context Builder with Multi-Thread Support

## Context

The web app's AI chat panel currently only supports asking AI about **one thread** or **one contact** at a time. The Chrome extension already has a richer "Ask AI" with multi-provider model selection and optional context input. The user wants the web app to support:

- **Searching** for contacts by email/name
- **Selecting multiple threads** (or all threads for a contact) as AI context
- **Choosing AI provider + model** from a dropdown (synced with Chrome extension's list)
- **Asking questions** about the combined context (e.g. "how much to quote?", "deception level?")

Additionally:
- **Token budget management** – don't exceed model context limits
- **Strip quoted/duplicate content** from emails (reuse existing `stripQuotedContent()`)
- **Persist chat sessions** server-side so conversations can be reloaded and continued

## Files to Modify

| File | Changes |
|------|---------|
| [public/api.php](public/api.php) | Accept `thread_ids[]` + `provider` + `session_id`; strip quoted content; token budget; save/load chat sessions |
| [public/assets/app.js](public/assets/app.js) | Extend `aiChat` state; add `PROVIDER_MODELS`; context builder functions; modify `sendAiChat()`; session persistence |
| [public/index.php](public/index.php) | Replace static model dropdown with provider+model selectors; add context builder HTML; add session list UI |
| [src/ThreadAnalyzer.php](src/ThreadAnalyzer.php) | Make `stripQuotedContent()` public static so it can be reused |

## Implementation Steps

### Step 1: Backend – api.php `ai_chat` action (lines 2310-2491)

**1a.** Parse new fields from JSON input:
- `thread_ids` (array of thread IDs) alongside existing `thread_id`
- `provider` (explicit provider code like "claude", "openai")
- `session_id` (optional – to continue an existing chat session)

**1b.** Strip quoted content from email bodies using `ThreadAnalyzer::stripQuotedContent()`:
- Currently the `ai_chat` sends raw `body_text` truncated to 800 chars (line 2347)
- Each email reply typically contains the entire previous conversation quoted below
- A 5-email thread sends ~5x duplicate content because email #5 quotes emails #1-4
- **Fix**: call `stripQuotedContent($body)` before truncating, to keep only the new content in each email
- This alone can reduce context size by 60-80% for long threads

**1c.** Replace single-thread context build (lines 2331-2363) with a multi-thread loop:
- If `thread_ids` array provided, loop and fetch emails for each thread
- Label each as "THREAD 1 OF N", "THREAD 2 OF N"
- Include per-thread analysis summaries if available
- If only `thread_id` (single) is sent, keep existing behavior (backward compat)

**1d.** Add token budget management after building context:
- Define model context limits map: `{'gpt-4o': 128000, 'claude-sonnet-4-5': 200000, ...}`
- Estimate tokens: `strlen($contextBlock) / 3.5` (rough chars-to-tokens ratio)
- Reserve ~30% for AI response + conversation history
- If context exceeds budget: truncate oldest emails first (keep newest), add "[X earlier emails omitted]" note
- Return `context_stats` in response: `{threads, emails, estimated_tokens, truncated}`

**1e.** Fix provider detection (lines 2419-2424):
- Use explicit `provider` field from frontend when available
- Add `deepseek` detection for model name fallback

**1f.** Fix proxy payload model format (line 2431):
- Change `'models' => [$chatModel]` to `'models' => [$chatProvider => $chatModel]`
- Fixes bug where the AI proxy ignores the selected model

### Step 2: Chat Session Persistence (new)

**2a.** Create `ai_chat_sessions` collection/table:
```
{
  session_id: string (uuid),
  created_at: datetime,
  updated_at: datetime,
  title: string (auto-generated from first question),
  mode: string,
  provider: string,
  model: string,
  context_thread_ids: array,
  contact_id: int|null,
  contact_name: string|null,
  messages: [
    { role: 'user'|'assistant', content: string, timestamp: datetime, model: string|null }
  ]
}
```

**2b.** New API actions in `api.php`:
- `ai_chat_save` – saves/updates a session (called after each AI response)
- `ai_chat_sessions` – lists recent sessions (for the session list UI)
- `ai_chat_session` – loads a specific session by `session_id`
- `ai_chat_session_delete` – deletes a session

**2c.** Modify existing `ai_chat` action:
- Accept `session_id` in payload
- After successful AI response, auto-save the message pair to the session
- Return `session_id` in response (create new if not provided)

### Step 3: Frontend JS – app.js

**3a.** Extend `aiChat` state (line 2672):
- Add `contextThreads: []` – array of `{thread_id, subject, email_count, contact_name}`
- Add `contextContact: null` – `{id, name, email}`
- Add `sessionId: null` – current chat session ID
- Add `sessions: []` – list of saved sessions

**3b.** Add `PROVIDER_MODELS` and `PROVIDER_NAMES` constants (after line 2684), synced with Chrome extension (`contentScript.js` lines 18-67):
- 8 providers: Claude, OpenAI, Gemini, Grok, DeepSeek, Qwen, Mistral, Cohere
- Same model IDs as the Chrome extension

**3c.** Add provider dropdown functions:
- `initAiProviderDropdown()` – populates provider select, calls `updateAiModelDropdown()`
- `updateAiModelDropdown()` – populates model select based on selected provider
- Call `initAiProviderDropdown()` from DOMContentLoaded

**3d.** Add context builder functions:
- `toggleContextBuilder()` – show/hide the search area
- `searchContextContacts()` – calls `apiCall('contacts', {search, limit:5})`, renders clickable results
- `selectContextContact(id, name, email)` – calls `apiCall('threads', {contact:id, limit:50})`, shows thread picker with "+ Add" buttons and an "Add All" button
- `addThreadToContext(threadId, subject, emailCount, contactName)` – pushes to `contextThreads[]`
- `removeThreadFromContext(threadId)` – removes from `contextThreads[]`
- `addAllContactThreads()` – adds all available threads at once
- `clearContextCart()` – clears all context
- `updateContextCart()` – renders the "cart" showing selected threads count and list with remove buttons

**3e.** Update `updateAiChatContextLabel()` (line 2728):
- When `contextThreads.length > 0`, show green text: "3 threads, ~15 emails from Stephanie"
- Otherwise keep existing mode-based label

**3f.** Add `multithread` quick prompts to `AI_QUICK_PROMPTS` (line 2686):
- "Summarize all conversations", "How much have we quoted?", "Deception level?", "Build a complete profile", "What patterns emerge?"
- Update `renderQuickPrompts()` to use `multithread` prompts when context builder has threads

**3g.** Modify `sendAiChat()` (line 2756):
- Read provider from `#ai-chat-provider` select
- If `contextThreads.length > 0`, send `thread_ids: [...]` array
- Otherwise send single `thread_id` as before (backward compat)
- Add `provider` and `session_id` to payload
- On response: store returned `session_id`, display `context_stats` if truncation happened

**3h.** Add session management functions:
- `loadChatSessions()` – fetches recent sessions from API, renders in a dropdown/list
- `loadChatSession(sessionId)` – loads a session, restores messages + context + model selection
- `startNewSession()` – clears chat and starts fresh (generates new session_id on first message)
- `deleteChatSession(sessionId)` – deletes from server

### Step 4: HTML – index.php

**4a.** Replace static model dropdown (lines 538-561) with two selects:
- Provider dropdown (`#ai-chat-provider`) with `onchange="updateAiModelDropdown()"`
- Model dropdown (`#ai-chat-model`) – populated dynamically by JS

**4b.** Add session controls to the chat header (line 531 area):
- "New Chat" button
- "History" button that toggles a session list overlay

**4c.** Insert context builder section between mode tabs (line 570) and messages div (line 574):
- Context summary bar (always visible, clickable to expand) showing "No context loaded" or "3 threads, ~15 emails"
- Collapsible search area with:
  - Contact/email search input + search button
  - Search results container
  - Thread picker container (shows after contact selected)
  - Context cart (selected threads with remove buttons and "Clear All")

### Step 5: Make `stripQuotedContent()` reusable – ThreadAnalyzer.php

**5a.** Change `private function stripQuotedContent(string $body): string` (line 1616) to `public static function stripQuotedContent(string $body): string`
- No other changes needed – the method is pure/stateless, no `$this` references

## Key Design Decisions

### Email deduplication strategy
Each email in a thread typically quotes all previous emails in the body. By calling `stripQuotedContent()` we keep only the **new content** per email. Since we send emails in chronological order, AI sees the full conversation without duplication. For a 10-email thread this can reduce context from ~40KB to ~8KB.

### Token budget approach
Rather than hard-failing, we use **smart truncation**:
1. Strip quoted content first (biggest win)
2. Calculate estimated tokens
3. If over budget: reduce per-email body length proportionally
4. If still over: omit oldest emails per thread, noting "[X earlier emails omitted for context budget]"
5. Never truncate the user's question or conversation history

### Chat session persistence
- Sessions are saved server-side (MongoDB `ai_chat_sessions` collection)
- Auto-saved after each AI response (not manually)
- Session title auto-generated from first user message
- Frontend sends `session_id` with each request; server appends to message history
- Loading a session restores: messages, context thread IDs, provider/model, mode
- This replaces the Chrome extension's `localStorage` approach with a proper server-side solution

## Verification

1. Open https://mssql.danmarcrm.com/dev1/extension2/public/
2. Click the AI chat FAB button (bottom-right)
3. Verify provider dropdown shows all 8 providers, model updates on provider change
4. Click the context summary bar → search area expands
5. Search for a contact email (e.g. "stephanie" or "david.zolanz")
6. Click a contact → threads load with "+ Add" and "Add All" buttons
7. Add individual threads or click "Add All"
8. Verify context label turns green showing thread/email count
9. Verify quick prompts switch to multi-thread prompts
10. Ask a question → verify response has context from ALL selected threads
11. Check response includes `context_stats` showing token count and if truncation happened
12. Ask a follow-up question → verify conversation continues with history
13. Refresh the page → click "History" → verify the session appears in the list
14. Click to reload the session → verify messages, context, and model selection are restored
15. Test backward compat: open a single thread view, click "Ask AI About This Thread" → still works
16. Test with a contact that has many threads (e.g. 30+) → verify token budget truncation kicks in and AI still responds coherently

+- **Token budget management** – don't exceed model context limits

+- **Strip quoted/duplicate content** from emails (reuse existing `stripQuotedContent()`)

+- **Persist chat sessions** server-side so conversations can be reloaded and continued

-| [public/index.php](public/index.php) | Replace static model dropdown with provider+model selectors; add context builder HTML section |

-| [public/assets/app.js](public/assets/app.js) | Extend `aiChat` state; add `PROVIDER_MODELS`; add context builder functions; modify `sendAiChat()` for multi-thread; update `renderQuickPrompts()` |

-| [public/api.php](public/api.php) | Accept `thread_ids[]` + `provider`; build multi-thread context; fix model override format for proxy |

+| [public/api.php](public/api.php) | Accept `thread_ids[]` + `provider` + `session_id`; strip quoted content; token budget; save/load chat sessions |

+| [public/assets/app.js](public/assets/app.js) | Extend `aiChat` state; add `PROVIDER_MODELS`; context builder functions; modify `sendAiChat()`; session persistence |

+| [public/index.php](public/index.php) | Replace static model dropdown with provider+model selectors; add context builder HTML; add session list UI |

+| [src/ThreadAnalyzer.php](src/ThreadAnalyzer.php) | Make `stripQuotedContent()` public static so it can be reused |

**1a.** Parse new fields from JSON input:

- `thread_ids` (array of thread IDs) alongside existing `thread_id`

- `provider` (explicit provider code like "claude", "openai")

+- `session_id` (optional – to continue an existing chat session)

-**1b.** Replace single-thread context build (lines 2331-2363) with a loop over all `thread_ids`:

-- If `thread_ids` is provided, loop and fetch emails for each thread

-- Label each as "THREAD 1 OF N", "THREAD 2 OF N" etc.

-- Dynamically reduce per-email body length based on total email count: `min(800, max(200, 60000 / totalEmailCount))`

-- Prepend summary line: "MULTI-THREAD CONTEXT: X total emails across Y threads"

+**1b.** Strip quoted content from email bodies using `ThreadAnalyzer::stripQuotedContent()`:

+- Currently the `ai_chat` sends raw `body_text` truncated to 800 chars (line 2347)

+- Each email reply typically contains the entire previous conversation quoted below

+- A 5-email thread sends ~5x duplicate content because email #5 quotes emails #1-4

+- **Fix**: call `stripQuotedContent($body)` before truncating, to keep only the new content in each email

+- This alone can reduce context size by 60-80% for long threads

+**1c.** Replace single-thread context build (lines 2331-2363) with a multi-thread loop:

+- If `thread_ids` array provided, loop and fetch emails for each thread

+- Label each as "THREAD 1 OF N", "THREAD 2 OF N"

- Include per-thread analysis summaries if available

-- If only `thread_id` (single) is sent, keep existing behavior

+- If only `thread_id` (single) is sent, keep existing behavior (backward compat)

-**1c.** Fix provider detection (lines 2419-2424):

+**1d.** Add token budget management after building context:

+- Define model context limits map: `{'gpt-4o': 128000, 'claude-sonnet-4-5': 200000, ...}`

+- Estimate tokens: `strlen($contextBlock) / 3.5` (rough chars-to-tokens ratio)

+- Reserve ~30% for AI response + conversation history

+- If context exceeds budget: truncate oldest emails first (keep newest), add "[X earlier emails omitted]" note

+- Return `context_stats` in response: `{threads, emails, estimated_tokens, truncated}`

+**1e.** Fix provider detection (lines 2419-2424):

- Use explicit `provider` field from frontend when available

- Add `deepseek` detection for model name fallback

-**1d.** Fix proxy payload model format (line 2431):

+**1f.** Fix proxy payload model format (line 2431):

-- This fixes a bug where the AI proxy ignores the selected model

+- Fixes bug where the AI proxy ignores the selected model

+### Step 2: Chat Session Persistence (new)

-**2a.** Extend `aiChat` state (line 2672):

+**2a.** Create `ai_chat_sessions` collection/table:

+  title: string (auto-generated from first question),

+    { role: 'user'|'assistant', content: string, timestamp: datetime, model: string|null }

+- `ai_chat_save` – saves/updates a session (called after each AI response)

+- `ai_chat_sessions` – lists recent sessions (for the session list UI)

+- `ai_chat_session` – loads a specific session by `session_id`

+- `ai_chat_session_delete` – deletes a session

+**2c.** Modify existing `ai_chat` action:

+- After successful AI response, auto-save the message pair to the session

+- Return `session_id` in response (create new if not provided)

+**3a.** Extend `aiChat` state (line 2672):

- Add `contextContact: null` – `{id, name, email}`

+- Add `sessionId: null` – current chat session ID

+- Add `sessions: []` – list of saved sessions

-**2b.** Add `PROVIDER_MODELS` and `PROVIDER_NAMES` constants (after line 2684), synced with Chrome extension's list from `contentScript.js` lines 18-67:

+**3b.** Add `PROVIDER_MODELS` and `PROVIDER_NAMES` constants (after line 2684), synced with Chrome extension (`contentScript.js` lines 18-67):

- Same model IDs as the Chrome extension

-**2c.** Add provider dropdown functions:

+**3c.** Add provider dropdown functions:

- Call `initAiProviderDropdown()` from DOMContentLoaded

- `toggleContextBuilder()` – show/hide the search area

- `clearContextCart()` – clears all context

-**2e.** Update `updateAiChatContextLabel()` (line 2728):

+**3e.** Update `updateAiChatContextLabel()` (line 2728):

- Otherwise keep existing mode-based label

-**2f.** Add `multithread` quick prompts to `AI_QUICK_PROMPTS` (line 2686):

+**3f.** Add `multithread` quick prompts to `AI_QUICK_PROMPTS` (line 2686):

-**2g.** Modify `sendAiChat()` (line 2756):

+**3g.** Modify `sendAiChat()` (line 2756):

- Read provider from `#ai-chat-provider` select

- If `contextThreads.length > 0`, send `thread_ids: [...]` array

- Otherwise send single `thread_id` as before (backward compat)

+- Add `provider` and `session_id` to payload

+- On response: store returned `session_id`, display `context_stats` if truncation happened

+**3h.** Add session management functions:

+- `loadChatSessions()` – fetches recent sessions from API, renders in a dropdown/list

+- `loadChatSession(sessionId)` – loads a session, restores messages + context + model selection

+- `startNewSession()` – clears chat and starts fresh (generates new session_id on first message)

+- `deleteChatSession(sessionId)` – deletes from server

-**3a.** Replace static model dropdown (lines 538-561) with two selects:

+**4a.** Replace static model dropdown (lines 538-561) with two selects:

- Model dropdown (`#ai-chat-model`) – populated dynamically by JS

-**3b.** Insert context builder section between mode tabs (line 570) and messages div (line 574):

+**4b.** Add session controls to the chat header (line 531 area):

+- "History" button that toggles a session list overlay

+**4c.** Insert context builder section between mode tabs (line 570) and messages div (line 574):

- Contact/email search input + search button

- Thread picker container (shows after contact selected)

- Context cart (selected threads with remove buttons and "Clear All")

+### Step 5: Make `stripQuotedContent()` reusable – ThreadAnalyzer.php

+**5a.** Change `private function stripQuotedContent(string $body): string` (line 1616) to `public static function stripQuotedContent(string $body): string`

+- No other changes needed – the method is pure/stateless, no `$this` references

+Each email in a thread typically quotes all previous emails in the body. By calling `stripQuotedContent()` we keep only the **new content** per email. Since we send emails in chronological order, AI sees the full conversation without duplication. For a 10-email thread this can reduce context from ~40KB to ~8KB.

+Rather than hard-failing, we use **smart truncation**:

+1. Strip quoted content first (biggest win)

+3. If over budget: reduce per-email body length proportionally

+4. If still over: omit oldest emails per thread, noting "[X earlier emails omitted for context budget]"

+5. Never truncate the user's question or conversation history

+- Sessions are saved server-side (MongoDB `ai_chat_sessions` collection)

+- Auto-saved after each AI response (not manually)

+- Session title auto-generated from first user message

+- Frontend sends `session_id` with each request; server appends to message history

+- Loading a session restores: messages, context thread IDs, provider/model, mode

+- This replaces the Chrome extension's `localStorage` approach with a proper server-side solution

7. Add individual threads or click "Add All"

8. Verify context label turns green showing thread/email count

9. Verify quick prompts switch to multi-thread prompts

-10. Ask a question → verify the AI response has context from ALL selected threads

-11. Test backward compat: open a single thread view, click "Ask AI About This Thread" → still works with single thread mode

-12. Test different providers/models work correctly

+10. Ask a question → verify response has context from ALL selected threads

+11. Check response includes `context_stats` showing token count and if truncation happened

+12. Ask a follow-up question → verify conversation continues with history

+13. Refresh the page → click "History" → verify the session appears in the list

+14. Click to reload the session → verify messages, context, and model selection are restored

+15. Test backward compat: open a single thread view, click "Ask AI About This Thread" → still works

+16. Test with a contact that has many threads (e.g. 30+) → verify token budget truncation kicks in and AI still responds coherently

# Plan: AI Context Builder with Multi-Thread Support

## Context

The web app's AI chat panel currently only supports asking AI about **one thread** or **one contact** at a time. The Chrome extension already has a richer "Ask AI" with multi-provider model selection and optional context input. The user wants the web app to support:

- **Searching** for contacts by email/name
- **Selecting multiple threads** (or all threads for a contact) as AI context
- **Choosing AI provider + model** from a dropdown (synced with Chrome extension's list)
- **Asking questions** about the combined context (e.g. "how much to quote?", "deception level?")

## Files to Modify

| File | Changes |
|------|---------|
| [public/index.php](public/index.php) | Replace static model dropdown with provider+model selectors; add context builder HTML section |
| [public/assets/app.js](public/assets/app.js) | Extend `aiChat` state; add `PROVIDER_MODELS`; add context builder functions; modify `sendAiChat()` for multi-thread; update `renderQuickPrompts()` |
| [public/api.php](public/api.php) | Accept `thread_ids[]` + `provider`; build multi-thread context; fix model override format for proxy |

## Implementation Steps

### Step 1: Backend – api.php `ai_chat` action (lines 2310-2491)

**1a.** Parse new fields from JSON input:
- `thread_ids` (array of thread IDs) alongside existing `thread_id`
- `provider` (explicit provider code like "claude", "openai")

**1b.** Replace single-thread context build (lines 2331-2363) with a loop over all `thread_ids`:
- If `thread_ids` is provided, loop and fetch emails for each thread
- Label each as "THREAD 1 OF N", "THREAD 2 OF N" etc.
- Dynamically reduce per-email body length based on total email count: `min(800, max(200, 60000 / totalEmailCount))`
- Prepend summary line: "MULTI-THREAD CONTEXT: X total emails across Y threads"
- Include per-thread analysis summaries if available
- If only `thread_id` (single) is sent, keep existing behavior

**1c.** Fix provider detection (lines 2419-2424):
- Use explicit `provider` field from frontend when available
- Add `deepseek` detection for model name fallback

**1d.** Fix proxy payload model format (line 2431):
- Change `'models' => [$chatModel]` to `'models' => [$chatProvider => $chatModel]`
- This fixes a bug where the AI proxy ignores the selected model

### Step 2: Frontend JS – app.js

**2a.** Extend `aiChat` state (line 2672):
- Add `contextThreads: []` – array of `{thread_id, subject, email_count, contact_name}`
- Add `contextContact: null` – `{id, name, email}`

**2b.** Add `PROVIDER_MODELS` and `PROVIDER_NAMES` constants (after line 2684), synced with Chrome extension's list from `contentScript.js` lines 18-67:
- 8 providers: Claude, OpenAI, Gemini, Grok, DeepSeek, Qwen, Mistral, Cohere
- Same model IDs as the Chrome extension

**2c.** Add provider dropdown functions:
- `initAiProviderDropdown()` – populates provider select, calls `updateAiModelDropdown()`
- `updateAiModelDropdown()` – populates model select based on selected provider
- Call `initAiProviderDropdown()` from DOMContentLoaded

**2d.** Add context builder functions:
- `toggleContextBuilder()` – show/hide the search area
- `searchContextContacts()` – calls `apiCall('contacts', {search, limit:5})`, renders clickable results
- `selectContextContact(id, name, email)` – calls `apiCall('threads', {contact:id, limit:50})`, shows thread picker with "+ Add" buttons and an "Add All" button
- `addThreadToContext(threadId, subject, emailCount, contactName)` – pushes to `contextThreads[]`
- `removeThreadFromContext(threadId)` – removes from `contextThreads[]`
- `addAllContactThreads()` – adds all available threads at once
- `clearContextCart()` – clears all context
- `updateContextCart()` – renders the "cart" showing selected threads count and list with remove buttons

**2e.** Update `updateAiChatContextLabel()` (line 2728):
- When `contextThreads.length > 0`, show green text: "3 threads, ~15 emails from Stephanie"
- Otherwise keep existing mode-based label

**2f.** Add `multithread` quick prompts to `AI_QUICK_PROMPTS` (line 2686):
- "Summarize all conversations", "How much have we quoted?", "Deception level?", "Build a complete profile", "What patterns emerge?"
- Update `renderQuickPrompts()` to use `multithread` prompts when context builder has threads

**2g.** Modify `sendAiChat()` (line 2756):
- Read provider from `#ai-chat-provider` select
- If `contextThreads.length > 0`, send `thread_ids: [...]` array
- Otherwise send single `thread_id` as before (backward compat)
- Add `provider` to payload

### Step 3: HTML – index.php

**3a.** Replace static model dropdown (lines 538-561) with two selects:
- Provider dropdown (`#ai-chat-provider`) with `onchange="updateAiModelDropdown()"`
- Model dropdown (`#ai-chat-model`) – populated dynamically by JS

**3b.** Insert context builder section between mode tabs (line 570) and messages div (line 574):
- Context summary bar (always visible, clickable to expand) showing "No context loaded" or "3 threads, ~15 emails"
- Collapsible search area with:
  - Contact/email search input + search button
  - Search results container
  - Thread picker container (shows after contact selected)
  - Context cart (selected threads with remove buttons and "Clear All")

## Verification

1. Open https://mssql.danmarcrm.com/dev1/extension2/public/
2. Click the AI chat FAB button (bottom-right)
3. Verify provider dropdown shows all 8 providers, model updates on provider change
4. Click the context summary bar → search area expands
5. Search for a contact email (e.g. "stephanie" or "david.zolanz")
6. Click a contact → threads load with "+ Add" and "Add All" buttons
7. Add individual threads or click "Add All"
8. Verify context label turns green showing thread/email count
9. Verify quick prompts switch to multi-thread prompts
10. Ask a question → verify the AI response has context from ALL selected threads
11. Test backward compat: open a single thread view, click "Ask AI About This Thread" → still works with single thread mode
12. Test different providers/models work correctly


ExitPlanMode

# Plan: AI Context Builder with Multi-Thread Support

## Context

The web app's AI chat panel currently only supports asking AI about **one thread** or **one contact** at a time. The Chrome extension already has a richer "Ask AI" with multi-provider model selection and optional context input. The user wants the web app to support:

- **Searching** for contacts by email/name
- **Selecting multiple threads** (or all threads for a contact) as AI context
- **Choosing AI provider + model** from a dropdown (synced with Chrome extension's list)
- **Asking questions** about the combined context (e.g. "how much to quote?", "deception level?")

Additionally:
- **Token budget management** – don't exceed model context limits
- **Strip quoted/duplicate content** from emails (reuse existing `stripQuotedContent()`)
- **Persist chat sessions** server-side so conversations can be reloaded and continued

## Files to Modify

| File | Changes |
|------|---------|
| [public/api.php](public/api.php) | Accept `thread_ids[]` + `provider` + `session_id`; strip quoted content; token budget; save/load chat sessions |
| [public/assets/app.js](public/assets/app.js) | Extend `aiChat` state; add `PROVIDER_MODELS`; context builder functions; modify `sendAiChat()`; session persistence |
| [public/index.php](public/index.php) | Replace static model dropdown with provider+model selectors; add context builder HTML; add session list UI |
| [src/ThreadAnalyzer.php](src/ThreadAnalyzer.php) | Make `stripQuotedContent()` public static so it can be reused |

## Implementation Steps

### Step 1: Backend – api.php `ai_chat` action (lines 2310-2491)

**1a.** Parse new fields from JSON input:
- `thread_ids` (array of thread IDs) alongside existing `thread_id`
- `provider` (explicit provider code like "claude", "openai")
- `session_id` (optional – to continue an existing chat session)

**1b.** Strip quoted content from email bodies using `ThreadAnalyzer::stripQuotedContent()`:
- Currently the `ai_chat` sends raw `body_text` truncated to 800 chars (line 2347)
- Each email reply typically contains the entire previous conversation quoted below
- A 5-email thread sends ~5x duplicate content because email #5 quotes emails #1-4
- **Fix**: call `stripQuotedContent($body)` before truncating, to keep only the new content in each email
- This alone can reduce context size by 60-80% for long threads

**1c.** Replace single-thread context build (lines 2331-2363) with a multi-thread loop:
- If `thread_ids` array provided, loop and fetch emails for each thread
- Label each as "THREAD 1 OF N", "THREAD 2 OF N"
- Include per-thread analysis summaries if available
- If only `thread_id` (single) is sent, keep existing behavior (backward compat)

**1d.** Add token budget management after building context:
- Define model context limits map: `{'gpt-4o': 128000, 'claude-sonnet-4-5': 200000, ...}`
- Estimate tokens: `strlen($contextBlock) / 3.5` (rough chars-to-tokens ratio)
- Reserve ~30% for AI response + conversation history
- If context exceeds budget: truncate oldest emails first (keep newest), add "[X earlier emails omitted]" note
- Return `context_stats` in response: `{threads, emails, estimated_tokens, truncated}`

**1e.** Fix provider detection (lines 2419-2424):
- Use explicit `provider` field from frontend when available
- Add `deepseek` detection for model name fallback

**1f.** Fix proxy payload model format (line 2431):
- Change `'models' => [$chatModel]` to `'models' => [$chatProvider => $chatModel]`
- Fixes bug where the AI proxy ignores the selected model

### Step 2: Chat Session Persistence (new)

**2a.** Create `ai_chat_sessions` collection/table:
```
{
  session_id: string (uuid),
  created_at: datetime,
  updated_at: datetime,
  title: string (auto-generated from first question),
  mode: string,
  provider: string,
  model: string,
  context_thread_ids: array,
  contact_id: int|null,
  contact_name: string|null,
  messages: [
    { role: 'user'|'assistant', content: string, timestamp: datetime, model: string|null }
  ]
}
```

**2b.** New API actions in `api.php`:
- `ai_chat_save` – saves/updates a session (called after each AI response)
- `ai_chat_sessions` – lists recent sessions (for the session list UI)
- `ai_chat_session` – loads a specific session by `session_id`
- `ai_chat_session_delete` – deletes a session

**2c.** Modify existing `ai_chat` action:
- Accept `session_id` in payload
- After successful AI response, auto-save the message pair to the session
- Return `session_id` in response (create new if not provided)

### Step 3: Frontend JS – app.js

**3a.** Extend `aiChat` state (line 2672):
- Add `contextThreads: []` – array of `{thread_id, subject, email_count, contact_name}`
- Add `contextContact: null` – `{id, name, email}`
- Add `sessionId: null` – current chat session ID
- Add `sessions: []` – list of saved sessions

**3b.** Add `PROVIDER_MODELS` and `PROVIDER_NAMES` constants (after line 2684), synced with Chrome extension (`contentScript.js` lines 18-67):
- 8 providers: Claude, OpenAI, Gemini, Grok, DeepSeek, Qwen, Mistral, Cohere
- Same model IDs as the Chrome extension

**3c.** Add provider dropdown functions:
- `initAiProviderDropdown()` – populates provider select, calls `updateAiModelDropdown()`
- `updateAiModelDropdown()` – populates model select based on selected provider
- Call `initAiProviderDropdown()` from DOMContentLoaded

**3d.** Add context builder functions:
- `toggleContextBuilder()` – show/hide the search area
- `searchContextContacts()` – calls `apiCall('contacts', {search, limit:5})`, renders clickable results
- `selectContextContact(id, name, email)` – calls `apiCall('threads', {contact:id, limit:50})`, shows thread picker with "+ Add" buttons and an "Add All" button
- `addThreadToContext(threadId, subject, emailCount, contactName)` – pushes to `contextThreads[]`
- `removeThreadFromContext(threadId)` – removes from `contextThreads[]`
- `addAllContactThreads()` – adds all available threads at once
- `clearContextCart()` – clears all context
- `updateContextCart()` – renders the "cart" showing selected threads count and list with remove buttons

**3e.** Update `updateAiChatContextLabel()` (line 2728):
- When `contextThreads.length > 0`, show green text: "3 threads, ~15 emails from Stephanie"
- Otherwise keep existing mode-based label

**3f.** Add `multithread` quick prompts to `AI_QUICK_PROMPTS` (line 2686):
- "Summarize all conversations", "How much have we quoted?", "Deception level?", "Build a complete profile", "What patterns emerge?"
- Update `renderQuickPrompts()` to use `multithread` prompts when context builder has threads

**3g.** Modify `sendAiChat()` (line 2756):
- Read provider from `#ai-chat-provider` select
- If `contextThreads.length > 0`, send `thread_ids: [...]` array
- Otherwise send single `thread_id` as before (backward compat)
- Add `provider` and `session_id` to payload
- On response: store returned `session_id`, display `context_stats` if truncation happened

**3h.** Add session management functions:
- `loadChatSessions()` – fetches recent sessions from API, renders in a dropdown/list
- `loadChatSession(sessionId)` – loads a session, restores messages + context + model selection
- `startNewSession()` – clears chat and starts fresh (generates new session_id on first message)
- `deleteChatSession(sessionId)` – deletes from server

### Step 4: HTML – index.php

**4a.** Replace static model dropdown (lines 538-561) with two selects:
- Provider dropdown (`#ai-chat-provider`) with `onchange="updateAiModelDropdown()"`
- Model dropdown (`#ai-chat-model`) – populated dynamically by JS

**4b.** Add session controls to the chat header (line 531 area):
- "New Chat" button
- "History" button that toggles a session list overlay

**4c.** Insert context builder section between mode tabs (line 570) and messages div (line 574):
- Context summary bar (always visible, clickable to expand) showing "No context loaded" or "3 threads, ~15 emails"
- Collapsible search area with:
  - Contact/email search input + search button
  - Search results container
  - Thread picker container (shows after contact selected)
  - Context cart (selected threads with remove buttons and "Clear All")

### Step 5: Make `stripQuotedContent()` reusable – ThreadAnalyzer.php

**5a.** Change `private function stripQuotedContent(string $body): string` (line 1616) to `public static function stripQuotedContent(string $body): string`
- No other changes needed – the method is pure/stateless, no `$this` references

## Key Design Decisions

### Email deduplication strategy
Each email in a thread typically quotes all previous emails in the body. By calling `stripQuotedContent()` we keep only the **new content** per email. Since we send emails in chronological order, AI sees the full conversation without duplication. For a 10-email thread this can reduce context from ~40KB to ~8KB.

### Token budget approach
Rather than hard-failing, we use **smart truncation**:
1. Strip quoted content first (biggest win)
2. Calculate estimated tokens
3. If over budget: reduce per-email body length proportionally
4. If still over: omit oldest emails per thread, noting "[X earlier emails omitted for context budget]"
5. Never truncate the user's question or conversation history

### Chat session persistence
- Sessions are saved server-side (MongoDB `ai_chat_sessions` collection)
- Auto-saved after each AI response (not manually)
- Session title auto-generated from first user message
- Frontend sends `session_id` with each request; server appends to message history
- Loading a session restores: messages, context thread IDs, provider/model, mode
- This replaces the Chrome extension's `localStorage` approach with a proper server-side solution

## Verification

1. Open https://mssql.danmarcrm.com/dev1/extension2/public/
2. Click the AI chat FAB button (bottom-right)
3. Verify provider dropdown shows all 8 providers, model updates on provider change
4. Click the context summary bar → search area expands
5. Search for a contact email (e.g. "stephanie" or "david.zolanz")
6. Click a contact → threads load with "+ Add" and "Add All" buttons
7. Add individual threads or click "Add All"
8. Verify context label turns green showing thread/email count
9. Verify quick prompts switch to multi-thread prompts
10. Ask a question → verify response has context from ALL selected threads
11. Check response includes `context_stats` showing token count and if truncation happened
12. Ask a follow-up question → verify conversation continues with history
13. Refresh the page → click "History" → verify the session appears in the list
14. Click to reload the session → verify messages, context, and model selection are restored
15. Test backward compat: open a single thread view, click "Ask AI About This Thread" → still works
16. Test with a contact that has many threads (e.g. 30+) → verify token budget truncation kicks in and AI still responds coherently


The user doesn't want to proceed with this tool use. The tool use was rejected (eg. if it was a file edit, the new_string was NOT written to the file). STOP what you are doing and wait for the user to tell you how to proceed.


[Request interrupted by user for tool use]


for the above project should we create agents? if so what you recomend? would memory help?


The user is asking about architectural decisions for the AI chat feature - specifically whether to use an "agent" pattern and whether "memory" (like the Chrome extension's saved context) would be useful. This is still in plan mode, so I should discuss this and potentially update the plan.

Let me think about what "agents" and "memory" mean in this context:

**Agents**: In the context of AI-powered applications, agents are specialized AI configurations with specific system prompts, tools, and behaviors. For example:
- A "Quote Analyst" agent that specializes in analyzing pricing/quotes across threads
- A "Deception Detector" agent that focuses on inconsistencies and red flags
- A "Contact Profiler" agent that builds psychological profiles

- A "General Assistant" for open-ended questions

The current system has "modes" (thread, contact, quote, reply, general) which are essentially lightweight agents - they just swap the system prompt. Making these into proper agents could add persistent instructions, agent-specific memory, and different model preferences per agent depending on the task complexity.

**Memory**: The Chrome extension has `savedAIContext` which lets users save important Q&A pairs and notes that are always included in future AI requests. Server-side memory could work similarly - storing cross-session facts about contacts and agent-specific context that persists across conversations.


Great question. Let me think through this in the context of what you're building.

## Agents

The current system already has lightweight "agents" via the **mode tabs** (General, Thread, Contact, Quote, Reply) — each just swaps the system prompt. But proper agents would add real value here:

| Agent | Purpose | Why it helps |
|-------|---------|-------------|
| **Quote Analyst** | Analyzes pricing across threads, tracks what was quoted vs accepted | Could use a stronger model (Opus) and specialized instructions for financial pattern detection |
| **Deception Detector** | Focuses on inconsistencies, tone shifts, contradictions | Needs detailed system prompt with deception indicators — different from general chat |
| **Contact Profiler** | Builds psychological/behavioral profiles | Accumulates insights over multiple conversations about the same contact |
| **Email Composer** | Writes replies with awareness of negotiation position | Needs different temperature/style than analysis agents |

**My recommendation**: Yes, but keep it simple — agents as **named presets** (system prompt + preferred model + memory scope), not a complex framework. They're stored in the `ai_chat_sessions` collection alongside the session that uses them.

## Memory

This is where it gets really powerful. Two types:

**1. Contact Memory** (high value) — Facts learned about a contact that persist across sessions:
- "Stephanie always negotiates 20-30% below initial quote"
- "David contradicted himself about budget on Dec 15 vs Jan 3"
- Auto-extracted by AI after each analysis, or manually pinned by user

**2. Agent/Session Memory** (the Chrome extension's `savedAIContext` approach) — Key findings from the current session that carry forward:
- Previous analysis conclusions
- User-pinned insights
- Avoids re-analyzing the same data

**My recommendation**: Yes to both, stored in MongoDB. Contact memory is the biggest win — it means the AI gets smarter about each contact over time without re-reading all their emails every time.