Here is the complete implementation plan, organized by priority and grouped by file to minimize edits.

---

# Email Intelligence System -- Implementation Plan (10 Tasks)

## Execution Order (Quick Wins First)

The tasks are reordered to complete the simplest changes first, building momentum before tackling the larger features.

---

## PHASE 1: Trivial Changes (Tasks 9, 10)

### Task 9: Hide Products Tab

**Effort:** 1 minute. Single line change.

**File: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/index.php`** (line 167)

Change:
```php
<button class="tab-btn" data-tab="products">🛒 Products</button>
```
To:
```php
<!-- <button class="tab-btn" data-tab="products">🛒 Products</button> -->
```

Leave the `<div id="tab-products">` section intact (it just will not be reachable). No other files change.

---

### Task 10: Update AI Model Versions

**Effort:** 10 minutes. Three files.

**File 1: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/aiapi/api_opus/config/api_keys.php`** (lines 73-88)

In the `claude` provider config, add the new models to the `models` array and update `default_model`:

```php
'claude' => [
    'enabled' => true,
    'name' => 'Claude (Anthropic)',
    'models' => [
        'claude-opus-4-6' => 'Claude Opus 4.6 (Best)',
        'claude-sonnet-4-6' => 'Claude Sonnet 4.6',
        'claude-opus-4-5-20251101' => 'Claude Opus 4.5',
        'claude-sonnet-4-5-20250929' => 'Claude Sonnet 4.5',
        'claude-haiku-4-5-20251001' => 'Claude Haiku 4.5 (Fast)',
    ],
    'default_model' => 'claude-sonnet-4-6',
    'pricing' => [
        'claude-opus-4-6' => ['input' => 5.00, 'output' => 25.00],
        'claude-sonnet-4-6' => ['input' => 3.00, 'output' => 15.00],
        'claude-opus-4-5-20251101' => ['input' => 5.00, 'output' => 25.00],
        'claude-sonnet-4-5-20250929' => ['input' => 3.00, 'output' => 15.00],
        'claude-haiku-4-5-20251001' => ['input' => 1.00, 'output' => 5.00],
    ],
],
```

**File 2: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/thread_view.php`** (around line 1512)

Add these entries to the `MODEL_LIMITS` JavaScript constant:

```javascript
'claude-opus-4-6': 200000,
'claude-sonnet-4-6': 200000,
```

**File 3: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/config/config.php`** (line 182)

Update:
```php
define('CLAUDE_MODEL', 'claude-sonnet-4-6');
```

**Also check:** `/var/www/mssql.danmarcrm.com/dev1/extension2/aiapi/api_opus/config/api_keys.php` -- apply same claude model additions if that instance is still in use.

---

## PHASE 2: Delete Functionality (Tasks 1, 6)

### Task 1: Delete Contacts, Threads, and AI Responses

**Effort:** 30-45 minutes. Four files.

#### File 1: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/api.php`

Add three new cases in the switch statement (after existing `ai_contact_memory_delete` case around line 5601):

**Case `delete_contact`:**
```php
case 'delete_contact':
    if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
        jsonError('POST required', 405);
    }
    $input = json_decode(file_get_contents('php://input'), true);
    $contactId = (int)($input['contact_id'] ?? 0);
    if (!$contactId) jsonError('contact_id required');

    $adapter = QueryAdapter::getInstance();
    $result = $adapter->deleteContact($contactId);
    jsonResponse(['success' => true, 'deleted' => $result]);
    break;
```

**Case `delete_thread`:**
```php
case 'delete_thread':
    if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
        jsonError('POST required', 405);
    }
    $input = json_decode(file_get_contents('php://input'), true);
    $threadId = $input['thread_id'] ?? '';
    if (!$threadId) jsonError('thread_id required');
    $threadId = resolveThreadId($threadId);

    $adapter = QueryAdapter::getInstance();
    $result = $adapter->deleteThread($threadId);
    jsonResponse(['success' => true, 'deleted' => $result]);
    break;
```

**Case `delete_ai_response`:**
```php
case 'delete_ai_response':
    if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
        jsonError('POST required', 405);
    }
    $input = json_decode(file_get_contents('php://input'), true);
    $responseId = $input['response_id'] ?? '';
    if (!$responseId) jsonError('response_id required');

    $mongo = \EmailIntelligence\MongoDatabase::getInstance();
    $deleted = $mongo->delete('ai_query_responses', ['_id' => new \MongoDB\BSON\ObjectId($responseId)]);
    jsonResponse(['success' => true, 'deleted' => $deleted]);
    break;
```

#### File 2: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/src/QueryAdapter.php`

Add two new public methods:

**`deleteContact(int $contactId): array`**
```php
public function deleteContact(int $contactId): array
{
    $mongo = MongoDatabase::getInstance();
    $result = [];

    // Delete all emails for this contact
    $result['emails'] = $mongo->delete('emails', ['contact_id' => $contactId]);

    // Delete thread_analysis for threads owned by this contact
    // First find thread_ids for this contact's emails
    $emails = $mongo->findAll('emails', ['contact_id' => $contactId], ['projection' => ['thread_id' => 1]]);
    $threadIds = array_unique(array_column($emails, 'thread_id'));
    if (!empty($threadIds)) {
        $result['thread_analysis'] = $mongo->delete('thread_analysis', ['thread_id' => ['$in' => $threadIds]]);
    }

    // Delete contact profile
    $result['contact_profiles'] = $mongo->delete('contact_profiles', ['contact_id' => $contactId]);

    // Delete AI contact memories
    $result['ai_contact_memory'] = $mongo->delete('ai_contact_memory', ['contact_id' => $contactId]);

    // Delete ai_query_responses referencing this contact
    $result['ai_query_responses'] = $mongo->delete('ai_query_responses', ['contact_id' => $contactId]);

    // Delete the contact itself
    $result['contacts'] = $mongo->delete('contacts', ['id' => $contactId]);

    return $result;
}
```

**Note on `deleteContact`:** The emails query above retrieves emails BEFORE deleting them, because we need the thread_ids. The `$mongo->delete('emails', ...)` call uses `deleteMany` internally (confirmed from MongoDatabase.php line 333). However, we must be careful: if a thread has emails from multiple contacts, deleting thread_analysis for the thread could remove analysis shared with other contacts. A safer approach is to only delete thread_analysis entries whose threads have NO remaining emails from OTHER contacts after the deletion. For simplicity in V1, we delete the thread_analysis anyway since these threads are going away. If partial thread deletion is needed, refine this later.

**Alternative approach (safer):** Collect thread_ids first, delete emails, then check if any emails remain for those thread_ids, and only delete thread_analysis for threads with zero remaining emails.

**`deleteThread(string $threadId): array`**
```php
public function deleteThread(string $threadId): array
{
    $mongo = MongoDatabase::getInstance();
    $result = [];

    // Delete all emails in this thread
    $result['emails'] = $mongo->delete('emails', ['thread_id' => $threadId]);

    // Delete thread analysis
    $result['thread_analysis'] = $mongo->delete('thread_analysis', ['thread_id' => $threadId]);

    // Delete AI query responses referencing this thread
    $result['ai_query_responses'] = $mongo->delete('ai_query_responses', ['thread_ids' => $threadId]);

    return $result;
}
```

#### File 3: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/assets/app.js`

**In `loadContacts()` function** (around line 217-221), add a Delete button to the Actions `<td>`:

Currently it has only the Enrich button. Add after it:
```javascript
<button class="btn btn-sm btn-danger" onclick="event.stopPropagation(); deleteContact(${contact.id}, '${escapeHtml(contact.name || contact.email)}')" title="Delete contact and all data">
    🗑
</button>
```

**In `loadEmails()` function** (around line 686), add a Delete button column. First add `<th>Actions</th>` to the thread table header (after "Last Activity"). Then add a new `<td>` in each row:
```javascript
<td onclick="event.stopPropagation();">
    <button class="btn btn-sm btn-danger" onclick="deleteThread('${escapeHtml(thread.thread_id)}', '${escapeHtml(thread.subject || '(No Subject)')}')" title="Delete thread">
        🗑
    </button>
</td>
```

**Add new JS functions** at the bottom of app.js:
```javascript
async function deleteContact(contactId, contactName) {
    if (!confirm('Delete contact "' + contactName + '" and ALL associated data (emails, analysis, profiles, memories)? This cannot be undone.')) return;

    try {
        const response = await fetch('api.php?action=delete_contact', {
            method: 'POST',
            headers: {'Content-Type': 'application/json'},
            body: JSON.stringify({contact_id: contactId})
        });
        const data = await response.json();
        if (data.success) {
            alert('Contact deleted successfully');
            loadContacts(state.contacts.page);
        } else {
            alert('Error: ' + (data.message || 'Delete failed'));
        }
    } catch(e) {
        alert('Error deleting contact: ' + e.message);
    }
}

async function deleteThread(threadId, subject) {
    if (!confirm('Delete thread "' + subject + '" and all associated emails/analysis? This cannot be undone.')) return;

    try {
        const response = await fetch('api.php?action=delete_thread', {
            method: 'POST',
            headers: {'Content-Type': 'application/json'},
            body: JSON.stringify({thread_id: threadId})
        });
        const data = await response.json();
        if (data.success) {
            alert('Thread deleted successfully');
            loadEmails(state.emails.page);
        } else {
            alert('Error: ' + (data.message || 'Delete failed'));
        }
    } catch(e) {
        alert('Error deleting thread: ' + e.message);
    }
}
```

#### File 4: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/thread_view.php`

**In the saved response display** (around line 1244-1251), add a delete button alongside the existing Quick View and Full View buttons:

```php
<button class="btn-delete" onclick="event.stopPropagation(); deleteAiResponse('<?= $saved['_id'] ?>')" style="color: #ef4444; background: none; border: 1px solid #ef4444; padding: 4px 8px; border-radius: 4px; cursor: pointer; font-size: 0.85em;">
    🗑 Delete
</button>
```

**Add the JS function** in thread_view.php's inline `<script>` block:
```javascript
async function deleteAiResponse(responseId) {
    if (!confirm('Delete this AI response? This cannot be undone.')) return;

    try {
        const response = await fetch('api.php?action=delete_ai_response', {
            method: 'POST',
            headers: {'Content-Type': 'application/json'},
            body: JSON.stringify({response_id: responseId})
        });
        const data = await response.json();
        if (data.success) {
            var el = document.getElementById('saved-response-' + responseId);
            if (el) el.remove();
        } else {
            alert('Error: ' + (data.message || 'Delete failed'));
        }
    } catch(e) {
        alert('Error deleting response: ' + e.message);
    }
}
```

---

### Task 6: Bulk Delete Threads by Email Address

**Effort:** 15 minutes. Three files.

#### File 1: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/api.php`

Add new case:
```php
case 'delete_threads_by_email':
    if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
        jsonError('POST required', 405);
    }
    $input = json_decode(file_get_contents('php://input'), true);
    $email = trim($input['email'] ?? '');
    $domain = trim($input['domain'] ?? '');
    if (!$email && !$domain) jsonError('email or domain required');

    $adapter = QueryAdapter::getInstance();
    $result = $adapter->deleteThreadsByEmail($email, $domain);
    jsonResponse(['success' => true, 'deleted' => $result]);
    break;
```

#### File 2: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/src/QueryAdapter.php`

Add method:
```php
public function deleteThreadsByEmail(string $email = '', string $domain = ''): array
{
    $mongo = MongoDatabase::getInstance();

    // Find all emails matching the address or domain
    $filter = [];
    if ($email) {
        $filter['$or'] = [
            ['from_email' => strtolower($email)],
            ['to_email' => strtolower($email)]
        ];
    } elseif ($domain) {
        $domainPattern = '@' . preg_quote($domain, '/') . '$';
        $filter['$or'] = [
            ['from_email' => ['$regex' => $domainPattern, '$options' => 'i']],
            ['to_email' => ['$regex' => $domainPattern, '$options' => 'i']]
        ];
    }

    // Get unique thread_ids
    $pipeline = [
        ['$match' => $filter],
        ['$group' => ['_id' => '$thread_id']]
    ];
    $threadResults = $mongo->aggregate('emails', $pipeline);
    $threadIds = array_column($threadResults, '_id');

    $result = ['thread_count' => count($threadIds), 'emails' => 0, 'thread_analysis' => 0, 'ai_query_responses' => 0];

    foreach ($threadIds as $tid) {
        $r = $this->deleteThread($tid);
        $result['emails'] += $r['emails'] ?? 0;
        $result['thread_analysis'] += $r['thread_analysis'] ?? 0;
        $result['ai_query_responses'] += $r['ai_query_responses'] ?? 0;
    }

    return $result;
}
```

#### File 3: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/assets/app.js`

Add a "Bulk Delete" button in the contacts table actions column or as a standalone cleanup tool. The simplest approach is to add a button in the Contacts toolbar (in index.php) or call it from the existing contact delete flow. For a contact-linked approach, add a small icon next to each contact's delete button:

```javascript
async function deleteThreadsByEmail(email) {
    if (!confirm('Delete ALL threads involving ' + email + '? This removes all emails, analysis, and AI responses for those threads. Cannot be undone.')) return;

    try {
        const response = await fetch('api.php?action=delete_threads_by_email', {
            method: 'POST',
            headers: {'Content-Type': 'application/json'},
            body: JSON.stringify({email: email})
        });
        const data = await response.json();
        if (data.success) {
            alert('Deleted ' + data.deleted.thread_count + ' threads, ' + data.deleted.emails + ' emails');
            loadContacts(state.contacts.page);
            loadEmails(state.emails.page);
        } else {
            alert('Error: ' + (data.message || 'Delete failed'));
        }
    } catch(e) {
        alert('Error: ' + e.message);
    }
}
```

---

## PHASE 3: Sorting and Filtering (Tasks 2, 3)

### Task 2: Sorting by Last Activity

**Effort:** 20 minutes. Four files.

#### File 1: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/api.php`

**Contacts endpoint** (line 294): Already accepts `sort` and `order` GET params. The `getContactsWithProfiles` method already passes these through. Need to ensure `last_contact` is a valid sort column.

**Threads endpoint** (line 355): Add `sort` and `order` params:
```php
$sort = $_GET['sort'] ?? 'last_date';
$order = strtoupper($_GET['order'] ?? 'DESC') === 'ASC' ? 'ASC' : 'DESC';
```
Pass to `getThreads()` -- method signature needs updating.

#### File 2: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/src/QueryAdapter.php`

**`getContactsWithProfilesMongo()`:** The `$allowedColumns` in the SQL version (line 96) does not include `last_contact`. Add it. In the Mongo version, add sort support for `last_contact` -- this requires a `$lookup` to the `emails` collection to get the max date, or store `last_contact` on the contact document itself. The simpler approach: sort by the profile's `updated_at` or add a `$lookup` + `$max` in the aggregation.

Recommended approach: In the Mongo aggregation pipeline within `getContactsWithProfilesMongo()`, add a `$lookup` to `emails` that gets the max date:
```php
if ($sortColumn === 'last_contact') {
    // Add lookup to get latest email date for sorting
    $pipeline[] = ['$lookup' => [
        'from' => 'emails',
        'localField' => 'id',
        'foreignField' => 'contact_id',
        'as' => 'contact_emails'
    ]];
    $pipeline[] = ['$addFields' => [
        'last_contact' => ['$max' => '$contact_emails.date']
    ]];
    $pipeline[] = ['$project' => ['contact_emails' => 0]]; // Remove the joined array
}
```

**`getThreads()` / `getThreadsMongo()`:** Update method signature to accept `$sort` and `$order` params. The pipeline already sorts by `last_date` (line 650). Make it dynamic:
```php
$sortField = in_array($sort, ['last_date', 'first_date', 'email_count', 'subject']) ? $sort : 'last_date';
$sortDir = $order === 'ASC' ? 1 : -1;
// Replace line 650:
['$sort' => [$sortField => $sortDir]],
```

#### File 3: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/index.php`

**Contacts toolbar** (around line 181-189), add sort dropdown:
```html
<select id="contact-sort" style="padding: 8px; border-radius: 4px; border: 1px solid #d1d5db;">
    <option value="name:ASC">Name (A-Z)</option>
    <option value="name:DESC">Name (Z-A)</option>
    <option value="last_contact:DESC" selected>Last Activity (Newest)</option>
    <option value="last_contact:ASC">Last Activity (Oldest)</option>
</select>
```

**Threads toolbar** (around line 199-202), add sort dropdown:
```html
<select id="thread-sort" style="padding: 8px; border-radius: 4px; border: 1px solid #d1d5db;">
    <option value="last_date:DESC" selected>Last Activity (Newest)</option>
    <option value="last_date:ASC">Last Activity (Oldest)</option>
    <option value="email_count:DESC">Most Emails</option>
    <option value="first_date:DESC">Newest Thread</option>
</select>
```

#### File 4: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/assets/app.js`

**`loadContacts()` function** (line 172): Read the sort dropdown and pass to API:
```javascript
const sortVal = (document.getElementById('contact-sort')?.value || 'name:ASC').split(':');
const data = await apiCall('contacts', {
    page: page,
    search: state.contacts.search,
    type: state.contacts.type,
    sort: sortVal[0],
    order: sortVal[1],
    limit: 10
});
```

**`loadEmails()` function** (line 650): Same pattern:
```javascript
const sortVal = (document.getElementById('thread-sort')?.value || 'last_date:DESC').split(':');
const params = { page: page, limit: 10, sort: sortVal[0], order: sortVal[1] };
```

**Add event listeners** (at bottom of init, near line 1574-1580):
```javascript
document.getElementById('contact-sort')?.addEventListener('change', () => loadContacts(1));
document.getElementById('thread-sort')?.addEventListener('change', () => loadEmails(1));
```

---

### Task 3: Filters

**Effort:** 30 minutes. Two files (index.php + app.js), with minor api.php changes.

#### File 1: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/index.php`

**Contacts tab** (after toolbar, before `contacts-list` div): Add filter section:
```html
<div class="filter-bar" style="display: flex; gap: 10px; flex-wrap: wrap; margin-bottom: 15px; padding: 10px; background: #f9fafb; border-radius: 6px;">
    <input type="date" id="contact-date-from" placeholder="From date" title="Activity from">
    <input type="date" id="contact-date-to" placeholder="To date" title="Activity to">
    <button class="btn btn-sm" onclick="loadContacts(1)">Apply Filters</button>
    <button class="btn btn-sm btn-secondary" onclick="clearContactFilters()">Clear</button>
</div>
```

Note: The existing `contact-search` and `contact-type-filter` already handle search by name/email and contact type filtering. These are already wired up (lines 1574, 1580 in app.js).

**Threads tab** (after toolbar): Add filter section:
```html
<div class="filter-bar" style="display: flex; gap: 10px; flex-wrap: wrap; margin-bottom: 15px; padding: 10px; background: #f9fafb; border-radius: 6px;">
    <input type="text" id="thread-contact-filter" placeholder="Contact name..." style="padding: 8px; border-radius: 4px; border: 1px solid #d1d5db; width: 150px;">
    <input type="date" id="thread-date-from" title="From date">
    <input type="date" id="thread-date-to" title="To date">
    <select id="thread-status-filter" style="padding: 8px; border-radius: 4px; border: 1px solid #d1d5db;">
        <option value="">All Status</option>
        <option value="analyzed">Analyzed</option>
        <option value="not_analyzed">Not Analyzed</option>
        <option value="needs_reanalysis">Needs Re-analysis</option>
    </select>
    <button class="btn btn-sm" onclick="loadEmails(1)">Apply</button>
    <button class="btn btn-sm btn-secondary" onclick="clearThreadFilters()">Clear</button>
</div>
```

#### File 2: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/assets/app.js`

**`loadContacts()`:** Add date range params:
```javascript
const dateFrom = document.getElementById('contact-date-from')?.value || '';
const dateTo = document.getElementById('contact-date-to')?.value || '';
// Add to apiCall params:
if (dateFrom) params.date_from = dateFrom;
if (dateTo) params.date_to = dateTo;
```

**`loadEmails()`:** Add filter params:
```javascript
const contactFilter = document.getElementById('thread-contact-filter')?.value || '';
const dateFrom = document.getElementById('thread-date-from')?.value || '';
const dateTo = document.getElementById('thread-date-to')?.value || '';
const statusFilter = document.getElementById('thread-status-filter')?.value || '';
if (contactFilter) params.contact_name = contactFilter;
if (dateFrom) params.date_from = dateFrom;
if (dateTo) params.date_to = dateTo;
if (statusFilter) params.status = statusFilter;
```

**Add clear functions:**
```javascript
function clearContactFilters() {
    document.getElementById('contact-search').value = '';
    document.getElementById('contact-type-filter').value = '';
    document.getElementById('contact-date-from').value = '';
    document.getElementById('contact-date-to').value = '';
    loadContacts(1);
}

function clearThreadFilters() {
    document.getElementById('email-search').value = '';
    document.getElementById('thread-contact-filter').value = '';
    document.getElementById('thread-date-from').value = '';
    document.getElementById('thread-date-to').value = '';
    document.getElementById('thread-status-filter').value = '';
    loadEmails(1);
}
```

#### File 3: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/api.php`

**Contacts endpoint:** Pass date range params to QueryAdapter:
```php
$dateFrom = $_GET['date_from'] ?? '';
$dateTo = $_GET['date_to'] ?? '';
```

**Threads endpoint:** Pass additional filter params:
```php
$contactName = $_GET['contact_name'] ?? '';
$dateFrom = $_GET['date_from'] ?? '';
$dateTo = $_GET['date_to'] ?? '';
$status = $_GET['status'] ?? '';
```

#### File 4: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/src/QueryAdapter.php`

**`getThreadsMongo()`:** Extend filter building:
- Date range: add `'date' => ['$gte' => $dateFrom, '$lte' => $dateTo]` to `$matchFilter`
- Contact name filter: post-filter after aggregation (since contact names are resolved post-pipeline) -- or do a pre-lookup. Simplest: client-side filter the contact_name column. But for server-side: do a pre-query to get contact_ids matching the name, then add `contact_id => ['$in' => $ids]` to `$matchFilter`.
- Status filter: post-filter on `is_analyzed` / `needs_reanalysis` from the joined analysis.

**`getContactsWithProfilesMongo()`:** Add date range support. Similar to sort: `$lookup` emails, filter by max date range.

---

## PHASE 4: Bug Fix (Task 4)

### Task 4: Verify "Etienne Turner" Thread Linkage Fix

**Effort:** 15 minutes investigation only.

The fix in `getThreadsMongo()` (lines 657-698 of QueryAdapter.php) already:
1. Collects all contacts per thread via `$push`
2. Weights inbound contacts 2x
3. Picks the most frequent external contact
4. Falls back to first non-null contact if all emails are from user

**Edge cases to verify:**
- Thread where ALL emails are outbound (user sent, no replies) -- `$contactCounts` will be empty because inbound emails from user are skipped. The fallback at line 683-690 picks first non-null `cid`, which should be the recipient's contact_id.
- However, `from_email` for outbound emails is the user's email, so it gets skipped. The `cid` (contact_id) field is still set on the email record to the recipient's contact. So the fallback works correctly.

**Action items:**
1. Manually verify in MongoDB: `db.emails.find({thread_id: "..."})` for the specific Etienne Turner thread to confirm contact_ids
2. If the contact_id on outbound emails is null (because it was never set), that is the root cause -- need to ensure `EmailSync` sets `contact_id` for outbound emails too
3. Add a safety check: if the fallback also yields null, use `to_email` from the first outbound email to look up the contact

**Potential code fix in QueryAdapter.php `getThreadsMongo()`** (after line 690):
```php
// Additional fallback: if still null, try to find contact by to_email
if (!$thread['contact_id']) {
    foreach ($thread['all_contacts'] ?? [] as $c) {
        $email = strtolower($c['email'] ?? '');
        if ($email && $email !== $userEmail) {
            // Look up contact by email
            $contact = $mongo->findOne('contacts', ['email' => $email]);
            if ($contact) {
                $thread['contact_id'] = (int)($contact['id'] ?? 0);
                break;
            }
        }
    }
}
```

Wait -- the `all_contacts` array only has `from_email`, not `to_email`. For outbound emails, `from_email` is the user. We would need to also push `to_email` into the aggregation. This is the actual fix needed:

In the `$group` stage (line 613-628), change `all_contacts` to include to_email:
```php
'all_contacts' => ['$push' => [
    'cid' => '$contact_id',
    'from' => '$from_email',
    'to' => '$to_email',
    'dir' => '$direction'
]]
```

Then in the post-processing, also consider `to_email` for outbound messages:
```php
foreach ($thread['all_contacts'] ?? [] as $c) {
    $cid = $c['cid'] ?? null;
    $fromEmail = strtolower($c['from'] ?? '');
    $toEmail = strtolower($c['to'] ?? '');
    $dir = $c['dir'] ?? '';

    if (!$cid) continue;

    // For inbound: from_email is external (weight 2x)
    // For outbound: to_email is external (weight 1x)
    if ($dir === 'inbound' && $fromEmail !== $userEmail) {
        $contactCounts[$cid] = ($contactCounts[$cid] ?? 0) + 2;
    } elseif ($dir === 'outbound' && $toEmail !== $userEmail) {
        $contactCounts[$cid] = ($contactCounts[$cid] ?? 0) + 1;
    }
}
```

---

## PHASE 5: New Features (Tasks 5, 8)

### Task 5: Sync by Domain/Email Address

**Effort:** 45-60 minutes. Four files.

This is a two-step flow: **Search** then **Import**.

#### File 1: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/index.php`

Add a new section in the Contacts tab toolbar or as a modal. A simple inline panel works best:

```html
<!-- Sync by Domain Section -->
<div class="card" style="margin-bottom: 20px; padding: 15px;">
    <h3 style="margin-top: 0;">Import Threads by Email/Domain</h3>
    <div style="display: flex; gap: 10px; align-items: end;">
        <div style="flex: 1;">
            <input type="text" id="sync-domain-input" placeholder="e.g., delta-engineering.be or john@delta-engineering.be" class="search-input" style="width: 100%;">
        </div>
        <button class="btn btn-primary" onclick="searchByDomain()">Search Gmail</button>
    </div>
    <div id="sync-domain-results" style="margin-top: 15px;"></div>
</div>
```

Place this inside `tab-contacts` or create a new toolbar section.

#### File 2: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/api.php`

Two new endpoints:

**`search_gmail_threads`** -- searches Gmail for threads matching a query:
```php
case 'search_gmail_threads':
    $query = trim($_GET['query'] ?? '');
    if (!$query) jsonError('query required');
    $maxResults = min(50, (int)($_GET['max_results'] ?? 20));

    // Build Gmail search query
    if (strpos($query, '@') !== false) {
        // Specific email address
        $gmailQuery = "from:{$query} OR to:{$query}";
    } else {
        // Domain
        $gmailQuery = "from:@{$query} OR to:@{$query}";
    }

    $gmail = new GmailClient();
    $threads = $gmail->searchThreads($gmailQuery, $maxResults);

    // Check which threads are already imported
    $mongo = \EmailIntelligence\MongoDatabase::getInstance();
    foreach ($threads as &$t) {
        $existing = $mongo->count('emails', ['thread_id' => $t['id']]);
        $t['already_imported'] = $existing > 0;
        $t['email_count'] = $existing;
    }

    jsonResponse(['threads' => $threads, 'query' => $gmailQuery]);
    break;
```

**`import_gmail_threads`** -- bulk-imports selected threads:
```php
case 'import_gmail_threads':
    if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
        jsonError('POST required', 405);
    }
    $input = json_decode(file_get_contents('php://input'), true);
    $threadIds = $input['thread_ids'] ?? [];
    if (empty($threadIds)) jsonError('thread_ids required');

    $gmail = new GmailClient();
    $emailSync = new EmailSync();
    $results = [];

    foreach ($threadIds as $tid) {
        try {
            $messages = $gmail->getThread($tid);
            $imported = 0;
            foreach ($messages as $msg) {
                $emailSync->processAndStoreEmail($msg, $tid);
                $imported++;
            }
            $results[] = ['thread_id' => $tid, 'success' => true, 'imported' => $imported];
        } catch (Exception $e) {
            $results[] = ['thread_id' => $tid, 'success' => false, 'error' => $e->getMessage()];
        }
    }

    jsonResponse(['success' => true, 'results' => $results]);
    break;
```

**Note:** The `processAndStoreEmail` method needs to be verified in `EmailSync.php`. The existing `sync_thread` case in api.php (line 1843) has complex logic for importing a thread -- consider refactoring to call the same code path rather than reimplementing. Alternatively, call the `sync_thread` logic in a loop for each thread_id.

#### File 3: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/assets/app.js`

Add functions:
```javascript
async function searchByDomain() {
    const query = document.getElementById('sync-domain-input').value.trim();
    if (!query) { alert('Enter an email or domain'); return; }

    const container = document.getElementById('sync-domain-results');
    container.innerHTML = '<p class="loading">Searching Gmail...</p>';

    try {
        const data = await apiCall('search_gmail_threads', {query: query, max_results: 30});

        if (!data.threads || data.threads.length === 0) {
            container.innerHTML = '<p>No threads found for "' + escapeHtml(query) + '"</p>';
            return;
        }

        let html = '<p>Found ' + data.threads.length + ' threads. <button class="btn btn-sm btn-primary" onclick="importAllDomainThreads()">Import All New</button></p>';
        html += '<div style="max-height: 400px; overflow-y: auto;">';

        data.threads.forEach(t => {
            const imported = t.already_imported ? ' <span class="badge badge-positive">Already imported</span>' : '';
            const checkbox = t.already_imported ? '' : '<input type="checkbox" class="domain-thread-cb" value="' + escapeHtml(t.id) + '" checked>';
            html += '<div style="padding: 8px; border-bottom: 1px solid #eee;">' + checkbox + ' <strong>' + escapeHtml(t.snippet || t.id) + '</strong>' + imported + '</div>';
        });

        html += '</div>';
        container.innerHTML = html;
    } catch(e) {
        container.innerHTML = '<p>Error: ' + escapeHtml(e.message) + '</p>';
    }
}

async function importAllDomainThreads() {
    const checkboxes = document.querySelectorAll('.domain-thread-cb:checked');
    const threadIds = Array.from(checkboxes).map(cb => cb.value);

    if (threadIds.length === 0) { alert('No threads selected'); return; }
    if (!confirm('Import ' + threadIds.length + ' threads?')) return;

    document.getElementById('sync-domain-results').innerHTML += '<p class="loading">Importing ' + threadIds.length + ' threads...</p>';

    try {
        const response = await fetch('api.php?action=import_gmail_threads', {
            method: 'POST',
            headers: {'Content-Type': 'application/json'},
            body: JSON.stringify({thread_ids: threadIds})
        });
        const data = await response.json();

        const successful = data.results.filter(r => r.success).length;
        alert('Imported ' + successful + ' of ' + threadIds.length + ' threads');
        loadContacts(1);
        loadEmails(1);
    } catch(e) {
        alert('Import error: ' + e.message);
    }
}
```

#### File 4: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/src/GmailClient.php`

The `searchThreads()` method (line 577) already exists and returns `[id, snippet]` pairs. It should work for the search step. However, for the import step, we need to verify that `getThread()` (line 545) returns parsed messages that `EmailSync` can process.

**Verify in EmailSync.php:** Check what method is used for storing individual messages. The `sync_thread` case in api.php (lines 1843-2106) is very long and handles many edge cases. The cleanest approach for Task 5 is to reuse the `sync_thread` API endpoint internally by calling it for each thread_id from the import endpoint, or extract the core logic into a shared method.

---

### Task 8: Two-Round AI D3.js Chart

**Effort:** 45-60 minutes. One file (thread_view.php).

#### File: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/thread_view.php`

**Step 1:** Add D3.js CDN in the `<head>`:
```html
<script src="https://d3js.org/d3.v7.min.js"></script>
```

**Step 2:** Add a "Generate Chart" button after AI response display. In the response rendering section (where responses are displayed after AI calls), add:
```html
<button class="btn btn-secondary" onclick="generateChart()" style="margin-top: 10px;">
    📊 Generate Relationship Chart
</button>
<div id="d3-chart-container" style="display:none; margin-top: 15px; border: 1px solid #e5e7eb; border-radius: 8px; background: white; position: relative;">
    <div id="d3-chart" style="width: 100%; height: 600px;"></div>
    <div id="chart-tooltip" style="display:none; position: absolute; padding: 8px 12px; background: rgba(0,0,0,0.8); color: white; border-radius: 4px; font-size: 0.85em; pointer-events: none; z-index: 10;"></div>
</div>
```

**Step 3:** Add the `generateChart()` function in the `<script>` block:

```javascript
async function generateChart() {
    // Get the latest AI response text
    var responseText = document.querySelector('.ai-response-content')?.innerText || '';
    if (!responseText) { alert('No AI response to chart'); return; }

    // Send to AI asking for structured JSON
    var chartPrompt = 'Based on the following analysis, generate a JSON object for a relationship chart with this exact structure:\n' +
        '{"nodes": [{"id": "unique_id", "label": "Name", "type": "person|company|topic|event", "details": "brief description"}], ' +
        '"links": [{"source": "node_id", "target": "node_id", "label": "relationship", "strength": 1-10}]}\n' +
        'Only return valid JSON, no other text.\n\nAnalysis:\n' + responseText.substring(0, 4000);

    document.getElementById('d3-chart-container').style.display = 'block';
    document.getElementById('d3-chart').innerHTML = '<p class="loading" style="text-align:center; padding:40px;">Generating chart data...</p>';

    try {
        // Use the existing AI sending mechanism with a single provider
        var response = await fetch('api.php?action=ask_ai', {
            method: 'POST',
            headers: {'Content-Type': 'application/json'},
            body: JSON.stringify({
                question: chartPrompt,
                thread_id: THREAD_ID,
                provider: 'claude',
                model: 'claude-sonnet-4-6'
            })
        });
        var data = await response.json();

        // Parse JSON from AI response
        var responseContent = data.response || '';
        var jsonMatch = responseContent.match(/\{[\s\S]*\}/);
        if (!jsonMatch) throw new Error('No JSON found in AI response');

        var chartData = JSON.parse(jsonMatch[0]);
        renderD3Chart(chartData);
    } catch(e) {
        document.getElementById('d3-chart').innerHTML = '<p style="color:red; padding:20px;">Error: ' + e.message + '</p>';
    }
}

function renderD3Chart(data) {
    var container = document.getElementById('d3-chart');
    container.innerHTML = '';
    var width = container.clientWidth;
    var height = 600;

    var typeColors = {
        person: '#3b82f6',
        company: '#10b981',
        topic: '#f59e0b',
        event: '#ef4444'
    };

    var svg = d3.select('#d3-chart').append('svg')
        .attr('width', width)
        .attr('height', height);

    // Add zoom behavior
    var g = svg.append('g');
    svg.call(d3.zoom().on('zoom', function(event) {
        g.attr('transform', event.transform);
    }));

    // Arrow markers for directed links
    svg.append('defs').selectAll('marker')
        .data(['arrow'])
        .enter().append('marker')
        .attr('id', 'arrow')
        .attr('viewBox', '0 -5 10 10')
        .attr('refX', 25)
        .attr('refY', 0)
        .attr('markerWidth', 6)
        .attr('markerHeight', 6)
        .attr('orient', 'auto')
        .append('path')
        .attr('d', 'M0,-5L10,0L0,5')
        .attr('fill', '#999');

    var simulation = d3.forceSimulation(data.nodes)
        .force('link', d3.forceLink(data.links).id(function(d) { return d.id; }).distance(120))
        .force('charge', d3.forceManyBody().strength(-300))
        .force('center', d3.forceCenter(width / 2, height / 2))
        .force('collision', d3.forceCollide().radius(40));

    var link = g.append('g').selectAll('line')
        .data(data.links)
        .enter().append('line')
        .attr('stroke', '#999')
        .attr('stroke-width', function(d) { return Math.max(1, (d.strength || 5) / 3); })
        .attr('marker-end', 'url(#arrow)');

    var linkLabel = g.append('g').selectAll('text')
        .data(data.links)
        .enter().append('text')
        .attr('text-anchor', 'middle')
        .attr('font-size', '10px')
        .attr('fill', '#666')
        .text(function(d) { return d.label || ''; });

    var node = g.append('g').selectAll('g')
        .data(data.nodes)
        .enter().append('g')
        .call(d3.drag()
            .on('start', function(event, d) { if (!event.active) simulation.alphaTarget(0.3).restart(); d.fx = d.x; d.fy = d.y; })
            .on('drag', function(event, d) { d.fx = event.x; d.fy = event.y; })
            .on('end', function(event, d) { if (!event.active) simulation.alphaTarget(0); d.fx = null; d.fy = null; })
        );

    node.append('circle')
        .attr('r', 20)
        .attr('fill', function(d) { return typeColors[d.type] || '#6b7280'; })
        .attr('stroke', '#fff')
        .attr('stroke-width', 2);

    node.append('text')
        .attr('dy', 30)
        .attr('text-anchor', 'middle')
        .attr('font-size', '11px')
        .attr('font-weight', 'bold')
        .text(function(d) { return d.label; });

    // Tooltip on hover
    var tooltip = d3.select('#chart-tooltip');
    node.on('mouseover', function(event, d) {
        tooltip.style('display', 'block')
            .html('<strong>' + d.label + '</strong><br>' + (d.details || d.type))
            .style('left', (event.offsetX + 10) + 'px')
            .style('top', (event.offsetY - 10) + 'px');
    }).on('mouseout', function() {
        tooltip.style('display', 'none');
    });

    simulation.on('tick', function() {
        link.attr('x1', function(d) { return d.source.x; })
            .attr('y1', function(d) { return d.source.y; })
            .attr('x2', function(d) { return d.target.x; })
            .attr('y2', function(d) { return d.target.y; });

        linkLabel.attr('x', function(d) { return (d.source.x + d.target.x) / 2; })
            .attr('y', function(d) { return (d.source.y + d.target.y) / 2; });

        node.attr('transform', function(d) { return 'translate(' + d.x + ',' + d.y + ')'; });
    });
}
```

**Note:** The `ask_ai` endpoint needs to be checked for compatibility. The existing AI sending in thread_view.php uses the `api_opus` proxy directly via JavaScript fetch calls to `../aiapi/api_opus/`. An alternative is to call that proxy directly for chart generation. Check the existing send pattern and match it.

---

## PHASE 6: Notes-Only (Task 7)

### Task 7: Chrome Extension Manual-Only Mode

**Cannot be implemented on this server.** The Chrome extension files are at `/Users/dan/Documents/extension2/extension2v1/chrome-extension` on the user's local Mac.

**Design notes for implementation:**

1. **`manifest.json`**: No changes needed for permissions.

2. **`popup.html` / `popup.js`**: Add a toggle switch for "Auto-sync" vs "Manual only":
   - Store in `chrome.storage.local` as `{syncMode: 'auto' | 'manual'}`
   - Default: `'auto'` (current behavior)

3. **`background.js` (or service worker)**: Check `syncMode` before auto-syncing:
   ```javascript
   chrome.storage.local.get('syncMode', function(data) {
       if (data.syncMode === 'manual') return; // Skip auto-sync
       // ... existing auto-sync logic
   });
   ```

4. **`content.js`**: In manual mode, inject a floating "Sync this thread" button when viewing a Gmail thread:
   ```javascript
   chrome.storage.local.get('syncMode', function(data) {
       if (data.syncMode === 'manual') {
           // Inject sync button overlay
           const btn = document.createElement('button');
           btn.textContent = 'Sync to Email Intelligence';
           btn.style.cssText = 'position:fixed; bottom:20px; right:20px; z-index:9999; padding:10px 20px; background:#3b82f6; color:white; border:none; border-radius:8px; cursor:pointer; font-size:14px; box-shadow:0 2px 10px rgba(0,0,0,0.2);';
           btn.onclick = function() { /* trigger sync_thread API call */ };
           document.body.appendChild(btn);
       }
   });
   ```

---

## Summary of File Changes by File

| File | Tasks | Changes |
|------|-------|---------|
| `public/api.php` | 1, 2, 3, 5, 6 | 6 new case blocks (delete_contact, delete_thread, delete_ai_response, delete_threads_by_email, search_gmail_threads, import_gmail_threads), modify contacts/threads cases for sort/filter params |
| `src/QueryAdapter.php` | 1, 2, 3, 4, 6 | 3 new methods (deleteContact, deleteThread, deleteThreadsByEmail), modify getThreadsMongo signature + sort, modify getContactsWithProfilesMongo for last_contact sort, fix all_contacts aggregation |
| `public/assets/app.js` | 1, 2, 3, 5, 6 | New functions (deleteContact, deleteThread, deleteThreadsByEmail, searchByDomain, importAllDomainThreads, clearContactFilters, clearThreadFilters), modify loadContacts/loadEmails for sort/filter params |
| `public/index.php` | 2, 3, 5, 9 | Add sort dropdowns, filter bars, sync-by-domain panel, comment out Products tab |
| `public/thread_view.php` | 1, 8, 10 | Add delete button for saved responses, D3.js chart feature, update MODEL_LIMITS |
| `config/config.php` | 10 | Update CLAUDE_MODEL constant |
| `aiapi/api_opus/config/api_keys.php` | 10 | Add new Claude model entries |

---

### Critical Files for Implementation

- `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/api.php` - Central API router; needs 6 new endpoint cases for delete, search, and import operations
- `/var/www/mssql.danmarcrm.com/dev1/extension2v1/src/QueryAdapter.php` - Data access layer; needs 3 new delete methods, sorting/filtering modifications, and the Etienne Turner fix in getThreadsMongo
- `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/assets/app.js` - All frontend interaction logic; needs delete functions, sort/filter wiring, and domain search UI handlers
- `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/index.php` - Dashboard HTML; needs sort dropdowns, filter bars, sync-by-domain panel, and Products tab hiding
- `/var/www/mssql.danmarcrm.com/dev1/extension2v1/public/thread_view.php` - Thread view page; needs delete button for AI responses, D3.js chart feature, and updated MODEL_LIMITS

agentId: a5da3ef (for resuming to continue this agent's work if needed)
<usage>total_tokens: 65269
tool_uses: 39
duration_ms: 299395</usage>

Prompt:
Design a detailed implementation plan for the following tasks in the Email Intelligence System at /var/www/mssql.danmarcrm.com/dev1/extension2v1/

## Context
This is a PHP/MongoDB web app with a dashboard (index.php + app.js), API (api.php), data layer (QueryAdapter.php), and a thread view page (thread_view.php). Chrome extension syncs Gmail threads. AI proxy at aiapi/api_opus/.

## Tasks (in priority order)

### TASK 1: Delete Functionality
Add ability to delete:
- **Contacts**: Already has a Delete button in dashboard UI. Need API endpoint `delete_contact` in api.php + QueryAdapter method. Must also delete associated emails, thread_analysis, contact_profiles, ai_contact_memory for that contact.
- **Email Threads**: Add delete button to threads list. API endpoint `delete_thread` + QueryAdapter. Must delete all emails with that thread_id, thread_analysis, ai_query_responses referencing that thread.
- **AI Responses**: Add delete button next to saved AI responses in thread_view.php. API endpoint `delete_ai_response`. Delete from ai_query_responses collection.

Files: api.php, QueryAdapter.php, app.js, index.php, thread_view.php

### TASK 2: Sorting by Last Activity
- Contacts list: Add sort by `last_contact` date (from contact profiles or max email date)
- Threads list: Add sort by `last_date` (already tracked in thread aggregation)
- Add sort dropdown/buttons in the UI for both tabs

Files: app.js, index.php, api.php (may need sort param support), QueryAdapter.php

### TASK 3: Filters
- **Contacts tab**: Add filters for contact type (client/supplier/other), search by name/email, last activity date range
- **Threads tab**: Add filters for contact name, date range, email count range, analysis status
- **Last Activity date filter**: Date range picker for both tabs
- Add filter UI section above each table

Files: app.js, index.php

### TASK 4: Fix "Etienne Turner" Thread Linkage
The contact name resolution was fixed in QueryAdapter.php but may still have issues:
- MongoDB `getThreadsMongo()` collects all_contacts, weights inbound 2x, picks most frequent
- But if contact_id is NULL or email matches user, it skips — edge case: if ALL emails in thread are from user (outbound only), falls back to first contact
- Need to verify the fix works and potentially handle edge cases

Files: QueryAdapter.php

### TASK 5: Sync by Domain/Email Address
New feature: Dashboard UI panel where user enters a domain (e.g. delta-engineering.be) or email address. System queries Gmail API to find all threads involving that domain/email, then imports them.

Implementation:
- New UI section in dashboard or modal
- New API endpoint `sync_by_domain` that:
  1. Uses GmailClient to search: `from:@domain.com OR to:@domain.com` (or specific email)
  2. Lists matching threads with preview (subject, date, count)
  3. User selects which to import (or "Import All")
  4. Imports selected threads using existing sync_thread logic
- Need two-step flow: search → preview → import

Files: api.php, index.php, app.js, GmailClient.php (may need search method)

### TASK 6: Bulk Delete Threads by Email Address
- Add ability to delete all threads for a specific email/domain
- UI: In contacts list or a cleanup tool
- API: `delete_threads_by_email` — finds all emails from that address, gets unique thread_ids, deletes all data

Files: api.php, QueryAdapter.php, app.js, index.php

### TASK 7: Chrome Extension Manual-Only Mode
- Add toggle in Chrome extension popup/options for "Auto-sync" vs "Manual sync only"
- In manual mode, show a "Sync this thread" button overlay in Gmail
- Store preference in chrome.storage.local
- Extension files are at /Users/dan/Documents/extension2/extension2v1/chrome-extension (local Mac) — we can only prepare the code changes

NOTE: Chrome extension is on user's local Mac, not this server. We should note what needs to change but can't edit those files directly.

### TASK 8: Two-Round AI D3.js Chart
- After Round 2 (or any AI response), add "Generate Chart" button
- AI generates structured JSON (entities, relationships, events)
- D3.js force-directed graph renders: nodes as balloons (color-coded by type), arrows showing relationships, labels
- Interactive: drag, zoom, pan, click for details

Files: thread_view.php (inline JS + D3.js CDN)

### TASK 9: Hide Products Tab
- Remove Products tab from navigation in index.php
- Keep product_search.php file intact

Files: index.php

### TASK 10: Update AI Model Versions
Update to latest models:
- Claude: Add `claude-opus-4-6`, `claude-sonnet-4-6` (current has claude-sonnet-4-5-20250929)
- Other providers: verify current versions

Files to update:
- /extension2v1/aiapi/api_opus/config/api_keys.php — provider models list + defaults
- /extension2v1/public/thread_view.php — MODEL_LIMITS constant
- /extension2v1/config/config.php — CLAUDE_MODEL constant
- Also check /extension2/aiapi/api_opus/config/api_keys.php

Current model config in api_keys.php uses arrays like:
```php
'claude' => [
    'default_model' => 'claude-sonnet-4-5-20250929',
    'models' => ['claude-sonnet-4-5-20250929', 'claude-3-5-sonnet-20241022', ...]
]
```

## Architecture Patterns to Follow
- Use `MongoDatabase::insertOrUpdate()` for upserts (NOT `upsert()`)
- Use `QueryAdapter` for all data access (NOT raw SQL)
- AI proxy models keyed by provider: `[$provider => $model]`
- MongoDB `delete()` method available on MongoDatabase class
- Node.js on server is v10.19 — can't use for JS syntax checks
- `ThreadAnalyzer::stripQuotedContent()` for email body processing

## Available MongoDB Methods (MongoDatabase.php)
- `insert($collection, $data)`
- `insertOrUpdate($collection, $filter, $data)`
- `update($collection, $filter, $data)`
- `delete($collection, $filter)`
- `findAll($collection, $filter, $options)`
- `findOne($collection, $filter)`
- `count($collection, $filter)`
- `aggregate($collection, $pipeline)`

## Existing API patterns
- JSON responses via `jsonResponse(['success' => true, ...])`
- Error handling via `jsonResponse(['success' => false, 'error' => '...'])`
- POST data via `$input = json_decode(file_get_contents('php://input'), true)`
- GET params via `$_GET['param']`

Please design a detailed, step-by-step plan with specific code changes needed for each task. Group related changes by file to minimize edits. Prioritize the tasks that are quickest wins first.


Now I have comprehensive exploration results and a detailed plan from the Plan agent. Let me write the final plan to the plan file, incorporating the key decisions from the user's answers and the plan agent's design. Let me read the current plan file first, then write the final version.


Good, I have thorough exploration and a detailed design. Let me read the current plan file and write the final plan.


     1→# Plan: Enhance thread_view.php + Fix Main Dashboard Issues
     2→
     3→## Context
     4→
     5→The user found `thread_view.php` already has a working AI form with multi-provider support, conversation history (sessionStorage), saved responses, and quick view. They want the new AI features (agents, context builder, token estimation, debug logging) merged into this existing page. Additionally, the main dashboard has 3 bugs: slow loading, wrong contact names on threads, and a useless AI Thread Analysis section.
     6→
     7→## 4 Tasks (in execution order)
     8→
     9→---
    10→
    11→### TASK 1: Hide AI Thread Analysis section (index.php + app.js)
    12→
    13→**Problem:** The "AI Thread Analysis" section shows "Unknown" contacts, runs heavy N+1 queries, and auto-refreshes every 30 seconds — slowing the page.
    14→
    15→**Files:**
    16→
    17→| File | Change |
    18→|------|--------|
    19→| [public/index.php:136](public/index.php#L136) | Add `style="display:none"` to `#analysis-status-section` div |
    20→| [public/assets/app.js:1589](public/assets/app.js#L1589) | Comment out `refreshAnalysisStatus()` call |
    21→| [public/assets/app.js:1592](public/assets/app.js#L1592) | Comment out `setInterval(refreshAnalysisStatus, 30000)` |
    22→
    23→---
    24→
    25→### TASK 2: Speed up main page (app.js)
    26→
    27→**Problem:** Threads and contacts load 20 records each — too many for initial page load.
    28→
    29→**Files:**
    30→
    31→| File | Change |
    32→|------|--------|
    33→| [public/assets/app.js:176](public/assets/app.js#L176) | `loadContacts` → change `limit: 20` to `limit: 10` |
    34→| [public/assets/app.js:650](public/assets/app.js#L650) | `loadEmails` → change `limit: 20` to `limit: 10` |
    35→
    36→---
    37→
    38→### TASK 3: Fix thread contact name bug (QueryAdapter.php)
    39→
    40→**Problem:** `getThreadsMongo()` line 609 uses `'$max' => '$contact_id'` which picks the numerically highest contact_id (always "Etienne Turner"). Same issue in `getThreadsSQL()` line 549-550 with `MAX(e.contact_id)` and `MAX(c.name)`.
    41→
    42→**Fix approach:**
    43→
    44→**MongoDB** ([src/QueryAdapter.php:569-650](src/QueryAdapter.php#L569)):
    45→1. In the `$group` stage, collect all contacts instead of `$max`:
    46→   - Replace `'contact_id' => ['$max' => '$contact_id']` with `'contacts' => ['$push' => ['cid' => '$contact_id', 'email' => '$from_email', 'dir' => '$direction']]`
    47→2. After aggregation, post-process each thread in PHP:
    48→   - Filter out contacts where `email === GMAIL_USER_EMAIL` (the user's own email)
    49→   - Filter to only `direction === 'inbound'` contacts
    50→   - Count occurrences of each remaining `contact_id`
    51→   - Pick the most frequent one
    52→   - If none found, fall back to the first non-user contact_id
    53→3. Use the resolved contact_id for the `$lookup` (or do a separate contact name lookup after post-processing)
    54→
    55→**SQL** ([src/QueryAdapter.php:514-567](src/QueryAdapter.php#L514)):
    56→1. Replace `MAX(e.contact_id)` with a subquery that finds the most frequent inbound sender:
    57→   ```sql
    58→   (SELECT e2.contact_id FROM emails e2
    59→    LEFT JOIN contacts c2 ON e2.contact_id = c2.id
    60→    WHERE e2.thread_id = e.thread_id
    61→    AND e2.direction = 'inbound'
    62→    AND LOWER(e2.from_email) != ?
    63→    GROUP BY e2.contact_id
    64→    ORDER BY COUNT(*) DESC LIMIT 1) as contact_id
    65→   ```
    66→2. Similar subquery for contact_name.
    67→3. Add `GMAIL_USER_EMAIL` as bind parameter.
    68→
    69→---
    70→
    71→### TASK 4: Enhance thread_view.php (the main task)
    72→
    73→**Problem:** thread_view.php needs agents, cross-thread context builder, token estimation, and debug logging — while keeping existing multi-provider simultaneous sending.
    74→
    75→**Current architecture (keep):**
    76→- Multi-provider checkboxes → sends to `api_opus/api/query.php` simultaneously
    77→- Conversation history in sessionStorage
    78→- Saved responses in MongoDB `ai_query_responses`
    79→- Quick view modal, load more responses
    80→- Content processing modes (full/stripped/smart)
    81→
    82→**New features to add:**
    83→
    84→#### 4A. Full-width layout
    85→- [thread_view.php:174](public/thread_view.php#L174): Change `.page-container { max-width: 1600px }` to `max-width: 100%; padding: 20px 30px;`
    86→- [thread_view.php:213-217](public/thread_view.php#L213): Change `.main-grid` from `grid-template-columns: 1fr 400px` to `grid-template-columns: 1fr 500px` (wider AI panel)
    87→
    88→#### 4B. Agent buttons with quick prompts
    89→Add above the provider grid in the right column (after line 975):
    90→
    91→```html
    92→<!-- Agent Selector -->
    93→<div class="ai-form-section">
    94→    <label>AI Agent</label>
    95→    <div class="agent-buttons" id="agent-buttons">
    96→        <!-- Populated by JS -->
    97→    </div>
    98→    <div class="agent-quick-prompts" id="agent-quick-prompts" style="display:none;">
    99→        <!-- Quick prompt buttons per agent -->
   100→    </div>
   101→</div>
   102→```
   103→
   104→**JS agent presets** (add as constants near top of `<script>` block):
   105→```js
   106→const AI_AGENTS = {
   107→    general: { name: 'General', icon: '💬', systemPrompt: '...', quickPrompts: [...] },
   108→    quote_analyst: { name: 'Quotes', icon: '💰', systemPrompt: '...', quickPrompts: [...] },
   109→    deception_detector: { name: 'Deception', icon: '🔍', systemPrompt: '...', quickPrompts: [...] },
   110→    contact_profiler: { name: 'Profiler', icon: '👤', systemPrompt: '...', quickPrompts: [...] },
   111→    email_composer: { name: 'Composer', icon: '✍', systemPrompt: '...', quickPrompts: [...] }
   112→};
   113→```
   114→
   115→When an agent is selected:
   116→- Highlight the active agent button
   117→- Show agent-specific quick prompts below the buttons
   118→- Clicking a quick prompt fills the question textarea
   119→- The agent's `systemPrompt` is prepended to the content sent to AI
   120→
   121→#### 4C. Context builder (search + add threads from any contact)
   122→Add a collapsible section between "Include Other Threads" and "Ask AI" cards:
   123→
   124→```html
   125→<!-- Context Builder -->
   126→<div class="card">
   127→    <div class="card-header">
   128→        <h2>+ Add More Context</h2>
   129→        <button class="btn btn-small" onclick="toggleContextBuilder()">Search</button>
   130→    </div>
   131→    <div id="context-builder" style="display:none;">
   132→        <input type="text" id="context-search" placeholder="Search contacts or threads...">
   133→        <div id="context-search-results"></div>
   134→    </div>
   135→    <div id="context-cart"></div>
   136→</div>
   137→```
   138→
   139→**JS functions:**
   140→- `toggleContextBuilder()` — show/hide search
   141→- `searchContext(query)` — calls `api.php?action=contacts&search=...` AND `api.php?action=threads&search=...` in parallel
   142→- Results show contacts (click to see their threads with "+Add" buttons) and threads (with "+Add" directly)
   143→- `addThreadToContext(threadId, subject, emailCount)` — adds to `contextThreads[]` array
   144→- `removeThreadFromContext(threadId)` — removes
   145→- `updateContextCart()` — renders green summary: "3 additional threads, ~25 emails"
   146→- When `submitToAI()` runs, fetches email content for each context thread (via `api.php?action=thread&thread_id=...`) and appends to the content
   147→
   148→#### 4D. Token estimation
   149→Add a live token counter near the submit button:
   150→
   151→```html
   152→<div id="token-estimate" style="font-size:0.85em; color:var(--text-secondary); margin-bottom:10px;">
   153→    Estimated: ~0 tokens
   154→</div>
   155→```
   156→
   157→**JS:**
   158→- `MODEL_LIMITS` constant mapping model names to max context (e.g. `{'claude-sonnet-4-5-20250929': 200000, 'gpt-4o': 128000, ...}`)
   159→- `estimateTokens()` — called on: agent change, context change, question typing, provider toggle
   160→  - Calculates `Math.ceil(totalContentLength / 3.5)`
   161→  - Shows warning if approaching any selected model's limit
   162→  - Compares against the **smallest** selected model's limit (since multi-provider)
   163→  - Display: "Estimated: ~12,500 tokens (OK for all selected models)" or "Warning: exceeds gpt-4o limit (128K)"
   164→- In `submitToAI()`: if over limit, confirm before sending
   165→
   166→#### 4E. System prompt injection per agent
   167→In `submitToAI()`, prepend the selected agent's system prompt:
   168→
   169→```js
   170→let systemPrompt = '';
   171→if (selectedAgent && AI_AGENTS[selectedAgent]) {
   172→    systemPrompt = AI_AGENTS[selectedAgent].systemPrompt + '\n\n';
   173→}
   174→// Prepend to fullContent
   175→fullContent = systemPrompt + fullContent;
   176→```
   177→
   178→This works because `api_opus/query.php` accepts `user_content` as one big string — the system prompt at the top guides the AI's behavior.
   179→
   180→#### 4F. Debug file logging
   181→**New API endpoint** in [public/api.php](public/api.php) — `log_ai_debug`:
   182→
   183→```php
   184→case 'log_ai_debug':
   185→    $logDir = dirname(__DIR__) . '/logs';
   186→    if (!is_dir($logDir)) mkdir($logDir, 0755, true);
   187→    $logFile = $logDir . '/ai_debug.log';
   188→    $logData = json_decode(file_get_contents('php://input'), true);
   189→    $logEntry = [
   190→        'timestamp' => date('Y-m-d H:i:s'),
   191→        'thread_id' => $logData['thread_id'] ?? '',
   192→        'agent' => $logData['agent'] ?? '',
   193→        'providers' => $logData['providers'] ?? [],
   194→        'content_length' => $logData['content_length'] ?? 0,
   195→        'estimated_tokens' => $logData['estimated_tokens'] ?? 0,
   196→        'question' => $logData['question'] ?? '',
   197→        'type' => $logData['type'] ?? 'request',  // 'request' or 'response'
   198→        'response_preview' => substr($logData['response'] ?? '', 0, 500),
   199→        'full_content' => $logData['full_content'] ?? '' // full payload for debug
   200→    ];
   201→    file_put_contents($logFile, json_encode($logEntry) . "\n", FILE_APPEND | LOCK_EX);
   202→    echo json_encode(['success' => true]);
   203→    break;
   204→```
   205→
   206→**JS:** Call this endpoint in `submitToAI()`:
   207→1. **Before** sending to AI: log the request (type: 'request', full_content, providers, tokens)
   208→2. **After** receiving response: log the response (type: 'response', response preview, costs, timing)
   209→
   210→Log file: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/logs/ai_debug.log` — one JSON line per entry.
   211→
   212→#### 4G. Contact memory injection (bonus)
   213→If `primaryContactId` exists, fetch memories from `api.php?action=ai_contact_memories&contact_id=X` and inject as "KNOWN FACTS ABOUT [CONTACT]" section in the content sent to AI. Display memories as pills in the context area.
   214→
   215→---
   216→
   217→## Implementation Order
   218→
   219→1. **TASK 1** — Hide AI Thread Analysis (3 line changes in index.php + app.js)
   220→2. **TASK 2** — Speed up main page (2 line changes in app.js)
   221→3. **TASK 3** — Fix thread contact name bug (QueryAdapter.php — MongoDB + SQL)
   222→4. **TASK 4** — Enhance thread_view.php:
   223→   - 4A: Full-width layout (CSS changes)
   224→   - 4B: Agent buttons + quick prompts (HTML + JS)
   225→   - 4C: Context builder (HTML + JS)
   226→   - 4D: Token estimation (HTML + JS)
   227→   - 4E: System prompt injection (JS modification to submitToAI)
   228→   - 4F: Debug logging endpoint (api.php + JS calls)
   229→   - 4G: Contact memory injection (JS)
   230→
   231→## Files Modified (summary)
   232→
   233→| File | Tasks |
   234→|------|-------|
   235→| [public/index.php](public/index.php) | T1: hide analysis section |
   236→| [public/assets/app.js](public/assets/app.js) | T1: disable refresh, T2: reduce limits |
   237→| [src/QueryAdapter.php](src/QueryAdapter.php) | T3: fix contact name bug |
   238→| [public/api.php](public/api.php) | T4F: add `log_ai_debug` endpoint |
   239→| [public/thread_view.php](public/thread_view.php) | T4: all thread_view enhancements |
   240→
   241→## Verification
   242→
   243→1. **Main dashboard:** Load page — should be fast, no analysis section visible, threads show correct contact names (not all "Etienne Turner")
   244→2. **Thread view:** Open a thread — should be full-width, agent buttons visible, context builder expandable
   245→3. **Agents:** Click "Quotes" agent → quick prompts appear → click one → fills question box → submit → AI response guided by quote analysis system prompt
   246→4. **Context builder:** Click "Search" → type contact name → see their threads → click "+Add" → green cart updates → submit → AI sees all added threads
   247→5. **Token counter:** Shows live estimate → add more context → count increases → shows warning if near limit
   248→6. **Multi-provider:** Select Claude + GPT → submit → both respond in tabs (existing behavior preserved)
   249→7. **Debug log:** Check `/var/www/mssql.danmarcrm.com/dev1/extension2v1/logs/ai_debug.log` → has request + response JSON lines
   250→8. **Conversation continuity:** Ask a follow-up question → AI sees previous Q&A in context (existing behavior preserved)
   251→9. **Saved responses:** Response auto-saves to MongoDB → appears in "Previous AI Analyses" (existing behavior preserved)
   252→

# Plan: Enhance thread_view.php + Fix Main Dashboard Issues

## Context

The user found `thread_view.php` already has a working AI form with multi-provider support, conversation history (sessionStorage), saved responses, and quick view. They want the new AI features (agents, context builder, token estimation, debug logging) merged into this existing page. Additionally, the main dashboard has 3 bugs: slow loading, wrong contact names on threads, and a useless AI Thread Analysis section.

## 4 Tasks (in execution order)

---

### TASK 1: Hide AI Thread Analysis section (index.php + app.js)

**Problem:** The "AI Thread Analysis" section shows "Unknown" contacts, runs heavy N+1 queries, and auto-refreshes every 30 seconds — slowing the page.

**Files:**

| File | Change |
|------|--------|
| [public/index.php:136](public/index.php#L136) | Add `style="display:none"` to `#analysis-status-section` div |
| [public/assets/app.js:1589](public/assets/app.js#L1589) | Comment out `refreshAnalysisStatus()` call |
| [public/assets/app.js:1592](public/assets/app.js#L1592) | Comment out `setInterval(refreshAnalysisStatus, 30000)` |

---

### TASK 2: Speed up main page (app.js)

**Problem:** Threads and contacts load 20 records each — too many for initial page load.

**Files:**

| File | Change |
|------|--------|
| [public/assets/app.js:176](public/assets/app.js#L176) | `loadContacts` → change `limit: 20` to `limit: 10` |
| [public/assets/app.js:650](public/assets/app.js#L650) | `loadEmails` → change `limit: 20` to `limit: 10` |

---

### TASK 3: Fix thread contact name bug (QueryAdapter.php)

**Problem:** `getThreadsMongo()` line 609 uses `'$max' => '$contact_id'` which picks the numerically highest contact_id (always "Etienne Turner"). Same issue in `getThreadsSQL()` line 549-550 with `MAX(e.contact_id)` and `MAX(c.name)`.

**Fix approach:**

**MongoDB** ([src/QueryAdapter.php:569-650](src/QueryAdapter.php#L569)):
1. In the `$group` stage, collect all contacts instead of `$max`:
   - Replace `'contact_id' => ['$max' => '$contact_id']` with `'contacts' => ['$push' => ['cid' => '$contact_id', 'email' => '$from_email', 'dir' => '$direction']]`
2. After aggregation, post-process each thread in PHP:
   - Filter out contacts where `email === GMAIL_USER_EMAIL` (the user's own email)
   - Filter to only `direction === 'inbound'` contacts
   - Count occurrences of each remaining `contact_id`
   - Pick the most frequent one
   - If none found, fall back to the first non-user contact_id
3. Use the resolved contact_id for the `$lookup` (or do a separate contact name lookup after post-processing)

**SQL** ([src/QueryAdapter.php:514-567](src/QueryAdapter.php#L514)):
1. Replace `MAX(e.contact_id)` with a subquery that finds the most frequent inbound sender:
   ```sql
   (SELECT e2.contact_id FROM emails e2
    LEFT JOIN contacts c2 ON e2.contact_id = c2.id
    WHERE e2.thread_id = e.thread_id
    AND e2.direction = 'inbound'
    AND LOWER(e2.from_email) != ?
    GROUP BY e2.contact_id
    ORDER BY COUNT(*) DESC LIMIT 1) as contact_id
   ```
2. Similar subquery for contact_name.
3. Add `GMAIL_USER_EMAIL` as bind parameter.

---

### TASK 4: Enhance thread_view.php (the main task)

**Problem:** thread_view.php needs agents, cross-thread context builder, token estimation, and debug logging — while keeping existing multi-provider simultaneous sending.

**Current architecture (keep):**
- Multi-provider checkboxes → sends to `api_opus/api/query.php` simultaneously
- Conversation history in sessionStorage
- Saved responses in MongoDB `ai_query_responses`
- Quick view modal, load more responses
- Content processing modes (full/stripped/smart)

**New features to add:**

#### 4A. Full-width layout
- [thread_view.php:174](public/thread_view.php#L174): Change `.page-container { max-width: 1600px }` to `max-width: 100%; padding: 20px 30px;`
- [thread_view.php:213-217](public/thread_view.php#L213): Change `.main-grid` from `grid-template-columns: 1fr 400px` to `grid-template-columns: 1fr 500px` (wider AI panel)

#### 4B. Agent buttons with quick prompts
Add above the provider grid in the right column (after line 975):

```html
<!-- Agent Selector -->
<div class="ai-form-section">
    <label>AI Agent</label>
    <div class="agent-buttons" id="agent-buttons">
        <!-- Populated by JS -->
    </div>
    <div class="agent-quick-prompts" id="agent-quick-prompts" style="display:none;">
        <!-- Quick prompt buttons per agent -->
    </div>
</div>
```

**JS agent presets** (add as constants near top of `<script>` block):
```js
const AI_AGENTS = {
    general: { name: 'General', icon: '💬', systemPrompt: '...', quickPrompts: [...] },
    quote_analyst: { name: 'Quotes', icon: '💰', systemPrompt: '...', quickPrompts: [...] },
    deception_detector: { name: 'Deception', icon: '🔍', systemPrompt: '...', quickPrompts: [...] },
    contact_profiler: { name: 'Profiler', icon: '👤', systemPrompt: '...', quickPrompts: [...] },
    email_composer: { name: 'Composer', icon: '✍', systemPrompt: '...', quickPrompts: [...] }
};
```

When an agent is selected:
- Highlight the active agent button
- Show agent-specific quick prompts below the buttons
- Clicking a quick prompt fills the question textarea
- The agent's `systemPrompt` is prepended to the content sent to AI

#### 4C. Context builder (search + add threads from any contact)
Add a collapsible section between "Include Other Threads" and "Ask AI" cards:

```html
<!-- Context Builder -->
<div class="card">
    <div class="card-header">
        <h2>+ Add More Context</h2>
        <button class="btn btn-small" onclick="toggleContextBuilder()">Search</button>
    </div>
    <div id="context-builder" style="display:none;">
        <input type="text" id="context-search" placeholder="Search contacts or threads...">
        <div id="context-search-results"></div>
    </div>
    <div id="context-cart"></div>
</div>
```

**JS functions:**
- `toggleContextBuilder()` — show/hide search
- `searchContext(query)` — calls `api.php?action=contacts&search=...` AND `api.php?action=threads&search=...` in parallel
- Results show contacts (click to see their threads with "+Add" buttons) and threads (with "+Add" directly)
- `addThreadToContext(threadId, subject, emailCount)` — adds to `contextThreads[]` array
- `removeThreadFromContext(threadId)` — removes
- `updateContextCart()` — renders green summary: "3 additional threads, ~25 emails"
- When `submitToAI()` runs, fetches email content for each context thread (via `api.php?action=thread&thread_id=...`) and appends to the content

#### 4D. Token estimation
Add a live token counter near the submit button:

```html
<div id="token-estimate" style="font-size:0.85em; color:var(--text-secondary); margin-bottom:10px;">
    Estimated: ~0 tokens
</div>
```

**JS:**
- `MODEL_LIMITS` constant mapping model names to max context (e.g. `{'claude-sonnet-4-5-20250929': 200000, 'gpt-4o': 128000, ...}`)
- `estimateTokens()` — called on: agent change, context change, question typing, provider toggle
  - Calculates `Math.ceil(totalContentLength / 3.5)`
  - Shows warning if approaching any selected model's limit
  - Compares against the **smallest** selected model's limit (since multi-provider)
  - Display: "Estimated: ~12,500 tokens (OK for all selected models)" or "Warning: exceeds gpt-4o limit (128K)"
- In `submitToAI()`: if over limit, confirm before sending

#### 4E. System prompt injection per agent
In `submitToAI()`, prepend the selected agent's system prompt:

```js
let systemPrompt = '';
if (selectedAgent && AI_AGENTS[selectedAgent]) {
    systemPrompt = AI_AGENTS[selectedAgent].systemPrompt + '\n\n';
}
// Prepend to fullContent
fullContent = systemPrompt + fullContent;
```

This works because `api_opus/query.php` accepts `user_content` as one big string — the system prompt at the top guides the AI's behavior.

#### 4F. Debug file logging
**New API endpoint** in [public/api.php](public/api.php) — `log_ai_debug`:

```php
case 'log_ai_debug':
    $logDir = dirname(__DIR__) . '/logs';
    if (!is_dir($logDir)) mkdir($logDir, 0755, true);
    $logFile = $logDir . '/ai_debug.log';
    $logData = json_decode(file_get_contents('php://input'), true);
    $logEntry = [
        'timestamp' => date('Y-m-d H:i:s'),
        'thread_id' => $logData['thread_id'] ?? '',
        'agent' => $logData['agent'] ?? '',
        'providers' => $logData['providers'] ?? [],
        'content_length' => $logData['content_length'] ?? 0,
        'estimated_tokens' => $logData['estimated_tokens'] ?? 0,
        'question' => $logData['question'] ?? '',
        'type' => $logData['type'] ?? 'request',  // 'request' or 'response'
        'response_preview' => substr($logData['response'] ?? '', 0, 500),
        'full_content' => $logData['full_content'] ?? '' // full payload for debug
    ];
    file_put_contents($logFile, json_encode($logEntry) . "\n", FILE_APPEND | LOCK_EX);
    echo json_encode(['success' => true]);
    break;
```

**JS:** Call this endpoint in `submitToAI()`:
1. **Before** sending to AI: log the request (type: 'request', full_content, providers, tokens)
2. **After** receiving response: log the response (type: 'response', response preview, costs, timing)

Log file: `/var/www/mssql.danmarcrm.com/dev1/extension2v1/logs/ai_debug.log` — one JSON line per entry.

#### 4G. Contact memory injection (bonus)
If `primaryContactId` exists, fetch memories from `api.php?action=ai_contact_memories&contact_id=X` and inject as "KNOWN FACTS ABOUT [CONTACT]" section in the content sent to AI. Display memories as pills in the context area.

---

## Implementation Order

1. **TASK 1** — Hide AI Thread Analysis (3 line changes in index.php + app.js)
2. **TASK 2** — Speed up main page (2 line changes in app.js)
3. **TASK 3** — Fix thread contact name bug (QueryAdapter.php — MongoDB + SQL)
4. **TASK 4** — Enhance thread_view.php:
   - 4A: Full-width layout (CSS changes)
   - 4B: Agent buttons + quick prompts (HTML + JS)
   - 4C: Context builder (HTML + JS)
   - 4D: Token estimation (HTML + JS)
   - 4E: System prompt injection (JS modification to submitToAI)
   - 4F: Debug logging endpoint (api.php + JS calls)
   - 4G: Contact memory injection (JS)

## Files Modified (summary)

| File | Tasks |
|------|-------|
| [public/index.php](public/index.php) | T1: hide analysis section |
| [public/assets/app.js](public/assets/app.js) | T1: disable refresh, T2: reduce limits |
| [src/QueryAdapter.php](src/QueryAdapter.php) | T3: fix contact name bug |
| [public/api.php](public/api.php) | T4F: add `log_ai_debug` endpoint |
| [public/thread_view.php](public/thread_view.php) | T4: all thread_view enhancements |

## Verification

1. **Main dashboard:** Load page — should be fast, no analysis section visible, threads show correct contact names (not all "Etienne Turner")
2. **Thread view:** Open a thread — should be full-width, agent buttons visible, context builder expandable
3. **Agents:** Click "Quotes" agent → quick prompts appear → click one → fills question box → submit → AI response guided by quote analysis system prompt
4. **Context builder:** Click "Search" → type contact name → see their threads → click "+Add" → green cart updates → submit → AI sees all added threads
5. **Token counter:** Shows live estimate → add more context → count increases → shows warning if near limit
6. **Multi-provider:** Select Claude + GPT → submit → both respond in tabs (existing behavior preserved)
7. **Debug log:** Check `/var/www/mssql.danmarcrm.com/dev1/extension2v1/logs/ai_debug.log` → has request + response JSON lines
8. **Conversation continuity:** Ask a follow-up question → AI sees previous Q&A in context (existing behavior preserved)
9. **Saved responses:** Response auto-saves to MongoDB → appears in "Previous AI Analyses" (existing behavior preserved)


Write

# Plan: Dashboard Enhancements — Delete, Filters, Sync, Charts, Model Updates

## Context

User tested the Two-Round AI feature and is happy with it. Now requests 10 enhancements covering: delete functionality, sorting, filters, contact name fix, sync-by-domain, bulk delete, Chrome extension manual mode, D3.js charts, Products tab hiding, and AI model version updates.

## Execution Order (6 Phases, quick wins first)

---

### PHASE 1: Trivial Changes

#### Task 9: Hide Products Tab
- **File:** `public/index.php`
- Comment out the Products tab button (`<button class="tab-btn" data-tab="products">`)
- Leave `product_search.php` and the tab content div intact

#### Task 10: Update AI Model Versions
- **File:** `aiapi/api_opus/config/api_keys.php` — Add `claude-opus-4-6` and `claude-sonnet-4-6` to Claude provider models, update `default_model` to `claude-sonnet-4-6`, add pricing entries
- **File:** `public/thread_view.php` — Add new models to `MODEL_LIMITS` JS constant
- **File:** `config/config.php` — Update `CLAUDE_MODEL` constant to `claude-sonnet-4-6`
- Also update `extension2/aiapi/api_opus/config/api_keys.php` to stay in sync

---

### PHASE 2: Delete Functionality (Tasks 1 + 6)

#### Task 1: Delete Contacts, Threads, AI Responses

**Backend (api.php + QueryAdapter.php):**
- Add `delete_contact` endpoint → calls new `QueryAdapter::deleteContact($contactId)` which cascades: deletes emails, thread_analysis (for threads with no remaining emails), contact_profiles, ai_contact_memory, ai_query_responses, and the contact record
- Add `delete_thread` endpoint → calls new `QueryAdapter::deleteThread($threadId)` which deletes all emails for that thread_id, thread_analysis, and ai_query_responses referencing the thread
- Add `delete_ai_response` endpoint → deletes single document from `ai_query_responses` by `_id`

**Frontend:**
- `app.js` — Add delete buttons in contacts list actions column and threads list. Add `deleteContact()` and `deleteThread()` JS functions with confirmation dialogs
- `thread_view.php` — Add delete button next to each saved AI response. Add `deleteAiResponse()` JS function that removes the DOM element on success

#### Task 6: Bulk Delete Threads by Email/Domain
- **api.php** — Add `delete_threads_by_email` endpoint accepting `email` or `domain` param
- **QueryAdapter.php** — Add `deleteThreadsByEmail($email, $domain)` method: finds all emails matching address/domain via aggregation, collects unique thread_ids, calls `deleteThread()` for each
- **app.js** — Add `deleteThreadsByEmail()` JS function, accessible from contact actions or a cleanup section

---

### PHASE 3: Sorting & Filters (Tasks 2 + 3)

#### Task 2: Sort by Last Activity
- **index.php** — Add sort dropdowns to Contacts toolbar (`Name A-Z / Z-A / Last Activity Newest / Oldest`) and Threads toolbar (`Last Activity / Most Emails / Newest Thread`)
- **app.js** — Read sort dropdown values in `loadContacts()` and `loadEmails()`, pass `sort` and `order` params to API
- **api.php** — Pass sort/order params through to QueryAdapter methods
- **QueryAdapter.php** — `getThreadsMongo()`: make sort field dynamic (already sorts by `last_date`, make it accept param). `getContactsWithProfilesMongo()`: add `$lookup` to emails to compute `last_contact` date for sorting

#### Task 3: Filters
- **index.php** — Add filter bar to Contacts tab (date range picker for last activity) and Threads tab (contact name, date range, analysis status dropdown)
- **app.js** — Read filter values in `loadContacts()` and `loadEmails()`, pass as API params. Add `clearContactFilters()` and `clearThreadFilters()` functions
- **api.php** — Pass filter params (date_from, date_to, contact_name, status) to QueryAdapter
- **QueryAdapter.php** — Add date range filtering to `getThreadsMongo()` match stage. Add contact_name pre-query (find matching contact_ids, add to `$in` filter). Add status filtering

---

### PHASE 4: Bug Fix (Task 4)

#### Task 4: Fix "Etienne Turner" Thread Linkage
The previous fix weights inbound contacts 2x, but has a gap: for outbound-only threads, `from_email` is always the user so all contacts get skipped.

**Fix in QueryAdapter.php `getThreadsMongo()`:**
- In `$group` stage: push `to_email` alongside `from_email` into `all_contacts` array:
  `'all_contacts' => ['$push' => ['cid' => '$contact_id', 'from' => '$from_email', 'to' => '$to_email', 'dir' => '$direction']]`
- In post-processing: for outbound emails, consider `to_email` (the recipient) as the external contact, weight 1x. For inbound, consider `from_email` (the sender), weight 2x
- Add fallback: if still null, look up contact by `to_email` from first outbound email

---

### PHASE 5: New Features (Tasks 5 + 8)

#### Task 5: Sync by Domain/Email Address
Two-step flow: Search Gmail → Preview results → Import selected threads.

- **index.php** — Add "Import Threads by Email/Domain" panel in dashboard with input field and search button, results container showing found threads with checkboxes and "Import All New" button
- **api.php** — Add `search_gmail_threads` endpoint: builds Gmail search query (`from:@domain OR to:@domain`), calls `GmailClient::searchThreads()`, checks which are already imported. Add `import_gmail_threads` endpoint: loops through selected thread_ids, reuses existing sync_thread logic to import each
- **app.js** — Add `searchByDomain()` and `importAllDomainThreads()` functions
- **GmailClient.php** — Verify `searchThreads()` method exists and returns thread ID + snippet (confirmed: line 577)

#### Task 8: D3.js Interactive Chart for AI Responses
- **thread_view.php** — Add D3.js v7 CDN in `<head>`
- Add "Generate Chart" button after AI response display
- Add `#d3-chart-container` div with 600px height, tooltip overlay
- `generateChart()` function: takes latest AI response text, sends to AI (via api_opus proxy) asking for structured JSON `{nodes: [{id, label, type, details}], links: [{source, target, label, strength}]}`
- `renderD3Chart(data)` function: creates SVG with force-directed graph, color-coded nodes (person=blue, company=green, topic=yellow, event=red), directional arrows, labels, drag/zoom/pan, hover tooltips

---

### PHASE 6: Notes Only (Task 7)

#### Task 7: Chrome Extension Manual-Only Mode
Chrome extension is on user's local Mac — cannot edit directly from server. Will provide code specifications:
- `popup.html/popup.js`: Add "Auto-sync / Manual only" toggle, stored in `chrome.storage.local`
- `background.js`: Check syncMode before auto-syncing, skip if manual
- `content.js`: In manual mode, inject floating "Sync this thread" button in Gmail UI

---

## Files Modified (Summary)

| File | Tasks | Key Changes |
|------|-------|-------------|
| `public/api.php` | 1, 2, 3, 5, 6 | 6 new endpoints (delete_contact, delete_thread, delete_ai_response, delete_threads_by_email, search_gmail_threads, import_gmail_threads), sort/filter param passing |
| `src/QueryAdapter.php` | 1, 2, 3, 4, 6 | 3 new methods (deleteContact, deleteThread, deleteThreadsByEmail), sort/filter support, fix all_contacts aggregation |
| `public/assets/app.js` | 1, 2, 3, 5, 6 | Delete functions, sort/filter wiring, domain search/import, clear filter functions |
| `public/index.php` | 2, 3, 5, 9 | Sort dropdowns, filter bars, sync-by-domain panel, hide Products tab |
| `public/thread_view.php` | 1, 8, 10 | Delete button for saved responses, D3.js chart feature, MODEL_LIMITS update |
| `config/config.php` | 10 | CLAUDE_MODEL constant update |
| `aiapi/api_opus/config/api_keys.php` | 10 | New Claude model entries + pricing |

## Verification

1. **Products tab**: Hidden from navigation, product_search.php still accessible directly
2. **AI models**: Claude Opus 4.6 / Sonnet 4.6 appear in provider dropdowns
3. **Delete contact**: Click delete → confirm → contact + all associated data removed, lists refresh
4. **Delete thread**: Click delete → confirm → thread + emails + analysis removed
5. **Delete AI response**: Click delete in thread_view → response row removed
6. **Bulk delete**: Enter email/domain → all matching threads deleted
7. **Sort**: Change sort dropdown → contacts/threads re-sort by last activity
8. **Filters**: Set date range / status → filtered results appear, Clear resets
9. **Contact name**: Threads show correct primary contact (not all "Etienne Turner")
10. **Sync by domain**: Enter domain → Gmail searched → preview shown → import selected → threads appear in dashboard
11. **D3.js chart**: Click "Generate Chart" → AI produces JSON → interactive force graph rendered with drag/zoom
12. **PHP syntax**: `php -l` passes on all modified PHP files