# Booking.com Scraper System - Complete Documentation

**Last Updated:** November 1, 2025  
**System Version:** v14With1CRM  
**Location:** `/var/www/mssql.danmarcrm.com/openai/v14With1CRM/`

---

## Table of Contents

1. [System Overview](#system-overview)
2. [Core Files](#core-files)
3. [How to Trigger Functions](#how-to-trigger-functions)
4. [Automated Scripts](#automated-scripts)
5. [Database Structure](#database-structure)
6. [Configuration](#configuration)
7. [Troubleshooting](#troubleshooting)

---

## System Overview

This system scrapes hotel and room data from Booking.com for the Otopeni, Romania area. It consists of two main components:

1. **Hotel & Room Details Scraper** - Fetches detailed information about individual hotels
2. **Hotel Position Tracker** - Tracks hotel search rankings/positions

### Recent Changes (November 2025)

Files have been renamed for better clarity:

| Old Name | New Name | Purpose |
|----------|----------|---------|
| `daily_booking_scraper_v14.php` | `getHotelAndRoomDetails.php` | Main hotel data scraper |
| `scraper.js` | `getHotelAndRoomDetails.js` | Puppeteer script for individual hotels |
| `fetch_positions_cli.php` | `getHotelsPosition.php` | CLI wrapper for position tracking |
| `search_results_fetcher.php` | `searchHotelsPosition.php` | Position fetcher class |
| `__dailystats.js` | `searchHotelsPosition.js` | Puppeteer script for search results |

---

## Core Files

### 1. getHotelAndRoomDetails.php

**Purpose:** Main scraper that fetches detailed hotel and room information from Booking.com.

**Features:**
- Fetches multiple days of data in one run
- Saves to `properties` and `room_details` tables
- Tracks room availability, pricing, and inventory
- Automatically calls position tracker for each date

**Database Tables:**
- `properties` - Main hotel information
- `room_details` - Individual room type details
- `propsid` - Hotel ID mapping

**Class:** `BookingScraperV14`

**Key Methods:**
- `fetchAndSaveSearchResults()` - Fetches hotel positions
- `scrapeBooking()` - Scrapes individual hotel details
- `saveProperty()` - Saves hotel data
- `saveRooms()` - Saves room details

---

### 2. getHotelAndRoomDetails.js

**Purpose:** Node.js Puppeteer script that fetches individual hotel details by navigating to each hotel's page.

**Features:**
- Anti-bot detection evasion
- User-agent rotation
- Request blocking for faster loading
- Extracts room types, pricing, availability

**Usage:**
```bash
node getHotelAndRoomDetails.js <hotel_id> <checkin> <checkout> <persons> [blockResources]
```

**Example:**
```bash
node getHotelAndRoomDetails.js 5199691 2025-11-01 2025-11-03 2 true
```

**Output:** JSON with hotel and room details

---

### 3. getHotelsPosition.php

**Purpose:** Command-line wrapper for fetching hotel search positions (rankings).

**Features:**
- Fetches search results with position tracking
- Saves to `properties_order` table
- Logs to `logs2/fetch_positions.log`

**Database Table:**
- `properties_order` - Hotel positions in search results

**Class Used:** `SearchResultsFetcher`

---

### 4. searchHotelsPosition.php

**Purpose:** PHP class that handles fetching and parsing search results with position tracking.

**Features:**
- Executes `searchHotelsPosition.js`
- Parses GraphQL API responses
- Saves hotel positions to database
- Tracks price and availability in search results

**Class:** `SearchResultsFetcher`

**Key Methods:**
- `fetchAndSave()` - Main execution method
- `parseDailystatsOutput()` - Parses JSON from JS script
- `saveHotel()` - Saves hotel with position

---

### 5. searchHotelsPosition.js

**Purpose:** Node.js Puppeteer script that fetches Booking.com search results using GraphQL API.

**Features:**
- Uses Booking.com GraphQL endpoint
- Fetches all pages of search results
- Anti-bot evasion techniques
- Returns hotels with positions

**Search Configuration:**
- **Location:** Otopeni, Romania
- **destId:** -1165889
- **destType:** CITY

**Usage:**
```bash
node searchHotelsPosition.js <checkin> <checkout>
```

**Example:**
```bash
node searchHotelsPosition.js 2025-11-01 2025-11-03
```

**Output:** JSON with hotel list and positions

---

## How to Trigger Functions

### Method 1: Direct PHP Execution

#### A. Fetch Hotel Details for a Date Range

```bash
php getHotelAndRoomDetails.php 'from=today&days=7&stay_nights=2&pers=2'
```

**Parameters:**
- `from` - Start date (`today`, `onemonth`, `twomonth`, or specific date)
- `days` - Number of days to fetch (default: 7)
- `stay_nights` - Length of stay in nights (default: 2)
- `pers` - Number of persons (default: 2)
- `limit` - Max hotels per date (default: 200)
- `forceReset` - Force re-fetch (1 or 0)

**Examples:**
```bash
# Fetch today + 31 days, 2-night stays, 2 persons
php getHotelAndRoomDetails.php 'from=today&days=31&stay_nights=2&pers=2'

# Fetch starting 1 month from now, 1-night stays, 4 persons
php getHotelAndRoomDetails.php 'from=onemonth&days=14&stay_nights=1&pers=4'

# Specific date range
php getHotelAndRoomDetails.php 'checkin=2025-12-01&checkout=2025-12-03&pers=2'
```

#### B. Fetch Hotel Positions Only

```bash
php getHotelsPosition.php <checkin> <checkout> <persons>
```

**Examples:**
```bash
# Fetch positions for Nov 1-3, 2 persons
php getHotelsPosition.php 2025-11-01 2025-11-03 2

# Fetch positions for Dec 6-8, 4 persons
php getHotelsPosition.php 2025-12-06 2025-12-08 4
```

---

### Method 2: Automated Shell Scripts

#### A. Daily Scheduler (Cron Job)

**Script:** `__dailyBookingScheduler.sh`

**What it does:**
- Runs automatically via cron
- Fetches 2 different configurations simultaneously

**Configurations:**
1. 2 persons, 2-night stays, 31 days ahead
2. 4 persons, 1-night stays, 31 days ahead

**How to trigger manually:**
```bash
cd /var/www/mssql.danmarcrm.com/openai/v14With1CRM
./__dailyBookingScheduler.sh
```

**Cron Setup:**
```bash
# Run daily at 2 AM
0 2 * * * /var/www/mssql.danmarcrm.com/openai/v14With1CRM/__dailyBookingScheduler.sh
```

---

#### B. Fetch Next 15 Days

**Script:** `fetch_next_15_days.sh`

**What it does:**
- Fetches data for the next 15 days
- 2-night stays, 2 persons
- All 200+ properties
- Comprehensive logging

**How to trigger:**
```bash
cd /var/www/mssql.danmarcrm.com/openai/v14With1CRM
./fetch_next_15_days.sh
```

**Configuration in script:**
- `START_DATE`: Starting date (default: today)
- `DAYS`: Number of days (default: 15)
- `STAY_NIGHTS`: Length of stay (default: 2)
- `PERSONS`: Number of guests (default: 2)
- `LIMIT`: Max properties (default: 200)

**Logs:** `logs/fetch_15days_YYYYMMDD_HHMMSS.log`

---

#### C. Fetch Both Types (Comprehensive)

**Script:** `fetch_both_types.sh`

**What it does:**
- Fetches BOTH hotel details AND positions
- Runs in parallel for efficiency
- Customizable parameters

**How to trigger:**
```bash
cd /var/www/mssql.danmarcrm.com/openai/v14With1CRM
./fetch_both_types.sh [from] [days] [stay_nights] [pers] [limit]
```

**Examples:**
```bash
# Default: today + 7 days, 2-night stays, 2 persons
./fetch_both_types.sh

# Custom: 1 month ahead, 14 days, 1-night stays, 4 persons
./fetch_both_types.sh onemonth 14 1 4 200
```

**What happens:**
1. Starts hotel details fetch in background
2. Fetches positions for each day sequentially
3. Logs everything

**Logs:**
- Hotel details: `logs2/fetch_booking.log`
- Positions: `logs2/fetch_positions.log`
- Main script: `logs2/fetch_both_types.log`

---

#### D. Daily Otopeni Fetcher

**Script:** `daily_otopeni_fetcher.sh`

**What it does:**
- Similar to fetch_next_15_days.sh
- Legacy script, still functional

**How to trigger:**
```bash
cd /var/www/mssql.danmarcrm.com/openai/v14With1CRM
./daily_otopeni_fetcher.sh
```

---

### Method 3: Direct Node.js Execution

#### A. Fetch Search Results with Positions

```bash
cd /var/www/mssql.danmarcrm.com/openai/v14With1CRM
node searchHotelsPosition.js 2025-11-01 2025-11-03
```

**Output:** JSON with all hotels in search results, including positions

**Use Cases:**
- Testing search results
- Debugging GraphQL queries
- Verifying search configuration

---

#### B. Fetch Individual Hotel Details

```bash
cd /var/www/mssql.danmarcrm.com/openai/v14With1CRM
node getHotelAndRoomDetails.js 5199691 2025-11-01 2025-11-03 2 true
```

**Parameters:**
1. Hotel ID (e.g., 5199691)
2. Check-in date (YYYY-MM-DD)
3. Check-out date (YYYY-MM-DD)
4. Number of persons
5. Block resources (true/false) - speeds up loading

**Output:** JSON with hotel and room details

**Use Cases:**
- Testing individual hotel fetching
- Debugging room extraction
- Verifying data structure

---

### Method 4: Web Interface (Dashboard)

#### A. Booking Dashboard

**URL:** `https://mssql.danmarcrm.com/openai/v14With1CRM/booking_dashboard.php`

**Features:**
- Start/stop fetches via web UI
- Set parameters via form
- View progress in real-time

**How to use:**
1. Open URL in browser
2. Fill in parameters (dates, persons, etc.)
3. Click "Start Fetch"
4. Monitor progress

---

#### B. Booking Monitor

**URL:** `https://mssql.danmarcrm.com/openai/v14With1CRM/booking_monitor.php`

**Features:**
- View fetch status (running/stopped)
- See latest statistics
- Start/stop automated fetches
- View recent fetch results

**Dashboard Metrics:**
- Total dates tracked
- Total hotels
- Total rooms
- Average price

---

#### C. Booking Analytics

**URL:** `https://mssql.danmarcrm.com/openai/v14With1CRM/booking_analytics.php`

**Features:**
- Detailed analytics and charts
- Price trends
- Availability analysis
- Booking predictions

---

## Automated Scripts

### Complete Script Reference

| Script | Purpose | Trigger Method |
|--------|---------|----------------|
| `__dailyBookingScheduler.sh` | Daily automated fetch (cron) | Automatic or `./script.sh` |
| `fetch_next_15_days.sh` | Fetch 15 days ahead | `./fetch_next_15_days.sh` |
| `fetch_next_14_days.sh` | Fetch 14 days ahead | `./fetch_next_14_days.sh` |
| `fetch_both_types.sh` | Fetch details + positions | `./fetch_both_types.sh [params]` |
| `daily_otopeni_fetcher.sh` | Legacy daily fetcher | `./daily_otopeni_fetcher.sh` |

---

## Database Structure

### Tables Used

#### 1. properties
Stores main hotel information from search results.

**Key Fields:**
- `id` - Hotel ID (primary key)
- `name` - Hotel name
- `checkin` - Check-in date
- `checkout` - Check-out date
- `total_rooms` - Total available rooms
- `no_availability` - 0=available, 1=sold out
- `query_date` - When data was fetched

---

#### 2. room_details
Stores individual room type information.

**Key Fields:**
- `id` - Auto-increment ID
- `hotel_id` - Foreign key to properties
- `checkin` - Check-in date
- `checkout` - Check-out date
- `room_name` - Room type name
- `max_persons` - Maximum occupancy
- `has_inventory` - 1=available, 0=not available
- `avg_price_per_night_eur` - Price per night
- `raw_price` - Original price
- `created_at` - Timestamp

---

#### 3. properties_order
Stores hotel positions in search results.

**Key Fields:**
- `id` - Hotel ID
- `name` - Hotel name
- `pos` - Position in search results (1-based)
- `checkin` - Check-in date
- `checkout` - Check-out date
- `staying_nights` - Length of stay
- `price_per_stay` - Total price for stay
- `is_sold_out` - 0=available, 1=sold out
- `query_date` - When data was fetched

---

#### 4. propsid
Hotel ID mapping and metadata.

**Key Fields:**
- `id` - Hotel ID
- `type` - Property type (apartment/hotel)
- Additional metadata fields

---

## Configuration

### Database Configuration

**Location:** Defined in `getHotelAndRoomDetails.php`

```php
define('STANDALONE_DB_CONFIG', [
    'host' => 'localhost',
    'name' => 'booking_analysis',
    'user' => 'root',
    'pass' => 'danmar1234'
]);
```

---

### Search Location Configuration

**File:** `searchHotelsPosition.js` (lines 83-88)

```javascript
"location": {
    "hotelIds": [2681021],
    "searchString": "Otopeni, Romania",
    "destType": "CITY",
    "destId": -1165889
}
```

**Current Configuration:**
- **Area:** Otopeni, Romania
- **Destination ID:** -1165889
- **Type:** City search

**To Change Location:**
1. Edit `searchHotelsPosition.js`
2. Update `searchString`, `destId`, and `destType`
3. Find correct destId by searching Booking.com manually

---

### Logging Configuration

**Main Logs Directory:** `logs2/`

**Log Files:**
- `fetch_booking.log` - Hotel details fetch log
- `fetch_positions.log` - Position tracking log
- `searchHotelsPosition_error.log` - JS errors from position script
- `scraper_error.log` - JS errors from hotel scraper

**Daily Logs:** `logs/fetch_15days_YYYYMMDD_HHMMSS.log`

---

## Troubleshooting

### Common Issues and Solutions

#### 1. "No output from searchHotelsPosition.js"

**Cause:** Node.js or Puppeteer error

**Solutions:**
```bash
# Check Node.js version
node --version  # Should be v14+

# Check if Puppeteer is installed
cd /var/www/mssql.danmarcrm.com/openai/v14With1CRM
npm list puppeteer

# Reinstall if needed
npm install puppeteer
```

**Check logs:**
```bash
tail -f logs2/searchHotelsPosition_error.log
```

---

#### 2. Database Connection Errors

**Error:** "Access denied for user 'root'@'localhost'"

**Solutions:**
```bash
# Test database connection
mysql -u root -p'danmar1234' booking_analysis -e "SELECT COUNT(*) FROM properties;"

# Check if database exists
mysql -u root -p'danmar1234' -e "SHOW DATABASES LIKE 'booking_analysis';"
```

---

#### 3. PHP Memory Limit Exceeded

**Error:** "Allowed memory size exhausted"

**Solutions:**
```bash
# Increase memory limit in command
php -d memory_limit=512M getHotelAndRoomDetails.php 'from=today&days=7&pers=2'

# Or edit php.ini
memory_limit = 512M
```

---

#### 4. Script Permissions

**Error:** "Permission denied"

**Solutions:**
```bash
# Make scripts executable
chmod +x fetch_both_types.sh
chmod +x fetch_next_15_days.sh
chmod +x __dailyBookingScheduler.sh
```

---

#### 5. Puppeteer Browser Not Found

**Error:** "Could not find expected browser"

**Solutions:**
```bash
# Run Puppeteer installation script
cd /var/www/mssql.danmarcrm.com/openai/v14With1CRM
./install-puppeteer-universal.sh
```

---

### Monitoring Active Fetches

#### Check if fetch is running:
```bash
ps aux | grep -E 'getHotelAndRoomDetails|fetch_next_15_days'
```

#### Check last 50 log entries:
```bash
tail -50 logs2/fetch_booking.log
```

#### Watch live updates:
```bash
tail -f logs2/fetch_booking.log
```

#### Kill stuck processes:
```bash
# Kill all booking fetches
pkill -f 'getHotelAndRoomDetails'
pkill -f 'fetch_next_15_days'
```

---

## Quick Reference Commands

### Most Common Operations

#### 1. Fetch Today + 30 Days (2 persons, 2 nights)
```bash
cd /var/www/mssql.danmarcrm.com/openai/v14With1CRM
php getHotelAndRoomDetails.php 'from=today&days=30&stay_nights=2&pers=2'
```

#### 2. Fetch Positions for Specific Date
```bash
php getHotelsPosition.php 2025-12-25 2025-12-27 2
```

#### 3. Run Complete Fetch (Details + Positions)
```bash
./fetch_both_types.sh today 7 2 2 200
```

#### 4. Check Latest Database Entries
```bash
mysql -u root -p'danmar1234' booking_analysis -e "
SELECT checkin, COUNT(DISTINCT id) as hotels 
FROM properties 
WHERE checkin >= CURDATE() 
GROUP BY checkin 
ORDER BY checkin 
LIMIT 10;
"
```

#### 5. View Recent Logs
```bash
# Last 100 lines of main fetch log
tail -100 logs2/fetch_booking.log

# Last 100 lines of position log
tail -100 logs2/fetch_positions.log
```

---

## API Integration

### REST API Endpoints

#### 1. Trigger Fetch via API

**File:** `booking_fetch_controller.php`

**POST Endpoint:**
```
POST /openai/v14With1CRM/booking_fetch_controller.php
```

**Parameters:**
- `action=start_fetch`
- `from=today`
- `days=7`
- `stay_nights=2`
- `pers=2`
- `limit=200`
- `force_reset=1` (optional)

**Example using curl:**
```bash
curl -X POST https://mssql.danmarcrm.com/openai/v14With1CRM/booking_fetch_controller.php \
  -d "action=start_fetch" \
  -d "from=today" \
  -d "days=7" \
  -d "stay_nights=2" \
  -d "pers=2" \
  -d "limit=200"
```

---

#### 2. Trigger Position Fetch via API

**POST Endpoint:**
```
POST /openai/v14With1CRM/booking_fetch_controller.php
```

**Parameters:**
- `action=start_position_fetch`
- `checkin=2025-11-01`
- `checkout=2025-11-03`
- `pers=2`

**Example using curl:**
```bash
curl -X POST https://mssql.danmarcrm.com/openai/v14With1CRM/booking_fetch_controller.php \
  -d "action=start_position_fetch" \
  -d "checkin=2025-11-01" \
  -d "checkout=2025-11-03" \
  -d "pers=2"
```

---

## System Requirements

### Software Requirements

- **PHP:** 7.4 or higher
- **Node.js:** 14.x or higher
- **MySQL:** 5.7 or higher
- **Puppeteer:** Latest version
- **Operating System:** Linux (Ubuntu/Debian recommended)

### Package Dependencies

**PHP Packages:**
- PDO MySQL extension
- JSON extension
- cURL extension

**Node.js Packages:**
- puppeteer
- (all dependencies in package.json)

---

## Support and Maintenance

### Regular Maintenance Tasks

#### 1. Clean Old Logs (Monthly)
```bash
cd /var/www/mssql.danmarcrm.com/openai/v14With1CRM/logs2
# Archive logs older than 30 days
find . -name "*.log" -mtime +30 -exec gzip {} \;
```

#### 2. Database Optimization (Weekly)
```bash
mysql -u root -p'danmar1234' booking_analysis << EOF
OPTIMIZE TABLE properties;
OPTIMIZE TABLE room_details;
OPTIMIZE TABLE properties_order;
EOF
```

#### 3. Update Puppeteer (As Needed)
```bash
cd /var/www/mssql.danmarcrm.com/openai/v14With1CRM
npm update puppeteer
```

---

## File Structure Summary

```
/var/www/mssql.danmarcrm.com/openai/v14With1CRM/
├── Core PHP Files
│   ├── getHotelAndRoomDetails.php        # Main scraper
│   ├── getHotelsPosition.php             # Position CLI wrapper
│   └── searchHotelsPosition.php          # Position fetcher class
│
├── Core JavaScript Files
│   ├── getHotelAndRoomDetails.js         # Hotel details Puppeteer script
│   └── searchHotelsPosition.js           # Search results Puppeteer script
│
├── Automation Scripts
│   ├── __dailyBookingScheduler.sh        # Daily cron job
│   ├── fetch_both_types.sh               # Fetch details + positions
│   ├── fetch_next_15_days.sh             # Fetch 15 days ahead
│   ├── fetch_next_14_days.sh             # Fetch 14 days ahead
│   └── daily_otopeni_fetcher.sh          # Legacy fetcher
│
├── Web Interfaces
│   ├── booking_dashboard.php             # Main dashboard
│   ├── booking_monitor.php               # Monitor interface
│   ├── booking_analytics.php             # Analytics interface
│   └── booking_fetch_controller.php      # API controller
│
├── Test Files
│   ├── test_fetcher.php
│   ├── test_position_scraping.php
│   ├── test_search_fetcher.php
│   └── test_search_results.php
│
└── Logs
    ├── logs/                              # Daily fetch logs
    └── logs2/                             # Main log directory
        ├── fetch_booking.log
        ├── fetch_positions.log
        ├── searchHotelsPosition_error.log
        └── scraper_error.log
```

---

## Version History

### November 1, 2025 - v14 File Rename
- Renamed all core files for clarity
- Updated searchString to "Otopeni, Romania"
- Updated all 19 files with references
- Created comprehensive documentation

### Previous Versions
- v14With1CRM - Integration with 1CRM
- v14 - GraphQL API implementation
- v13 - Enhanced room details
- v12 and earlier - Legacy versions

---

## Contact and Support

For issues, questions, or feature requests:
- Check logs in `logs2/` directory
- Review this documentation
- Test with individual scripts before running automated jobs

---

**End of Documentation**



