# DaemonControl

A Linux daemon system for job scheduling and system monitoring with a Qt-based GUI.

## Project Status

**Current Milestone: 4 of 6 - GUI Complete** ✅

### Roadmap

- [x] **Milestone 1**: Foundation Infrastructure (Database, Config, Logging)
- [x] **Milestone 2**: Daemon Process (Scheduler, Job Execution)
- [x] **Milestone 3**: System Tray Integration
- [x] **Milestone 3.5**: Fleet Monitoring System (Docker, Tunnel, Backup, Disk)
- [x] **Milestone 4**: GUI - Job Management & Dashboard
- [ ] **Milestone 5**: Advanced Features (Alerts, Export, History Graphs)
- [ ] **Milestone 6**: Performance Optimization & Polish

## Features

### Milestone 1: Foundation Infrastructure ✅
- SQLite database with WAL mode and schema versioning
- JSON configuration with path expansion
- Structured logging with daily rotation
- Singleton pattern for thread safety

### Milestone 2: Daemon Base ✅
- **APScheduler Integration**: Job scheduling with cron expressions
- **Job Execution Engine**: Subprocess management with timeout handling
- **Execution Tracking**: Complete execution history with logs
- **Concurrent Execution**: Configurable max concurrent jobs
- **Graceful Shutdown**: Ctrl+C handling with cleanup
- **Venv Detection**: Automatic Python interpreter discovery
- **Log Management**: Separate log files per execution

### Milestone 3: System Tray Integration ✅
- **System Tray Icon**: Visual status indicator (green/gray/red)
- **Context Menu**: Right-click menu for all controls
- **Start/Stop Daemon**: Control daemon from tray
- **Reload Jobs**: Reload job configurations without full restart
- **Quick Access**: Open logs folder with one click
- **Desktop Notifications**: Status change notifications
- **Background Operation**: Daemon runs in background

### Milestone 3.5: Fleet Monitoring System ✅
- **Docker Monitoring**: Track all 14 containers status
- **WireGuard Tunnel**: TCP-based connectivity check (no root required)
- **Backup Monitoring**: Lightweight filesystem-based Restic snapshot check
- **Disk Monitoring**: Track usage on SSD, HDD, and root partitions
- **Database-Cached Status**: Daemon writes, GUI reads for instant display
- **Smart Tray Icon**: Color changes based on overall fleet health
- **Scheduled Jobs**: Automatic monitoring every 5-15 minutes
- **Fleet Tab GUI**: Real-time dashboard with all system metrics

### Milestone 4: GUI - Job Management & Dashboard ✅
- **Main GUI Window**: Full PyQt6 application with tabbed interface
- **Fleet Tab**: Complete fleet monitoring dashboard with real-time status
- **Jobs Tab**: Full job management (create, edit, delete, view)
- **Dashboard Tab**: Overview and statistics
- **History Tab**: Execution history viewer with filtering
- **Job Dialog**: Rich form for creating/editing jobs with validation
- **Menu Bar**: File, Daemon, View, Help menus with shortcuts
- **Status Bar**: Real-time status updates
- **Signal Integration**: Cross-tab communication and updates

## Installation

### Prerequisites
- Python 3.8 or higher
- Linux operating system

### Setup

1. Clone the repository:
```bash
cd /mnt/ssd/data/python-lab/DaemonControl
```

2. Create virtual environment:
```bash
python3 -m venv venv
source venv/bin/activate
```

3. Install dependencies:
```bash
pip install -r requirements.txt
pip install pytest  # For running tests
```

4. Generate system tray icons:
```bash
python3 gui/resources/create_icons.py
```

## Usage Modes

DaemonControl can run in two modes:

### System Tray Mode (Recommended)

Start with system tray icon for easy control:

```bash
python3 start_tray.py
```

Features:
- Tray icon shows daemon status (🟢 running, ⚪ stopped, 🔴 error)
- Right-click menu for all controls
- Desktop notifications for status changes
- Daemon runs in background

**Menu Options:**
- **Start Daemon** - Start background daemon
- **Stop Daemon** - Gracefully stop daemon
- **Reload Jobs** - Reload job configurations
- **Open Logs Folder** - Quick access to logs
- **Exit** - Stop daemon and close tray

### Foreground Mode (For Testing/Debugging)

Run daemon directly in terminal:

```bash
python3 start_daemon.py
```

Use this mode for:
- Debugging daemon issues
- Viewing real-time logs
- Testing without GUI dependencies
- Ctrl+C to stop

## Quick Start

### 1. Create and Schedule a Job

```python
from core import DatabaseManager
from datetime import datetime

db = DatabaseManager()

# Create a job
job_id = db.create_job(
    name="backup_job",
    job_type="script",
    executable_path="/usr/local/bin/backup.sh",
    working_directory="/home/user/backups",
    timeout=7200,
    description="Daily backup script"
)

# Add a schedule (cron expression: daily at 2 AM)
now = datetime.now().isoformat()
with db.get_connection() as conn:
    conn.execute(
        "INSERT INTO schedules (job_id, cron_expression, enabled, created_at, updated_at) "
        "VALUES (?, ?, 1, ?, ?)",
        (job_id, "0 2 * * *", now, now)
    )

print(f"Created and scheduled job '{job_id}'")
```

### 2. Start the Daemon

```bash
# Start daemon (runs in foreground)
python3 start_daemon.py

# You should see:
# - "Starting DaemonControl daemon..."
# - "Loaded X jobs"
# - Jobs will execute on schedule
# Press Ctrl+C to stop
```

### 3. Monitor Execution

```bash
# Watch daemon log
tail -f ~/.config/daemon-control/logs/daemon.log

# Watch job execution logs
tail -f ~/.config/daemon-control/logs/executions/*.log

# Query execution history
python3 -c "
from core import DatabaseManager
db = DatabaseManager()
with db.get_connection() as conn:
    cursor = conn.execute('SELECT * FROM executions ORDER BY start_time DESC LIMIT 5')
    for row in cursor:
        print(dict(row))
"
```

### Example: Simple Test Job

```python
from core import DatabaseManager
from datetime import datetime

db = DatabaseManager()

# Create a test job that runs every minute
job_id = db.create_job(
    name="test_echo",
    job_type="script",
    executable_path="/usr/bin/echo",
    description="Test job that echoes hello"
)

# Schedule to run every minute
now = datetime.now().isoformat()
with db.get_connection() as conn:
    conn.execute(
        "INSERT INTO schedules (job_id, cron_expression, enabled, created_at, updated_at) "
        "VALUES (?, ?, 1, ?, ?)",
        (job_id, "* * * * *", now, now)
    )

print("Test job created. Start daemon to see it run!")
```

## Project Structure

```
DaemonControl/
├── requirements.txt          # Python dependencies
├── README.md                 # This file
├── INSTALL.md               # Installation guide
├── .gitignore               # Git ignore patterns
├── start_tray.py            # System tray entry point (M3)
├── start_daemon.py          # Foreground daemon entry point (M2)
├── run_gui.sh               # GUI launcher script (M3.5)
├── create_test_jobs.py      # Test job creation script (M2)
├── example_usage.py         # Usage examples (M1)
├── config/
│   └── default_config.json  # Default configuration template
├── core/                    # Core infrastructure modules (M1)
│   ├── __init__.py         # Package exports
│   ├── database.py         # DatabaseManager - SQLite management
│   ├── config.py           # ConfigManager - Configuration handling
│   └── logger.py           # setup_logger - Logging setup
├── daemon/                  # Daemon modules (M2)
│   ├── __init__.py         # Package exports
│   ├── scheduler_daemon.py # Main daemon orchestrator
│   └── job_executor.py     # Job execution engine
├── modules/                 # Feature modules ⭐ NEW (M3.5)
│   └── fleet/              # Fleet monitoring modules
│       ├── __init__.py     # Package exports
│       ├── docker_monitor.py    # Docker container monitoring
│       ├── tunnel_monitor.py    # WireGuard tunnel monitoring
│       ├── backup_monitor.py    # Restic backup monitoring
│       ├── disk_monitor.py      # Disk usage monitoring
│       └── fleet_status.py      # Unified fleet status orchestrator
├── jobs/                    # Job scripts ⭐ NEW (M3.5)
│   ├── fleet_monitor.py    # Fleet monitoring job (every 5 min)
│   ├── disk_monitor.py     # Disk monitoring job (every 15 min)
│   └── backup_monitor.py   # Backup monitoring job (hourly)
├── gui/                     # GUI modules (M3, M3.5, M4)
│   ├── __init__.py         # Package exports
│   ├── main_window.py      # Main GUI window (M3.5)
│   ├── fleet_tab.py        # Fleet monitoring tab (M3.5)
│   ├── jobs_tab.py         # Jobs management tab ⭐ NEW (M4)
│   ├── dashboard_tab.py    # Dashboard overview tab ⭐ NEW (M4)
│   ├── history_tab.py      # Execution history tab ⭐ NEW (M4)
│   ├── job_dialog.py       # Job create/edit dialog ⭐ NEW (M4)
│   ├── system_tray.py      # System tray application (M3)
│   └── resources/          # GUI resources
│       ├── create_icons.py # Icon generation script
│       ├── icon_active.png # Green icon (daemon running)
│       ├── icon_inactive.png # Gray icon (daemon stopped)
│       └── icon_error.png  # Red icon (daemon error)
└── tests/                   # Test suite
    ├── test_foundation.py  # Foundation tests (M1)
    ├── test_daemon.py      # Daemon tests (M2)
    └── test_system_tray.py # System tray tests (M3)
```

## Configuration

Configuration file location: `~/.config/daemon-control/config.json`

Default configuration:
```json
{
  "database": {
    "path": "~/.config/daemon-control/daemon.db"
  },
  "logging": {
    "level": "INFO",
    "daemon_log_file": "~/.config/daemon-control/logs/daemon.log",
    "execution_log_dir": "~/.config/daemon-control/logs/executions",
    "retention_days": 30,
    "rotation": "daily"
  },
  "daemon": {
    "max_concurrent_jobs": 3,
    "default_timeout": 3600,
    "check_interval": 60
  }
}
```

## Testing

Run the test suite:

```bash
# Activate virtual environment
source venv/bin/activate

# Run all tests
pytest tests/ -v

# Run foundation tests only
pytest tests/test_foundation.py -v

# Run daemon tests only
pytest tests/test_daemon.py -v

# Run with coverage
pytest tests/ --cov=core --cov=daemon --cov-report=html
```

### Test Coverage

**Foundation Tests** (`test_foundation.py`):
- Database creation and schema validation
- Singleton pattern enforcement
- Job CRUD operations
- Execution tracking
- Configuration loading and path expansion
- Logger file creation and levels
- Integration workflows

**Daemon Tests** (`test_daemon.py`):
- JobExecutor initialization and execution
- Python executable detection (venv support)
- Command building for different job types
- SchedulerDaemon initialization
- Job loading and scheduling with APScheduler
- Integration testing

## Validation

Verify the installation:

```bash
# Test module imports
python3 -c "from core import DatabaseManager; db = DatabaseManager(); print('✓ Database OK')"

# Test configuration
python3 -c "from core import ConfigManager; cfg = ConfigManager(); print(f'✓ Config OK: {cfg.get(\"database\", \"path\")}')"

# Test logger
python3 -c "from core import setup_logger; logger = setup_logger('test'); logger.info('✓ Logger OK')"
```

## Fleet Monitoring

The Fleet Monitoring system provides real-time visibility into your infrastructure health.

### What is Monitored

**Docker Containers (every 5 minutes)**
- Status of all 14 containers
- Container names and states
- Overall health: healthy if all running

**WireGuard Tunnel (every 5 minutes)**
- TCP connectivity check to VPS (10.0.0.1:22)
- Latency measurement (connection time)
- Packet loss detection (3 attempts)
- No root privileges required (uses TCP socket instead of ICMP ping)

**Restic Backups (every hour)**
- Lightweight filesystem-based check
- Last snapshot age (healthy if < 30 hours)
- Snapshot count
- No password or restic command needed (just file modification time)

**Disk Usage (every 15 minutes)**
- Hub SSD (`/mnt/ssd`)
- Cassaforte HDD (`/mnt/backup-hdd`)
- Root partition (`/`)
- Warning thresholds: 80% (warning), 90% (critical)

### How It Works

1. **Daemon Background Jobs**: Three monitoring jobs run automatically:
   - `fleet_monitor` - Docker + Tunnel (every 5 min)
   - `disk_monitor` - Disk usage (every 15 min)
   - `backup_monitor` - Backup status (hourly)

2. **Database Cache**: Jobs write results to `fleet_status` table

3. **GUI Display**: Fleet tab reads from database for instant display (no live checks)

4. **Smart Tray Icon**:
   - 🟢 Green = All systems healthy
   - ⚪ Gray = Warning or data stale
   - 🔴 Red = Critical issues

### Viewing Fleet Status

**GUI Method:**
```bash
./run_gui.sh
# Click "Fleet" tab
```

**Command Line:**
```bash
python3 -c "
from core.database import DatabaseManager
db = DatabaseManager()
status = db.get_latest_fleet_status()
for check_type, data in status.items():
    print(f'{check_type}: {data[\"status\"]} - {data[\"message\"]}'
)
"
```

### Fleet Monitoring Jobs

All jobs are pre-configured in the database:

```bash
# View job status
python3 -c "
import sqlite3
conn = sqlite3.connect('~/.config/daemon-control/daemon.db'.replace('~', '$HOME'))
cursor = conn.execute('SELECT name, enabled FROM jobs WHERE name LIKE \"%monitor\"')
for row in cursor:
    print(f'{row[0]}: {\"enabled\" if row[1] else \"disabled\"}'
)
"
```

Jobs are automatically scheduled when daemon starts. No manual configuration needed.

## GUI Application

The full GUI application provides comprehensive management and monitoring capabilities.

### Launching the GUI

```bash
./run_gui.sh
```

Or with the virtual environment:
```bash
venv/bin/python gui/main_window.py
```

### GUI Tabs

**Fleet Tab (🚢)**
- Real-time fleet monitoring dashboard
- Docker containers status (all 14 containers)
- WireGuard tunnel connectivity and latency
- Restic backup status and snapshot age
- Disk usage with visual progress bars
- Overall system health indicator
- Data refreshed from database (instant load)

**Dashboard Tab (📊)**
- System overview and statistics
- Job execution summaries
- Quick status indicators
- Performance metrics

**Jobs Tab (⚙️)**
- View all scheduled jobs in table format
- Create new jobs with validation
- Edit existing job configurations
- Delete jobs with confirmation
- Enable/disable jobs
- View job schedules and next run time
- Double-click to edit, right-click for context menu

**History Tab (📜)**
- Complete execution history
- Filter by job name, status, date range
- View execution logs and output
- Sort by any column
- Export execution data

### GUI Features

**Job Management:**
- **Create Job**: File → New Job (Ctrl+N) or Jobs tab → Create button
- **Edit Job**: Select job, click Edit or double-click
- **Delete Job**: Select job, click Delete (with confirmation)
- **Job Dialog**: Rich form with validation for all job properties

**Menu Bar:**
- **File**: New Job (Ctrl+N), Exit (Ctrl+Q)
- **Daemon**: Start, Stop, Reload, Status
- **View**: Refresh (F5), Open Logs Folder
- **Help**: About

**Status Bar:**
- Real-time status updates
- Daemon connection state
- Last update timestamp

## Database Schema

### Tables

**schema_version**
- `version` (INTEGER PRIMARY KEY): Schema version number
- `applied_at` (TEXT): When version was applied

**jobs**
- `id` (INTEGER PRIMARY KEY): Unique job ID
- `name` (TEXT UNIQUE): Job name
- `job_type` (TEXT): 'script' or 'python_module'
- `executable_path` (TEXT): Path to executable
- `working_directory` (TEXT): Working directory
- `timeout` (INTEGER): Timeout in seconds
- `enabled` (INTEGER): 1=enabled, 0=disabled
- `created_at` (TEXT): Creation timestamp
- `updated_at` (TEXT): Last update timestamp
- `description` (TEXT): Job description

**schedules**
- `id` (INTEGER PRIMARY KEY): Schedule ID
- `job_id` (INTEGER FK): Associated job
- `cron_expression` (TEXT): Cron schedule
- `enabled` (INTEGER): 1=enabled, 0=disabled
- `created_at` (TEXT): Creation timestamp
- `updated_at` (TEXT): Last update timestamp

**executions**
- `id` (INTEGER PRIMARY KEY): Execution ID
- `job_id` (INTEGER FK): Associated job
- `status` (TEXT): 'queued', 'running', 'success', 'failed', 'timeout'
- `start_time` (TEXT): Start timestamp
- `end_time` (TEXT): End timestamp
- `exit_code` (INTEGER): Process exit code
- `log_output` (TEXT): Captured output
- `error_message` (TEXT): Error details
- `created_at` (TEXT): Creation timestamp

**fleet_status** ⭐ NEW (M3.5)
- `id` (INTEGER PRIMARY KEY): Status record ID
- `check_time` (TIMESTAMP): When check was performed
- `check_type` (TEXT): 'docker', 'tunnel', 'backup', 'disk', 'overall'
- `status` (TEXT): 'healthy', 'warning', 'critical', 'unknown'
- `data` (TEXT): JSON with detailed check results
- `message` (TEXT): Human-readable status message

## Development

### Code Style
- PEP 8 compliant
- Type hints on all functions
- Docstrings for all classes and methods
- Maximum line length: 100 characters

### Adding New Features
1. Update the appropriate core module
2. Add tests to `tests/test_foundation.py`
3. Update this README
4. Run tests to ensure nothing breaks

## Troubleshooting

### Database Locked Error
- SQLite uses WAL mode for better concurrency
- Ensure only one DatabaseManager instance (singleton)
- Check file permissions on `~/.config/daemon-control/`

### Import Errors
- Ensure virtual environment is activated
- Verify all dependencies installed: `pip install -r requirements.txt`
- Check Python version: `python3 --version` (requires 3.8+)

### Log Files Not Created
- Check directory permissions
- Verify path expansion in config
- Ensure parent directories exist (created automatically)

## License

MIT License - See LICENSE file for details

## Contributing

This is currently a personal project. For bugs or suggestions, please open an issue.

## Cron Expression Examples

The daemon uses standard cron format: `minute hour day month day_of_week`

```python
# Every minute
"* * * * *"

# Every hour at minute 0
"0 * * * *"

# Daily at 2:30 AM
"30 2 * * *"

# Every Monday at 9:00 AM
"0 9 * * 1"

# First day of every month at midnight
"0 0 1 * *"

# Every 15 minutes
"*/15 * * * *"
```

## Autostart (Optional)

To automatically start the system tray on login:

### Option 1: XDG Autostart (Simple)

Create `~/.config/autostart/daemon-control.desktop`:

```ini
[Desktop Entry]
Type=Application
Name=DaemonControl
Comment=Job Scheduling Daemon System Tray
Exec=/usr/bin/python3 /path/to/DaemonControl/start_tray.py
Icon=/path/to/DaemonControl/gui/resources/icon_active.png
Hidden=false
NoDisplay=false
X-GNOME-Autostart-enabled=true
```

Replace `/path/to/DaemonControl` with your actual path.

### Option 2: systemd User Service

Create `~/.config/systemd/user/daemon-control.service`:

```ini
[Unit]
Description=DaemonControl System Tray
After=graphical-session.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /path/to/DaemonControl/start_tray.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target
```

Enable and start:
```bash
systemctl --user enable daemon-control.service
systemctl --user start daemon-control.service
```

Check status:
```bash
systemctl --user status daemon-control.service
```

## Troubleshooting

### System Tray Icon Not Visible

**GNOME Desktop:**
- Install AppIndicator extension: `sudo apt install gnome-shell-extension-appindicator`
- Enable in Extensions app
- Log out and back in

**Wayland:**
- Some Wayland sessions may not support system tray
- Try X11 session instead
- Or use foreground mode: `python3 start_daemon.py`

### Daemon Won't Start from Tray

Check logs:
```bash
cat ~/.config/daemon-control/logs/daemon.log
```

Verify daemon works in foreground:
```bash
python3 start_daemon.py
```

### Dependencies Missing

Install all requirements:
```bash
pip install -r requirements.txt
```

Or install individually:
```bash
pip install apscheduler PyQt6 Pillow python-dateutil
```

## Technical Notes

### Fleet Monitoring Implementation Details

**Tunnel Monitor - TCP Socket Approach**

The tunnel monitor uses TCP socket connections instead of ICMP ping to avoid requiring raw socket capabilities (cap_net_raw). This design choice enables:
- No root/sudo privileges needed
- No capability requirements
- Works in systemd user services
- Tests connectivity to SSH port (22) on VPS tunnel IP

**Backup Monitor - Filesystem-Based Check**

Instead of calling `restic snapshots --json` (which requires password and root access), the backup monitor:
- Reads snapshot directory modification times
- Checks most recent snapshot file age
- No password or restic command needed
- Executes in <100ms vs 30-60+ seconds
- No permission errors (directory is readable by user)

**Database-Cached Architecture**

Fleet monitoring follows a producer-consumer pattern:
- **Producers**: Background daemon jobs write status to database
- **Consumers**: GUI reads from database for instant display
- Benefits: GUI loads instantly, no heavy checks on UI thread, historical data available

## Next Steps

**Milestone 5** will implement:
- Desktop notifications for critical fleet alerts
- Export functionality (CSV, JSON) for execution history
- Historical graphs and trend analysis for fleet metrics
- Advanced filtering and search in history tab
- Email/webhook alerts for critical system events

**Milestone 6** will focus on:
- Performance optimization for large execution histories
- Database cleanup and archiving strategies
- UI/UX polish and accessibility improvements
- Comprehensive test coverage for GUI components
- Documentation and user guide

Stay tuned!