VPS /var/log Monitoring & Security Analytics Guide
Full technical reference covering system architecture, Keycloak OIDC Single Sign-On, real-time log parsing, SSH/Fail2ban/UFW intrusion analytics, Server-Sent Events (SSE) live tailing, binary/gzipped log decoding, and VPS operational runbooks.
1. Platform Architecture & Technology Stack
How logs.ibnuhajar.io processes, analyzes, and streams /var/log data
The monitoring dashboard is built as a full-stack Server-Side Rendered (SSR) application powered by React Router 8 (`v8.5.1`) running on Node.js (`127.0.0.1:3005`) behind an Nginx TLS 1.3 Reverse Proxy. All filesystem access to /var/log executes strictly inside server-only loaders and resource routes (app/services/logs.server.ts), ensuring zero raw filesystem exposure to unauthenticated clients.
┌──────────────────────┐ HTTPS (TLS 1.3) ┌──────────────────────────────┐
│ Browser Client UI │ ◄─────────────────────► │ Nginx Reverse Proxy (:443) │
│ React Router 8 DOM │ SSE (Unbuffered) │ logs.ibnuhajar.io │
└──────────┬───────────┘ └──────────────┬───────────────┘
│ │
│ OAuth2 / OIDC (PKCE S256) │ HTTP/1.1 (127.0.0.1:3005)
▼ ▼
┌──────────────────────┐ ┌──────────────────────────────┐
│ Keycloak 26.8.2 SSO │ ◄─────────────────────► │ React Router 8 SSR Server │
│ auth.ibnuhajar.io │ Token & UserInfo │ Loaders & Resource Routes │
└──────────────────────┘ └──────────────┬───────────────┘
│
┌───────────────────────┼───────────────────────┐
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Tail Byte Reader│ │ zlib .gz Gunzip│ │ Binary Decoders│
│ syslog/auth/ufw │ │ *.log.*.gz │ │ last/lastb │
└─────────────────┘ └─────────────────┘ └─────────────────┘Framework mode with React 19, Vite 8 Environment API, Tailwind CSS v4, and Lucide icons.
Centralized SSO via auth.ibnuhajar.io using PKCE S256 and signed HTTP-only cookies.
Fixed-window byte tailing, native zlib gunzip decompression, and utmp binary decoders.
Managed via ibnuhajar-logs.service on port 3005 with Let's Encrypt auto-renewing SSL.
2. Authentication & Keycloak OIDC SSO Integration
How Single Sign-On with auth.ibnuhajar.io protects sensitive /var/log data
Because /var/log contains sensitive IP addresses, SSH usernames, and system telemetry, every dashboard view and API endpoint is guarded by requireUser(request) in app/services/auth.server.ts.
- PKCE S256 Protection: Every login generates a cryptographic 32-byte
code_verifier, SHA-256code_challenge, and anti-CSRFstatetoken stored in an encrypted session cookie. - Dual-Realm Support: Users can sign in on /login using either the exploration-lab application realm or the master administrator realm.
- Federated Single Sign-Out: Clicking the logout button calls
/auth/logout, destroying the local session cookie and redirecting to Keycloak'send_session_endpointwithid_token_hint.
3. Feature Guide: System & Security Overview (/)
Real-time executive summary of VPS health, log volume, and intrusion telemetry
Calculates total bytes across all discovered log files, counts files modified in the last 24 hours, and queries df -h /var/log for partition utilization percentage and available free space.
Displays the VPS hostname, system uptime, RAM usage percentage, and Linux 1m / 5m / 15m load averages via Node's node:os module, with a manual Refresh Metrics revalidation button.
Buckets log timestamps over the past 12 hours into stacked bars segmented by INFO, SECURITY, and ERROR/WARN with interactive hover tooltips.
Presents a unified feed of the latest security and critical error events alongside a ranked progress-bar list of the largest files inside /var/log.
4. Feature Guide: /var/log File Explorer & Inspector (/explorer)
Deep search, regex filtering, .gz archive decompression, and binary login log decoding
Rotated gzipped logs (such as php8.5-fpm.log.2.gz) are automatically decompressed in-memory using zlib.gunzipSync and can be searched or downloaded as plain text with one click.
Clicking binary files like btmp, btmp.1, wtmp, or lastlog automatically runs lastb, last, or lastlog to render human-readable login history.
Toggle between parsed structured rows (with severity badges, timestamps, service tags, and clickable IP filter pills) and raw terminal output, or export the file via /api/download.
Interactive Regex Search Cheatsheet
Click any pattern below to run it directly inside the Log Explorer with Regex mode enabled
| Use Case | Target File | Regex Pattern | Action |
|---|---|---|---|
SSH Brute-Force & Auth Failures Matches all failed SSH login attempts, PAM auth failures, and invalid username probes. | auth.log | (Failed password|Invalid user|authentication failure) | Try Pattern |
Specific Attacker Subnet or IP Finds all events originating from a specific /16 IPv4 CIDR block. | auth.log | \b143\.246\.\d+\.\d+\b | Try Pattern |
Fail2ban Jail Ban / Unban Events Isolates active IP ban and unban enforcement actions triggered by Fail2ban. | fail2ban.log | \[sshd\]\s+(Ban|Unban)\s+\d+\.\d+\.\d+\.\d+ | Try Pattern |
UFW Blocked SYN Scans on Sensitive Ports Detects external TCP SYN packets targeting SSH, MySQL, PostgreSQL, Redis, or admin ports. | ufw.log | PROTO=TCP.*DPT=(22|3306|5432|6379|8080|8082).*SYN | Try Pattern |
Nginx 4xx / 5xx Error Status Codes Matches HTTP 401/403/404/429 client errors and 500/502/503/504 upstream errors. | nginx/access.log | "\s+(40[134]|429|50[0234])\s+\d+ | Try Pattern |
WordPress / Env / Git Scanner Probes Identifies automated vulnerability scanners probing common web paths. | nginx/access.log | (wp-admin|wp-login|\.env|\.git|xmlrpc\.php|phpmyadmin) | Try Pattern |
Kernel Out-of-Memory (OOM) or Segfaults Finds critical Linux kernel memory exhaustion or process crash events. | kern.log | (Out of memory|Killed process|segfault|oom-kill) | Try Pattern |
5. Feature Guide: Security, Firewall & Intrusion Center (/security)
Automated correlation across auth.log, fail2ban.log, ufw.log, and nginx logs
Extracts every Failed password and Invalid user event, ranking the Top Attacking IPv4 Addresses along with the specific usernames each IP attempted to compromise (e.g., root, admin, postgres). Clicking any attacker IP pivots directly to the filtered auth.log audit trail.
Tracks total Found offender matches, Ban actions, and Unban expirations across active jails ([sshd]), providing a live chronological feed of banned IPs.
Parses Linux kernel [UFW BLOCK] packets by protocol (TCP, UDP, ICMP), maps destination ports (DPT=) to well-known services (SSH, HTTP/S, PostgreSQL, MySQL, Redis, RDP), and ranks the most aggressive port-scanner IPs.
Aggregates HTTP response status codes (2xx, 3xx, 4xx, 5xx), identifies top client IPs and requested URL paths, and surfaces recent [crit] / [error] entries from nginx/error.log.
6. Feature Guide: Real-Time Live Tail SSE Stream (/live)
Sub-second Server-Sent Events streaming for active /var/log files
Upon connecting to /api/stream?file=..., the server immediately reads the last 24 KB (~40 lines) of the file, parses each line, and pushes an initial snapshot so the terminal is populated right away.
Every 1,000ms, the server checks fsp.stat(fullPath). When new bytes are appended, only the newly written byte range (stat.size - currentSize) is read and streamed. If the file is rotated (stat.size < currentSize), the offset automatically resets to 0.
Includes Pause / Resume Stream, Auto-Scroll ON/OFF, instant client-side keyword and severity filtering (ERROR, SECURITY, WARN, INFO), and a 500-line rolling ring buffer.
7. /var/log Monitored File Dictionary
Reference guide to system, security, web, database, and binary log files on this VPS
| File Path | Category | Format / Parser | Description & Telemetry Purpose |
|---|---|---|---|
| /var/log/syslog | System | ISO-8601 Syslog | Primary system activity log recording systemd service events, cron jobs, daemon messages, and kernel netfilter logs. |
| /var/log/auth.log | Security | ISO-8601 PAM/SSHD | Authentication and authorization events including OpenSSH logins, sudo usage, PAM checks, and brute-force attempts. |
| /var/log/fail2ban.log | Security | Fail2ban Timestamped | Intrusion prevention daemon log tracking IP pattern matches (Found), temporary firewall bans (Ban), and expirations (Unban). |
| /var/log/ufw.log | Security | Kernel Netfilter | Uncomplicated Firewall packet log capturing blocked incoming packets ([UFW BLOCK]) with SRC/DST IP, PROTO, SPT, and DPT. |
| /var/log/kern.log | System | ISO-8601 Kernel | Linux kernel ring buffer and subsystem messages, hardware/driver events, firewall drops, and OOM killer notices. |
| /var/log/nginx/access.log | Web & Proxy | Nginx Combined | Every HTTP/HTTPS request processed by Nginx across ibnuhajar.io, auth.ibnuhajar.io, and logs.ibnuhajar.io. |
| /var/log/nginx/error.log | Web & Proxy | Nginx Error | Nginx reverse proxy upstream failures, TLS/SSL handshake errors, and configuration warnings. |
| /var/log/btmp & btmp.1 | Security (Binary) | utmp Binary (lastb) | Binary record of bad/failed login attempts. Automatically decoded into human-readable rows via `lastb -f`. |
| /var/log/wtmp | Security (Binary) | utmp Binary (last) | Binary history of all successful user logins, logouts, and system reboots. Automatically decoded via `last -f`. |
| /var/log/lastlog | Security (Binary) | lastlog Binary | Binary database of the most recent login timestamp for every system account. Decoded automatically via `lastlog`. |
| /var/log/postgresql/*.log | Database & Apps | PostgreSQL Log | PostgreSQL 18 cluster startup, checkpoint, connection, and query error logs (including Keycloak backing DB). |
| /var/log/php8.5-fpm.log* | Database & Apps | PHP-FPM / Gzip (.gz) | PHP 8.5 FastCGI Process Manager worker pool lifecycle events, including rotated `.gz` compressed archives. |
| /var/log/dpkg.log & apt/* | Package & Updates | Debian Package Log | Audit trail of apt/dpkg package installations, upgrades, and unattended security updates. |
| /var/log/letsencrypt/letsencrypt.log | Web & Proxy | Certbot Python Log | ACME certificate issuance and automated renewal logs for *.ibnuhajar.io domains. |
8. HTTP Resource Routes & Security Controls
Internal API endpoints and sandbox protections enforcing read-only /var/log safety
Streams real-time log lines via text/event-stream with 15s keep-alive heartbeats and X-Accel-Buffering: no.
Downloads any log file inside /var/log. Passing decompress=1 on a .gz file gunzips it on the fly and serves a .txt attachment.
Handles OIDC PKCE redirect, token exchange, UserInfo lookup, and federated logout.
- Strict Path Traversal Prevention:
resolveSafeLogPath()resolves all requested paths against/var/logand throws an immediate error if a path attempts../traversal outside/var/log. - Read-Only Non-Destructive Access: The application never modifies, truncates, or deletes any file in
/var/log. - Hardened Nginx Headers: Enforces
Strict-Transport-Security,X-Content-Type-Options: nosniff,X-Frame-Options: SAMEORIGIN, andContent-Security-Policyvia/etc/nginx/snippets/site-security.conf.
9. VPS Operations, Systemd & Nginx Administration Runbook
Commands for managing, rebuilding, and monitoring logs.ibnuhajar.io on the VPS
Systemd Service Management (ibnuhajar-logs.service)
# Check service status & resource consumption systemctl status ibnuhajar-logs.service # Restart service after configuration changes systemctl restart ibnuhajar-logs.service # View real-time application stdout/stderr logs journalctl -u ibnuhajar-logs.service -f
Building & Deploying Code Updates
cd /root/AIEngineer/LogsMonitoring # Run React Router 8 typegen & TypeScript validation npm run typecheck # Build production client & SSR bundles npm run build # Restart the production systemd service systemctl restart ibnuhajar-logs.service
Nginx Reverse Proxy & Let's Encrypt SSL
# Configuration file location cat /etc/nginx/sites-available/logs.ibnuhajar.io # Test Nginx syntax & reload gracefully nginx -t && systemctl reload nginx # Test automated Let's Encrypt certificate renewal certbot renew --dry-run
Keycloak OIDC Container (auth.ibnuhajar.io)
cd /opt/keycloak # Check Keycloak container status & health docker compose ps curl -fsS http://127.0.0.1:9002/health/ready # Verify OIDC discovery endpoint curl -s https://auth.ibnuhajar.io/realms/exploration-lab/.well-known/openid-configuration | jq .