پرش به محتویات

MoaV Monitoring Stack

Real-time observability for your MoaV deployment with Grafana dashboards.

A tour of the ten dashboards — 22s, 668 KB.

Overview

The monitoring profile provides:

  • Prometheus - Time-series database for metrics storage (15-day retention)
  • Grafana - Beautiful dashboards for visualization
  • Node Exporter - System metrics (CPU, RAM, disk, network)
  • cAdvisor - Container metrics per service
  • Clash Exporter - sing-box proxy metrics via Clash API
  • sing-box Exporter - Per-user connections, protocol breakdown, GeoIP country stats
  • Xray Exporter - Per-user connections and traffic (upload/download), GeoIP country stats
  • WireGuard Exporter - VPN peer and traffic metrics, GeoIP country labels
  • AmneziaWG Exporter - Per-peer traffic metrics, GeoIP country labels
  • Telemt Exporter - MTProxy pool health, DC availability, upstream quality
  • Snowflake Exporter - Tor donation metrics (people served, bandwidth donated)
  • GeoIP Database - DB-IP Lite country database for offline IP-to-country lookups

Quick Start

# Start with monitoring profile
moav start monitoring proxy admin

# Or add to existing deployment
moav start monitoring

Access

Grafana is on port 9444 over HTTPS: https://your-domain:9444 (or the server IP). Log in as admin with your ADMIN_PASSWORD — the same one as the admin dashboard.

For a faster, CDN-fronted route see Cloudflare CDN for Grafana below.

Pre-built dashboards

Ten dashboards are provisioned automatically — nothing to import or build. Grafana groups them under the MoaV folder.

Dashboard What it shows
System CPU, memory, disk, network throughput for the host
Containers Per-container CPU, memory, restarts (via cAdvisor)
sing-box Per-user connections and traffic for Reality, Trojan, AnyTLS, Hysteria2, Shadowsocks, CDN
WireGuard Peer handshakes, transfer per peer, last-seen
AmneziaWG The same, for the obfuscated interface
Xray (XHTTP) XHTTP / XDNS connections and throughput
Telegram MTProxy telemt connections, pool health, config-change activity
DNS Tunnels & GooseRelay Traffic across dnstt, Slipstream, MasterDNS, XDNS and GooseRelay. dnstt has no protocol-level metrics API, so its panels show container and network telemetry only
Conduit Bandwidth donated to Psiphon users, lifetime totals
Snowflake Bandwidth donated to Tor users, client connections

Per-user series come from the Clash API via clash-exporter; WireGuard and AmneziaWG read interface state published by their containers, so the exporters never need the Docker socket.

Conduit lifetime bandwidth

Conduit's live bandwidth gauges reset every time the container restarts. The Lifetime Download / Lifetime Upload panels work around that by adding a persistent offset back — so your cumulative donation totals keep growing across restarts.

This is automatic once monitoring and Conduit are both running. A systemd watcher banks the pre-restart total and reloads Prometheus the moment Conduit comes back up. Manage it with moav conduit-offsets; set CONDUIT_OFFSETS_AUTOUPDATE=false in .env to opt out. Hosts without systemd can run scripts/update-conduit-offsets.sh from cron.

GeoIP Country Distribution

All four protocol dashboards (sing-box, Xray, WireGuard, AmneziaWG) include a "Geographic Distribution" row showing user connections by country.

How it works

  • A shared DB-IP Lite Country database (~5MB) provides offline IP-to-country lookups — no external API calls at runtime
  • sing-box: polls the Clash API (/connections) for source IPs of active connections
  • Xray: extracts source IPs from Xray access logs
  • WireGuard / AmneziaWG: reads endpoint IPs from wg show / awg show
  • Country codes are ISO 3166-1 alpha-2 (e.g., IR, DE, US)

Setup

The GeoIP database must be downloaded once before country metrics appear:

# Download the GeoIP database (run once, or monthly to refresh)
docker compose --profile setup run --rm geoip-updater

The database is stored in a Docker volume (moav_geoip) and shared read-only with all exporters.

Refreshing the database

DB-IP Lite is updated monthly. To refresh:

docker compose --profile setup run --rm geoip-updater
docker compose restart singbox-exporter xray-exporter wireguard-exporter amneziawg-exporter

Using MaxMind GeoLite2 instead

The database format is MMDB — compatible with both DB-IP and MaxMind. To use MaxMind GeoLite2-Country instead:

# Download GeoLite2-Country.mmdb from maxmind.com (requires free account)
# Copy into the volume:
docker run --rm -v moav_geoip:/geoip -v /path/to/GeoLite2-Country.mmdb:/src/db.mmdb alpine \
  cp /src/db.mmdb /geoip/dbip-country-lite.mmdb

Graceful degradation

If the GeoIP database is not downloaded, all country lookups return "XX" (unknown) and existing metrics continue to work normally. The exporters log a warning on startup:

GeoIP: could not load /geoip/dbip-country-lite.mmdb: [Errno 2] No such file or directory

Configuration

Port Configuration

# .env
PORT_GRAFANA=9444    # External Grafana port (default: 9444)

Grafana can be slow to load over high-latency connections due to large JS/CSS assets. You can use Cloudflare's CDN to cache static assets for much faster loading.

Step 1: Add DNS Record

In Cloudflare Dashboard, add:

Type Name Content Proxy
A grafana YOUR_SERVER_IP Proxied (orange cloud)

Step 2: Configure Environment

Add to your .env file: 

GRAFANA_ROOT_URL=https://grafana.yourdomain.xyz:2083

Step 3: Restart Services

moav restart grafana grafana-proxy

Step 4: Access Grafana

Access via https://grafana.yourdomain.xyz:2083 instead of :9444.

Note: Port 2083 is used because Cloudflare only proxies specific HTTPS ports (443, 2053, 2083, 2087, 2096, 8443). The grafana-proxy service handles SSL termination and caching headers.

Benefits:

  • Static assets (JS, CSS, images) cached at Cloudflare edge
  • Gzip compression
  • Faster global access
  • WebSocket support for live dashboard updates

Retention

Prometheus retains data for 15 days by default. To change this, modify the --storage.tsdb.retention.time flag in docker-compose.yml:

prometheus:
  command:
    - '--storage.tsdb.retention.time=30d'  # 30 days

Enabling/Disabling

# .env
ENABLE_MONITORING=true   # Set to false to disable

Resource Usage

Warning: The monitoring stack nearly doubles MoaV's resource requirements. While MoaV alone runs on 1 vCPU / 1GB RAM, adding monitoring requires at least 2 vCPU / 2GB RAM for stable operation.

Approximate additional resources when monitoring is enabled:

Component CPU RAM Disk
Prometheus 0.1-0.3 cores 200-500 MB ~50 MB/day
Grafana 0.1-0.2 cores 100-200 MB ~50 MB
Node Exporter <0.1 cores ~20 MB -
cAdvisor 0.1-0.3 cores 50-150 MB -
Clash Exporter <0.1 cores ~30 MB -
WireGuard Exporter <0.1 cores ~10 MB -
Snowflake Exporter <0.1 cores ~10 MB -
Total ~0.5-1 cores ~400-900 MB ~1 GB/15 days

Minimum Requirements

Configuration vCPU RAM Notes
MoaV only 1 1 GB Bare minimum
MoaV + Monitoring 2 2 GB Recommended (for stats & monitoring)
MoaV + Monitoring + All Profiles 2 4 GB Production recommended

If you have only 1GB RAM, do not enable monitoring - it could cause hangs and crashes.

Security

  • Prometheus is internal only (no external port exposed)
  • Grafana requires authentication via ADMIN_PASSWORD
  • All exporters expose metrics only to the internal Docker network

What's Not Included

The following services do not currently expose metrics:

Service Reason
TrustTunnel No metrics API available
dnstt No metrics API available

Container-level metrics (CPU, memory, network) are still available for these services via cAdvisor.

Troubleshooting

Run moav doctor services to quickly check if monitoring services are running. For detailed issues, see TROUBLESHOOTING.md.

Common issues covered:

  • System hangs after starting monitoring (RAM issues)
  • Grafana shows "No Data"
  • High memory usage from cAdvisor
  • Snowflake metrics showing zeros
  • WireGuard exporter not starting

Blocked Registries (Iran, Russia, etc.)

If you're in a region where container registries are blocked (gcr.io, ghcr.io, Docker Hub), you can build images locally:

# Build commonly blocked images (cAdvisor from gcr.io, clash-exporter from ghcr.io)
moav build --local

# Build a specific image
moav build --local prometheus

# Build EVERYTHING locally (all services + all external images, no registry pulls)
moav build --local all

Available images for local build:

  • cadvisor - Container metrics (gcr.io) ← commonly blocked
  • clash-exporter - Clash API exporter (ghcr.io) ← commonly blocked
  • prometheus - Time-series database (Docker Hub)
  • grafana - Visualization dashboards (Docker Hub)
  • node-exporter - System metrics (Docker Hub)
  • nginx - Web server (Docker Hub)
  • certbot - Let's Encrypt client (Docker Hub)

The build command automatically updates your .env to use the local images.

Alternatively, configure mirror registries manually in .env: 

IMAGE_CADVISOR=mirror.example.com/cadvisor/cadvisor:latest
IMAGE_PROMETHEUS=mirror.example.com/prom/prometheus:latest

CLI Commands

# Start monitoring only
moav start monitoring

# Start with other profiles
moav start monitoring proxy admin

# Build images locally (for blocked registries)
moav build --local

# View monitoring logs
moav logs prometheus
moav logs grafana

# Stop monitoring
moav stop prometheus grafana node-exporter cadvisor clash-exporter

Customization

Adding Custom Dashboards

Place JSON dashboard files in: 

configs/monitoring/grafana/provisioning/dashboards/

Grafana automatically loads new dashboards within 30 seconds.

Custom Prometheus Scrape Targets

Edit configs/monitoring/prometheus.yml to add new targets: 

scrape_configs:
  - job_name: 'my-custom-exporter'
    static_configs:
      - targets: ['my-service:9999']

Then reload Prometheus: 

docker exec moav-prometheus wget -qO- --post-data='' http://localhost:9091/-/reload

Architecture

 ┌─────────────────┐
 │  Cloudflare CDN │ (optional, for faster loading)
 └────────┬────────┘
          │                  
          ▼                         
 ┌─────────────┐  ┌─────────────┐       
 │grafana-proxy│  │   Grafana   │
 │ :2083 (CDN) │  │ :9444 direct│
 └──────┬──────┘  └──────┬──────┘
        │                │
        └───────┬────────┘
                │
         ┌──────▼──────┐
         │  Prometheus │ :9091 (internal)
         │ (time-series│
         │   storage)  │
         └──────┬──────┘
   ┌─────────┬──┴─────┬─────────┬─────────┐
   │         │        │         │         │
┌──▼────┐ ┌──▼───┐ ┌──▼────┐ ┌──▼────┐ ┌──▼──────┐
│ node  │ │cAdvsr│ │ clash │ │  wg   │ │snowflke │
│export │ │(cntr)│ │(sngbx)│ │(vpn)  │ │ (tor)   │
└───────┘ └──────┘ └───────┘ └───────┘ └─────────┘