user@elrise.io:~$ 
2026-07-2825 min readvpnhysteriaxrayvlessrealityquictlssysadminnetworking

TLS masquerading in Hysteria 2 and XTLS-Reality: deploying proxies on a self-hosted VPS

0. What this is and why

This material covers the architecture and deployment of two modern proxy protocols with TLS masquerading — Hysteria 2 and XRay with XTLS-Reality — on a self-hosted VPS (Debian 12 / Ubuntu 22.04+):

Both protocols are interesting from an engineering perspective as examples of how QUIC, ACME, x25519 keys, and systemd assemble into a production-ready proxy stack. Deployment takes ~20 minutes per protocol.

This guide assumes you already have:

1. Prerequisites

1.1. Update the system and install base packages

apt update && apt -y upgrade
apt -y install curl wget tar socat cron nginx-light dnsutils

1.2. Open ports in the firewall

UFW (Ubuntu):

ufw allow 22/tcp comment 'ssh'
ufw allow 80/tcp comment 'acme + http redirect'
ufw allow 443/tcp comment 'vless reality'
ufw allow 443/udp comment 'hysteria2 quic'
ufw --force enable
ufw status numbered

iptables-nft (Debian 12 default):

apt -y install iptables-persistent
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -p udp --dport 443 -j ACCEPT
netfilter-persistent save

1.3. Get an LE certificate via acme.sh

curl -fsSL https://get.acme.sh | sh -s email="<USER_EMAIL>"
source ~/.bashrc
mkdir -p /etc/hysteria/certs
~/.acme.sh/acme.sh --issue -d <SERVER_DOMAIN> --standalone --httpport 80 --server letsencrypt
~/.acme.sh/acme.sh --install-cert -d <SERVER_DOMAIN> \
    --cert-file      /etc/hysteria/certs/server.crt \
    --key-file       /etc/hysteria/certs/server.key \
    --fullchain-file /etc/hysteria/certs/server.fullchain.pem \
    --reloadcmd      "systemctl reload hysteria-server.service 2>/dev/null || true"

<SERVER_DOMAIN> is your domain, e.g. vpn.example.com. <USER_EMAIL> is the address used to register the ACME account.

Verify the certificate:

openssl x509 -in /etc/hysteria/certs/server.crt -noout -subject -issuer -dates

Expected issuer: O = Let's Encrypt. The certificate is valid for 90 days; acme.sh renews it automatically roughly every 60 days via cron.

2. Hysteria 2

2.1. Installation

Latest release: https://github.com/apernet/hysteria/releases

HY_VER="2.6.0"   # check the latest version on GitHub
cd /tmp
wget -q "https://github.com/apernet/hysteria/releases/download/app/v${HY_VER}/hysteria-linux-amd64" \
     -O /usr/local/bin/hysteria
chmod +x /usr/local/bin/hysteria
hysteria version

2.2. Configuration

Create /etc/hysteria/config.yaml:

log:
  level: info

listen: 0.0.0.0:8443

tls:
  cert: /etc/hysteria/certs/server.crt
  key:  /etc/hysteria/certs/server.key

auth:
  type: password
  password: "<HY2_PASSWORD>"   # replace with a random 24-char string

masquerade:
  type: proxy
  proxy:
    url: https://www.microsoft.com/
    rewriteHost: true

disableUDP: false

Generate the password:

openssl rand -base64 24 | tr -d '+/=' | head -c 24

masquerade.url is the site that Hysteria will respond with when someone probes UDP 8443 without the correct password. Pick a high-traffic HTTPS site with stable TLS — a mainstream international service is usually a good default.

2.3. systemd unit

Create /etc/systemd/system/hysteria-server.service:

[Unit]
Description=Hysteria 2 Server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
WorkingDirectory=/etc/hysteria
ExecStart=/usr/local/bin/hysteria server --config /etc/hysteria/config.yaml
Restart=on-failure
RestartSec=5s
LimitNOFILE=65535
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

Activate:

systemctl daemon-reload
systemctl enable --now hysteria-server.service
systemctl status hysteria-server.service

Expected output: Active: active (running). In the log:

server up and running {"listen": "0.0.0.0:8443"}

DOMAIN=<SERVER_DOMAIN>
PASS="<HY2_PASSWORD>"

printf "hysteria2://%s@%s:8443/?sni=%s&insecure=0&obfs=none&protocol=udp#hysteria-%s\n" \
       "$PASS" "$DOMAIN" "$DOMAIN" "$DOMAIN"

Or by IP, if the domain has not propagated in client DNS yet:

printf "hysteria2://%s@<SERVER_IP>:8443/?sni=%s&insecure=0&obfs=none&protocol=udp#hysteria-%s\n" \
       "$PASS" "$DOMAIN" "$DOMAIN"

2.5. Testing

From the local machine:

ss -ulnp | grep 8443          # Hysteria should be listening
ss -tlnp | grep 8443          # nothing should be here (Hysteria is UDP)

# From a desktop (via VPN or a different geo):
curl -k https://<SERVER_DOMAIN>:8443/   # should return 400 or the masquerade page (active probing)

From an Android client — the connection should come up in under a second.

3. XRay + VLESS+Reality

3.1. Installing XRay

Latest release: https://github.com/XTLS/Xray-core/releases

XR_VER="26.1.13"   # check on GitHub
wget -q "https://github.com/XTLS/Xray-core/releases/download/v${XR_VER}/Xray-linux-64.zip" \
     -O /tmp/xray.zip
apt -y install unzip
unzip -o /tmp/xray.zip -d /usr/local/bin/
chmod +x /usr/local/bin/xray
/usr/local/bin/xray version

3.2. Generating keys and UUID

# x25519 keys for Reality
/usr/local/bin/xray x25519

# Note down:
#   PrivateKey: ...
#   Password (PublicKey): ...

# UUID for the client
/usr/local/bin/xray uuid
# Note down: e.g. 00000000-0000-0000-0000-000000000000

# ShortID (any random 16 hex chars)
openssl rand -hex 8
# Note down: e.g. abcdef0123456789

3.3. Configuration

Create /usr/local/etc/xray/config.json:

{
  "log": {
    "loglevel": "warning",
    "access": "/var/log/xray/access.log",
    "error":  "/var/log/xray/error.log"
  },
  "inbounds": [
    {
      "tag": "vless-reality-443",
      "listen": "0.0.0.0",
      "port": 443,
      "protocol": "vless",
      "settings": {
        "decryption": "none",
        "clients": [
          {
            "id": "<XR_UUID>",
            "flow": "xtls-rprx-vision",
            "email": "main-client"
          }
        ]
      },
      "streamSettings": {
        "network": "tcp",
        "security": "reality",
        "realitySettings": {
          "show": false,
          "dest": "www.microsoft.com:443",
          "xver": 0,
          "serverNames": [
            "www.microsoft.com",
            "www.apple.com",
            "www.samsung.com",
            "www.cloudflare.com"
          ],
          "privateKey": "<XR_PRIVKEY>",
          "shortIds": [
            "<XR_SHORTID>"
          ]
        }
      },
      "sniffing": {
        "enabled": true,
        "destOverride": ["http", "tls", "quic"],
        "routeOnly": true
      }
    }
  ],
  "outbounds": [
    { "protocol": "freedom", "tag": "direct" },
    { "protocol": "blackhole", "tag": "block" }
  ]
}

Parameters:

3.4. systemd unit

Create /etc/systemd/system/xray.service:

[Unit]
Description=Xray Service
Documentation=https://github.com/XTLS/xray-core
After=network-online.target nss-lookup.target
Wants=network-online.target nss-lookup.target

[Service]
Type=simple
User=nobody
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
ExecStart=/usr/local/bin/xray run -config /usr/local/etc/xray/config.json
Restart=on-failure
RestartPreventExitStatus=23
LimitNPROC=65535
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Activate:

mkdir -p /var/log/xray
useradd -r -s /usr/sbin/nologin nobody 2>/dev/null || true
chown -R nobody:nogroup /var/log/xray
systemctl daemon-reload
systemctl enable --now xray.service
systemctl status xray.service

3.5. Freeing port 443 (if nginx is using it)

Reality must listen on 443 to look like ordinary HTTPS. If nginx is already running on the server, move it to a different port (8443 or 8080):

# In nginx configs, find listen 443 ssl and change it to listen 8443 ssl
grep -rn "listen.*443" /etc/nginx/

# After editing:
nginx -t && systemctl reload nginx

PUB="<XR_PUBKEY>"      # Password from step 3.2
SID="<XR_SHORTID>"     # from step 3.2
UUID="<XR_UUID>"
SERVER="<SERVER_IP>"   # or <SERVER_DOMAIN>

# Primary link (SNI = microsoft.com)
printf "vless://%s@%s:443?type=tcp&security=reality&pbk=%s&fp=chrome&sni=www.microsoft.com&sid=%s&flow=xtls-rprx-vision#main-microsoft\n" \
       "$UUID" "$SERVER" "$PUB" "$SID"

# Alternative (SNI = apple.com)
printf "vless://%s@%s:443?type=tcp&security=reality&pbk=%s&fp=chrome&sni=www.apple.com&sid=%s&flow=xtls-rprx-vision#main-apple\n" \
       "$UUID" "$SERVER" "$PUB" "$SID"

# Alternative (SNI = cloudflare)
printf "vless://%s@%s:443?type=tcp&security=reality&pbk=%s&fp=chrome&sni=www.cloudflare.com&sid=%s&flow=xtls-rprx-vision#main-cloudflare\n" \
       "$UUID" "$SERVER" "$PUB" "$SID"

3.7. Testing

ss -tlnp | grep 443          # xray should be listening

# Active probing test: Reality should return the certificate of the dest site
timeout 10 openssl s_client -connect <SERVER_IP>:443 -servername www.microsoft.com -brief 2>&1 | head -10

Expected output: Peer certificate: C = US, O = Microsoft Corporation, CN = ..., Verification: OK. Empty output or an error means Reality is not working.

4. Android client

  1. Install Hiddify Next from GitHub Releases (the .apk).
  2. Open the app → ”+” in the top-right corner → “Add from clipboard”.
  3. Copy the share link (from step 2.4 or 3.6) into the clipboard.
  4. Tap the added entry — Hiddify will show a profile card.
  5. In the profile settings, verify the parameters:
    • For Hysteria 2: address, port, password, SNI.
    • For VLESS+Reality: address, port, UUID, flow=xtls-rprx-vision, pbk, sid, sni. If the “SNI” field is editable, it should match sni= in the share link.
  6. On the main screen, tap the big “Connect” button — the status changes to “Connected”, a notification with a VPN icon appears.

4.3. Auto-selecting the closest server

If you run multiple VPN servers on different IPs, Hiddify can pick the best one by latency:

Hiddify will probe every profile and connect to the fastest one.

5. Technical details

5.1. Why VLESS+Reality resists DPI

Reality does not use its own TLS certificate. At the TLS handshake the client sends a ClientHello with SNI=microsoft.com (or any other entry in serverNames). The Reality server returns the certificate of the destination site on the fly — the same certificate the real microsoft.com would issue. Passive DPI sees “client connects to microsoft.com, receives a valid microsoft certificate” — a standard pattern.

During active probing (when an operator connects directly to our IP:443), if they do not have the correct PublicKey and ShortID, Reality proxies the TCP connection to dest = microsoft.com:443 and that site replies normally. The prober cannot tell our server apart from the real microsoft.com.

5.2. Why Hysteria 2 works

Hysteria uses QUIC (UDP). QUIC is a relatively recent transport, and many DPI stacks cannot analyze it deeply. The certificate is valid (LE), the fingerprint matches an ordinary QUIC client. DPI sees “UDP 443 with a QUIC handshake and a valid TLS” — and lets it through. Hysteria ships additional obfuscation plugins, but the base mode with an ACME cert handles the bulk of passive DPI deployments in practice.

6. Service management

# Hysteria 2
systemctl status hysteria-server.service
systemctl restart hysteria-server.service
journalctl -u hysteria-server.service -n 50 --no-pager

# XRay
systemctl status xray.service
systemctl restart xray.service
journalctl -u xray.service -n 50 --no-pager

# XRay logs (access / errors)
tail -f /var/log/xray/access.log
tail -f /var/log/xray/error.log

# Certificate check (has it expired?)
openssl x509 -in /etc/hysteria/certs/server.crt -noout -dates

7. Troubleshooting

7.1. Hysteria: client connects but no internet

Verify DNS works through the VPN:

# On the server
tcpdump -i any -n udp port 53 | head -20
# On the client, open https://1.1.1.1 — should display the Cloudflare page

If DNS does not work, add this section to the Hysteria config:

tcpForwarding:
  enabled: true
udpForwarding:
  enabled: true

7.2. Reality: client gets invalid request

7.3. Reality: certificate CN shows a different site (for example, microsoft.com from a regional edge)

This is normal. CDNs and large providers serve different certificates from regional edges. Reality picks the certificate on the fly from the dest site and does not forge it. The client does not raise an error, because the TLS handshake goes through Reality, not as an ordinary HTTPS client.

7.4. Active probing from network operators

When a Reality server is probed and the prober sees the certificate of microsoft.com — that is expected and safe. The prober receives the real microsoft.com response when connecting without the correct keys. That is the active-probing defence.

7.5. Hysteria certificate expired

acme.sh renews automatically. If it did not, force-renew manually:

~/.acme.sh/acme.sh --renew -d <SERVER_DOMAIN> --force
~/.acme.sh/acme.sh --install-cert -d <SERVER_DOMAIN> \
    --cert-file      /etc/hysteria/certs/server.crt \
    --key-file       /etc/hysteria/certs/server.key \
    --fullchain-file /etc/hysteria/certs/server.fullchain.pem
systemctl restart hysteria-server.service

7.6. Connection works but sites don’t open

Verify on the server:

# Should return a response
curl -I https://www.google.com
# Should return our IP
curl --interface eth0 ifconfig.me
# XRay/Hysteria logs
journalctl -u xray.service -n 20 --no-pager
journalctl -u hysteria-server.service -n 20 --no-pager

If curl -I https://www.google.com does not work from the server itself, the problem is not the VPN — it is the server’s network connectivity.

8. Hardening and best practices

8.1. Rotating passwords / UUIDs / keys

8.2. Minimizing the attack surface

In the XRay config:

"policy": {
  "levels": {
    "0": {
      "handshake": 3,
      "connIdle": 300,
      "uplinkOnly": 2,
      "downlinkOnly": 5
    }
  },
  "system": {
    "statsInboundUplink": false,
    "statsInboundDownlink": false
  }
}

This limits handshake counts and idle connections, making active scanning harder.

8.3. Logs and audit

XRay by default logs warning and above — access.log is written only for errors. Do not switch info on permanently — logs will grow.

logrotate -f /etc/logrotate.conf

8.4. Monitoring

A minimal health-check via systemd timer:

cat > /etc/systemd/system/vpn-healthcheck.timer << 'EOF'
[Unit]
Description=VPN health check timer

[Timer]
OnBootSec=5min
OnUnitActiveSec=15min

[Install]
WantedBy=timers.target
EOF

cat > /etc/systemd/system/vpn-healthcheck.service << 'EOF'
[Unit]
Description=VPN health check

[Service]
Type=oneshot
ExecStart=/usr/local/bin/vpn-healthcheck.sh
EOF

cat > /usr/local/bin/vpn-healthcheck.sh << 'EOF'
#!/bin/bash
set -e
ss -tlnp | grep -q ':443 .*xray'        || { echo "XRay down"; exit 1; }
ss -ulnp | grep -q ':8443 .*hysteria'  || { echo "Hysteria down"; exit 1; }
curl -fsSI https://www.google.com -m 5 > /dev/null || { echo "No internet"; exit 1; }
echo OK
EOF
chmod +x /usr/local/bin/vpn-healthcheck.sh

systemctl daemon-reload
systemctl enable --now vpn-healthcheck.timer

8.5. Backing up configs

mkdir -p /root/vpn-config-backups
cd /etc && tar czf /root/vpn-config-backups/hysteria-$(date +%F).tar.gz hysteria
cd /usr/local/etc && tar czf /root/vpn-config-backups/xray-$(date +%F).tar.gz xray

Disclaimer

Setting up a personal VPN is non-trivial and carries real costs: VPS bills, certificate maintenance, security risks, and a regulatory grey area that varies by jurisdiction. If you only need to read news or work with code, you probably do not need this. If you do need it — the guide above will help, but the responsibility is yours.

All information in this guide is sourced from public documentation (GitHub, official protocol docs, public reviews). All statements may no longer be valid at any moment: DPI implementations evolve, protocols ship new versions, deployment environments change. Before deploying, cross-check against the current version of the upstream sources linked in §9.

References

Discussion

Comments (0)

No comments yet.