# Operations and Security ## Threat model in one paragraph NetworkCore expects Redis to be reachable only by trusted servers. If Redis sits on a different host than the Hytale servers (the common case with hosted game servers), enable `redis.tls` so the connection is encrypted in transit. The shared auth token guards against accidents (a misconfigured staging server talking to prod) and casual mischief (someone with bus access spoofing a server id). It does *not* protect against an attacker who already has both Redis credentials and the token - at that point they can craft any envelope they want. The defenses in depth: TLS for the wire, the auth token for accidents, and network controls (firewall rules, private Redis interfaces, hosting-provider ACLs) for the actual attacker scenario. ## Auth token rollout ``` NETWORK_AUTH_TOKEN= ``` Pick something long. 32+ chars of randomness is fine. Set it the same way on every server. When rotating: do a rolling restart. Mid-rotation, servers running the old token will reject messages from servers running the new token (and vice versa). `messagebus_rejected_total` will tick up, and you'll see warning logs at most once per 10s per server. Finish the rollout fast. If you want zero rejection windows, you'd need a multi-token scheme where receivers accept either the old or new token during transition. That's not in v1. ## What is and isn't encrypted or signed - **Redis traffic.** Optional TLS (`redis.tls` in config). Turn it on whenever Redis runs on a different host than the Hytale server. Without TLS, anyone sniffing the network between the Hytale server and Redis sees every payload and the auth token. With TLS, the connection is encrypted end to end. Use `verify_peer: true` against any production Redis with a real cert. - **Messages themselves.** No HMAC, no signing. The auth token in the envelope is checked for equality, not authenticity. An attacker with Redis credentials can craft any envelope and stamp it with the token (if they have it). TLS prevents the token from leaking on the wire, but doesn't prevent forgery by anyone who legitimately has Redis access. - **The metrics endpoint.** No auth at all. Defaults to `127.0.0.1` so only the local machine can hit it. If you bind to a non-loopback address, put a reverse proxy with auth in front. ## The metrics endpoint `http://127.0.0.1:9100/metrics` by default. Three ways to consume it remotely: 1. **Run Prometheus on the same host.** Simplest. Prometheus hits 127.0.0.1:9100 and forwards via its own remote-write to a central store. 2. **SSH tunnel from the scraper host.** `ssh -L 9100:127.0.0.1:9100 hytale-server`, then point Prometheus at `127.0.0.1:9100`. 3. **Reverse proxy with auth.** Nginx in front of `127.0.0.1:9100`, requires HTTP Basic or mTLS. Change `metrics.bind` to `127.0.0.1` (default) and have nginx be the public face. Do not change `metrics.bind` to `0.0.0.0` without one of the above. The endpoint leaks server identity, player counts, JVM internals, plugin state, and roughly what you're doing operationally. ## Ops checks at a glance | Question | How to answer | |---|---| | Is this server alive in the network? | `/networkcore status` shows "Network: not connected" if Redis is down | | Are other servers seeing me? | On another server: `/networkcore servers` - your id should be listed within a few seconds | | Is the metrics endpoint up? | `curl http://127.0.0.1:9100/metrics` | | Are servers communicating? | `redis-cli -h MONITOR` and look for PUBLISH activity | | Is Redis in the expected state? | `redis-cli -h KEYS 'network:*'` shows registry hashes | | Are auth rejects happening? | `messagebus_rejected_total` in `/metrics`; warning logs `Rejected message on ...` | | What's the per-server RPC latency? | PromQL: `rate(rpc_request_duration_seconds_sum_seconds[1m]) / rate(rpc_request_duration_seconds_count[1m])` | ## Common failure modes ### "Failed to connect to Redis at host:port" at startup Redis is unreachable. Check `redis.host` / `redis.port`. Check `REDIS_PASSWORD`. Check whether Redis is actually running. `/networkcore status` will show "Network: not connected" and the rest of the plugin keeps running with the message bus / registry / RPC services unregistered. ### `messagebus_rejected_total` is climbing One of your servers has a different `NETWORK_AUTH_TOKEN` than the rest. Find it (the warning log includes the `from` server id) and fix. ### A server isn't appearing in `/networkcore servers` on other hosts 1. Did its `start()` complete? Check its log for "Connected to Redis ... auth=on/off". 2. Is its auth token the same as everyone else's? 3. `redis-cli -h HGETALL network:servers:` - if empty, it's not heartbeating. 4. If the hash is there but isn't appearing on other hosts, the receiver has the wrong token or the receiver's Redis is a different instance entirely (check `redis.database`). ### Servers stay listed for ~15s after crash Expected. Hard crashes don't send the goodbye. The 15s TTL is what removes them. ### RPC requests time out 1. Is anyone actually serving the named service? `registerService("")` must have been called. 2. Is the handler slow? Check `rpc_request_duration_seconds`. 3. Is the responder server reachable on the bus? If `/networkcore servers` shows it, yes. ## Sizing notes - One Redis instance handles thousands of messages/second easily. The bottleneck is almost always subscriber processing, not Redis. - The MessageBus worker pool is sized 2 core / 8 max. If you're doing heavy fan-in (lots of subscribers all doing real work on the same channel), you may want to bump this. It's a constant in `RedisMessageBus` right now; promote to config if needed. - Heartbeat interval (5s) and TTL (15s) are constants. Tighter values mean faster failure detection at the cost of more Redis writes. 5s/15s is fine up to hundreds of servers. - The local server-registry cache rescans Redis every 5s using `SCAN` - safe even with many keys, no `KEYS` blocking. Scales linearly with server count.