This commit is contained in:
2026-05-26 00:11:45 -04:00
commit 296c233157
41 changed files with 2829 additions and 0 deletions
+30
View File
@@ -0,0 +1,30 @@
# Build output
target/
out/
build/
*.class
*.jar
dependency-reduced-pom.xml
# IDE
.idea/
*.iml
*.iws
*.ipr
.vscode/
.project
.classpath
.settings/
# Logs
*.log
logs/
# OS
.DS_Store
Thumbs.db
# Local config / secrets
config.json.local
*.env
.env
+35
View File
@@ -0,0 +1,35 @@
# NetworkCore
The foundational plugin for KweebecNet. Every other plugin on the network depends on this one.
## What it does
- **Cross-server messaging** over Redis pub/sub, so plugins running on different Hytale servers can talk to each other (party invites, queue handoffs, global chat, etc.)
- **Server presence**, so every server knows which other servers are online and what role each one plays
- **Metrics**, so we can graph and alert on what the network is doing in real time (player counts, RPC latency, message volume, JVM health)
- **RPC**, so a plugin can ask another server a question and get an answer back
If this plugin isn't running, **nothing else on the network works correctly**. Chat won't cross servers. Parties break. Queues can't allocate matches. Metrics go dark. Treat it like the load-bearing wall it is.
## Why this is critical
This isn't a feature plugin you can disable to debug something else. If you stop NetworkCore on one server, that server falls off the network from everyone else's point of view within ~15 seconds. If Redis goes down, the entire mesh stops communicating until it's back.
Things that look fine in isolation but break the network if you get them wrong:
- Different `NETWORK_AUTH_TOKEN` on one server vs the others (silent message rejection)
- Pointing one server at a different Redis instance than the rest (servers can't see each other)
- Disabling the plugin "just to test" on a production server (leaves a hole in the mesh)
- Changing `redis.key_prefix` on one server (lives in its own private network)
## Before you touch this
**Read the docs in [docs/](docs/) first.** The full reading order is in [docs/README.md](docs/README.md). At minimum, read:
- [01_Overview.md](docs/01_Overview.md) - what this owns and why it's split out
- [03_Configuration.md](docs/03_Configuration.md) - every config field and what changing it does
- [09_Operations_And_Security.md](docs/09_Operations_And_Security.md) - threat model, common failure modes, ops checks
**If you're making changes to NetworkCore itself**, or to how a dependent plugin uses it, talk to a higher-up before merging. This is one of those rare pieces of infrastructure where a "small fix" can take down the whole network in ways that are not obvious until production traffic hits it.
If you're new and unsure, the safe move is to ask first. Nobody will mind. Everyone will mind if you break the bus.
+65
View File
@@ -0,0 +1,65 @@
# Overview
## What NetworkCore is
A single Hytale plugin that every other KweebecNet plugin depends on. It owns three concerns that don't belong in any individual gamemode or feature plugin:
1. **Metrics** - counters, gauges, and timers exposed over a Prometheus-compatible HTTP endpoint
2. **Cross-server messaging** - Redis pub/sub with a small JSON envelope, plus a request/response RPC layer on top
3. **Server presence** - every server in the network heartbeats to Redis, every other server learns about it within a few seconds and is notified via local Hytale events
## Why it's split out
If every gamemode plugin opened its own Redis connection, registered its own metrics, and rolled its own service-discovery, you'd have N copies of the same logic, N config blocks to keep in sync, and N places to add TLS when the time comes. Putting it in one plugin means:
- Plugins don't carry Redis as a dependency. They depend on NetworkCore's `MessageBus` interface.
- A single config file owns Redis + network secrets.
- When we add TLS, an audit log, or a different backend, only NetworkCore changes.
## What lives in NetworkCore (and what doesn't)
**In:**
- The MessageBus, ServerRegistry, MetricsService, RpcClient
- Auth-token verification on inbound messages
- The /networkcore admin command
- Built-in metrics (player count, JVM memory)
**Out:**
- Player profiles, stats, currency - those belong in a separate plugin (probably backed by MySQL)
- Chat, parties, queues - separate plugins that *use* the MessageBus
- World/match management - the existing MinigameCore-style plugins keep that role
The rule of thumb: if it's something every plugin should be able to use, it's in NetworkCore. If it's a feature you'd ship to players, it's its own plugin.
## How plugins use it
Two access patterns, both backed by the same internal service registry:
```java
NetworkCore core = NetworkCore.getInstance();
// Convenience getters for the bundled services
core.getMetrics().counter("matches_started", Map.of("game", "bridge_duel")).inc();
core.getMessageBus().publish("party.invite", invite);
// Generic lookup for anything (including services other plugins contribute)
MetricsService metrics = core.getService(MetricsService.class);
```
A plugin that wants to contribute its own service to the registry can do:
```java
core.registerService(MyPartyService.class, new MyPartyServiceImpl());
```
Other plugins then resolve it via `core.getService(MyPartyService.class)`. No DI framework, just a typed map.
## Lifecycle
NetworkCore uses the standard Hytale plugin lifecycle:
- `setup()` - load config, generate or read the server id, register the admin command, register MetricsService (built-ins included). Singleton becomes available here.
- `start()` - open the Redis connection, bind the Prometheus HTTP server, start the heartbeat, publish a hello to the network.
- `shutdown()` - publish a goodbye, stop the heartbeat, close Redis and the HTTP server.
Dependent plugins should resolve services in their own `start()`, not `setup()`. Hytale's dependency ordering guarantees we finish `setup()` before any dependent plugin's `start()` runs, but it does not guarantee we finish `start()` first - so a `getMessageBus()` call from another plugin's `setup()` may return null.
+104
View File
@@ -0,0 +1,104 @@
# Plugin Integration
How another plugin depends on NetworkCore and uses it.
## 1. Declare the dependency
In your plugin's `src/main/resources/manifest.json`:
```json
{
"Group": "yourgroup",
"Name": "YourPlugin",
"Version": "1.0.0",
"Main": "your.package.YourPlugin",
"Dependencies": {
"kewwbec:NetworkCore": "*"
},
"OptionalDependencies": {},
"DisabledByDefault": false
}
```
The dependency string is `kewwbec:NetworkCore`. Hytale's plugin loader uses this to load NetworkCore before your plugin and to refuse to load your plugin if NetworkCore isn't installed.
If your plugin can run without NetworkCore (degraded), use `OptionalDependencies` instead and null-check before calling.
## 2. Get a reference
Two access patterns. Use whichever fits.
### Singleton (most common)
```java
import net.kewwbec.networkcore.NetworkCore;
NetworkCore core = NetworkCore.getInstance();
core.getMessageBus().publish("party.invite", invite);
```
This is the same pattern MinigameCoreV2 uses (`MinigameCore.getInstance()`).
### Service registry (when you want loose coupling)
```java
NetworkCore core = NetworkCore.getInstance();
MetricsService metrics = core.getService(MetricsService.class);
```
`getService` throws `IllegalStateException` if nothing is registered. Convenience getters like `getMetrics()` / `getMessageBus()` / `getServerRegistry()` / `getRpc()` return null instead, which is what you want for graceful degradation.
## 3. When to resolve services
Resolve in your plugin's `start()`, not `setup()`. The order is:
```
NetworkCore.setup() -> instance assigned, MetricsService registered
[your plugin].setup() -> NetworkCore.getInstance() works, getMetrics() works,
but getMessageBus() / getServerRegistry() / getRpc() return null
NetworkCore.start() -> Redis connects, MessageBus / ServerRegistry / RpcClient registered
[your plugin].start() -> everything is available
```
So:
```java
public final class YourPlugin extends JavaPlugin {
private MetricsService metrics;
private MessageBus bus;
@Override
protected void setup() {
// Safe: MetricsService is registered during NetworkCore.setup()
this.metrics = NetworkCore.getInstance().getMetrics();
}
@Override
protected void start() {
// Safe here: NetworkCore has finished start()
this.bus = NetworkCore.getInstance().getMessageBus();
if (bus != null) {
bus.subscribe("party.invite", PartyInvite.class, this::onInvite);
}
}
}
```
## 4. Contributing your own service
If your plugin offers an API other plugins might want to consume through the same registry:
```java
@Override
protected void start() {
NetworkCore.getInstance().registerService(PartyService.class, new MyPartyService());
}
```
Other plugins do `core.getService(PartyService.class)`. There's no requirement to do this - plugins can also expose their own singleton. But registering with the central registry is useful when you don't want consumers to import your plugin's classes directly.
## 5. Verify the integration
In-game, run `/networkcore status`. You should see the local server id, role, known servers count, and metrics endpoint. If any of those are missing, NetworkCore failed to start (check the log).
`curl http://127.0.0.1:9100/metrics` should return Prometheus output that includes `players_online` and JVM memory gauges.
+96
View File
@@ -0,0 +1,96 @@
# Configuration
## Location
`<plugin data dir>/config.json`. NetworkCore writes a default file on first boot if none exists, with the values shown below.
## Full config
```json
{
"server": {
"id": null,
"role": "lobby"
},
"redis": {
"host": "127.0.0.1",
"port": 6379,
"password": null,
"database": 0,
"key_prefix": "network:"
},
"network": {
"auth_token": null
},
"metrics": {
"enabled": true,
"bind": "127.0.0.1",
"port": 9100,
"path": "/metrics"
},
"rpc": {
"timeout_ms": 5000
}
}
```
## Fields
### server
| Field | Default | Meaning |
|---|---|---|
| `id` | `null` | Optional fixed server id. If null, auto-generated as `<role>-<8 hex chars>` (e.g. `lobby-3f2a91bd`) on every boot. Set this when you want a stable id across restarts. |
| `role` | `"lobby"` | Logical role of this server. Used to filter via `ServerRegistry.getServersByRole(...)`. Examples: `lobby`, `bridge_duel`, `arena`. Use snake_case. |
### redis
| Field | Default | Meaning |
|---|---|---|
| `host` | `"127.0.0.1"` | Redis host |
| `port` | `6379` | Redis port |
| `password` | `null` | Redis AUTH password. Prefer the env var (below). |
| `database` | `0` | Redis logical database (0-15 on stock Redis) |
| `key_prefix` | `"network:"` | All keys NetworkCore writes start with this prefix. Server registry keys end up at `network:servers:<id>`. Change this only if you're sharing a Redis with something unrelated. |
### network
| Field | Default | Meaning |
|---|---|---|
| `auth_token` | `null` | Shared secret. When set, every outgoing envelope is stamped with this token and every inbound message is rejected unless its token matches. When null, no auth is enforced. Use the env var. |
### metrics
| Field | Default | Meaning |
|---|---|---|
| `enabled` | `true` | Turn the whole metrics system on/off. When off, `getMetrics()` returns null. |
| `bind` | `"127.0.0.1"` | Address to bind the Prometheus HTTP server. Defaults to loopback so it's not exposed to the public internet. Set to `0.0.0.0` only if you've put auth / a reverse proxy in front. |
| `port` | `9100` | Port for the Prometheus exporter |
| `path` | `"/metrics"` | URL path. Standard convention is `/metrics`. |
### rpc
| Field | Default | Meaning |
|---|---|---|
| `timeout_ms` | `5000` | How long an RPC request waits for a response before completing the future exceptionally with `TimeoutException`. |
## Environment variable overrides
These take precedence over the config file at boot time. Use them for secrets so the config file stays safe to check into version control.
| Env var | Overrides |
|---|---|
| `REDIS_PASSWORD` | `redis.password` |
| `NETWORK_AUTH_TOKEN` | `network.auth_token` |
## Recommended setup per environment
**Local dev (single server):**
- Defaults are fine. Run Redis on localhost, don't set the auth token.
**Multi-server staging or production:**
- `server.id` left null is fine if you're OK with the id changing on every boot. Set it to something stable like `lobby-prod-01` if you want logs and dashboards to keep continuity across restarts.
- `server.role` set explicitly per server. This is what dashboards and routing use.
- `redis.host` pointed at your shared Redis. `redis.password` not in the file; set `REDIS_PASSWORD` env var.
- `network.auth_token` not in the file; set `NETWORK_AUTH_TOKEN` env var to the same value on every server.
- `metrics.bind` stays `127.0.0.1`. Run Prometheus on the same host (or use an SSH tunnel / reverse proxy with auth to expose it elsewhere).
+105
View File
@@ -0,0 +1,105 @@
# Metrics
## What you get
A `MetricsService` with three metric types and a Prometheus-compatible HTTP endpoint that scrapers (Prometheus, Grafana Agent, Vector) can read every N seconds.
Metric state lives in JVM memory only. It resets when the server restarts. Persistence is the scraper's job - see [09_Operations_And_Security.md](09_Operations_And_Security.md) for the rationale.
## API
```java
public interface MetricsService {
Counter counter(String name, Map<String, String> labels);
Gauge gauge(String name, Map<String, String> labels);
Timer timer(String name, Map<String, String> labels);
}
```
All three are interned by `(name, labels)`. Calling `counter("matches_started", Map.of("game", "bridge_duel"))` twice returns the same instance, so it's safe to call inside a hot path.
### Counter
Monotonically increasing.
```java
Counter started = metrics.counter("matches_started", Map.of("game", "bridge_duel"));
started.inc();
started.inc(5);
double v = started.value();
```
### Gauge
Point-in-time value. Set explicitly or bind to a supplier that's called whenever Prometheus scrapes.
```java
Gauge queueLen = metrics.gauge("queue_length", Map.of("game", "bridge_duel"));
queueLen.set(42);
Gauge memUsed = metrics.gauge("custom_memory_bytes", Map.of("pool", "off_heap"));
memUsed.bind(() -> myAllocator.currentBytes());
```
`bind` is preferred when the value is computed from somewhere else - it dodges write-paths and gets the current value at scrape time.
### Timer
Records durations. Tracks count and total seconds.
```java
Timer dbQuery = metrics.timer("db_query_seconds", Map.of("query", "find_player"));
long start = System.nanoTime();
runQuery();
dbQuery.recordNanos(System.nanoTime() - start);
```
The Prometheus output for a timer is two series: `<name>_count` and `<name>_sum_seconds`. To get the average in PromQL: `rate(db_query_seconds_sum_seconds[5m]) / rate(db_query_seconds_count[5m])`.
## Naming
Metric names get sanitized to `[a-zA-Z0-9_]` (anything else becomes `_`). Stick to lowercase snake_case to avoid surprises. Don't include label values in the name - use labels for that. Example:
Good: `matches_started{game="bridge_duel"}`
Bad: `matches_started_bridge_duel`
Labels with high cardinality (player UUIDs, match IDs, timestamps) will blow up Prometheus storage. Stick to bounded values: game id, world name, role.
## Built-in metrics
NetworkCore registers these on its own. You'll see them in the output without doing anything.
| Name | Type | Labels | Source |
|---|---|---|---|
| `players_online` | gauge | none | `Universe.get().getPlayerCount()` |
| `jvm_memory_used_bytes` | gauge | `pool=<pool name>` | JVM `MemoryPoolMXBean` |
| `messagebus_published_total` | counter | none | every `MessageBus.publish` call |
| `messagebus_received_total` | counter | none | every accepted incoming message |
| `messagebus_rejected_total` | counter | none | every message rejected due to bad auth token |
| `rpc_request_total` | counter | `service`, `method`, `result` | every RPC completion (`result` is `ok`, `error`, `timeout`, or `decode_error`) |
| `rpc_request_duration_seconds` | timer | `service`, `method` | every RPC completion |
## The endpoint
```
GET http://<bind>:<port><path>
```
Defaults: `http://127.0.0.1:9100/metrics`. Response is Prometheus text format 0.0.4. Bind defaults to loopback - you have to opt into exposing it externally (see [09_Operations_And_Security.md](09_Operations_And_Security.md)).
## Prometheus scrape config
```yaml
scrape_configs:
- job_name: 'kweebec_servers'
static_configs:
- targets:
- '10.0.0.10:9100'
- '10.0.0.11:9100'
- '10.0.0.12:9100'
```
## When *not* to use this
Don't store anything you want to query later (like "what was Bob's match count yesterday"). That's a database, not a metrics system. Use MetricsService for things you want to graph or alert on - rates, latencies, queue depths, error counts.
+139
View File
@@ -0,0 +1,139 @@
# MessageBus
A thin wrapper over Redis pub/sub. Use it for fire-and-forget cross-server events (party invites, chat broadcasts, "player joined queue X"). For request/response, see [07_RPC.md](07_RPC.md).
## API
```java
public interface MessageBus {
void publish(String channel, Object payload);
<T> Subscription subscribe(String channel, Class<T> type, Consumer<T> handler);
interface Subscription extends AutoCloseable {
@Override void close();
}
}
```
## Publishing
```java
PartyInvite invite = new PartyInvite(fromUuid, toUuid);
core.getMessageBus().publish("party.invite", invite);
```
`payload` can be any class Gson can serialize. The bus wraps it in an envelope before sending.
The call is non-blocking - it hands off to Lettuce's async API and returns immediately. There is no acknowledgement that another server received it. Pub/sub is at-most-once delivery: if no server is subscribed when you publish, the message disappears.
## Subscribing
```java
MessageBus.Subscription sub = core.getMessageBus().subscribe(
"party.invite",
PartyInvite.class,
invite -> handleInvite(invite)
);
```
Save the subscription if you want to unsubscribe later (`sub.close()`). NetworkCore's own subscriptions are cleaned up on shutdown, but yours aren't tied to anything by default.
### Thread safety - read this
Callbacks run on an internal worker pool (`networkcore-bus-*` threads), **not** on a Hytale world thread. If your handler touches world or entity state, hop to the right thread first:
```java
core.getMessageBus().subscribe("party.invite", PartyInvite.class, invite -> {
World world = playerRef.getWorld();
world.execute(() -> {
// Now safe to touch entities, components, etc.
playerRef.sendMessage(Message.raw("Party invite from " + invite.from));
});
});
```
Failing to hop will mostly work most of the time and then crash unpredictably under load. Always hop.
The worker pool is sized 2 core / 8 max, queue 1024. A slow handler will not block other subscribers - it'll just back up its own queue. If the queue fills, the publisher thread runs the task itself (CallerRunsPolicy), which slows down message intake until the backlog clears.
## The envelope
What actually goes over Redis:
```json
{
"v": 1,
"from": "lobby-3f2a91bd",
"auth": "<token from config>",
"ts": 1716610000000,
"type": "com.example.PartyInvite",
"body": { ... your payload ... }
}
```
You don't construct this. The bus does it. `type` is informational only - dispatch is by channel, not by class.
## Channel naming
A loose convention to keep things grep-able:
- `party.invite`, `party.disband` - simple cross-server events
- `chat.global` - broadcast channels
- `rpc:req:<service>`, `rpc:resp:<replyTo>` - reserved by RPC, don't touch
- `network:events` - reserved by NetworkCore for hello/goodbye, don't touch
## Auth filtering
If `network.auth_token` is set (see [03_Configuration.md](03_Configuration.md)), inbound messages whose envelope `auth` doesn't match are dropped silently (with a rate-limited warning log and an increment of `messagebus_rejected_total`). Outbound messages get stamped automatically.
If a misconfigured server keeps spamming the bus with the wrong token, you'll see one log line every 10s rather than one per message.
## Performance and limits
- Lettuce uses one connection for publish, one for subscribe. They share threading via Netty.
- Pub/sub is in-memory in Redis. No persistence. If Redis restarts, in-flight messages are lost.
- Redis pub/sub bandwidth is real but small - the practical bottleneck is usually subscriber-side processing, not network.
- Payloads are JSON. Keep them small. Don't put 10MB worlds through the bus - put them in MySQL or object storage and bus a reference.
## Common patterns
### Broadcast to everyone
Just publish - every server subscribed to the channel gets it.
```java
core.getMessageBus().publish("chat.global", chatMessage);
```
### Target a specific server
Encode the target in the payload, let receivers filter:
```java
class TargetedAction {
String targetServerId;
String action;
Map<String, Object> data;
}
bus.subscribe("admin.action", TargetedAction.class, action -> {
if (!action.targetServerId.equals(core.getServerId())) return;
apply(action);
});
```
Or use a per-server channel: `admin.action.<server-id>`.
### Avoid receiving your own messages
Filter on `serverId`:
```java
String myId = core.getServerId();
bus.subscribe("chat.global", ChatMessage.class, msg -> {
if (myId.equals(msg.fromServerId)) return;
relay(msg);
});
```
The bus does NOT filter your own publishes - all subscribers (including the publisher) see the message.
+106
View File
@@ -0,0 +1,106 @@
# ServerRegistry
Every server in the network heartbeats its presence to Redis. Every server keeps a local snapshot of who's online and reacts to changes via Hytale events.
## API
```java
public interface ServerRegistry {
ServerInfo getSelf();
Collection<ServerInfo> getServers();
Collection<ServerInfo> getServersByRole(String role);
@Nullable ServerInfo getServer(String id);
}
```
All four are local cache reads. No Redis round-trip per call.
```java
public final class ServerInfo {
String id(); // e.g. "lobby-3f2a91bd"
String role(); // e.g. "lobby", "bridge_duel"
String address(); // currently unused, reserved for future
int players(); // last reported player count
long startedAt(); // epoch millis when the server booted
long lastSeen(); // epoch millis of last heartbeat we observed
}
```
## How it works
On `start()`:
1. Subscribe to `network:events` (presence channel).
2. Write own heartbeat to `network:servers:<id>` (TTL 15s).
3. Scan `network:servers:*` to populate the local cache and fire join events for everyone discovered.
4. Publish a `hello` on `network:events`.
5. Schedule a heartbeat task every 5s. Each tick rewrites the heartbeat (resetting TTL) and re-scans.
On `stop()` (clean shutdown only):
1. Publish a `goodbye` on `network:events`.
2. Delete the registry key.
If a server hard-crashes (kill -9, OOM, network partition), no goodbye is sent. Other servers learn it's gone when the 15s TTL expires the key and the next scan doesn't see it.
## Detection latency
| Event | When other servers learn |
|---|---|
| New server boots | Within a few hundred ms (immediate hello over pub/sub) |
| Server clean-shutdown | Within a few hundred ms (immediate goodbye over pub/sub) |
| Server hard-crash | Up to 20s (TTL 15s + up to 5s for the next scan) |
## Events
When the registry's view of the network changes, it dispatches these on the Hytale event bus:
```java
import net.kewwbec.networkcore.api.events.ServerJoinedNetworkEvent;
import net.kewwbec.networkcore.api.events.ServerLeftNetworkEvent;
getEventRegistry().registerGlobal(ServerJoinedNetworkEvent.class, event -> {
ServerInfo server = event.getServer();
LOGGER.at(Level.INFO).log("Server joined: %s (role=%s)", server.id(), server.role());
});
getEventRegistry().registerGlobal(ServerLeftNetworkEvent.class, event -> {
ServerInfo server = event.getServer();
LOGGER.at(Level.INFO).log("Server left: %s", server.id());
});
```
Event handlers run on the Hytale event-bus dispatch thread. Same world-thread caveat as MessageBus subscribers: if you touch entities, hop to `world.execute(...)`.
## Server id and role
The `id` is either the value in `server.id` in `config.json`, or auto-generated as `<role>-<8 hex chars>` on every boot.
The `role` is a free-form string from `server.role`. Use snake_case. Examples:
- `lobby` - hub/main world
- `bridge_duel` - one Bridge Duel match host
- `arena` - PvP arena
- `staff` - admin-only world
Roles are how plugins route. Example: a queue plugin running on a lobby finds an available `bridge_duel` server with:
```java
Collection<ServerInfo> available = core.getServerRegistry()
.getServersByRole("bridge_duel")
.stream()
.filter(s -> s.players() < 20)
.toList();
```
## Redis layout
For reference, what NetworkCore writes:
```
network:servers:<id> HASH server heartbeat record, TTL 15s
fields: id, role, players, started_at, last_seen
network:events CHANNEL pub/sub for presence (hello/goodbye)
```
`network:` is the default `redis.key_prefix`. Change in config if needed.
+139
View File
@@ -0,0 +1,139 @@
# RPC
Request/response between servers, built on top of MessageBus. Use this when you need an answer; use MessageBus when fire-and-forget is enough.
## API
```java
public interface RpcClient {
<Req, Resp> CompletableFuture<Resp> request(
String service,
String method,
Req payload,
Class<Resp> responseType
);
RpcService registerService(String service);
}
public interface RpcService {
<Req, Resp> RpcService method(
String method,
Class<Req> requestType,
Class<Resp> responseType,
Function<Req, Resp> handler
);
void close();
}
```
## Calling
```java
record PlayerLookupReq(UUID uuid) {}
record PlayerLookupResp(String username, int kills) {}
CompletableFuture<PlayerLookupResp> future = core.getRpc().request(
"player-stats", // service name (matches what the server registered)
"lookup", // method name
new PlayerLookupReq(uuid),
PlayerLookupResp.class
);
future.whenComplete((resp, err) -> {
if (err != null) {
// err can be TimeoutException, RpcException (handler threw), or IllegalStateException (closed)
return;
}
use(resp);
});
```
The future completes:
- Normally with the decoded response if the handler returned successfully
- Exceptionally with `TimeoutException` if no response arrives within `rpc.timeout_ms` (default 5000)
- Exceptionally with `RpcClientImpl.RpcException` if the handler on the other side threw
- Exceptionally with `IllegalStateException("RpcClient closed")` if NetworkCore is shutting down
Don't block on `.get()` from a Hytale thread - hop to your own executor or use `.thenAccept` / `.whenComplete`.
## Serving
```java
RpcService stats = core.getRpc().registerService("player-stats");
stats.method("lookup", PlayerLookupReq.class, PlayerLookupResp.class, req -> {
Player p = lookupInDb(req.uuid());
return new PlayerLookupResp(p.username(), p.kills());
});
stats.method("topKills", TopKillsReq.class, TopKillsResp.class, this::handleTopKills);
```
The handler is called on the MessageBus worker pool (not a Hytale world thread). Same caveat as a subscriber - hop to world thread if you touch entities.
Handlers should return reasonably fast (under the RPC timeout, with margin). If a handler is slow or might block, run the work inside the handler and don't block the worker pool indefinitely - the pool is shared with regular MessageBus subscribers.
Throwing from a handler is captured and turned into an error response on the client side. Don't rely on this for control flow.
## How any-server-can-respond works
There's no service registry for RPC. The mechanism is:
1. Client publishes `RpcMessages.Request` on channel `rpc:req:<service>` with a unique correlation id and the client's reply channel.
2. Every server that has called `registerService("<service>")` is subscribed to that channel. They all receive the request.
If multiple servers register the same service name, all of them will process the request and reply. The client takes the first response that arrives and discards the rest (the future has already completed). For a real load-balanced setup, use distinct service names per instance, or have only one server register each service.
## Service naming
Use lowercase kebab-case. Pick names that describe the *capability*, not the server instance. Examples:
- `player-stats` - read player stats from MySQL
- `party-state` - the canonical party data store
- `match-allocator` - pick a server to host a match on
Don't:
- `player-stats-server-1` (couples the name to a specific instance)
- `lookup` (too generic, will collide)
## When RPC is the wrong tool
Use MessageBus instead when:
- You don't need a reply.
- You're broadcasting to all servers.
Use a database (MySQL) instead when:
- Multiple readers need consistent data.
- You need history or transactions.
Use the Hytale event bus (in-process) when:
- The consumer is on the same server.
RPC is for: "I need server X to do something for me and tell me how it went."
## Wire format
For reference. You don't construct these.
Request, published on `rpc:req:<service>` wrapped in the standard `Envelope`:
```json
{
"correlationId": "<uuid>",
"replyTo": "rpc:resp:<callerServerId>",
"method": "lookup",
"body": { ... }
}
```
Response, published on the `replyTo` channel:
```json
{
"correlationId": "<same uuid>",
"ok": true,
"error": null,
"body": { ... }
}
```
+60
View File
@@ -0,0 +1,60 @@
# Commands
Currently one in-game admin command: `/networkcore`.
All subcommands are run by an in-world player. Console support isn't wired up yet.
## /networkcore status
Prints local server identity, Redis state, and the metrics endpoint.
Example output:
```
=== NetworkCore ===
Server id: lobby-3f2a91bd
Role: lobby
Players: 12
Known servers: 4
Metrics: http://127.0.0.1:9100/metrics
```
If you see `Network: not connected (Redis unavailable)`, NetworkCore failed to open the Redis connection at start. Check the server log for the underlying error and verify `redis.host` / `redis.port` / `REDIS_PASSWORD`.
## /networkcore servers
Lists every server NetworkCore currently knows about (including itself).
Example:
```
=== Network Servers ===
lobby-3f2a91bd | role=lobby | players=12
bridge_duel-9c1b22a4 | role=bridge_duel | players=8
bridge_duel-d8e09131 | role=bridge_duel | players=0
arena-1f4e0bcc | role=arena | players=3
```
This reads from the local cache - same source as `ServerRegistry.getServers()`. If a server is missing here, either it hasn't joined the network yet or it failed to heartbeat (see [06_ServerRegistry.md](06_ServerRegistry.md) for detection latency).
## /networkcore publish &lt;channel&gt; &lt;json&gt;
Publishes a raw JSON payload to a MessageBus channel. Dev/debug helper - useful for poking subscribers without writing test code.
```
/networkcore publish chat.global {"from":"console","text":"test"}
```
The JSON is parsed and wrapped in the standard envelope (with your auth token if configured) before being sent. If the JSON is malformed you'll get an error in chat.
This is dangerous in production - it lets anyone with command access spoof any message on any channel. Lock it down with permissions before exposing to non-admins.
## Permissions
Subcommand permissions aren't enforced yet. If your server runs HytalePerms, gate `/networkcore` behind an admin group at the command-routing layer.
## Things this command does *not* do (and why)
- **Restart Redis / reload config.** Restart the server. Reload is fragile - half-loaded state is worse than no state.
- **Kick a server out of the network.** Servers self-register and self-deregister. If you want a server gone, stop its process. If you need to "evict" it administratively, that's a separate ops tool.
- **Send a chat message to all servers.** This belongs in a chat plugin that uses MessageBus, not in the core admin command.
+74
View File
@@ -0,0 +1,74 @@
# Operations and Security
## Threat model in one paragraph
NetworkCore assumes Redis is on a private network reachable only by trusted servers. The shared auth token guards against accidents (a misconfigured staging server talking to prod) and casual mischief (someone with bus access spoofing a server id). It does *not* protect against an attacker who already has Redis credentials - at that point they can read the token off any envelope and start signing their own. For hard isolation, rely on network controls: VPC peering, firewall rules, Redis bound to a private interface.
## Auth token rollout
```
NETWORK_AUTH_TOKEN=<long random string>
```
Pick something long. 32+ chars of randomness is fine. Set it the same way on every server.
When rotating: do a rolling restart. Mid-rotation, servers running the old token will reject messages from servers running the new token (and vice versa). `messagebus_rejected_total` will tick up, and you'll see warning logs at most once per 10s per server. Finish the rollout fast.
If you want zero rejection windows, you'd need a multi-token scheme where receivers accept either the old or new token during transition. That's not in v1.
## What is *not* encrypted or signed
- **Redis traffic.** Plain TCP. Anyone sniffing the wire between Hytale servers and Redis sees every payload and every token. Mitigations: keep Redis on a private network, run it on the same host as the Hytale server, or use SSH tunnels / a VPC. TLS (rediss://) is a future addition.
- **Messages themselves.** No HMAC, no signing. A token in the envelope is checked for equality, not for authenticity. An attacker on the bus can craft any envelope.
- **The metrics endpoint.** No auth at all. Defaults to `127.0.0.1` so only the local machine can hit it. If you bind to a non-loopback address, put a reverse proxy with auth in front.
## The metrics endpoint
`http://127.0.0.1:9100/metrics` by default. Three ways to consume it remotely:
1. **Run Prometheus on the same host.** Simplest. Prometheus hits 127.0.0.1:9100 and forwards via its own remote-write to a central store.
2. **SSH tunnel from the scraper host.** `ssh -L 9100:127.0.0.1:9100 hytale-server`, then point Prometheus at `127.0.0.1:9100`.
3. **Reverse proxy with auth.** Nginx in front of `127.0.0.1:9100`, requires HTTP Basic or mTLS. Change `metrics.bind` to `127.0.0.1` (default) and have nginx be the public face.
Do not change `metrics.bind` to `0.0.0.0` without one of the above. The endpoint leaks server identity, player counts, JVM internals, plugin state, and roughly what you're doing operationally.
## Ops checks at a glance
| Question | How to answer |
|---|---|
| Is this server alive in the network? | `/networkcore status` shows "Network: not connected" if Redis is down |
| Are other servers seeing me? | On another server: `/networkcore servers` - your id should be listed within a few seconds |
| Is the metrics endpoint up? | `curl http://127.0.0.1:9100/metrics` |
| Are servers communicating? | `redis-cli -h <host> MONITOR` and look for PUBLISH activity |
| Is Redis in the expected state? | `redis-cli -h <host> KEYS 'network:*'` shows registry hashes |
| Are auth rejects happening? | `messagebus_rejected_total` in `/metrics`; warning logs `Rejected message on ...` |
| What's the per-server RPC latency? | PromQL: `rate(rpc_request_duration_seconds_sum_seconds[1m]) / rate(rpc_request_duration_seconds_count[1m])` |
## Common failure modes
### "Failed to connect to Redis at host:port" at startup
Redis is unreachable. Check `redis.host` / `redis.port`. Check `REDIS_PASSWORD`. Check whether Redis is actually running. `/networkcore status` will show "Network: not connected" and the rest of the plugin keeps running with the message bus / registry / RPC services unregistered.
### `messagebus_rejected_total` is climbing
One of your servers has a different `NETWORK_AUTH_TOKEN` than the rest. Find it (the warning log includes the `from` server id) and fix.
### A server isn't appearing in `/networkcore servers` on other hosts
1. Did its `start()` complete? Check its log for "Connected to Redis ... auth=on/off".
2. Is its auth token the same as everyone else's?
3. `redis-cli -h <host> HGETALL network:servers:<that-id>` - if empty, it's not heartbeating.
4. If the hash is there but isn't appearing on other hosts, the receiver has the wrong token or the receiver's Redis is a different instance entirely (check `redis.database`).
### Servers stay listed for ~15s after crash
Expected. Hard crashes don't send the goodbye. The 15s TTL is what removes them.
### RPC requests time out
1. Is anyone actually serving the named service? `registerService("<name>")` must have been called.
2. Is the handler slow? Check `rpc_request_duration_seconds`.
3. Is the responder server reachable on the bus? If `/networkcore servers` shows it, yes.
## Sizing notes
- One Redis instance handles thousands of messages/second easily. The bottleneck is almost always subscriber processing, not Redis.
- The MessageBus worker pool is sized 2 core / 8 max. If you're doing heavy fan-in (lots of subscribers all doing real work on the same channel), you may want to bump this. It's a constant in `RedisMessageBus` right now; promote to config if needed.
- Heartbeat interval (5s) and TTL (15s) are constants. Tighter values mean faster failure detection at the cost of more Redis writes. 5s/15s is fine up to hundreds of servers.
- The local server-registry cache rescans Redis every 5s using `SCAN` - safe even with many keys, no `KEYS` blocking. Scales linearly with server count.
+29
View File
@@ -0,0 +1,29 @@
# NetworkCore Docs
NetworkCore is the foundational plugin every other KweebecNet plugin depends on. It owns metrics, cross-server messaging, and the server-discovery layer.
If you're building a plugin that needs to talk to other servers, record stats, or know who else is online, you depend on this.
## Reading order
1. [01_Overview.md](01_Overview.md) - what NetworkCore actually does and why it exists
2. [02_Plugin_Integration.md](02_Plugin_Integration.md) - how another plugin depends on it and gets a reference
3. [03_Configuration.md](03_Configuration.md) - config.json + env vars
4. [04_Metrics.md](04_Metrics.md) - MetricsService API and the Prometheus endpoint
5. [05_MessageBus.md](05_MessageBus.md) - Redis pub/sub for cross-server events
6. [06_ServerRegistry.md](06_ServerRegistry.md) - discovery and presence events
7. [07_RPC.md](07_RPC.md) - request/response between servers
8. [08_Commands.md](08_Commands.md) - the /networkcore admin command
9. [09_Operations_And_Security.md](09_Operations_And_Security.md) - auth token, what's not secured, ops checks
## Quick reference
| Thing | Class |
|---|---|
| Plugin entry | [NetworkCore](../src/main/java/net/kewwbec/networkcore/NetworkCore.java) |
| Singleton | `NetworkCore.getInstance()` |
| Manifest dependency string | `kewwbec:NetworkCore` |
| Default Redis | `127.0.0.1:6379` |
| Default Prometheus endpoint | `http://127.0.0.1:9100/metrics` |
| Auth env var | `NETWORK_AUTH_TOKEN` |
| Redis password env var | `REDIS_PASSWORD` |
+109
View File
@@ -0,0 +1,109 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>net.kewwbec</groupId>
<artifactId>networkcore</artifactId>
<version>0.1.0</version>
<packaging>jar</packaging>
<name>NetworkCore</name>
<description>Cross-server messaging and metrics core for KweebecNet</description>
<properties>
<maven.compiler.source>21</maven.compiler.source>
<maven.compiler.target>21</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<hytale.server.jar>${user.home}/AppData/Roaming/Hytale/install/release/package/game/latest/Server/HytaleServer.jar</hytale.server.jar>
<shade.base>net.kewwbec.networkcore.shaded</shade.base>
</properties>
<dependencies>
<dependency>
<groupId>com.hypixel.hytale</groupId>
<artifactId>hytale-server</artifactId>
<version>local</version>
<scope>system</scope>
<systemPath>${hytale.server.jar}</systemPath>
</dependency>
<dependency>
<groupId>io.lettuce</groupId>
<artifactId>lettuce-core</artifactId>
<version>6.3.2.RELEASE</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.11.0</version>
</dependency>
<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
<version>3.0.2</version>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<finalName>${project.artifactId}-${project.version}</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.5.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<minimizeJar>false</minimizeJar>
<artifactSet>
<excludes>
<exclude>com.hypixel.hytale:*</exclude>
<exclude>com.google.code.findbugs:*</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>module-info.class</exclude>
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>io.lettuce</pattern>
<shadedPattern>${shade.base}.lettuce</shadedPattern>
</relocation>
<relocation>
<pattern>io.netty</pattern>
<shadedPattern>${shade.base}.netty</shadedPattern>
</relocation>
<relocation>
<pattern>reactor</pattern>
<shadedPattern>${shade.base}.reactor</shadedPattern>
</relocation>
<relocation>
<pattern>org.reactivestreams</pattern>
<shadedPattern>${shade.base}.reactivestreams</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
@@ -0,0 +1,157 @@
package net.kewwbec.networkcore;
import com.hypixel.hytale.logger.HytaleLogger;
import com.hypixel.hytale.server.core.plugin.JavaPlugin;
import com.hypixel.hytale.server.core.plugin.JavaPluginInit;
import net.kewwbec.networkcore.api.MessageBus;
import net.kewwbec.networkcore.api.MetricsService;
import net.kewwbec.networkcore.api.RpcClient;
import net.kewwbec.networkcore.api.ServerRegistry;
import net.kewwbec.networkcore.bus.RedisMessageBus;
import net.kewwbec.networkcore.command.NetworkCoreCommand;
import net.kewwbec.networkcore.config.ConfigLoader;
import net.kewwbec.networkcore.config.CoreConfig;
import net.kewwbec.networkcore.internal.ServerIdGenerator;
import net.kewwbec.networkcore.internal.ServiceRegistry;
import net.kewwbec.networkcore.metrics.BuiltInMetrics;
import net.kewwbec.networkcore.metrics.MetricsServiceImpl;
import net.kewwbec.networkcore.metrics.PrometheusExporter;
import net.kewwbec.networkcore.registry.RedisServerRegistry;
import net.kewwbec.networkcore.rpc.RpcClientImpl;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.logging.Level;
public final class NetworkCore extends JavaPlugin {
private static final HytaleLogger LOGGER = HytaleLogger.forEnclosingClass();
private static volatile NetworkCore instance;
private final ServiceRegistry services = new ServiceRegistry();
private CoreConfig config;
private String serverId;
@Nullable private MetricsServiceImpl metricsImpl;
@Nullable private PrometheusExporter exporter;
@Nullable private RedisMessageBus messageBus;
@Nullable private RedisServerRegistry serverRegistry;
@Nullable private RpcClientImpl rpcClient;
public NetworkCore(@Nonnull JavaPluginInit init) {
super(init);
}
@Nonnull
public static NetworkCore getInstance() {
NetworkCore i = instance;
if (i == null) throw new IllegalStateException("NetworkCore not initialized");
return i;
}
@Override
protected void setup() {
instance = this;
try {
this.config = ConfigLoader.loadOrCreate(getDataDirectory());
} catch (IOException e) {
((HytaleLogger.Api) LOGGER.at(Level.SEVERE).withCause(e)).log("Failed to load NetworkCore config");
this.config = new CoreConfig();
}
this.serverId = (config.server.id == null || config.server.id.isBlank())
? ServerIdGenerator.generate(config.server.role)
: config.server.id;
if (config.metrics.enabled) {
this.metricsImpl = new MetricsServiceImpl();
services.register(MetricsService.class, metricsImpl);
BuiltInMetrics.register(metricsImpl);
}
getCommandRegistry().registerCommand(new NetworkCoreCommand(this));
LOGGER.at(Level.INFO).log("NetworkCore setup complete (server id %s, role %s)", serverId, config.server.role);
}
@Override
protected void start() {
if (metricsImpl != null && config.metrics.enabled) {
try {
this.exporter = new PrometheusExporter(metricsImpl, config.metrics.bind, config.metrics.port, config.metrics.path);
exporter.start();
} catch (IOException e) {
((HytaleLogger.Api) LOGGER.at(Level.SEVERE).withCause(e)).log("Failed to start Prometheus exporter on %s:%d", config.metrics.bind, config.metrics.port);
this.exporter = null;
}
}
try {
this.messageBus = new RedisMessageBus(config.redis, config.network.auth_token, serverId, metricsImpl);
services.register(MessageBus.class, messageBus);
} catch (RuntimeException e) {
((HytaleLogger.Api) LOGGER.at(Level.SEVERE).withCause(e)).log("Failed to connect to Redis at %s:%d", config.redis.host, config.redis.port);
return;
}
this.serverRegistry = new RedisServerRegistry(messageBus, config.redis, serverId, config.server.role);
serverRegistry.start();
services.register(ServerRegistry.class, serverRegistry);
this.rpcClient = new RpcClientImpl(messageBus, serverId, config.rpc.timeout_ms, metricsImpl);
services.register(RpcClient.class, rpcClient);
LOGGER.at(Level.INFO).log("NetworkCore started");
}
@Override
protected void shutdown() {
if (rpcClient != null) {
try { rpcClient.close(); } catch (RuntimeException ignored) {}
rpcClient = null;
}
if (serverRegistry != null) {
try { serverRegistry.stop(); } catch (RuntimeException ignored) {}
serverRegistry = null;
}
if (messageBus != null) {
try { messageBus.close(); } catch (RuntimeException ignored) {}
messageBus = null;
}
if (exporter != null) {
try { exporter.stop(); } catch (RuntimeException ignored) {}
exporter = null;
}
services.clear();
instance = null;
LOGGER.at(Level.INFO).log("NetworkCore shutdown complete");
}
@Nonnull
public <T> T getService(@Nonnull Class<T> type) {
return services.require(type);
}
public <T> void registerService(@Nonnull Class<T> type, @Nonnull T impl) {
services.register(type, impl);
}
@Nullable
public MetricsService getMetrics() { return services.get(MetricsService.class); }
@Nullable
public MessageBus getMessageBus() { return services.get(MessageBus.class); }
@Nullable
public ServerRegistry getServerRegistry() { return services.get(ServerRegistry.class); }
@Nullable
public RpcClient getRpc() { return services.get(RpcClient.class); }
@Nonnull
public CoreConfig getConfigSnapshot() {
return config;
}
@Nonnull
public String getServerId() {
return serverId;
}
}
@@ -0,0 +1,10 @@
package net.kewwbec.networkcore.api;
public interface Counter {
void inc();
void inc(double amount);
double value();
}
@@ -0,0 +1,13 @@
package net.kewwbec.networkcore.api;
import javax.annotation.Nonnull;
import java.util.function.DoubleSupplier;
public interface Gauge {
void set(double value);
void bind(@Nonnull DoubleSupplier supplier);
double value();
}
@@ -0,0 +1,21 @@
package net.kewwbec.networkcore.api;
import javax.annotation.Nonnull;
import java.util.function.Consumer;
public interface MessageBus {
void publish(@Nonnull String channel, @Nonnull Object payload);
/**
* Callbacks run on the bus's worker pool, NOT on a Hytale world thread.
* If you touch world or entity state from the handler, hop via World.execute.
*/
@Nonnull
<T> Subscription subscribe(@Nonnull String channel, @Nonnull Class<T> type, @Nonnull Consumer<T> handler);
interface Subscription extends AutoCloseable {
@Override
void close();
}
}
@@ -0,0 +1,16 @@
package net.kewwbec.networkcore.api;
import javax.annotation.Nonnull;
import java.util.Map;
public interface MetricsService {
@Nonnull
Counter counter(@Nonnull String name, @Nonnull Map<String, String> labels);
@Nonnull
Gauge gauge(@Nonnull String name, @Nonnull Map<String, String> labels);
@Nonnull
Timer timer(@Nonnull String name, @Nonnull Map<String, String> labels);
}
@@ -0,0 +1,16 @@
package net.kewwbec.networkcore.api;
import javax.annotation.Nonnull;
import java.util.concurrent.CompletableFuture;
public interface RpcClient {
@Nonnull
<Req, Resp> CompletableFuture<Resp> request(@Nonnull String service,
@Nonnull String method,
@Nonnull Req payload,
@Nonnull Class<Resp> responseType);
@Nonnull
RpcService registerService(@Nonnull String service);
}
@@ -0,0 +1,15 @@
package net.kewwbec.networkcore.api;
import javax.annotation.Nonnull;
import java.util.function.Function;
public interface RpcService {
@Nonnull
<Req, Resp> RpcService method(@Nonnull String method,
@Nonnull Class<Req> requestType,
@Nonnull Class<Resp> responseType,
@Nonnull Function<Req, Resp> handler);
void close();
}
@@ -0,0 +1,59 @@
package net.kewwbec.networkcore.api;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.Objects;
public final class ServerInfo {
private final String id;
private final String role;
@Nullable
private final String address;
private final int players;
private final long startedAt;
private final long lastSeen;
public ServerInfo(@Nonnull String id,
@Nonnull String role,
@Nullable String address,
int players,
long startedAt,
long lastSeen) {
this.id = id;
this.role = role;
this.address = address;
this.players = players;
this.startedAt = startedAt;
this.lastSeen = lastSeen;
}
@Nonnull public String id() { return id; }
@Nonnull public String role() { return role; }
@Nullable public String address() { return address; }
public int players() { return players; }
public long startedAt() { return startedAt; }
public long lastSeen() { return lastSeen; }
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof ServerInfo other)) return false;
return players == other.players
&& startedAt == other.startedAt
&& lastSeen == other.lastSeen
&& id.equals(other.id)
&& role.equals(other.role)
&& Objects.equals(address, other.address);
}
@Override
public int hashCode() {
return Objects.hash(id, role, address, players, startedAt, lastSeen);
}
@Override
public String toString() {
return "ServerInfo[" + id + " role=" + role + " players=" + players + "]";
}
}
@@ -0,0 +1,20 @@
package net.kewwbec.networkcore.api;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.Collection;
public interface ServerRegistry {
@Nonnull
ServerInfo getSelf();
@Nonnull
Collection<ServerInfo> getServers();
@Nonnull
Collection<ServerInfo> getServersByRole(@Nonnull String role);
@Nullable
ServerInfo getServer(@Nonnull String id);
}
@@ -0,0 +1,12 @@
package net.kewwbec.networkcore.api;
public interface Timer {
void recordNanos(long nanos);
void recordSeconds(double seconds);
long count();
double totalSeconds();
}
@@ -0,0 +1,20 @@
package net.kewwbec.networkcore.api.events;
import com.hypixel.hytale.event.IEvent;
import net.kewwbec.networkcore.api.ServerInfo;
import javax.annotation.Nonnull;
public final class ServerJoinedNetworkEvent implements IEvent<Object> {
private final ServerInfo server;
public ServerJoinedNetworkEvent(@Nonnull ServerInfo server) {
this.server = server;
}
@Nonnull
public ServerInfo getServer() {
return server;
}
}
@@ -0,0 +1,20 @@
package net.kewwbec.networkcore.api.events;
import com.hypixel.hytale.event.IEvent;
import net.kewwbec.networkcore.api.ServerInfo;
import javax.annotation.Nonnull;
public final class ServerLeftNetworkEvent implements IEvent<Object> {
private final ServerInfo server;
public ServerLeftNetworkEvent(@Nonnull ServerInfo server) {
this.server = server;
}
@Nonnull
public ServerInfo getServer() {
return server;
}
}
@@ -0,0 +1,15 @@
package net.kewwbec.networkcore.bus;
import com.google.gson.JsonElement;
import javax.annotation.Nullable;
public final class Envelope {
public int v = 1;
@Nullable public String from;
@Nullable public String auth;
public long ts;
@Nullable public String type;
@Nullable public JsonElement body;
}
@@ -0,0 +1,207 @@
package net.kewwbec.networkcore.bus;
import com.google.gson.Gson;
import com.hypixel.hytale.logger.HytaleLogger;
import io.lettuce.core.RedisClient;
import io.lettuce.core.RedisURI;
import io.lettuce.core.api.StatefulRedisConnection;
import io.lettuce.core.api.sync.RedisCommands;
import io.lettuce.core.pubsub.RedisPubSubAdapter;
import io.lettuce.core.pubsub.StatefulRedisPubSubConnection;
import io.lettuce.core.pubsub.api.sync.RedisPubSubCommands;
import net.kewwbec.networkcore.api.Counter;
import net.kewwbec.networkcore.api.MessageBus;
import net.kewwbec.networkcore.api.MetricsService;
import net.kewwbec.networkcore.config.CoreConfig;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.time.Duration;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Consumer;
import java.util.logging.Level;
public final class RedisMessageBus implements MessageBus {
private static final HytaleLogger LOGGER = HytaleLogger.forEnclosingClass();
private final Gson gson = new Gson();
private final String serverId;
@Nullable private final String authToken;
private final RedisClient client;
private final StatefulRedisConnection<String, String> commands;
private final StatefulRedisPubSubConnection<String, String> pubSub;
private final RedisPubSubCommands<String, String> pubSubSync;
private final ThreadPoolExecutor workers;
private final Map<String, List<Subscriber<?>>> subscribers = new ConcurrentHashMap<>();
private final AtomicLong lastRejectLogMillis = new AtomicLong(0L);
@Nullable private final Counter publishedCounter;
@Nullable private final Counter receivedCounter;
@Nullable private final Counter rejectedCounter;
public RedisMessageBus(@Nonnull CoreConfig.RedisSection redis,
@Nullable String authToken,
@Nonnull String serverId,
@Nullable MetricsService metrics) {
this.serverId = serverId;
this.authToken = (authToken == null || authToken.isEmpty()) ? null : authToken;
RedisURI.Builder b = RedisURI.Builder.redis(redis.host, redis.port)
.withDatabase(redis.database)
.withTimeout(Duration.ofSeconds(5));
if (redis.password != null && !redis.password.isEmpty()) {
b.withPassword(redis.password.toCharArray());
}
RedisURI uri = b.build();
this.client = RedisClient.create(uri);
this.commands = client.connect();
this.pubSub = client.connectPubSub();
this.pubSubSync = pubSub.sync();
this.workers = new ThreadPoolExecutor(
2, 8,
60L, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(1024),
r -> {
Thread t = new Thread(r, "networkcore-bus-" + UUID.randomUUID().toString().substring(0, 4));
t.setDaemon(true);
return t;
},
new ThreadPoolExecutor.CallerRunsPolicy()
);
pubSub.addListener(new RedisPubSubAdapter<>() {
@Override
public void message(String channel, String message) {
handleMessage(channel, message);
}
});
this.publishedCounter = (metrics == null) ? null : metrics.counter("messagebus_published_total", Map.of());
this.receivedCounter = (metrics == null) ? null : metrics.counter("messagebus_received_total", Map.of());
this.rejectedCounter = (metrics == null) ? null : metrics.counter("messagebus_rejected_total", Map.of());
LOGGER.at(Level.INFO).log("Connected to Redis at %s:%d (db %d, auth=%s)",
redis.host, redis.port, redis.database, this.authToken != null ? "on" : "off");
}
@Nonnull
public RedisCommands<String, String> sync() {
return commands.sync();
}
@Override
public void publish(@Nonnull String channel, @Nonnull Object payload) {
Envelope env = new Envelope();
env.from = serverId;
env.auth = authToken;
env.ts = System.currentTimeMillis();
env.type = payload.getClass().getName();
env.body = gson.toJsonTree(payload);
String json = gson.toJson(env);
commands.async().publish(channel, json);
if (publishedCounter != null) publishedCounter.inc();
}
@Override
@Nonnull
public <T> Subscription subscribe(@Nonnull String channel, @Nonnull Class<T> type, @Nonnull Consumer<T> handler) {
Subscriber<T> sub = new Subscriber<>(type, handler);
List<Subscriber<?>> list = subscribers.computeIfAbsent(channel, k -> new CopyOnWriteArrayList<>());
boolean firstForChannel = list.isEmpty();
list.add(sub);
if (firstForChannel) {
pubSubSync.subscribe(channel);
}
return () -> {
list.remove(sub);
if (list.isEmpty()) {
try {
pubSubSync.unsubscribe(channel);
} catch (RuntimeException ignored) {
}
subscribers.remove(channel, list);
}
};
}
private void handleMessage(@Nonnull String channel, @Nonnull String message) {
List<Subscriber<?>> list = subscribers.get(channel);
if (list == null || list.isEmpty()) {
return;
}
Envelope env;
try {
env = gson.fromJson(message, Envelope.class);
} catch (RuntimeException e) {
LOGGER.at(Level.WARNING).log("Failed to parse envelope on %s: %s", channel, e.getMessage());
return;
}
if (env == null || env.body == null) return;
if (authToken != null && !authToken.equals(env.auth)) {
if (rejectedCounter != null) rejectedCounter.inc();
long now = System.currentTimeMillis();
long last = lastRejectLogMillis.get();
if (now - last > 10_000L && lastRejectLogMillis.compareAndSet(last, now)) {
LOGGER.at(Level.WARNING).log("Rejected message on %s from %s: bad auth token", channel, env.from);
}
return;
}
if (receivedCounter != null) receivedCounter.inc();
for (Subscriber<?> sub : list) {
dispatch(sub, env);
}
}
private <T> void dispatch(@Nonnull Subscriber<T> sub, @Nonnull Envelope env) {
T payload;
try {
payload = gson.fromJson(env.body, sub.type);
} catch (RuntimeException e) {
LOGGER.at(Level.WARNING).log("Failed to decode payload as %s: %s", sub.type.getName(), e.getMessage());
return;
}
workers.execute(() -> {
try {
sub.handler.accept(payload);
} catch (Throwable t) {
((HytaleLogger.Api) LOGGER.at(Level.SEVERE).withCause(t)).log("Subscriber for %s threw", sub.type.getName());
}
});
}
public void close() {
try { pubSub.close(); } catch (RuntimeException ignored) {}
try { commands.close(); } catch (RuntimeException ignored) {}
try { client.shutdown(); } catch (RuntimeException ignored) {}
workers.shutdown();
try {
if (!workers.awaitTermination(3, TimeUnit.SECONDS)) {
workers.shutdownNow();
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
workers.shutdownNow();
}
}
private static final class Subscriber<T> {
final Class<T> type;
final Consumer<T> handler;
Subscriber(@Nonnull Class<T> type, @Nonnull Consumer<T> handler) {
this.type = type;
this.handler = handler;
}
}
}
@@ -0,0 +1,141 @@
package net.kewwbec.networkcore.command;
import com.google.gson.Gson;
import com.google.gson.JsonElement;
import com.google.gson.JsonSyntaxException;
import com.hypixel.hytale.component.Ref;
import com.hypixel.hytale.component.Store;
import com.hypixel.hytale.server.core.Message;
import com.hypixel.hytale.server.core.command.system.CommandContext;
import com.hypixel.hytale.server.core.command.system.arguments.system.RequiredArg;
import com.hypixel.hytale.server.core.command.system.arguments.types.ArgTypes;
import com.hypixel.hytale.server.core.command.system.basecommands.AbstractCommandCollection;
import com.hypixel.hytale.server.core.command.system.basecommands.AbstractPlayerCommand;
import com.hypixel.hytale.server.core.universe.PlayerRef;
import com.hypixel.hytale.server.core.universe.world.World;
import com.hypixel.hytale.server.core.universe.world.storage.EntityStore;
import net.kewwbec.networkcore.NetworkCore;
import net.kewwbec.networkcore.api.MessageBus;
import net.kewwbec.networkcore.api.ServerInfo;
import net.kewwbec.networkcore.api.ServerRegistry;
import net.kewwbec.networkcore.config.CoreConfig;
import javax.annotation.Nonnull;
import java.awt.Color;
public final class NetworkCoreCommand extends AbstractCommandCollection {
public NetworkCoreCommand(@Nonnull NetworkCore plugin) {
super("networkcore", "NetworkCore admin commands");
addSubCommand(new StatusCommand(plugin));
addSubCommand(new ServersCommand(plugin));
addSubCommand(new PublishCommand(plugin));
}
public static final class StatusCommand extends AbstractPlayerCommand {
private final NetworkCore plugin;
public StatusCommand(@Nonnull NetworkCore plugin) {
super("status", "Show local server status");
this.plugin = plugin;
}
@Override
protected void execute(@Nonnull CommandContext context,
@Nonnull Store<EntityStore> store,
@Nonnull Ref<EntityStore> ref,
@Nonnull PlayerRef playerRef,
@Nonnull World world) {
CoreConfig.MetricsSection m = plugin.getConfigSnapshot().metrics;
context.sendMessage(Message.raw("=== NetworkCore ===").color(Color.YELLOW));
context.sendMessage(Message.raw("Server id: " + plugin.getServerId()).color(Color.WHITE));
context.sendMessage(Message.raw("Role: " + plugin.getConfigSnapshot().server.role).color(Color.WHITE));
ServerRegistry registry = plugin.getServerRegistry();
if (registry == null) {
context.sendMessage(Message.raw("Network: not connected (Redis unavailable)").color(Color.RED));
} else {
ServerInfo self = registry.getSelf();
context.sendMessage(Message.raw("Players: " + self.players()).color(Color.WHITE));
context.sendMessage(Message.raw("Known servers: " + registry.getServers().size()).color(Color.WHITE));
}
if (m.enabled) {
context.sendMessage(Message.raw("Metrics: http://" + m.bind + ":" + m.port + m.path).color(Color.WHITE));
} else {
context.sendMessage(Message.raw("Metrics: disabled").color(Color.WHITE));
}
}
}
public static final class ServersCommand extends AbstractPlayerCommand {
private final NetworkCore plugin;
public ServersCommand(@Nonnull NetworkCore plugin) {
super("servers", "List known servers in the network");
this.plugin = plugin;
}
@Override
protected void execute(@Nonnull CommandContext context,
@Nonnull Store<EntityStore> store,
@Nonnull Ref<EntityStore> ref,
@Nonnull PlayerRef playerRef,
@Nonnull World world) {
ServerRegistry registry = plugin.getServerRegistry();
if (registry == null) {
context.sendMessage(Message.raw("Network not connected (Redis unavailable).").color(Color.RED));
return;
}
context.sendMessage(Message.raw("=== Network Servers ===").color(Color.YELLOW));
for (ServerInfo s : registry.getServers()) {
context.sendMessage(Message.raw(
s.id() + " | role=" + s.role() + " | players=" + s.players()
).color(Color.WHITE));
}
}
}
public static final class PublishCommand extends AbstractPlayerCommand {
private static final Gson GSON = new Gson();
private final NetworkCore plugin;
private final RequiredArg<String> channelArg = withRequiredArg("channel", "Channel name", ArgTypes.STRING);
private final RequiredArg<String> jsonArg = withRequiredArg("json", "JSON payload", ArgTypes.STRING);
public PublishCommand(@Nonnull NetworkCore plugin) {
super("publish", "Publish a raw JSON payload to a channel");
this.plugin = plugin;
}
@Override
protected void execute(@Nonnull CommandContext context,
@Nonnull Store<EntityStore> store,
@Nonnull Ref<EntityStore> ref,
@Nonnull PlayerRef playerRef,
@Nonnull World world) {
String channel = channelArg.get(context);
String json = jsonArg.get(context);
if (channel == null || channel.isBlank() || json == null) {
context.sendMessage(Message.raw("Usage: /networkcore publish <channel> <json>").color(Color.YELLOW));
return;
}
JsonElement parsed;
try {
parsed = GSON.fromJson(json, JsonElement.class);
} catch (JsonSyntaxException e) {
context.sendMessage(Message.raw("Invalid JSON: " + e.getMessage()).color(Color.RED));
return;
}
MessageBus bus = plugin.getMessageBus();
if (bus == null) {
context.sendMessage(Message.raw("MessageBus not initialized.").color(Color.RED));
return;
}
bus.publish(channel, parsed);
context.sendMessage(Message.raw("Published to " + channel).color(Color.GREEN));
}
}
}
@@ -0,0 +1,53 @@
package net.kewwbec.networkcore.config;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
public final class ConfigLoader {
private static final Gson GSON = new GsonBuilder().setPrettyPrinting().create();
private ConfigLoader() {
}
@Nonnull
public static CoreConfig loadOrCreate(@Nonnull Path dataDirectory) throws IOException {
Path file = dataDirectory.resolve("config.json");
if (!Files.exists(file)) {
Files.createDirectories(dataDirectory);
CoreConfig fresh = new CoreConfig();
Files.writeString(file, GSON.toJson(fresh), StandardCharsets.UTF_8);
return applyEnvOverrides(fresh);
}
String json = Files.readString(file, StandardCharsets.UTF_8);
CoreConfig parsed = GSON.fromJson(json, CoreConfig.class);
if (parsed == null) {
parsed = new CoreConfig();
}
if (parsed.server == null) parsed.server = new CoreConfig.ServerSection();
if (parsed.redis == null) parsed.redis = new CoreConfig.RedisSection();
if (parsed.network == null) parsed.network = new CoreConfig.NetworkSection();
if (parsed.metrics == null) parsed.metrics = new CoreConfig.MetricsSection();
if (parsed.rpc == null) parsed.rpc = new CoreConfig.RpcSection();
return applyEnvOverrides(parsed);
}
@Nonnull
private static CoreConfig applyEnvOverrides(@Nonnull CoreConfig config) {
String envPw = System.getenv("REDIS_PASSWORD");
if (envPw != null && !envPw.isEmpty()) {
config.redis.password = envPw;
}
String envToken = System.getenv("NETWORK_AUTH_TOKEN");
if (envToken != null && !envToken.isEmpty()) {
config.network.auth_token = envToken;
}
return config;
}
}
@@ -0,0 +1,43 @@
package net.kewwbec.networkcore.config;
import javax.annotation.Nullable;
public final class CoreConfig {
public ServerSection server = new ServerSection();
public RedisSection redis = new RedisSection();
public NetworkSection network = new NetworkSection();
public MetricsSection metrics = new MetricsSection();
public RpcSection rpc = new RpcSection();
public static final class ServerSection {
@Nullable
public String id = null;
public String role = "lobby";
}
public static final class RedisSection {
public String host = "127.0.0.1";
public int port = 6379;
@Nullable
public String password = null;
public int database = 0;
public String key_prefix = "network:";
}
public static final class NetworkSection {
@Nullable
public String auth_token = null;
}
public static final class MetricsSection {
public boolean enabled = true;
public String bind = "127.0.0.1";
public int port = 9100;
public String path = "/metrics";
}
public static final class RpcSection {
public long timeout_ms = 5000L;
}
}
@@ -0,0 +1,20 @@
package net.kewwbec.networkcore.internal;
import javax.annotation.Nonnull;
import java.util.UUID;
public final class ServerIdGenerator {
private ServerIdGenerator() {
}
@Nonnull
public static String generate(@Nonnull String role) {
String shortUuid = UUID.randomUUID().toString().substring(0, 8);
String safeRole = role.trim().toLowerCase().replaceAll("[^a-z0-9_]", "_");
if (safeRole.isEmpty()) {
safeRole = "server";
}
return safeRole + "-" + shortUuid;
}
}
@@ -0,0 +1,41 @@
package net.kewwbec.networkcore.internal;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
public final class ServiceRegistry {
private final Map<Class<?>, Object> services = new ConcurrentHashMap<>();
public <T> void register(@Nonnull Class<T> type, @Nonnull T impl) {
services.put(type, impl);
}
public <T> void unregister(@Nonnull Class<T> type) {
services.remove(type);
}
@Nullable
public <T> T get(@Nonnull Class<T> type) {
Object value = services.get(type);
if (value == null) {
return null;
}
return type.cast(value);
}
@Nonnull
public <T> T require(@Nonnull Class<T> type) {
T value = get(type);
if (value == null) {
throw new IllegalStateException("No service registered for " + type.getName());
}
return value;
}
public void clear() {
services.clear();
}
}
@@ -0,0 +1,29 @@
package net.kewwbec.networkcore.metrics;
import com.hypixel.hytale.server.core.universe.Universe;
import net.kewwbec.networkcore.api.Gauge;
import javax.annotation.Nonnull;
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryPoolMXBean;
import java.util.Map;
public final class BuiltInMetrics {
private BuiltInMetrics() {
}
public static void register(@Nonnull MetricsServiceImpl metrics) {
Gauge players = metrics.gauge("players_online", Map.of());
players.bind(() -> {
Universe universe = Universe.get();
return (universe == null) ? 0 : universe.getPlayerCount();
});
for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) {
String name = pool.getName();
Gauge used = metrics.gauge("jvm_memory_used_bytes", Map.of("pool", name));
used.bind(() -> pool.getUsage() == null ? 0.0D : pool.getUsage().getUsed());
}
}
}
@@ -0,0 +1,137 @@
package net.kewwbec.networkcore.metrics;
import net.kewwbec.networkcore.api.Counter;
import net.kewwbec.networkcore.api.Gauge;
import net.kewwbec.networkcore.api.MetricsService;
import net.kewwbec.networkcore.api.Timer;
import javax.annotation.Nonnull;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.TreeMap;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.DoubleAdder;
import java.util.concurrent.atomic.LongAdder;
import java.util.function.DoubleSupplier;
public final class MetricsServiceImpl implements MetricsService {
private final Map<MetricKey, CounterImpl> counters = new ConcurrentHashMap<>();
private final Map<MetricKey, GaugeImpl> gauges = new ConcurrentHashMap<>();
private final Map<MetricKey, TimerImpl> timers = new ConcurrentHashMap<>();
@Override
@Nonnull
public Counter counter(@Nonnull String name, @Nonnull Map<String, String> labels) {
return counters.computeIfAbsent(new MetricKey(name, labels), CounterImpl::new);
}
@Override
@Nonnull
public Gauge gauge(@Nonnull String name, @Nonnull Map<String, String> labels) {
return gauges.computeIfAbsent(new MetricKey(name, labels), GaugeImpl::new);
}
@Override
@Nonnull
public Timer timer(@Nonnull String name, @Nonnull Map<String, String> labels) {
return timers.computeIfAbsent(new MetricKey(name, labels), TimerImpl::new);
}
@Nonnull
public Collection<CounterImpl> counterSamples() { return Collections.unmodifiableCollection(counters.values()); }
@Nonnull
public Collection<GaugeImpl> gaugeSamples() { return Collections.unmodifiableCollection(gauges.values()); }
@Nonnull
public Collection<TimerImpl> timerSamples() { return Collections.unmodifiableCollection(timers.values()); }
public static final class MetricKey {
final String name;
final Map<String, String> labels;
MetricKey(@Nonnull String name, @Nonnull Map<String, String> labels) {
this.name = name;
this.labels = new TreeMap<>(labels);
}
@Nonnull public String name() { return name; }
@Nonnull public Map<String, String> labels() { return labels; }
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof MetricKey other)) return false;
return name.equals(other.name) && labels.equals(other.labels);
}
@Override
public int hashCode() {
int r = name.hashCode();
r = 31 * r + labels.hashCode();
return r;
}
}
public static final class CounterImpl implements Counter {
private final MetricKey key;
private final DoubleAdder value = new DoubleAdder();
CounterImpl(@Nonnull MetricKey key) {
this.key = key;
}
@Override public void inc() { value.add(1.0D); }
@Override public void inc(double amount) { value.add(amount); }
@Override public double value() { return value.sum(); }
@Nonnull public MetricKey key() { return key; }
}
public static final class GaugeImpl implements Gauge {
private final MetricKey key;
private volatile double explicit = 0.0D;
private volatile DoubleSupplier supplier;
GaugeImpl(@Nonnull MetricKey key) {
this.key = key;
}
@Override public void set(double value) { this.supplier = null; this.explicit = value; }
@Override public void bind(@Nonnull DoubleSupplier s) { this.supplier = s; }
@Override
public double value() {
DoubleSupplier s = supplier;
return (s != null) ? s.getAsDouble() : explicit;
}
@Nonnull public MetricKey key() { return key; }
}
public static final class TimerImpl implements Timer {
private final MetricKey key;
private final LongAdder count = new LongAdder();
private final DoubleAdder totalSeconds = new DoubleAdder();
TimerImpl(@Nonnull MetricKey key) {
this.key = key;
}
@Override
public void recordNanos(long nanos) {
count.add(1);
totalSeconds.add(nanos / 1_000_000_000.0D);
}
@Override
public void recordSeconds(double seconds) {
count.add(1);
totalSeconds.add(seconds);
}
@Override public long count() { return count.sum(); }
@Override public double totalSeconds() { return totalSeconds.sum(); }
@Nonnull public MetricKey key() { return key; }
}
}
@@ -0,0 +1,127 @@
package net.kewwbec.networkcore.metrics;
import com.sun.net.httpserver.HttpServer;
import com.hypixel.hytale.logger.HytaleLogger;
import javax.annotation.Nonnull;
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.util.Map;
import java.util.concurrent.Executors;
import java.util.logging.Level;
public final class PrometheusExporter {
private static final HytaleLogger LOGGER = HytaleLogger.forEnclosingClass();
private final MetricsServiceImpl metrics;
private final String bindHost;
private final int port;
private final String path;
private HttpServer server;
public PrometheusExporter(@Nonnull MetricsServiceImpl metrics,
@Nonnull String bindHost,
int port,
@Nonnull String path) {
this.metrics = metrics;
this.bindHost = bindHost;
this.port = port;
this.path = path;
}
public void start() throws IOException {
HttpServer s = HttpServer.create(new InetSocketAddress(bindHost, port), 16);
s.createContext(path, exchange -> {
byte[] body = render().getBytes(StandardCharsets.UTF_8);
exchange.getResponseHeaders().set("Content-Type", "text/plain; version=0.0.4; charset=utf-8");
exchange.sendResponseHeaders(200, body.length);
try (OutputStream os = exchange.getResponseBody()) {
os.write(body);
}
});
s.setExecutor(Executors.newFixedThreadPool(2, r -> {
Thread t = new Thread(r, "networkcore-metrics-http");
t.setDaemon(true);
return t;
}));
s.start();
this.server = s;
LOGGER.at(Level.INFO).log("Prometheus exporter listening on http://%s:%d%s", bindHost, port, path);
}
public void stop() {
if (server != null) {
server.stop(0);
server = null;
}
}
@Nonnull
String render() {
StringBuilder sb = new StringBuilder(4096);
for (MetricsServiceImpl.CounterImpl c : metrics.counterSamples()) {
appendMetric(sb, c.key(), "counter", c.value());
}
for (MetricsServiceImpl.GaugeImpl g : metrics.gaugeSamples()) {
appendMetric(sb, g.key(), "gauge", g.value());
}
for (MetricsServiceImpl.TimerImpl t : metrics.timerSamples()) {
appendMetric(sb, withSuffix(t.key(), "_count"), "counter", t.count());
appendMetric(sb, withSuffix(t.key(), "_sum_seconds"), "counter", t.totalSeconds());
}
return sb.toString();
}
@Nonnull
private static MetricsServiceImpl.MetricKey withSuffix(@Nonnull MetricsServiceImpl.MetricKey key, @Nonnull String suffix) {
return new MetricsServiceImpl.MetricKey(sanitize(key.name()) + suffix, key.labels());
}
private static void appendMetric(@Nonnull StringBuilder sb,
@Nonnull MetricsServiceImpl.MetricKey key,
@Nonnull String type,
double value) {
String name = sanitize(key.name());
sb.append("# TYPE ").append(name).append(' ').append(type).append('\n');
sb.append(name);
Map<String, String> labels = key.labels();
if (!labels.isEmpty()) {
sb.append('{');
boolean first = true;
for (Map.Entry<String, String> e : labels.entrySet()) {
if (!first) sb.append(',');
first = false;
sb.append(sanitize(e.getKey())).append("=\"").append(escape(e.getValue())).append('"');
}
sb.append('}');
}
sb.append(' ').append(formatDouble(value)).append('\n');
}
@Nonnull
private static String sanitize(@Nonnull String s) {
StringBuilder out = new StringBuilder(s.length());
for (int i = 0; i < s.length(); i++) {
char c = s.charAt(i);
boolean ok = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || c == '_';
out.append(ok ? c : '_');
}
return out.toString();
}
@Nonnull
private static String escape(@Nonnull String s) {
return s.replace("\\", "\\\\").replace("\"", "\\\"").replace("\n", "\\n");
}
@Nonnull
private static String formatDouble(double v) {
if (v == Math.floor(v) && !Double.isInfinite(v)) {
return Long.toString((long) v);
}
return Double.toString(v);
}
}
@@ -0,0 +1,15 @@
package net.kewwbec.networkcore.registry;
public final class PresencePayload {
public static final String CHANNEL = "network:events";
public static final String KIND_HELLO = "hello";
public static final String KIND_GOODBYE = "goodbye";
public String kind;
public String id;
public String role;
public int players;
public long startedAt;
public long ts;
}
@@ -0,0 +1,249 @@
package net.kewwbec.networkcore.registry;
import com.hypixel.hytale.event.EventBus;
import com.hypixel.hytale.event.IBaseEvent;
import com.hypixel.hytale.logger.HytaleLogger;
import com.hypixel.hytale.server.core.HytaleServer;
import com.hypixel.hytale.server.core.universe.Universe;
import io.lettuce.core.api.sync.RedisCommands;
import net.kewwbec.networkcore.api.MessageBus;
import net.kewwbec.networkcore.api.ServerInfo;
import net.kewwbec.networkcore.api.ServerRegistry;
import net.kewwbec.networkcore.api.events.ServerJoinedNetworkEvent;
import net.kewwbec.networkcore.api.events.ServerLeftNetworkEvent;
import net.kewwbec.networkcore.bus.RedisMessageBus;
import net.kewwbec.networkcore.config.CoreConfig;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
import java.util.logging.Level;
public final class RedisServerRegistry implements ServerRegistry {
private static final HytaleLogger LOGGER = HytaleLogger.forEnclosingClass();
private static final long HEARTBEAT_SECONDS = 5L;
private static final long TTL_SECONDS = 15L;
private final RedisMessageBus bus;
private final String keyPrefix;
private final String id;
private final String role;
private final long startedAt;
private final Map<String, ServerInfo> cache = new ConcurrentHashMap<>();
private final EventBus eventBus;
@Nullable private ScheduledFuture<?> heartbeatTask;
@Nullable private MessageBus.Subscription presenceSub;
public RedisServerRegistry(@Nonnull RedisMessageBus bus,
@Nonnull CoreConfig.RedisSection redis,
@Nonnull String id,
@Nonnull String role) {
this.bus = bus;
this.keyPrefix = redis.key_prefix + "servers:";
this.id = id;
this.role = role;
this.startedAt = System.currentTimeMillis();
this.eventBus = HytaleServer.get().getEventBus();
}
public void start() {
presenceSub = bus.subscribe(PresencePayload.CHANNEL, PresencePayload.class, this::onPresence);
heartbeat();
refreshCache();
announce(PresencePayload.KIND_HELLO);
heartbeatTask = HytaleServer.SCHEDULED_EXECUTOR.scheduleAtFixedRate(() -> {
try {
heartbeat();
refreshCache();
} catch (Throwable t) {
((HytaleLogger.Api) LOGGER.at(Level.WARNING).withCause(t)).log("Heartbeat tick failed");
}
}, HEARTBEAT_SECONDS, HEARTBEAT_SECONDS, TimeUnit.SECONDS);
}
public void stop() {
if (heartbeatTask != null) {
heartbeatTask.cancel(false);
heartbeatTask = null;
}
try {
announce(PresencePayload.KIND_GOODBYE);
} catch (RuntimeException e) {
LOGGER.at(Level.WARNING).log("Failed to publish goodbye: %s", e.getMessage());
}
if (presenceSub != null) {
try { presenceSub.close(); } catch (RuntimeException ignored) {}
presenceSub = null;
}
try {
bus.sync().del(keyFor(id));
} catch (RuntimeException e) {
LOGGER.at(Level.WARNING).log("Failed to remove server registry entry on shutdown: %s", e.getMessage());
}
}
private void announce(@Nonnull String kind) {
PresencePayload p = new PresencePayload();
p.kind = kind;
p.id = id;
p.role = role;
p.players = currentPlayerCount();
p.startedAt = startedAt;
p.ts = System.currentTimeMillis();
bus.publish(PresencePayload.CHANNEL, p);
}
private void onPresence(@Nonnull PresencePayload p) {
if (p.id == null || p.kind == null) return;
if (id.equals(p.id)) return;
if (PresencePayload.KIND_HELLO.equals(p.kind)) {
ServerInfo info = new ServerInfo(p.id, p.role != null ? p.role : "unknown", null, p.players, p.startedAt, p.ts);
ServerInfo previous = cache.put(p.id, info);
if (previous == null) {
try {
eventBus.dispatchFor(ServerJoinedNetworkEvent.class).dispatch((IBaseEvent) new ServerJoinedNetworkEvent(info));
} catch (RuntimeException ignored) {
}
}
} else if (PresencePayload.KIND_GOODBYE.equals(p.kind)) {
ServerInfo removed = cache.remove(p.id);
if (removed != null) {
try {
eventBus.dispatchFor(ServerLeftNetworkEvent.class).dispatch((IBaseEvent) new ServerLeftNetworkEvent(removed));
} catch (RuntimeException ignored) {
}
}
}
}
private void heartbeat() {
RedisCommands<String, String> r = bus.sync();
String key = keyFor(id);
Map<String, String> fields = new HashMap<>();
fields.put("id", id);
fields.put("role", role);
fields.put("players", Integer.toString(currentPlayerCount()));
fields.put("started_at", Long.toString(startedAt));
fields.put("last_seen", Long.toString(System.currentTimeMillis()));
r.hset(key, fields);
r.expire(key, TTL_SECONDS);
}
private void refreshCache() {
RedisCommands<String, String> r = bus.sync();
Set<String> liveIds = new HashSet<>();
Map<String, ServerInfo> latest = new HashMap<>();
String cursor = "0";
do {
io.lettuce.core.KeyScanCursor<String> step = r.scan(io.lettuce.core.ScanCursor.of(cursor), io.lettuce.core.ScanArgs.Builder.matches(keyPrefix + "*").limit(100));
for (String key : step.getKeys()) {
Map<String, String> hash = r.hgetall(key);
if (hash == null || hash.isEmpty()) continue;
String otherId = hash.getOrDefault("id", key.substring(keyPrefix.length()));
ServerInfo info = new ServerInfo(
otherId,
hash.getOrDefault("role", "unknown"),
hash.get("address"),
parseInt(hash.get("players"), 0),
parseLong(hash.get("started_at"), 0L),
parseLong(hash.get("last_seen"), 0L)
);
liveIds.add(otherId);
latest.put(otherId, info);
}
cursor = step.getCursor();
if (step.isFinished()) break;
} while (true);
Set<String> previous = new HashSet<>(cache.keySet());
Set<String> joined = new HashSet<>(liveIds);
joined.removeAll(previous);
Set<String> left = new HashSet<>(previous);
left.removeAll(liveIds);
cache.keySet().retainAll(liveIds);
cache.putAll(latest);
for (String j : joined) {
ServerInfo info = latest.get(j);
if (info != null) {
try {
eventBus.dispatchFor(ServerJoinedNetworkEvent.class).dispatch((IBaseEvent) new ServerJoinedNetworkEvent(info));
} catch (RuntimeException ignored) {
}
}
}
for (String l : left) {
ServerInfo info = new ServerInfo(l, "unknown", null, 0, 0L, 0L);
try {
eventBus.dispatchFor(ServerLeftNetworkEvent.class).dispatch((IBaseEvent) new ServerLeftNetworkEvent(info));
} catch (RuntimeException ignored) {
}
}
}
@Override
@Nonnull
public ServerInfo getSelf() {
ServerInfo cached = cache.get(id);
if (cached != null) return cached;
return new ServerInfo(id, role, null, currentPlayerCount(), startedAt, System.currentTimeMillis());
}
@Override
@Nonnull
public Collection<ServerInfo> getServers() {
return Collections.unmodifiableCollection(new ArrayList<>(cache.values()));
}
@Override
@Nonnull
public Collection<ServerInfo> getServersByRole(@Nonnull String role) {
List<ServerInfo> out = new ArrayList<>();
for (ServerInfo s : cache.values()) {
if (s.role().equalsIgnoreCase(role)) out.add(s);
}
return Collections.unmodifiableCollection(out);
}
@Override
@Nullable
public ServerInfo getServer(@Nonnull String id) {
return cache.get(id);
}
@Nonnull
private String keyFor(@Nonnull String id) {
return keyPrefix + id;
}
private static int currentPlayerCount() {
Universe universe = Universe.get();
return (universe == null) ? 0 : universe.getPlayerCount();
}
private static int parseInt(@Nullable String s, int fallback) {
if (s == null) return fallback;
try { return Integer.parseInt(s); } catch (NumberFormatException e) { return fallback; }
}
private static long parseLong(@Nullable String s, long fallback) {
if (s == null) return fallback;
try { return Long.parseLong(s); } catch (NumberFormatException e) { return fallback; }
}
}
@@ -0,0 +1,153 @@
package net.kewwbec.networkcore.rpc;
import com.google.gson.Gson;
import com.hypixel.hytale.logger.HytaleLogger;
import com.hypixel.hytale.server.core.HytaleServer;
import net.kewwbec.networkcore.api.Counter;
import net.kewwbec.networkcore.api.MessageBus;
import net.kewwbec.networkcore.api.MetricsService;
import net.kewwbec.networkcore.api.RpcClient;
import net.kewwbec.networkcore.api.RpcService;
import net.kewwbec.networkcore.api.Timer;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.logging.Level;
public final class RpcClientImpl implements RpcClient {
private static final HytaleLogger LOGGER = HytaleLogger.forEnclosingClass();
private final MessageBus bus;
private final Gson gson = new Gson();
private final String replyChannel;
private final long timeoutMs;
private final MessageBus.Subscription replySub;
private final Map<String, Pending<?>> pending = new ConcurrentHashMap<>();
@Nullable private final MetricsService metrics;
public RpcClientImpl(@Nonnull MessageBus bus, @Nonnull String serverId, long timeoutMs, @Nullable MetricsService metrics) {
this.bus = bus;
this.replyChannel = "rpc:resp:" + serverId;
this.timeoutMs = timeoutMs;
this.metrics = metrics;
this.replySub = bus.subscribe(replyChannel, RpcMessages.Response.class, this::onResponse);
}
@Override
@Nonnull
public <Req, Resp> CompletableFuture<Resp> request(@Nonnull String service,
@Nonnull String method,
@Nonnull Req payload,
@Nonnull Class<Resp> responseType) {
RpcMessages.Request req = new RpcMessages.Request();
req.correlationId = UUID.randomUUID().toString();
req.replyTo = replyChannel;
req.method = method;
req.body = gson.toJsonTree(payload);
CompletableFuture<Resp> future = new CompletableFuture<>();
long startNanos = System.nanoTime();
Pending<Resp> p = new Pending<>(responseType, future, service, method, startNanos);
pending.put(req.correlationId, p);
ScheduledFuture<?> timeout = HytaleServer.SCHEDULED_EXECUTOR.schedule(() -> {
Pending<?> removed = pending.remove(req.correlationId);
if (removed != null) {
removed.future.completeExceptionally(new TimeoutException("RPC " + service + "/" + method + " timed out after " + timeoutMs + "ms"));
recordOutcome(service, method, "timeout", startNanos);
}
}, timeoutMs, TimeUnit.MILLISECONDS);
p.timeout = timeout;
bus.publish("rpc:req:" + service, req);
return future;
}
@Override
@Nonnull
public RpcService registerService(@Nonnull String service) {
return new RpcServiceImpl(bus, gson, service);
}
public void close() {
replySub.close();
for (Pending<?> p : pending.values()) {
if (p.timeout != null) p.timeout.cancel(false);
p.future.completeExceptionally(new IllegalStateException("RpcClient closed"));
}
pending.clear();
}
private void onResponse(@Nonnull RpcMessages.Response resp) {
if (resp.correlationId == null) return;
Pending<?> p = pending.remove(resp.correlationId);
if (p == null) return;
if (p.timeout != null) p.timeout.cancel(false);
complete(p, resp);
}
@SuppressWarnings("unchecked")
private <Resp> void complete(@Nonnull Pending<?> raw, @Nonnull RpcMessages.Response resp) {
Pending<Resp> p = (Pending<Resp>) raw;
if (!resp.ok) {
p.future.completeExceptionally(new RpcException(resp.error == null ? "unknown error" : resp.error));
recordOutcome(p.service, p.method, "error", p.startNanos);
return;
}
try {
Resp body = gson.fromJson(resp.body, p.responseType);
p.future.complete(body);
recordOutcome(p.service, p.method, "ok", p.startNanos);
} catch (RuntimeException e) {
((HytaleLogger.Api) LOGGER.at(Level.WARNING).withCause(e)).log("Failed to decode RPC response for %s/%s", p.service, p.method);
p.future.completeExceptionally(e);
recordOutcome(p.service, p.method, "decode_error", p.startNanos);
}
}
private void recordOutcome(@Nonnull String service, @Nonnull String method, @Nonnull String result, long startNanos) {
if (metrics == null) return;
Map<String, String> labels = Map.of("service", service, "method", method, "result", result);
Counter c = metrics.counter("rpc_request_total", labels);
c.inc();
Timer t = metrics.timer("rpc_request_duration_seconds", Map.of("service", service, "method", method));
t.recordNanos(System.nanoTime() - startNanos);
}
private static final class Pending<Resp> {
final Class<Resp> responseType;
final CompletableFuture<Resp> future;
final String service;
final String method;
final long startNanos;
@Nullable ScheduledFuture<?> timeout;
Pending(@Nonnull Class<Resp> type,
@Nonnull CompletableFuture<Resp> future,
@Nonnull String service,
@Nonnull String method,
long startNanos) {
this.responseType = type;
this.future = future;
this.service = service;
this.method = method;
this.startNanos = startNanos;
}
}
public static final class RpcException extends RuntimeException {
public RpcException(@Nonnull String message) {
super(message);
}
}
}
@@ -0,0 +1,26 @@
package net.kewwbec.networkcore.rpc;
import com.google.gson.JsonElement;
import javax.annotation.Nullable;
public final class RpcMessages {
private RpcMessages() {
}
public static final class Request {
public String correlationId;
public String replyTo;
public String method;
@Nullable
public JsonElement body;
}
public static final class Response {
public String correlationId;
public boolean ok;
@Nullable public String error;
@Nullable public JsonElement body;
}
}
@@ -0,0 +1,88 @@
package net.kewwbec.networkcore.rpc;
import com.google.gson.Gson;
import com.google.gson.JsonElement;
import com.hypixel.hytale.logger.HytaleLogger;
import net.kewwbec.networkcore.api.MessageBus;
import net.kewwbec.networkcore.api.RpcService;
import javax.annotation.Nonnull;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Function;
import java.util.logging.Level;
final class RpcServiceImpl implements RpcService {
private static final HytaleLogger LOGGER = HytaleLogger.forEnclosingClass();
private final MessageBus bus;
private final Gson gson;
private final String serviceName;
private final MessageBus.Subscription subscription;
private final Map<String, MethodEntry<?, ?>> methods = new ConcurrentHashMap<>();
RpcServiceImpl(@Nonnull MessageBus bus, @Nonnull Gson gson, @Nonnull String serviceName) {
this.bus = bus;
this.gson = gson;
this.serviceName = serviceName;
this.subscription = bus.subscribe("rpc:req:" + serviceName, RpcMessages.Request.class, this::handle);
}
@Override
@Nonnull
public <Req, Resp> RpcService method(@Nonnull String method,
@Nonnull Class<Req> requestType,
@Nonnull Class<Resp> responseType,
@Nonnull Function<Req, Resp> handler) {
methods.put(method, new MethodEntry<>(requestType, responseType, handler));
return this;
}
@Override
public void close() {
subscription.close();
methods.clear();
}
private void handle(@Nonnull RpcMessages.Request req) {
if (req.correlationId == null || req.replyTo == null || req.method == null) return;
MethodEntry<?, ?> entry = methods.get(req.method);
RpcMessages.Response resp = new RpcMessages.Response();
resp.correlationId = req.correlationId;
if (entry == null) {
resp.ok = false;
resp.error = "Unknown method: " + req.method;
bus.publish(req.replyTo, resp);
return;
}
try {
JsonElement body = invoke(entry, req.body);
resp.ok = true;
resp.body = body;
} catch (Throwable t) {
((HytaleLogger.Api) LOGGER.at(Level.WARNING).withCause(t)).log("RPC handler %s/%s threw", serviceName, req.method);
resp.ok = false;
resp.error = t.getClass().getSimpleName() + ": " + t.getMessage();
}
bus.publish(req.replyTo, resp);
}
private <Req, Resp> JsonElement invoke(@Nonnull MethodEntry<Req, Resp> entry, JsonElement body) {
Req parsed = gson.fromJson(body, entry.requestType);
Resp result = entry.handler.apply(parsed);
return gson.toJsonTree(result);
}
private static final class MethodEntry<Req, Resp> {
final Class<Req> requestType;
final Class<Resp> responseType;
final Function<Req, Resp> handler;
MethodEntry(@Nonnull Class<Req> req, @Nonnull Class<Resp> resp, @Nonnull Function<Req, Resp> handler) {
this.requestType = req;
this.responseType = resp;
this.handler = handler;
}
}
}
+15
View File
@@ -0,0 +1,15 @@
{
"Group": "kewwbec",
"Name": "NetworkCore",
"Version": "0.1.0",
"Description": "Cross-server messaging and metrics core",
"IncludesAssetsPack": false,
"Authors": [
{"Name": "kewwbec"}
],
"ServerVersion": "*",
"Dependencies": {},
"OptionalDependencies": {},
"DisabledByDefault": false,
"Main": "net.kewwbec.networkcore.NetworkCore"
}