Relay network usage metrics and configurable data cap

The Wireleap relay software now supports tracking the amount of traffic it relays, both globally and per contract. Using this data, it is also now possible to limit the amount of the traffic it will relay during a configured period, also, both globally and per contract.

This is particularly useful for two reasons. To not exceed any hosting bandwidth quota limits (which may incur unexpected charges), and to allow the allocation of specific amounts of bandwidth per Wireleap network they support.

This has been a long time in the making, and turned out to be way more complex and intricate than initially thought. It has also been one of the most requested features from existing and prospective relay operators.

Example use case

Joe has a server with a bandwidth quota of up to 5TB per month.

The first thing Joe wants to take care of is not going over his bandwidth quota, so he configures a global_limit and timeframe. Any timeframe measurement period could be configured, such as 30d, but Joe decides to set a weekly timeframe using 7d with a limit of 1TB.

Next, Joe would like his relay to support two Wireleap networks (i.e. contracts). The first being the free Wireleap Libre network, and the second being a paid contract. Joe would like to provide up a quarter of network traffic (256GB) for the Libre contract, and the rest to the paid contract as he will be compensated for the service he provides.

{
    "address": "0.0.0.0:13499",
    "archive_dir": "archive/sharetokens",
    "auto_submit_interval": "5m0s",
    "network_usage": {
        "global_limit": "1TB",
        "timeframe": "7d",
        "write_interval": "5m0s",
        "archive_dir": "archive/netstats"
    },
    "contracts": {
        "https://c.libre.wireleap.com": {
            "address": "wireleap://joe1.relaysinc.com:13499",
            "role": "entropic",
            "network_usage_limit": "256GB"
        },
        "https://contract1.example.com": {
            "address": "wireleap://joe1.relaysinc.com:13499",
            "role": "entropic"
        }
    }
}

With this configuration in place, if or when the network_usage_limit on the Libre contract is hit, the relay will automatically unenroll from Libre for the remainder of the timeframe. Additionally, if or when the global_limit is hit, even if the Libre contract network_usage_limit has not been hit, the relay will unenroll from all contracts for the remainder of the timeframe.

When a new timeframe measurement period begins, the relay will re-enroll into any contracts it may have been unenrolled from.

Timeframes and records

Timeframes are configured in the duration.T format. When a new timeframe is started, the previous stats are archived. During the timeframe, the network usage stats are written to disk at the specified interval. It is also possible to trigger a write by sending the SIGUSR2 signal:

su -l wireleap-relay
kill -USR2 $(cat wireleap_relay.pid)
cat stats.json | jq -r ".contract_stats | map(.network_bytes) | add"

What happens when a limit is reached

Enrollments

If a contract specific limit is reached, the relay will unenroll from that contract. If the global limit is reached irrelevant to specific contract limits, the relay will unenroll from all configured contracts. Once the current measurement period ends, the relay will re-enroll into the configured contracts.

Thresholds

Limits are configured in the datasize.ByteSize format. In order to not abruptly disconnect clients using the relay when a limit is hit, but rather initiate a smooth transition for clients to other relays, as well as account for a margin of error, two thresholds have been implemented:

The soft-limit is used to prevent new clients from reaching the relay by refusing new connections and unenrolling the relay from the contract. It will be activated when 90% of contract specified limit is reached.

If the hard-limit is reached and there are still active connections, they will be closed.

It’s important to note that the global limit does not support soft-limit, and if reached the relay will disconnect all clients and unenroll from all the contracts. If the relay is configured with only one contract, it is recommended to specify the contract limit in order to enable soft-limit functionality.

How network usage is measured

Or, what is and what is not measured?

With the 0.6.0 release, the relay measures routed TCP streams only (client-relay, relay-relay, relay-clearnet). H/2, TCP (layer 4), IP (layer 3), and lower headers (LAN and physical point-to-point link scopes) are not accounted for. Other traffic such as relay-contract, relay-dir, relay upgrades, and anything else outside of the scope of Wireleap is not accounted for.

In other words, the network usage in-scope is wireleap traffic flowing through the relay. It should be relatively accurate, within a margin of error.

Some of the finer details

Each IP packet has a header and a payload. The payload is a TCP packet, which also has a header and a payload. The TCP payload is, overall, what is currently being measured.

Depending on the length of the IP packets, the measured traffic value will deviate more or less from the real value. The IP packet length is as low as the contained payload and as large as the available MTU (Maximum Transmission Unit). If the MTU is reached, the payload is sent via multiple IP packets, each one containing an additional TCP header. The IPv4 header size is 20-24 bytes and the TCP header is 20 bytes. The network MTU is usually 1492-1500 bytes. For our calculations, we’re assuming the most common values:

The only remaining variable is the average packet length, which indirectly depends on the protocols and applications used by the client. The nature of packet size distribution is not generic, and usage evolves with time. As a result, certain educated assumptions are made: Small packets (100 bytes) are commonly used for signalisation purposes, audio streams or multiplayer video games. Large packets, hitting the MTU are used for heavy payloads requiring multiple packets.

So, the guiding assumption is that 30% of packets are small, and 70% are large. The remaining sizes are marginal. Average packet size is ~1075 bytes, and measured traffic is 95.8%.

As extra margin for error, the traffic limitation features activate at 90% (soft-limit) of the defined threshold and stop all kind of traffic at 93% (hard-limit).

Changes

wireleap-relay (0.6.0)

• Depends on wireleap/common v0.3.3.

• Network usage limiting feature added.

• Added network_usage config section and per-contract network_usage_limit field.

• Relay will now disenroll from a contract when the network usage for that contract exceeds the defined value or disenroll from all contracts when the global value is reached.

• Includes interface versions:

  • clientrelay v0.2.0
  • relaycontract v0.1.0
  • relayrelay v0.2.0
  • relaydir v0.2.0