The Wireleap relay software now supports tracking the amount of traffic it relays, both globally and per contract. Using this data, it is also now possible to limit the amount of the traffic it will relay during a configured period, also, both globally and per contract.
This is particularly useful for two reasons. To not exceed any hosting bandwidth quota limits (which may incur unexpected charges), and to allow the allocation of specific amounts of bandwidth per Wireleap network they support.
This has been a long time in the making, and turned out to be way more complex and intricate than initially thought. It has also been one of the most requested features from existing and prospective relay operators.
Joe has a server with a bandwidth quota of up to 5TB
per month.
The first thing Joe wants to take care of is not going over his bandwidth
quota, so he configures a global_limit
and timeframe
. Any timeframe
measurement period could be configured, such as 30d
, but Joe decides
to set a weekly timeframe using 7d
with a limit of 1TB
.
Next, Joe would like his relay to support two Wireleap networks (i.e.
contracts). The first being the free Wireleap Libre network, and the second being a paid contract. Joe would like to
provide up a quarter of network traffic (256GB
) for the Libre
contract, and the rest to the paid contract as he will be compensated
for the service he provides.
{
"address": "0.0.0.0:13499",
"archive_dir": "archive/sharetokens",
"auto_submit_interval": "5m0s",
"network_usage": {
"global_limit": "1TB",
"timeframe": "7d",
"write_interval": "5m0s",
"archive_dir": "archive/netstats"
},
"contracts": {
"https://c.libre.wireleap.com": {
"address": "wireleap://joe1.relaysinc.com:13499",
"role": "entropic",
"network_usage_limit": "256GB"
},
"https://contract1.example.com": {
"address": "wireleap://joe1.relaysinc.com:13499",
"role": "entropic"
}
}
}
Key | Type | Comment |
---|---|---|
network_usage.global_limit | string | maximum routed traffic in defined period |
network_usage.timeframe | string | routed traffic measurement fixed time window |
network_usage.write_interval | string | interval between autosaves |
network_usage.archive_dir | string | path of the archived statistics directory |
contracts.X.network_usage_limit | string | maximum routed traffic for this contract |
With this configuration in place, if or when the network_usage_limit
on the Libre contract is hit, the relay will automatically unenroll from
Libre for the remainder of the timeframe. Additionally, if or when the
global_limit
is hit, even if the Libre contract network_usage_limit
has not been hit, the relay will unenroll from all contracts for the
remainder of the timeframe.
When a new timeframe measurement period begins, the relay will re-enroll into any contracts it may have been unenrolled from.
Timeframes are configured in the
duration.T
format. When a new timeframe is started, the previous stats are
archived. During the timeframe, the network usage stats are written to
disk at the specified interval. It is also possible to trigger a write
by sending the SIGUSR2
signal:
su -l wireleap-relay
kill -USR2 $(cat wireleap_relay.pid)
cat stats.json | jq -r ".contract_stats | map(.network_bytes) | add"
If a contract specific limit is reached, the relay will unenroll from that contract. If the global limit is reached irrelevant to specific contract limits, the relay will unenroll from all configured contracts. Once the current measurement period ends, the relay will re-enroll into the configured contracts.
Limits are configured in the datasize.ByteSize
format. In order to not abruptly disconnect clients using the relay when
a limit is hit, but rather initiate a smooth transition for clients to
other relays, as well as account for a margin of error, two thresholds
have been implemented:
Threshold | Value | Comment |
---|---|---|
soft-limit | 90% | Refuse new connections |
hard-limit | 93% | Close active connections |
The soft-limit
is used to prevent new clients from reaching the relay
by refusing new connections and unenrolling the relay from the contract.
It will be activated when 90% of contract specified limit is
reached.
If the hard-limit
is reached and there are still active connections,
they will be closed.
It’s important to note that the global limit does not support
soft-limit
, and if reached the relay will disconnect all clients and
unenroll from all the contracts. If the relay is configured with only
one contract, it is recommended to specify the contract limit in order
to enable soft-limit
functionality.
Or, what is and what is not measured?
With the 0.6.0
release, the relay measures routed TCP streams only
(client-relay, relay-relay, relay-clearnet). H/2, TCP (layer 4), IP
(layer 3), and lower headers (LAN and physical point-to-point link
scopes) are not accounted for. Other traffic such as relay-contract,
relay-dir, relay upgrades, and anything else outside of the scope of
Wireleap is not accounted for.
In other words, the network usage in-scope is wireleap traffic flowing through the relay. It should be relatively accurate, within a margin of error.
Some of the finer details
Each IP packet has a header and a payload. The payload is a TCP packet, which also has a header and a payload. The TCP payload is, overall, what is currently being measured.
Depending on the length of the IP packets, the measured traffic value will deviate more or less from the real value. The IP packet length is as low as the contained payload and as large as the available MTU (Maximum Transmission Unit). If the MTU is reached, the payload is sent via multiple IP packets, each one containing an additional TCP header. The IPv4 header size is 20-24 bytes and the TCP header is 20 bytes. The network MTU is usually 1492-1500 bytes. For our calculations, we’re assuming the most common values:
Key | Value |
---|---|
IP header | 20 bytes |
TCP header | 20 bytes |
MTU | 1492 bytes |
The only remaining variable is the average packet length, which indirectly depends on the protocols and applications used by the client. The nature of packet size distribution is not generic, and usage evolves with time. As a result, certain educated assumptions are made: Small packets (100 bytes) are commonly used for signalisation purposes, audio streams or multiplayer video games. Large packets, hitting the MTU are used for heavy payloads requiring multiple packets.
So, the guiding assumption is that 30% of packets are small, and 70% are large. The remaining sizes are marginal. Average packet size is ~1075 bytes, and measured traffic is 95.8%.
Average packet size (bytes) | Traffic measure accuracy |
---|---|
X | = (X - headers(IP+TCP)) / X |
1500 | 97.33% |
1492 | 97.32% |
1300 | 96.92% |
1100 | 96.36% |
900 | 95.56% |
700 | 94.29% |
500 | 92.00% |
300 | 86.67% |
100 | 60.00% |
40 | 0% |
As extra margin for error, the traffic limitation features activate at
90% (soft-limit
) of the defined threshold and stop all kind of
traffic at 93% (hard-limit
).
wireleap-relay (0.6.0)
Depends on wireleap/common
v0.3.3.
Network usage limiting feature added.
Added network_usage
config section and per-contract
network_usage_limit
field.
Relay will now disenroll from a contract when the network usage for that contract exceeds the defined value or disenroll from all contracts when the global value is reached.
Includes interface versions:
clientrelay
v0.2.0relaycontract
v0.1.0relayrelay
v0.2.0relaydir
v0.2.0Version
0.6.0
Next Article
Client restructuring and APIPrevious Article
Introducing the Wireleap Libre networkNo results found
client
=dataflow
'circuit
!docs
^installation
paper$
White space acts as an AND operator, while a single pipe
(|
) character acts as an OR operator. To escape
white space, use double quotes such as ="wireleap
client"
for an exact match.