Webtonium peer protocol v1

Webtonium lets web browsers find and share files without a central search server. Browsers pass searches through a peer network, then transfer files over a direct WebRTC connection.

MUST and MUST NOT mark required behavior. Reference defaults describe configurable local policy.

1. How the network works

A peer is a browser running the protocol. Its catalog lists the files it shares. Each peer has a node ID and one of two roles:

• A leaf starts searches, answers searches, and transfers files.
• An ultrapeer does those things and also relays searches, results, peer discovery, and connection setup messages.

The overlay is the network of neighbor connections used to carry these messages. Leaves MUST NOT connect to other leaves or relay traffic between neighbors. Ultrapeers connect to leaves and other ultrapeers; connections between ultrapeers form the mesh. Each client selects its role locally; changing roles requires new neighbor connections.

A file search and download follows this sequence:

1. Join. Establish a neighbor connection, exchange hello messages, and exchange search summaries called QRP tables (Query Routing Protocol tables).
2. Search. Send a filename or whole-file SHA-256 query. Ultrapeers use QRP tables to avoid branches that cannot match. Hop and transmission limits keep searches bounded.
3. Return matches. File owners send results back along the search path. Each result includes the route to its owner.
4. Connect directly. Send WebRTC setup messages along that route to establish a separate connection to the owner.
5. Transfer and verify. Download the file directly from its owner, checking each chunk and the whole-file hash.

Peers can also discover more neighbors with ping and pong, then use the returned routes to set up new overlay connections. A peer with no neighbors needs external bootstrap; this protocol's discovery requires an existing route.

Paths, hop limits, and budgets

An edge is a connection between two neighbors. Crossing it counts as one transmission. Every overlay message may traverse at most five edges, so a complete route has at most six node IDs.

The origin starts a search or discovery probe. Its query or PING carries a partial path, a ttl, and a budget:

• path lists nodes already visited, ending at the sender and excluding the receiver.
• ttl limits how far the message can still travel, including the edge to the current receiver. It is a positive integer at most 5.
• budget limits additional transmissions after that edge, across all forwarding branches.

Results and connection setup messages instead carry a complete route and an at index identifying the receiver. Their routes obey the same five-edge maximum. The sections below define the direction and checks for each message.

Example: from search to download

Consider A—U—V—B: A and B are leaves; U and V are ultrapeers. A searches through U and V. B sends its result back through V and U. A then uses that route to arrange a direct connection to B. The file travels directly between A and B.

2. Connections and message format

WebRTC channels

An overlay connection is one reliable, ordered WebRTC data channel labeled webtonium-overlay. A direct transfer uses a separate WebRTC connection with one reliable, ordered channel labeled webtonium-transfer. Configure ordered delivery with unlimited retransmissions and accept only the channel for that connection's purpose. The peer proposing the connection (the offerer) creates the channel; the peer accepting it (the answerer) accepts the channel.

Overlay messages

Every overlay message is one UTF-8 JSON object in a WebRTC text message, at most 65,536 encoded bytes. Required fields must have the specified JSON types. Unknown properties are ignored; unknown top-level message types, malformed messages, and invalid state transitions close the incoming link. Senders MUST use unique object keys; receivers using last-key-wins JSON decoding are permitted. Property order and JSON whitespace have no meaning.

Shared values

File metadata

File metadata is { "id": ID, "name": string, "size": UInt, "modified": UInt, "sha256"?: SHA-256 }. name is a nonempty basename (the filename without a directory path), at most 512 units. size is at most 70,368,744,177,664 bytes (64 TiB, the transfer sequence limit). modified is Unix epoch milliseconds. id identifies a file in the owner's local catalog. sha256 is optional while content indexing is incomplete. A client MUST NOT advertise a hash it has not computed or verified.

Client extensions

Every overlay message may include an optional ext object (Record<string, string | number>) for client-specific fields. It contains at most 16 entries: keys are 1–32 units, string values at most 1,024 units, and numeric values finite JSON numbers. The compact JSON encoding of the entire object MUST fit 4,096 UTF-8 bytes; normal message limits still apply. Ignore unrecognized entries and preserve the block when relaying a message. Prefix client-specific keys with the vendor code, for example "ext": { "WEBT.example": "value" }.

3. Joining: hello and admission

Once an overlay channel opens, each endpoint decides whether to accept the other as a neighbor. Both endpoints send this as their first message:

{
  "type": "hello",
  "versions": [1],
  "node": "example-node",
  "role": "ultrapeer",
  "vendor": "WEBT"
}

Client identity

vendor is optional and, when present, exactly four uppercase ASCII letters/digits ([A-Z0-9]{4}). WEBT identifies this implementation. Other clients choose their own code. Display vendor advertising as peer-supplied client information. Apply the same protocol rules to every vendor.

Version and admission checks

Accept one hello per connection. versions is a list of 1–8 distinct supported protocol versions, each an integer in 1–65,535. Select the highest version present in both lists and use its message formats for the connection. The reference advertises [1]; a peer advertising [1,2] therefore uses version 1 with it. Close the connection if the lists have no common version.

Reject a self ID, an ID already admitted on another link, a leaf–leaf connection, or an exhausted role slot. If signaling supplied an expected node ID/role, hello MUST agree. No other overlay message may precede hello. Each endpoint independently admits the remote hello, then immediately sends its current QRP reset and snapshot.

Capacity and deadlines

The reference allows a leaf four ultrapeer neighbors; an ultrapeer allows 24 ultrapeers and 64 leaves. Clients may choose smaller capacities and advertise their available slots. Advertise actual free capacity in discovery and account for pending connections when admitting new ones.

The reference closes a missing hello after ten seconds and an unfinished initial/replacement QRP snapshot after ten seconds. A reset cannot extend an already pending QRP deadline.

4. Searching and returning matches

A query searches by filename tokens or by one exact content hash. First derive its search keys, then use those keys for QRP filtering and exact catalog matching.

Filename matching

Normalize both filenames and ordinary search text with this exact sequence:

1. Unicode NFKD compatibility decomposition.
2. Remove Unicode category M (marks).
3. Locale-independent Unicode lowercase, equivalent to ECMAScript toLowerCase(); not full case folding.
4. Extract maximal runs of Unicode categories L or N (letters/numbers).
5. Remove duplicate tokens, keeping first occurrence order.

A token is one word produced by these steps. A filename query is at most 256 units and must yield 1–16 distinct tokens. A file matches only if its basename has all query tokens as complete words. Use complete basename tokens for matching. caf does not match cafe; straße and strasse differ.

Content lookup and magnets

A hash query's transmitted text is exactly urn:sha256: followed by the lowercase 64-character whole-file digest. Its one search key is that entire string, including its prefix. Do not split it into words or normalize it as a filename. All other transmitted query text uses filename matching, including a bare hexadecimal digest; input aliases are converted by the originating client. Matching requires exact equality to the file's advertised sha256, regardless of filename.

Clients may accept a bare hexadecimal SHA-256, sha256:<hex>, or a magnet in their search box. Convert those inputs to the canonical URN before transmission. The supported magnet profile is:

magnet:?xt=urn:sha256:<64 hexadecimal characters>&dn=<percent-encoded suggested name>

Read xt using URL query decoding; require exactly one distinct supported SHA-256 value. dn is a display hint, never a matching requirement or filesystem path. Ignore other parameters; do not fetch embedded source/tracker URLs. Reject unsupported hash algorithms or magnets without a supported exact topic rather than silently searching their text. BitTorrent btih/btmh topics identify torrent metadata, not the whole-file digest defined here. Transmit the resulting canonical URN in an ordinary query message.

Filename search remains available before file hashing finishes. A newly indexed hash changes the advertised QRP table. A client searching a hash may need to retry after other clients finish indexing; hash queries MUST NOT fall back to matching a magnet's filename hint.

Sending and receiving a query

{
  "type": "query",
  "id": "query-id",
  "text": "sample recording",
  "path": ["origin"],
  "ttl": 4,
  "budget": 511
}

Validate each query before processing it:

• path ends at the incoming neighbor and does not contain the receiver. A leaf sender's path has length one.
• ttl is a positive integer with path.length + ttl <= 6.
• budget is an integer in 0–8,191. It counts additional transmissions available after this edge.

Drop a duplicate live query ID. Otherwise append self to the path, retain that path and the query, search the local catalog, and, if an ultrapeer, consider forwarding with TTL decreased by one and the received budget. Leaves never forward received queries. An origin can send through its ultrapeers using the initial TTL and its chosen total budget. An origin may allocate at most 8,192 query transmissions across all rounds of one search. The reference begins at TTL 4 with an allowance of 512.

Dividing the forwarding budget

Each forwarding node must pay for its outgoing edges and the budgets it passes on. With remaining budget B, k receiving neighbors (children), and additional budgets b_i, it MUST obey:

k + sum(b_i) <= B

Do not send with zero remaining TTL or budget. Do not revisit a path node. Every leaf destination consumes one transmission and receives budget zero.

Choosing search branches

A QRP table summarizes a neighbor's search keys: its own files for a leaf, or its files and attached leaves for an ultrapeer. It does not cover other ultrapeers reached through it. Possible matches still need exact catalog checks; unknown tables match every query. Section 5 defines the table format and exchange.

Filter leaf edges by their QRP table. For ultrapeer forwarding, use the remaining hop count:

• One hop: send only to leaves.
• Two hops: check a prospective ultrapeer's combined table before sending. It can search its own catalog and use the final hop for its leaves. Filter these branches before dividing the budget.
• More than two hops: use mesh ultrapeer tables to prioritize neighbors, while keeping routes through nonmatching intermediate neighborhoods.

The reference selects matching leaves first and at most four mesh neighbors. It pays for their edges, then divides the remaining budget evenly among mesh children. Clients may choose another selection policy that obeys the budget and QRP rules.

Expanding a search

To expand a sparse search, send a new query ID with the same search text, and aggregate results from all active rounds. Keep the sum of allocated round budgets at or below 8,192. The reference allocates 512, then 1,536, then 6,144, waiting ten seconds between rounds. It stops expanding after 50 distinct filename results or three distinct owners for a hash search. A new search or local cancellation stops further rounds; existing rounds retain their normal cache lifetime.

Returning results

Remember each forwarded neighbor and the TTL sent down that branch. Results return as:

{
  "type": "result",
  "queryId": "query-id",
  "route": ["origin", "hub", "owner"],
  "at": 1,
  "files": [
    {
      "id": "file-id",
      "name": "sample recording.flac",
      "size": 1234,
      "modified": 1000
    }
  ]
}

files contains 1–20 metadata objects. route is the complete origin-to-owner path; its last ID owns every file in the batch. at is the receiving node's zero-based index, from 0 through route.length-2.

Drop results for absent/expired query records. Otherwise require:

• at equals the retained query path length minus one, and that path exactly prefixes route.
• route[at+1] is the incoming neighbor, to which this query was actually forwarded.
• The suffix length route.length - retainedPath.length does not exceed that branch's sent TTL.
• A leaf sender is the route's owner; it cannot supply a downstream endpoint.

Invalid routes close the link.

Discard files that do not match the original filename/hash query before consuming any result allowance. Deduplicate by (owner, file ID).

At index zero deliver to the search UI. Otherwise only an ultrapeer may relay; send to route[at-1] with at-1. The reference retains at most 100 results per query at each node. Local catalog results use the accumulated query path as their route. Collect results until the query expires or its result cap is reached.

5. QRP: advertising searchable files

QRP tables let neighbors decide where to send searches without exchanging full file lists. Each bit represents one or more search keys. A reset and snapshot send a full table; a patch updates individual bits.

Building a table

A QRP table is exactly 65,536 presence bits packed into 8,192 bytes. Add a bit for every filename token and, when known, the file's one canonical hash URN. Calculate the zero-based slot as follows; arithmetic is modulo 2^32:

x = 0
for i, code in enumerate(UTF16_CODE_UNITS(key)):
    if 0x41 <= code <= 0x5a: code += 0x20
    x = x XOR ((code AND 0xff) << (8 * (i mod 4)))
slot = ((x * 0x4f1bbcdc) mod 2^32) >> 16   # unsigned shift
bytes[slot >> 3] |= 1 << (slot AND 7)

Hash the low byte of each UTF-16 code unit. Supplementary characters contribute two units. Bit zero is the low bit of byte zero. A known table matches a query if every key's bit is set. Collisions give false positives; exact catalog matching decides results. An unknown table MUST match every query.

What each peer advertises

A leaf advertises its own catalog table. An ultrapeer combines its table with all attached leaf tables using bitwise OR: a bit is set if any of those tables sets it. It includes no tables from mesh ultrapeers. If any attached leaf table is unknown, its aggregate is all ones. The aggregate describes that ultrapeer and its immediate leaves.

Exchanging tables

Track each neighbor's table separately. Start with an unknown table, committed revision 0, and no pending reset. The committed revision is the version of the last fully applied table update:

• Reset requires revision > committed and no pending reset. Mark the table unknown immediately and remember the pending revision.
• Snapshot requires that exact pending revision. Validate and atomically replace the entire table, commit its revision, and clear the pending reset.
• Patch requires a known table, no pending reset, and base == committed. Positive n sets bit n-1; negative -n clears bit n-1. Apply in order, then commit the new revision. Repeated slots use the last value. A jump in revision is permitted; a mismatched base is not.

Send reset plus snapshot on every new link. Start revisions at 1, increment when the advertised bit array changes, and use a patch for at most 1,024 changed slots or reset/snapshot for larger changes. Emit changed slots once each in ascending slot order. Close the link on an invalid QRP exchange state.

6. Finding more neighbors

Send a ping to find peers with room for a new neighbor. Suitable peers return a pong along the probe path. Discovery uses the same budget rule as search but ignores QRP.

Sending a probe

{
  "type": "ping",
  "id": "probe-id",
  "role": "leaf",
  "path": ["origin"],
  "ttl": 5,
  "budget": 63
}

role is the origin's role. Require a unique path ending at the sender and excluding self, positive TTL, path.length + ttl <= 6, and budget 0–63. A direct origin's role must match its hello; a leaf cannot relay PING. Cache/deduplicate the ID, append self, advertise if suitable, then only ultrapeers may forward with TTL minus one. The reference originates at TTL 5 with 64 total transmissions, at most four mesh destinations and two leaf destinations per forwarding node. A leaf-origin probe visits only ultrapeers.

Advertising an available peer

{
  "type": "pong",
  "pingId": "probe-id",
  "node": "candidate",
  "role": "ultrapeer",
  "leafSlots": 12,
  "ultrapeerSlots": 3,
  "trickle": true,
  "route": ["origin", "hub", "candidate"],
  "at": 1
}

route has at most six IDs, ends at node, and at is the receiver index, less than the last index. Slot counts are integers in 0–65,535; a leaf's leafSlots must be zero. trickle, if present, is exactly true and advertises support for incremental overlay ICE signaling.

Only advertise an endpoint with a slot for the origin's role: a leaf origin needs an ultrapeer with leafSlots > 0; an ultrapeer origin needs ultrapeerSlots > 0. Full nodes may still relay probes.

Checking and using replies

Correlate each PONG with a live PING, its exact retained prefix, the forwarded neighbor at route[at+1], and that branch's TTL. A leaf sender must be the endpoint; a direct endpoint's role must match hello. Deduplicate endpoint IDs and allow at most 1 + childBudget unique replies from each forwarded branch. Drop excess replies and expired probes. Invalid correlated return paths close the link. Deliver at index zero, otherwise relay with index minus one. These rules bound one honest probe to 64 endpoint responses and 320 return-hop transmissions.

The origin may use a PONG route to introduce that candidate. Reported slots may change; the candidate decides whether to admit the connection. A broken route requires rediscovery. Losing every neighbor requires external bootstrap. Probe scheduling, retry backoff, candidate caching, and connection targets are local policy.

7. Setting up connections through a route

WebRTC peers exchange setup information before opening a connection. Here, signaling means carrying that information through existing overlay neighbors. SDP describes the proposed connection; ICE candidates describe possible network paths. Relays carry the messages, while the endpoints create the new connection.

Use a route from discovery to add an overlay neighbor, or a route from a search result to start a direct transfer.

The routed envelope

{
  "type": "route",
  "id": "envelope-id",
  "route": ["origin", "hub", "target"],
  "at": 1,
  "payload": { "kind": "..." }
}

at is the receiving node's index, from 1 through the last index. A sender begins at 1. Require route[at] == self and route[at-1] == incomingNeighbor; a leaf sender can appear only at index zero. Drop duplicate live envelope IDs. Deliver the payload at the last index; otherwise only an ultrapeer may forward to route[at+1], increasing at by one. Preserve the envelope ID, route, and payload. Drop the envelope if its next hop is disconnected.

The entire envelope must fit 32,768 UTF-8 bytes after compact JSON serialization, including the route and escaped SDP. Raw overlay messages still have the 65,536-byte bound. payload is a JSON object with nesting depth at most 12 (root depth 0) and at most 2,048 object/array containers including its root. Unknown payload kinds or malformed endpoint signaling are ignored, rather than disconnecting a relay for forwarding an unsupported payload.

Overlay setup messages

Every overlay signal has kind, session: ID, sender: ID, target: ID, role: Role (the sender's role), and expires: UInt (Unix epoch milliseconds).

SDP starts with v=0 and is at most 24,576 units; the envelope byte bound also applies.

Offering an overlay connection

With trickle ICE, peers send SDP first and send candidates as they become available. Without it, peers gather candidates before sending the complete SDP.

The offerer selects an unused session ID and reserves capacity. If the destination advertised trickle support, set expiry to reservation time plus 90 seconds, send SDP immediately after setting the local description, then release queued candidates. Otherwise gather the complete offer first (reference gathering deadline: 60 seconds), then set advertised expiry to now plus 90 seconds. The reference also bounds pre-offer setup by 90 seconds.

Accepting an offer and its answer

At receipt require now <= expires <= now + 120000, sender to start the route, and both target and self to end it. Ignore malformed/stale signals. Before allocating, suppress repeated (sender, session) offers, reject duplicate peers/sessions, and reserve a compatible role slot. Decline with the reason above if unavailable. On success set the remote offer, produce the answer in the same mode, and send it over the exact reversed route, preserving session and expiry. Reserve slots through channel opening until hello admission. Release reservations on failure/cancellation.

Accept at most one answer/decline for a pending outgoing session. Require the expected sender, role, session, expiry, and exact reversed route. A mode mismatch or decline fails the attempt. Require the resulting connection's hello to match that sender and role. An expired/cancelled session cannot be revived by a delayed reply. Stop signaling deadlines when the channel opens; hello and QRP deadlines are separate.

Simultaneous offers

If two nodes simultaneously offer to each other, the lexicographically smaller ASCII node ID keeps its outgoing offer and declines the incoming one as glare. The larger closes its outgoing attempt and accepts the smaller's offer. Existing incoming/open sessions are duplicate conflicts.

Trickle ICE candidates

An ICE candidate object contains candidate (string, at most 2,048 units), optional/null sdpMid and usernameFragment (strings at most 256 units), and optional/null sdpMLineIndex (integer 0–65,535). A nonempty candidate string needs at least one non-null media identifier. Normalize omitted optional fields to null. A null candidate marks end-of-candidates.

Send at most 64 candidate messages per direction including duplicates and the null marker. Send your description first, then candidates serially in event order. Correlate each received candidate to the exact pending trickle session, sender/role, target, expiry, and reversed route. Queue early candidates until remote SDP is set; deduplicate normalized objects and apply serially. Ignore candidates after the null marker, channel opening, or cancellation. An unknown candidate session never allocates a connection. Exceeding its candidate allowance fails the pending session. Use a new session and connection for a new ICE generation.

Direct transfer setup

Use the source route returned by a search result:

{
  "kind": "transfer-offer",
  "session": "transfer-session",
  "description": { "type": "offer", "sdp": "v=0..." }
}

The owner replies with transfer-answer, the same session, an answer description, and the exact reversed route. Both kinds use complete gathered SDP, starting with v=0, at most 65,536 units and still constrained by the routed envelope byte bound. They create a webtonium-transfer channel; file data does not traverse ultrapeers.

Correlate an answer to one pending offer and its reversed route. Ignore unknown/duplicate replies. Reference direct connection setup times out after 90 seconds; cancellation or failure closes its connection. Retry a failed transfer introduction with a fresh search result and session.

8. Transferring and verifying a file

Once the direct channel opens, the downloader requests one file by its catalog ID. The uploader sends one chunk at a time and waits for an acknowledgment before continuing.

Control sequence

Transfer control frames are JSON text messages of at most 4,096 UTF-16 units. Chunks are WebRTC binary messages. Exactly one file is transferred per channel, in this sequence:

Binary chunks

Binary chunk layout:

Every chunk except the last carries exactly 16,384 payload bytes. The last carries exactly the remaining advertised bytes; there are no chunks for an empty file. Permit one unacknowledged chunk. Reject wrong sequence, size, chunk digest, ACK, control order, final byte count, final chunk count, or final digest.

Verification, completion, and retry

If the selected result has a sha256, the downloader MUST also compare the independently computed whole-file hash against that selected result, not just the sender's end. Abort uncommitted output on mismatch, error, timeout, or cancellation. Never interpret a remote name as a filesystem path. Filesystem APIs, destination naming, and local sharing policy are outside the wire protocol.

For an empty file, end has chunks: 0 and SHA-256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855. Completion receipt is advisory after local commit: losing it can leave a saved download while the uploader reports failure. The reference inactivity deadline is 30 seconds, including disk work. Start a new session and request the whole file when retrying an interrupted transfer.

9. Worked exchange and conformance vectors

Query, result, and transfer setup

For admitted neighbors A—U—V—B, with A/B leaves and U/V ultrapeers, a TTL-3, budget-8 search behaves as follows:

The answer uses a fresh envelope ID and the same transfer session. File chunks then go directly A↔B.

Filename normalization

QRP slots and updates

The last row is an intentional collision, resolved by exact matching. An empty snapshot is A repeated 10,923 times followed by =. Moving a table from only hello to only world at committed revision 1 uses { "type": "qrp-patch", "base": 1, "revision": 2, "changes": [-38933,53227] }.

File hash and binary chunk

The SHA-256 of the three ASCII bytes abc is ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad. Its single binary frame is the concatenation of these hexadecimal groups:

00000000
ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad
616263

ACK sequence is 0; end has chunks: 1 and that digest. A hash query is urn:sha256: plus that digest; a matching file may have any basename. Executable vectors and end-to-end examples live in content tests, QRP tests, hash search integration, and routing integration.

10. Reference limits and trust boundaries

These defaults bound local memory, traffic, and concurrent work.

Resource limits

Dropping traffic or closing a link

Query/PING admission overload and exhausted PONG allowances drop traffic without disconnecting a valid relay. Malformed overlay traffic, invalid state/path, and other resource-limit failures close the link. Expired/deduplicated messages are silently dropped. Error text is not a stable machine-readable API.

What a result or peer claim means

Present search results as the matches found within the query's budgets, hop limit, and lifetime.

Treat node IDs, vendor codes, routes, metadata, capacity claims, and introductions as peer-supplied assertions; WebRTC encrypts each transport. Verify downloaded bytes against the selected hash when available.

Unicode category/case data follows the host runtime; use the vectors above to check normalization across implementations. LIMITATIONS.md records further reference-implementation constraints.