NOTE: Welcome to my public notebook! There are many blogs here, but this is truly just a note. It is an AI re-write of the Gnutella 0.6 spec formatted for easier reading. It is not precise enough to be used in implementing the protocol, but it is a good introductory text for curious readers. I used this as a refresher when I built my own client a while back.

If you want content written by a human see my Gnutella Explanation

Gnutella is a decentralized peer-to-peer protocol for finding shared resources and arranging direct transfers between peers. A Gnutella participant connects to other participants, forwards searches and replies, advertises known hosts, and uses ordinary HTTP for the actual movement of file data.

Gnutella 0.6 is dated June 2002. Some areas are unfinished, especially parts of Ultrapeer behavior, Query Routing Protocol details, security analysis, upload queueing, and some appendices. Missing rules remain undefined.

1. What Gnutella is

Gnutella is a decentralized peer-to-peer system. In a centralized system, one main server might know where all the files or resources are. Gnutella does not work that way. Instead, each participant runs a Gnutella program, connects to other participants, and helps carry network traffic.

A Gnutella program is called a servent. The word blends server and client, because each participant acts like both. It can ask other nodes for things, and it can answer requests from other nodes. Here, node, peer, host, program, and servent usually refer to the same participating system, with servent emphasizing the Gnutella software itself.

The point of the network is discovery. A user shares resources from their own machine. Another user searches the network for resources. If the search finds something, the two machines can connect directly and transfer the resource. If the resource is a file, the actual file transfer uses HTTP. Gnutella itself is used mostly to find who has the file, not to carry the file contents.

Resources can be more than ordinary files. They might be cryptographic keys, mappings to other resources, metadata, or other kinds of information. Gnutella 0.6 mainly defines how to find and transfer files. It does not fully define how every other possible resource type should work.

The protocol assumes some knowledge of network programming, but it does not require previous Gnutella experience. Exact fields, byte sizes, flags, and protocol examples matter for implementation, even when the main ideas are straightforward.

2. How to read the protocol requirements

Protocol requirements use words like must, should, may, required, recommended, and optional in the formal RFC sense. Must means the behavior is required for compatibility. Should means the behavior is strongly recommended, but there might be valid reasons not to do it. May means the behavior is allowed, but not required.

If a node must do something, that behavior is part of staying compatible with the protocol. If a node should do something, that behavior is normally expected, but exceptions are possible.

The Gnutella Development Forum served as a place for additional documentation, proposals, protocol extensions, and discussion among implementers. Some referenced materials required forum membership.

3. Basic terms

A servent is the program participating in the Gnutella network. The words peer, node, and host are close in meaning, but they usually refer to the network participant rather than the software itself. Sometimes a peer clearly acts like a client, and sometimes it clearly acts like a server. But most of the time, it is both.

A message is a unit of information sent over the Gnutella network. Other Gnutella materials may call the same thing a packet or a descriptor. The basic idea is that nodes talk by sending messages to each other.

A GUID is a globally unique identifier. It is a sixteen-byte value, usually made from random bytes. It identifies messages and nodes. It is not a cryptographic signature. It does not prove identity. It is just a way to label something so that it is very unlikely to be confused with something else.

4. Extending the protocol

Gnutella 0.6 allows extensions, but extensions must not split the main network into incompatible islands. For example, a node may want to compress or encrypt its traffic. That is allowed only if it first confirms, during the connection handshake, that the other side can understand the compressed or encrypted form. If the other side cannot understand it, the node must fall back to ordinary compatible traffic or decline the connection without harming the larger network.

A node may refuse to connect to another node that lacks some feature. Implementers must not accidentally divide the Gnutella network into separate incompatible networks. Separate special-purpose networks are allowed, but if they are not compatible with Gnutella, they are no longer the Gnutella network.

The protocol also supports extensions inside messages. A node that does not understand a particular extension can often pass it along anyway. That lets newer features travel through older nodes until they reach a node that does understand them.

5. The major Gnutella messages

After two Gnutella nodes connect, they communicate by sending protocol messages. The main message types are Ping, Pong, Query, Query Hit, Push, and Bye.

A Ping message is used to discover hosts on the network. If a node receives a Ping, it can answer with one or more Pong messages.

A Pong message is the answer to a Ping. It tells the requester about a Gnutella node, including the node's address, listening port, and how much data it claims to be sharing.

A Query message is the search message. A user searches for something, and the node sends a Query into the network.

A Query Hit message is the answer to a Query. It means a peer has something that matches the search and is providing enough information for the searcher to request it.

A Push message helps with firewalled nodes. If a node has a file but cannot accept incoming connections, the downloader can ask that node to open an outgoing connection back to the downloader.

A Bye message is optional. It tells the other side that the connection is being closed, and it may include a reason.

6. Connecting to the Gnutella network

A Gnutella node joins the network by connecting to a node that is already connected. Techniques for finding the first node and deciding when to accept new connections are referenced as appendices, but those appendix bodies are not fully present in the available text.

The default Gnutella port is 6346. A node may use a different unused port. If the usual port is already taken, the node should try another port. The port it chooses is later advertised to other peers through Pong messages.

Once a node has the address of another Gnutella node, it opens a TCP connection. The side that opens the connection is called the client. The side that receives it is called the server. Before binary Gnutella messages are exchanged, the two sides perform a handshake.

The handshake works like this. The client connects and requests Gnutella 0.6. The client then sends headers describing what features it supports. The server answers with a success response if it accepts the connection, followed by its own feature headers. The client then sends a final success response if it still wants to continue after seeing the server's headers. After that, both sides may send binary Gnutella messages.

Both sides announce capabilities, both sides agree that the connection is acceptable, and then real Gnutella traffic begins.

Headers use a style similar to HTTP headers. A header has a name, then a value. Unknown headers must be ignored. This is important because ignoring unknown headers lets future extensions coexist with older software.

Standard HTTP-style header names should be used when those names already fit. For example, User-Agent is preferred over a custom vendor header that means the same thing.

A future version number should not automatically break compatibility. If a future client requests Gnutella 0.7, a Gnutella 0.6 server should not reject it just because the version is higher. The server should respond with the highest version it supports.

Older Gnutella 0.4 nodes use a simpler handshake. They do not use the Gnutella 0.6 header exchange. A 0.6 node may retry with the older 0.4 connection string if a 0.6 attempt is rejected.

If a node rejects a connection, it should, when possible, give the rejected peer a list of other nodes to try. This is done with the X-Try header. That header contains peer addresses and ports. This is sometimes called a connection pong, because it gives host information without requiring a normal Ping and Pong exchange.

If a node rejects a connection because it is busy, the usual response code is 503, followed by a short explanation such as Busy.

7. Message framing and the message header

Once the connection is established, Gnutella messages are sent in a binary stream. A single IP packet may contain multiple Gnutella messages, and a single Gnutella message may be split across multiple IP packets. So an implementation must not assume that one socket read equals one full Gnutella message.

Every Gnutella message begins with a fixed header containing five pieces of information.

First, it contains the message identifier. This is the GUID for the message. It lets nodes notice whether they have already seen the same message.

Second, it contains the message type. The type identifies the message as a Ping, Pong, Bye, Push, Query, or Query Hit.

Third, it contains the time to live, usually called TTL. TTL is a limit on how far a message can travel. Each time a node forwards the message, it decreases the TTL. When the TTL reaches zero, the message is no longer forwarded.

Fourth, it contains the hop count. Hops means how many times the message has already been forwarded. Each time a node forwards a message, it increases the hop count.

Fifth, it contains the payload length. The payload is the actual message body after the header. The payload length tells the receiver exactly where this message ends and where the next message begins.

The payload length is especially important. It is the only reliable way to find the next message in the stream. If a node loses synchronization with the stream, it should close the connection, because the sender is producing or forwarding invalid messages.

The protocol warns against abusing TTL. A high TTL can flood the network with too much traffic. For query messages, a node should reduce TTL when TTL plus hops would exceed the local maximum. A common maximum is seven hops. Messages with extremely high TTL values should be dropped.

Most messages should not be larger than four kilobytes. Query messages are expected to be much smaller, because they are broadcast to many nodes.

8. Ping messages

A Ping is a discovery message. A node sends a Ping to learn about other Gnutella hosts.

A Ping message usually has no ordinary payload. It may contain a GGEP extension block, but it should not contain other payload data.

In early Gnutella, Ping messages were broadcast widely across the network, and Pongs were routed back to the originator. That used a lot of bandwidth. Modern Gnutella nodes therefore use Pong caching or other bandwidth-saving methods.

9. Pong messages

A Pong is the answer to a Ping. A Pong describes a Gnutella host. It includes the host's port, the host's IP address, the number of files the host is sharing, and the number of kilobytes the host is sharing. It may also include extension data.

A node can send more than one Pong in response to one Ping. This is useful because a node may answer not only for itself, but also with cached information about other known nodes.

A Pong must use the same message identifier as the Ping it answers. That lets the requesting side connect the response to the request.

The shared-file count and shared-size fields were originally meant to estimate how much content was available on the network. In a large modern network, where Ping and Pong traffic is reduced, those fields are no longer a reliable way to measure the whole network. Even so, nodes should still fill them in honestly.

10. How Ping and Pong should be used

Ping and Pong handling follows general rules rather than one required caching algorithm.

When a node receives a Ping with enough TTL to travel farther, and it has not received another Ping on that connection in the last second, it should answer with several Pongs if possible. Ten Pongs is given as a reasonable number. The Pongs should use the same message identifier as the Ping.

Pongs should be high quality. That means the addresses should probably be connectable, and they should represent a useful spread of hosts from across the network.

Ping and Pong traffic should be kept low. A node must not emit huge quantities of Ping and Pong messages.

A Ping with TTL equal to one and a hop count of zero or one is treated as a probe of the immediate neighbor. It must be answered with information about the node that received it.

A Ping with TTL equal to two and a hop count of zero is treated as a crawler ping. A crawler uses it to scan the local network neighborhood. The receiver should answer with information about itself and about the nodes it is directly connected to.

A node that follows these Ping and Pong rules should announce support with a Pong-Caching header during the handshake. The header name is Pong-Caching. The version value is 0.1. The name is a little misleading, because a node can follow the rules even if it does not literally store cached Pongs in the simplest possible way.

When storing or forwarding Pong messages, a node should preserve any GGEP extension data. A sender cannot assume that a Ping will reach only its direct neighbor. Depending on the neighbor's implementation, it may travel farther.

11. A simple Pong caching scheme

One simple Pong caching scheme keeps recent Pong information per connection. It is not the only acceptable way, but it illustrates the idea.

For each connection, the node keeps a small collection of recently seen Pong messages. Ten entries per connection is suggested. When a new Pong arrives, it replaces the oldest stored Pong for that connection.

For each stored Pong, the node keeps the host IP address, port, shared file count, shared kilobyte count, any extension block, and the number of hops between this node and the host described by the Pong.

When a Ping arrives on one connection, the node answers using Pongs stored from other connections. It avoids sending Pongs back to the same direction they came from. The node should also include a Pong about itself if it can accept incoming Gnutella connections.

The outgoing Pong uses the message identifier of the incoming Ping, not the identifier from the stored Pong. The hop count is adjusted to show the extra distance. The TTL is chosen so that the Pong can travel back but does not live forever.

A good response chooses a mix of Pongs from different connections and different distances. That gives the requester a broader and more useful view of the network.

To keep the cache fresh, a node can send a Ping on each connection every few seconds. That sounds frequent, but the neighboring nodes answer from their caches, so the cost is limited. If a neighbor does not support Pong caching, the node should send these update Pings much less often, perhaps once per minute.

This simple scheme uses about one hundred thirty-one bytes per second per connection when there are no extensions. Extensions increase the cost, but the bandwidth remains manageable.

12. Query messages

A Query is the search message. Since it is broadcast to many nodes, it should be small. A Query should not be larger than two hundred fifty-six bytes. Nodes may drop larger Queries, and should drop Queries larger than four kilobytes.

A Query contains a minimum desired speed, a search string, and optional extension data.

The minimum speed tells other nodes not to answer unless they can communicate at least that fast. This is an old mechanism and depends on honest reporting.

The search string is the text the user is searching for. In the binary format, it is terminated by a null byte.

A Query may include extensions. The allowed extension families are GGEP, HUGE, and XML. HUGE is used for hash and URN-style identifiers. XML can express richer search information or metadata. GGEP is the preferred generic extension container.

If a Query carries extensions, nodes need to know where each extension begins and ends. Extension blocks are separated in a defined way, and GGEP must appear in the right place so older parsing rules do not break.

13. Query Hit messages

A Query Hit is the answer to a Query. It tells the searching node that a matching file or resource was found.

A Query Hit contains the number of results, the responding host's port, the responding host's IP address, a speed value, and a result set. The result set contains one entry for each matching file.

Each result entry identifies a file using a file index, a file size, and a file name. The file index is a number assigned by the responding host. It is used later when another node requests the file. The file size is given in bytes. The file name is the name that the downloader can show to the user and use in the HTTP request.

Each result entry can also include extension information. The extensions might include HUGE data, GGEP data, or plain text metadata. Plain text metadata was used, for example, to describe audio files with information such as bitrate, sample rate, and duration.

The Query Hit may include an extra block sometimes called EQHD. This block gives more information about the responding client. It has a vendor code, an open-data size, and several flags.

The flags can indicate whether the private data contains GGEP, whether the speed field is a measured upload speed, whether the node has successfully uploaded before, whether the node is currently busy, and whether the node needs Push because it is firewalled or cannot accept incoming TCP connections.

Vendors should avoid undocumented private data and use GGEP extensions instead. Private data can still exist, but if a node does not recognize it, it should ignore it.

The last part of the Query Hit is the servent identifier of the responding node. This identifier is used later to route Push messages back to the right node.

14. Forwarding Queries and routing Query Hits

When a node receives a Query, it normally forwards the Query to all directly connected peers except the peer it received the Query from. Flow control and the Ultrapeer system can change this behavior, but simple Gnutella forwarding works that way.

Before forwarding, the node decreases TTL and increases the hop count. If decreasing TTL makes TTL zero, the message is not forwarded.

A node must drop duplicate messages. If it has already seen a message with the same type and identifier, it should not process or forward it again. This prevents loops from endlessly circulating the same Query.

A Query Hit must travel back along the same path that carried the original Query. This is important. Only the nodes that routed the Query should route the response. If a node sees a Query Hit for a Query identifier it never saw, it should remove that Query Hit from the network.

15. Creating new Queries

Most Queries are created when a user searches. A node may also create automatic Queries, for example to find more sources for a file. Automatic Queries can easily overload the network, so they must be rare. A node should not send more than one automatic Query per hour.

Nodes should prevent users from creating excessive Queries by repeatedly clicking a button. The client should also watch immediate neighbors. If a neighbor sends too many Queries, sends duplicates, or otherwise behaves badly, the receiving node should drop those Queries or close the connection.

A new Query should normally use a TTL no higher than seven. It must not use a TTL higher than ten. The hop count starts at zero.

16. Answering Queries

When a node receives a Query, it compares the search criteria against its locally shared files.

The character encoding of search text was never fully specified. Nodes must assume plain ASCII unless an extension says otherwise. If the text contains non-ASCII bytes, the node may need to guess the encoding. The likely encodings are ISO Latin 1 and UTF 8.

The exact search rules are not fully specified. Interoperable behavior follows a few common guidelines.

A search is treated as a set of keywords. A node should usually answer only with files that contain all the keywords. Words should be split on non-alphanumeric characters, and spaces are the standard separators.

A node may choose to require the terms to appear in the same number and order as the query. Matching should be case insensitive.

Regular expressions are not supported by default. Characters like asterisk or period should either be treated as normal characters or ignored. Empty searches and searches made only of one-letter words should be ignored.

A node may ignore very short searches because they are too broad and would generate too much traffic.

There is a special index Query. A Query with local TTL and four spaces as the search criteria is used to request a listing of all files shared by a host. A node should answer with all its shared files, possibly using multiple Query Hit messages. It may refuse for privacy or bandwidth reasons.

A Query Hit must use the same message identifier as the Query it answers. Its TTL should be high enough to get back to the searcher. Some implementations use a larger TTL to be safe, and high-TTL replies should be allowed to pass.

17. Push messages

Push exists because not all nodes can accept incoming connections. A node might be behind a firewall or network address translation device. If that node has a file, a normal downloader may not be able to connect to it directly.

Here is the idea. The downloader receives a Query Hit from a node that appears unable to accept incoming connections. The downloader sends a Push message back through the network toward the node that produced the Query Hit. The Push message asks the target to connect back to the downloader at a specific IP address and port so the downloader can request the file.

A Push message includes the servent identifier of the target node, the file index, the IP address and port where the target should connect, and optional extension data.

A node should act on a Push only when the servent identifier in the Push matches its own identifier. A Push should use a new message identifier, not the same identifier as the Query Hit that led to it.

Push messages are routed using the servent identifier from the Query Hit. Since Push messages are not broadcast, duplicates should be rare. Two Push messages with the same servent identifier are not automatically duplicates. They are duplicates only if their message identifiers are the same.

18. Bye messages

Bye is an optional message used to close a connection politely. A node must not send Bye unless the other side announced support for Bye during the handshake. The header used for that announcement is Bye-Packet, version 0.1.

A Bye message should be local only. Its TTL is one and its hop count is zero, so it does not accidentally propagate through the network.

When a node receives a Bye, it must close the connection immediately. The node that sent Bye should wait briefly for the remote side to close before closing its own side completely. It should not send any more normal data after Bye. Its outgoing queue should be cleared first.

The Bye message includes a code and a description. The code follows the same broad idea as mail or HTTP-style status codes.

A code in the two hundreds means the remote node did not do anything wrong. The connection is closing for a normal reason, such as the program exiting or the local user closing the connection.

A code in the four hundreds means the remote node did something the local node considers wrong. Examples include sending oversized packets, sending too many duplicates, relaying improper queries, sending messages with excessive TTL, sending too many unknown message types, staying inactive too long, failing to answer a local Ping, or not sharing enough.

A code in the five hundreds means the local node noticed an internal error. Examples include an input or output error, protocol desynchronization, or a full send queue.

The description should be human readable. HTTP-like headers should be included when possible, especially a Server header, because this helps with debugging.

19. GGEP, the generic extension system

GGEP stands for Gnutella Generic Extension Protocol. It is a framework for adding new data to Gnutella messages. New packet extensions should be placed inside GGEP.

Some older extension types, such as XML metadata, existed before GGEP. Those are still allowed outside GGEP in some cases, but GGEP is the recommended container for new work.

Nodes are recommended to implement GGEP. Even when a node cannot understand a GGEP extension, it should pass GGEP blocks along inside Gnutella messages. A node that can forward messages containing GGEP should announce support during the Gnutella 0.6 handshake with a GGEP version header.

The current GGEP version is 0.5. Nodes should remove GGEP blocks from Ping, Pong, and Push messages before sending those messages to peers that did not announce GGEP support.

A GGEP block begins with a special marker byte. After that, it contains one or more extensions. Each extension has flags, an identifier, a data length, and extension data.

The flags say things like whether this is the last extension in the block, whether the data is encoded, and whether the data is compressed. The compression method mentioned is deflate. The encoding method mentioned is COBS, which is useful because in some contexts the data must avoid null bytes.

The identifier names the extension. It can be between one and fifteen bytes. Very short identifiers must be registered with the Gnutella Development Forum to avoid conflicts. Vendor-specific identifiers can be formed from a vendor code plus a private identifier, but vendor-specific names make it harder for other implementers to use the same extension.

The data length gives the number of bytes of extension data that follow. GGEP uses a compact length encoding that avoids zero bytes and can represent small or large extension data without wasting much space.

The extension data itself is defined by the specific extension. If a node does not recognize the extension, it can skip it because the length is known.

20. Flow control

Every node needs a way to regulate how much data flows through each connection. Without flow control, a slow or overloaded connection could build up a huge queue and harm the network.

The simplest response to overload is to close the connection. A better response is to drop broadcast messages when bandwidth is tight. A more refined queue-based approach works like this.

Each connection has an output queue. Messages waiting to be sent are placed in that queue. When the queue is mostly empty, everything works normally. When the queue grows beyond a middle threshold, the node enters flow-control mode. It stays in that mode until the queue drains below a lower threshold. This gap between entering and leaving flow-control mode is called hysteresis. It prevents the node from rapidly switching modes on and off.

While in flow-control mode, the node drops incoming Queries on that connection. The reason is that a connection already struggling with output should not be asked to carry large Query Hit replies.

Messages are prioritized. Broadcast messages that have traveled fewer hops are more important, because they are closer to their source. Replies, such as Query Hits, become more important if they have already traveled many hops, because dropping them would waste all the network effort already spent carrying them.

By message type, Push is most important, then Query Hit, then Pong, then Query, then Ping. Bye is special and should always be sent after clearing the queue.

If adding a new message would overflow the queue, the node should first drop less important queued messages. If that frees enough space, it sends the new message. If not, it may drop low-priority messages such as Query, Pong, or Ping. If the message is more important and still cannot be queued, the node can send a Bye message with a send-queue-overflow reason.

Other flow-control algorithms are also possible, but they are not defined in detail here.

21. Network structure and Ultrapeers

Originally, Gnutella nodes connected to each other more randomly. That worked better for broadband users than for slow modem users. A more structured network helps reduce load.

The Ultrapeer system creates a hierarchy. Some nodes are leaves. A leaf keeps only a small number of connections, and those connections are to Ultrapeers. An Ultrapeer acts as a proxy between its leaves and the wider Gnutella network.

This structure helps the network scale. Leaves do not have to see or route every message. Ultrapeers do more routing work, and they can decide which queries should be forwarded to which leaves.

An Ultrapeer forwards a query to a leaf only if it believes the leaf might answer it. Leaves do not relay queries between Ultrapeers. Ultrapeers connect to other Ultrapeers and to ordinary Gnutella hosts.

Ultrapeers use the Query Routing Protocol, or QRP, to decide which leaf nodes might have matching content. If both a leaf and an Ultrapeer support another filtering protocol, they may use it instead. QRP is not used between Ultrapeers or normal hosts.

Implementations should support the Ultrapeer system or some future equivalent system that reduces bandwidth load on slow users.

22. Becoming an Ultrapeer

Because Gnutella is decentralized, there is no central server that appoints Ultrapeers. Each node decides for itself whether it is capable of becoming one.

Ultrapeer capability depends on several basic requirements.

A candidate should not be firewalled. A node can often estimate this by checking whether it has received incoming connections.

It should run on an operating system that handles many sockets well. Linux, Windows 2000, Windows NT, Windows XP, and Mac OS X generally handle this role better than Windows 95, Windows 98, Windows ME, or classic Mac OS.

It should have enough bandwidth. A practical minimum is at least fifteen kilobytes per second downstream and ten kilobytes per second upstream.

It should have enough uptime. Ultrapeers should be stable. A simple rule of thumb is that a node's expected future uptime is related to how long it has already been running. A node should not become an Ultrapeer immediately after startup.

It should have enough memory and CPU. Ultrapeers store routing tables and process many incoming queries. The exact requirements depend on implementation quality.

If those criteria are met, the node is Ultrapeer-capable. That does not mean it must actually become an Ultrapeer. Whether it becomes one depends on network need and how strongly it satisfies the criteria.

23. Ultrapeer handshaking

Ultrapeer information is exchanged during the normal Gnutella 0.6 handshake. Several headers carry Ultrapeer information.

The X-Ultrapeer header indicates whether a node is acting as an Ultrapeer or wants to be a shielded leaf. True means the node is an Ultrapeer. False means it wants to be a leaf.

The X-Ultrapeer-Needed header is used to balance how many Ultrapeers exist, although its behavior is not fully explained.

The X-Try-Ultrapeers header is like X-Try, but it lists only Ultrapeer addresses.

The X-Query-Routing header announces support for QRP and gives the QRP version.

Header order does not matter. True and False are case-insensitive.

If a leaf connects to an Ultrapeer and the handshake succeeds, the leaf becomes shielded by that Ultrapeer. It should drop non-Ultrapeer connections and send a QRP routing table if QRP is being used.

If a shielded leaf receives an incoming connection request, it should refuse it and redirect the requester toward other nodes, preferably including Ultrapeers.

If a node wants to be a leaf but cannot find an Ultrapeer, it may behave like an older unrouted Gnutella 0.4 connection.

When two Ultrapeers connect, both announce that they are Ultrapeers. If both already have leaves, they remain Ultrapeers. No QRP table is sent between Ultrapeers after the connection is established.

Sometimes there are too many Ultrapeer-capable nodes. In that case, an Ultrapeer may tell another capable node that more Ultrapeers are not needed. If the other node has no leaves, it may become a leaf instead. If it already has leaves, it should continue acting as an Ultrapeer.

24. Query Routing Protocol

QRP is the Query Routing Protocol. It is part of the Ultrapeer system. Its purpose is to help an Ultrapeer avoid sending a leaf queries that the leaf cannot possibly answer.

QRP does not guarantee that every forwarded query will match. Its goal is more modest: avoid forwarding queries that definitely do not match.

At the leaf level, the node looks at the names of resources it is sharing. It breaks those names into words. A word is a sequence of letters and digits.

The leaf lowercases words, removes accents, and keeps only words that are at least three letters long. It hashes each word into a large table. The table does not store the word itself. It only records that some word ended up at a particular table position.

The leaf also hashes trimmed versions of words, removing one, two, or three trailing letters when the remaining word is still long enough. This is a simple way to handle plurals and related endings.

The leaf then sends this boolean table to its Ultrapeer. The table may be compressed and sent in pieces mixed with ordinary Gnutella traffic.

At the Ultrapeer level, the Ultrapeer forwards all queries to the leaf until it has received the full routing table. Once it has the table, it uses it to filter queries.

For each query, the Ultrapeer breaks the query into words, normalizes them in the same general way, hashes them, and checks the leaf's routing table. Depending on the matching rules, all words or some required words must appear in the table. If the table indicates that the leaf might have a match, the query is forwarded. If the table indicates no possible match, the query is not forwarded.

The QRP description is unfinished here and depends on external LimeWire documentation for the full protocol.

25. Normal file transfer

When a node receives a Query Hit, it may download one of the files described in that result. The file data is not sent through the Gnutella search network. Instead, the downloader makes a direct connection to the host that has the file.

The file transfer protocol is HTTP. HTTP 1.1 is recommended, but HTTP 1.0 is allowed. Implementations should accept HTTP 1.0 requests and may retry with HTTP 1.0 if the other side does not support HTTP 1.1.

The downloader asks the sharing node for a specific file by sending an HTTP GET request containing the file index and file name that appeared in the Query Hit.

The file name in the request must be URL encoded according to standard URI rules. However, because some old clients did not support encoding correctly, implementations should also accept non-encoded requests. They may retry with a non-encoded form if the encoded request fails with a not-found response.

The HTTP Host header is required by HTTP 1.1. It tells the server what address the client connected to. The receiving Gnutella node may not actually use the value, but the header is still required by the HTTP protocol.

The User-Agent header identifies the requesting program. Implementers should not make strong assumptions about its value.

The server answers with HTTP headers and then sends the file data. The downloader reads the number of bytes described by the Content-Length header.

The most important HTTP features for Gnutella file transfer are range requests and persistent connections. A range request lets a downloader ask for a specific byte range. This is needed for resuming interrupted downloads and for requesting file segments. A persistent connection lets the same TCP connection stay open for another request after one transfer finishes.

The Connection header controls whether the connection is closed after a transfer. If it says close, the connection must be closed after the transfer. If it says Keep-Alive, or if the header is absent, the connection stays open and the client may issue another request.

Unknown HTTP headers should be ignored.

A node should not try to download multiple files from the same source at once. Instead, it should queue those downloads locally.

HTTP-related HUGE extensions should also be supported.

26. File transfer from firewalled nodes

Sometimes the downloader cannot directly connect to the node that has the file. The sharing node may be behind a firewall that blocks incoming connections. In that case, the downloader can request a Push.

The downloader routes a Push request back through the Gnutella network to the node that sent the Query Hit. When the target node receives the Push, it should open a new TCP connection to the downloader's IP address and port.

If that connection cannot be established, the downloader is probably also behind a firewall. In that case, this file transfer method cannot complete the transfer.

If the connection succeeds, the firewalled sharing node sends a short GIV message indicating that it is ready to provide a file. The GIV message includes the file index, the servent identifier, and a file name. The downloader should not trust those fields too much. It should simply request the file it wants using the normal HTTP GET process.

The sharing node must allow the downloader to request any file, not only the file named in the Push. After the connection is established, the rest of the transfer is the same as a normal file transfer.

If the TCP connection is lost during a Push-based transfer, the sharing node should try to reconnect. That matters because the downloader might not be able to send another Push request to the sharing node.

27. Busy nodes and upload queues

A node may reject a download if its upload bandwidth is already full. The usual response is the HTTP 503 code, meaning the node is busy or temporarily unavailable.

A node may use a fixed number of upload slots, but a smarter system uses bandwidth more effectively. One example is allowing new downloads as long as at least twenty percent of upload bandwidth remains unused.

If a busy node receives a Push, it should still connect back to the requester. Then, after the requester asks for the file, it can return the busy response. This lets the requester learn the real status.

A downloader may retry after a busy response, but it must not retry more than once per minute for the same file source. In other words, it must not open new connections to the same remote host more often than once per minute.

Servers should prevent aggressive clients from gaining an advantage by retrying too often. For example, a server can refuse connection attempts from a host that has requested the same download too recently, or it can temporarily ban hosts that flood it.

Upload queues are allowed but mostly outside the core protocol. One copied, unfinished queueing approach works as follows.

In that queueing approach, a downloader can send a header saying it supports queues. If an upload slot is available, the download begins normally. If no slot is available, the request goes into a queue and the server returns a busy response with queue information.

The queue information can include the requester's position, the queue length, the upload limit, and minimum and maximum times before the client should check again. The requester should wait an appropriate amount of time before trying again. Polling too soon is considered flooding. Waiting too long may be treated as dropping out of the queue.

The queueing description is unfinished.

28. Sharing rules

A Gnutella node that can download files must also be able to share files. Clients should encourage users to share.

Nodes should try to prevent programs that cannot share from downloading. For example, a node should not allow ordinary web browsers or download accelerators to take files from the network without participating as Gnutella nodes. Many clients respond to such programs with an HTML page explaining how Gnutella works and where to get a real Gnutella client.

A node must not give preferential treatment to users running the same client software. It must answer Queries and accept downloads according to the same rules for all compatible nodes.

A node may block clients that seriously break protocol rules and harm the network experience for others.

By default, a node should share the directory where downloaded files are placed. It should also share newly completed downloads without requiring a restart. It should avoid changing shared file index numbers unnecessarily, because other nodes may use those numbers when requesting files.

A node must not share incomplete files as if they were complete. A common way to avoid that is to store incomplete downloads in a separate directory. When a download finishes, it can be moved into the shared downloads directory. Partial files may be shared only if they are clearly marked as incomplete.

29. Security considerations

The security analysis is incomplete.

For individual users, one risk is accidental oversharing. Inexperienced users might share sensitive files, such as cookie files or password files. Clients should warn users who try to share sensitive information.

Another risk is malicious content. Viruses, trojans, and other harmful files can be shared by malicious users or by users who do not realize the files are harmful. Clients should encourage users to scan downloaded files, but virus scanning is outside the Gnutella protocol itself.

For the network as a whole, the main concerns include query flooding, fake files, denial-of-service attacks, and fake random Pongs. Fair sharing of bandwidth across connections and message types may be part of the solution.

Attackers may be able to abuse the Gnutella network to affect unrelated Internet hosts. Several scenarios are possible.

An attacker might answer Ping messages with fake Pong messages pointing to a target host. Other Gnutella nodes might then try to connect to the target, thinking it is a Gnutella peer.

An attacker might answer Queries with fake Query Hits pointing to a target host. This could cause download attempts against the target.

An attacker might respond to Query Hits with Push messages pointing to a target host. That would cause the receiver of the Push to attempt a connection to the target. The severity of this threat is not fully analyzed.

30. Credits and named extensions

The named credits include several Gnutella features that were not in the original Gnutella 0.4 specification. The 0.6 handshake is credited to LimeWire. The X-Try header is credited to Mike Green. The Bye packet is credited to Raphael Manfredi. Pong caching is credited to LimeWire and others. The EQHD block is credited to Free Peers. GGEP is credited to Jason Thomas. HUGE is credited to Gordon Mohr. Ultrapeers and Query Routing Protocol are credited to LimeWire. XML queries are credited to LimeWire. Negotiation of Gnutella network compression is credited to Raphael Manfredi.

31. Appendix 1, HUGE

HUGE stands for Hash and URN Gnutella Extensions. It adds support for hashes and URNs in Gnutella. A hash is a value that can uniquely identify file content. A URN is a uniform resource name, which is a way to name a resource independently of where it is located.

HUGE should be implemented inside GGEP. It can also be used as a stand-alone extension block. When used inside GGEP, the GGEP identifier for HUGE information is the letter u. When used outside GGEP, HUGE blocks begin with the text urn.

A single Gnutella message may contain multiple HUGE blocks.

HUGE also extends direct file transfer. It can let hosts communicate URNs and build download meshes. A download mesh means a node can learn about other locations for the same file, making downloads more resilient and efficient.

HUGE is recommended.

32. Appendix 2, XML

XML blocks can be used for richer queries and for metadata in Query Hit messages. XML should be placed inside GGEP, but it is also allowed as a stand-alone extension block.

An XML block may be compressed. If a block starts with a marker meaning deflate, the XML is compressed using the deflate algorithm. If it starts with a plaintext marker, or no compression marker, it is uncompressed. Other markers would mean other compression algorithms, but no other algorithms are defined here.

XML blocks can be recognized because they start with an opening angle bracket or an opening brace. The available XML appendix cuts off or garbles part of the next rule, so that missing detail remains undefined.

33. Compression negotiation

Compression negotiation belongs to the material on Gnutella network traffic compression.

Compression is negotiated with headers during the handshake. A node can announce that it can receive compressed data using an Accept-Encoding header. A peer can then choose a supported compression method and announce that it will send compressed data using a Content-Encoding header.

The only compression method described here is deflate. A node may advertise multiple compression methods, separated by commas, but the peer chooses one method for the Content-Encoding value.

Compression is asymmetric. That means one direction can be compressed while the other direction is uncompressed. For example, node A might send compressed data to node B, while node B sends uncompressed data back to node A. Both sides have to negotiate their sending and receiving behavior.

Already-compressed GGEP payloads should not be decompressed before sending them into the connection compression layer. The deflate algorithm handles already-compressed data with little overhead. If connection compression becomes common, individual GGEP extensions should usually not be separately compressed.

34. What the whole protocol is doing, in one continuous story

A Gnutella node starts by finding another node and opening a TCP connection. During the handshake, both sides announce what protocol version and optional features they support. If they agree, they begin sending binary Gnutella messages.

Each message has a header that identifies the message, identifies the message type, limits how far it can travel, records how far it has already traveled, and tells the receiver how much payload follows.

The network uses Ping and Pong to discover hosts. A Ping asks for host information. A Pong provides it. Because broadcast Ping traffic can waste bandwidth, modern nodes cache Pongs and answer discovery requests from their caches when possible.

The network uses Query and Query Hit to search. A Query asks the network for matching resources. Nodes forward Queries outward, while avoiding duplicates and respecting TTL limits. A node that has a matching shared file sends a Query Hit back along the path the Query used. The Query Hit tells the searcher where the file is and how to request it.

The actual file transfer happens outside the Gnutella search network. The downloader connects directly to the file host and requests the file using HTTP. Range requests allow resumes and partial downloads. Persistent connections allow multiple requests on one TCP connection.

If the file host cannot accept incoming connections, Push gives it a way to participate anyway. The downloader asks the firewalled host to connect back. If the firewalled host can open an outgoing connection to the downloader, the normal HTTP transfer can proceed over that connection.

To keep the network healthy, nodes use flow control. They limit queues, drop lower-priority messages under pressure, and prioritize messages that preserve useful network work. They also avoid excessive Queries and control repeated download retries.

To help the network scale, nodes may use the Ultrapeer architecture. Leaves connect to Ultrapeers instead of routing everything themselves. Ultrapeers use QRP tables from leaves to avoid forwarding searches that the leaves cannot answer.

Extensions are handled mainly through GGEP. HUGE adds hashes and URNs. XML allows richer queries and metadata. Compression can be negotiated during handshaking.

The protocol is careful about compatibility. Unknown headers should be ignored. Unknown extensions can often be forwarded. New features should be negotiated before use. Implementers should not split the network by requiring incompatible behavior without fallback.

35. Developer summary

The central implementation lesson is that Gnutella is not just a search protocol and not just a file transfer protocol. It is a discovery and routing protocol wrapped around direct HTTP transfers.

The most important moving parts are connection handshakes, message framing, duplicate detection, TTL and hop handling, query routing, and out-of-band HTTP file transfer.

The most important network-health concerns are bandwidth, queue growth, excessive broadcast traffic, repeated retries, fake or stale host information, and compatibility with older or less capable nodes.

The most important design principle is graceful interoperability. Nodes may support newer features, but they need to announce those features, fall back when needed, ignore unknown headers, and avoid harming nodes that do not understand every extension.

Several areas remain incomplete. The Ultrapeer and QRP sections are partly unfinished. The security analysis is incomplete. The upload queueing section is copied and unfinished. Some appendices are listed but not fully present in the available body. Those gaps matter for exact implementation.

The simplest mental model is this: Gnutella nodes connect to each other, trade capability headers, send small search and routing messages, use Ping and Pong to learn about peers, use Query and Query Hit to find files, use Push for firewalled peers, use HTTP to move file bytes directly, and use extensions like GGEP, HUGE, XML, and compression to add features without breaking older clients.