Hash and URN Gnutella Extensions

HUGE Version 0.94

Abstract

HUGE is a set of extensions for Gnutella version 0.4.

These extensions allow Gnutella to identify and locate files by Uniform Resource Names, or URNs. A URN is a reliable and persistent name that does not depend on where a file is stored. Secure file hashes are one way to create these names.

With HUGE, Gnutella can identify a file by its contents instead of relying only on its filename or its location.

1. HUGE in a Nutshell

HUGE adds several related abilities to Gnutella.

First, a search can ask other clients to include URNs with their results. For example, a search for “Gnutella Protocol” can also include the prefix urn:. This means:

Find files that match these words. When possible, include a URN with each result.

Second, a search can look for one exact file by its URN. In this case, the ordinary search text may be empty. The query instead contains a complete identifier beginning with something such as urn:sha1:.

This means:

Find the file with this exact SHA-1 hash.

A SHA-1 hash contains 20 raw bytes. In a Gnutella URN, those bytes are written as 32 Base32 characters.

When a client returns a QueryHit, it can attach one or more URNs to each individual file result. The URN appears in the extension area after the filename.

This means:

This file matches your search, and this is its content identifier.

A client that advertises a file’s URN must also allow that file to be requested by its URN. The HTTP request path begins with /uri-res/N2R?, followed by the complete URN.

This method exists alongside the older Gnutella download method, which requests a file by its local file number and filename.

A compliant client must support SHA-1 URNs. It must also understand enough of the related bitprint format to convert a bitprint identifier into its SHA-1 portion.

URN types that a client does not understand should be ignored without causing an error.

2. Background

2.1 Motivation and Goals

Traditional Gnutella searches identify files mostly by their names. This creates several problems.

Two copies of the same file may have different filenames. A client may show them as unrelated results even though their contents are identical.

A filename also provides no reliable proof that a downloaded file is the one the user expected.

Identifying files by a secure hash solves many of these problems.

A client can combine search results that represent the same file, even when the filenames are different.

It can download different parts of the same file from several sources. This is often called parallel downloading or swarming. When the download finishes, the client can verify that the assembled file has the expected hash.

A client can also resume an interrupted download from a different source. The hash provides confidence that the new source is serving the same file.

Hashes also make it possible to connect Gnutella content with outside catalogs and other peer-to-peer systems. Examples include Bitzi, FastTrack, EDonkey2000, OpenCola, MojoNation, and Freenet.

The goal of HUGE is to add these abilities without breaking older Gnutella clients. Clients that do not support HUGE should continue operating normally. HUGE should also coexist with other Gnutella extensions.

The words must, must not, should, should not, may, and optional have specific meanings in this document.

A feature described as something a client must do is required for compliance.

A feature described as something a client should do is strongly recommended, although there may be valid reasons not to do it.

A feature described as something a client may do is optional.

2.2 Status of This Document

Several commercial and non-commercial Gnutella clients have expressed support for HUGE. Many have substantially implemented the main parts of the specification.

The sections covering URN formats, queries, QueryHits, and downloads have received broad support in developer discussions and informal polls.

However, the complete specification has not passed through a formal Gnutella Developer Forum approval process.

The GGEP design described later in this document is experimental. It is meant for discussion and trial implementations. It does not represent an established standard or an immediate replacement for the existing HUGE system.

Version 0.94 includes several important changes.

Clients are now allowed to include URNs in QueryHits even when the original query did not explicitly request them. This reflects what many implementations were already doing.

Optional timestamps on alternate file locations must use the W3C profile of the ISO 8601 date and time format.

This version also explains how traditional HUGE extensions can coexist with GGEP extensions. It includes an experimental way to place HUGE information inside GGEP.

2.3 Hash and URN Conventions

A URN follows this general structure:

urn, followed by a namespace name, followed by a value defined by that namespace.

The parts are separated by colons.

The examples in this document use the sha1 namespace. A SHA-1 URN begins with urn:sha1: and ends with a 32-character Base32 representation of a 20-byte SHA-1 hash.

Letter case does not matter for these SHA-1 identifiers. Other URN formats may have different case rules.

The Base32 value must use the canonical Base32 alphabet:

A through Z, followed by the digits 2 through 7.

The digits zero and one are not used.

To encode binary data as Base32, read the bits in groups of five, starting with the most significant bits. If the final group contains fewer than five bits, add zero bits to the end.

Each five-bit group represents a value between zero and thirty-one. Replace that value with the matching character from the Base32 alphabet.

The encoded result must not contain spaces, separators, or padding characters at the end.

For example, the two bytes with hexadecimal values 0F and F5 become the Base32 text B72Q.

HUGE also recognizes the proposed bitprint URN format.

A bitprint contains a 32-character SHA-1 value, followed by a period, followed by a 39-character TigerTree value.

The SHA-1 portion identifies the complete file. The TigerTree portion provides additional protection and can help verify smaller parts of a file.

A bitprint can be converted into a SHA-1 URN by keeping the first 32 characters and removing the period and TigerTree portion.

Every HUGE-compliant client must be able to calculate and report SHA-1 values when appropriate.

A client that does not calculate full bitprint values should still recognize incoming bitprints and convert them into SHA-1 values whenever possible.

2.4 Gnutella Version

HUGE was designed as an extension to Gnutella Protocol version 0.4, document revision 1.2.

It does not depend on the Gnutella version 0.6 handshake system. The two systems are independent and can be used together.

3. The HUGE General Extension Mechanism

HUGE needs to place extra information inside Query and QueryHit messages.

It does this by using space that already exists between null characters in those messages. This area is often described as the space “between the nulls.”

In a Query, the ordinary search text ends with a null character. HUGE information can appear after that null and before the final null.

In a QueryHit, each filename ends with a null character. HUGE information can appear after the filename and before the next null.

This creates a problem when several unrelated extensions want to use the same area.

The HUGE General Extension Mechanism, or GEM, solves this problem by dividing the extension area into separate strings.

The strings are separated by the ASCII file-separator character. This is character number 28, or hexadecimal value 1C.

This character was chosen because it should not appear in normal human-readable text. It is also not allowed inside XML text.

A message may contain as many separate extension strings as will fit inside a legal Gnutella message.

Every extension must have a recognizable prefix. That prefix allows a client to decide which specification controls the extension.

A client must ignore extension strings that it does not understand.

Any extension string beginning with urn:, regardless of letter case, must be interpreted according to HUGE.

Future GEM extensions should avoid prefixes that could be confused with urn:. For example, another extension should not claim broad prefixes such as u, ur, or urn.

Coexisting with GGEP

The Gnutella Generic Extension Protocol, or GGEP, later became a widely supported way to add named fields to many kinds of Gnutella messages.

GGEP uses a different format from HUGE GEM. However, the GGEP specification describes a way for the two formats to coexist.

A client that implements both HUGE and GGEP must follow those coexistence rules.

A client that implements HUGE without fully implementing GGEP must still recognize the GGEP magic byte, hexadecimal C3. When an apparent extension begins with that byte, the client must treat it as data outside the HUGE GEM format.

Because GGEP provides a more general extension system, it is possible that few additional GEM-style extensions will be created.

However, existing HUGE GEM support is already deployed. A client must continue to understand and use the GEM form of HUGE to comply with this version of the specification.

4. Query Extensions

HUGE adds two main abilities to Query messages.

A Query can ask for URNs to be included with ordinary search results.

A Query can also search for a file by an exact URN.

Requesting URNs with Search Results

To request URNs, a client adds a GEM extension beginning with either the general prefix urn: or a more specific prefix such as urn:sha1:.

Using urn: means that the responding client may return any URN type it knows.

Using urn:sha1: specifically asks for SHA-1 URNs.

A client may request several URN types at once. However, the general urn: request is recommended unless the client has a reason to require a particular format.

When a Query requests URNs, a responding client should include matching URNs whenever possible.

A client must not discard an otherwise valid search result simply because it cannot provide the requested URN. It should still return the file result without the URN.

A client may also include URNs when they were not explicitly requested.

Searching by Exact URN

To search for a specific file, a client includes the complete URN as a GEM extension.

The ordinary text search may be empty.

A Query may contain several complete URNs. It may also contain both URNs and ordinary search text.

A complete URN in a Query also acts as a request for that kind of URN to appear in the returned results.

The matching rules use an or, not an and.

A responding client should return a file when it matches the ordinary search text, when it matches any supplied URN, or when it matches both.

It does not need to satisfy every part of the Query at the same time.

5. QueryHit Extensions

When a client includes URNs in a QueryHit, each URN belongs to one individual file result.

The URN must appear in that result’s extension area, after the filename.

It should not be treated as a single value for the entire QueryHit message, because one QueryHit may describe several different files.

A client may include a URN because the original Query requested it. It may also include URNs by default.

Each URN must be a valid GEM extension string beginning with urn:.

6. Download Extensions

6.1 Requesting a File by URN

A client that advertises URNs must allow another client to request the same file using its URN.

Traditional Gnutella downloads use an HTTP path beginning with /get/. The path contains a local file number and a filename.

HUGE adds another request form. Its path begins with /uri-res/N2R?, followed by the complete URN.

The URN-based request does not replace the traditional request format. A compliant client that reports URNs must support both.

The existing Gnutella PUSH and GIV systems are not changed by HUGE.

6.2 HTTP Headers

HUGE defines two HTTP headers.

The first identifies the content being transferred.

The second shares other known locations for the same content.

6.2.1 X-Gnutella-Content-URN

When responding to an HTTP file request, a compliant client should include the X-Gnutella-Content-URN header whenever it knows a reliable URN for the file.

The URN must describe the entire file.

This remains true when the response contains only part of the file because of an HTTP Range request.

A client may sometimes include this header even when it cannot currently provide the file. For example, it may recognize which file was requested but be temporarily unable to send it.

A downloading client may also send the X-Gnutella-Content-URN header in its request. This tells the remote client which content it expects to receive.

The client may send this header whether it used the old filename-based request path or the new URN-based path.

When the responding client is certain that the requested URN does not match the file it would otherwise return, it may answer with an HTTP 404 Not Found response.

A file may have several valid URNs. These may be placed in one header as a comma-separated list or sent as several headers with the same name.

Every URN in the list must identify the same file.

6.2.2 X-Gnutella-Alternate-Location

The X-Gnutella-Alternate-Location header shares other places where the same file may be found.

This header has a defined meaning only when it appears together with X-Gnutella-Content-URN.

The content header identifies the file. The alternate-location header lists other sources that are believed to provide that same file.

This header may appear in either an HTTP request or an HTTP response.

Each alternate location must contain a complete URL that can be used to retrieve the file.

A location may also include a timestamp. The timestamp records when that location was last known to work. This means that some kind of successful request was made to that location at that time.

The timestamp must use the W3C profile of ISO 8601.

Several alternate locations may be placed in one comma-separated header. They may also be sent as several headers with the same name.

The normal HTTP rules for splitting long headers across lines also apply.

A client may include alternate locations in an error response. For example, a client that is busy or does not currently have the file may still point the downloader toward other possible sources.

Alternate locations can travel in both directions. A client that already has a file may learn about new sources when another client includes alternate locations in an incoming request.

When an alternate-location header appears without a matching content URN, its meaning is not defined by this specification. A client should carefully verify such locations before trusting them.

Future versions may add more information after each alternate-location URL. Implementations should therefore tolerate extra space-separated values and ignore values they do not understand.

7. Recommended Implementation Order

A client does not need to add every HUGE feature at once. The features can be implemented in stages.

Stage One: Understand Extensions

The client should recognize the extension area in Queries and QueryHits.

It should separate GEM extensions, ignore unknown extensions, and continue processing the rest of the message.

It must also forward a Query whose traditional search text is empty when the Query still contains extensions. An empty text search may actually be an exact URN search.

After this stage, HUGE messages should not interfere with normal Gnutella behavior.

Stage Two: Report and Serve URNs

The client should calculate and report URNs for shared files.

It should accept downloads using the URN-based request path.

It should also send the X-Gnutella-Content-URN header when possible.

After this stage, other clients can use hashes to improve their downloads even if the local client does not yet generate URN searches.

Stage Three: Request URNs

The client should add URN requests to the Queries it creates.

This allows it to receive stable content identifiers with normal search results.

It can then group duplicate results, verify downloads, and prepare to use alternate sources.

Stage Four: Share Alternate Locations

The client should remember alternate file locations and exchange them through HTTP headers.

Ordinary download activity will then help create a mesh of redundant sources for each file.

Stage Five: Search by Exact URN

The client should generate exact URN Queries when needed.

It may use these Queries to safely resume an interrupted download, locate another source for a known file, or respond to a user selecting a specific file from a catalog or web page.

After this stage, a client can locate the same file long after the original download began, even when the filename or source has changed.

8. Experimental HUGE Support in GGEP

Developers adding GGEP support may also experiment with carrying HUGE information inside GGEP blocks.

This proposal is not an approved Gnutella Developer Forum standard. It is provided for discussion and testing.

The proposed GGEP identifier for HUGE information is the lowercase letter u.

The letter stands in place of the urn: prefix used by GEM.

Where a client would normally add a HUGE GEM string, it may instead add a GGEP block with the identifier u.

The GGEP payload contains the rest of the URN text, without the leading urn:.

For a general request asking responders to include any available URNs, the client would use a u block with an empty payload.

For a search by an exact SHA-1 URN, the payload would begin with sha1: and continue with the Base32 hash.

A QueryHit could report a file’s SHA-1 URN using the same u block and payload format.

When reading one of these blocks, the receiver must treat the missing urn: prefix as implied.

This design keeps the GEM and GGEP forms similar. It also leaves room for additional URN namespaces in the future.

Traditional HUGE GEM is already widely deployed. Even if the GGEP form becomes popular, clients will still need to send GEM-style Queries and respond to them with GEM-style QueryHits for the foreseeable future.

Adding GGEP support now could prepare clients for a possible future change. It does not remove the need to support the existing GEM format.