Text version here.

The Edsu Protocol, DRAFT Version 0.1, Rev. B 2018-11-04

1 ESON

The ESON format underlies much of Edsu. It doesn't stand for anything: it's a pun on JSON, with Edsu's E substituted. It was created to be simpler to emit and parse than JSON, but still be able to express the minimum amount of structure required by the higher-level protocols in Edsu that use it.

1.1 Format

An ESON document is composed of zero or more item lines, followed by an empty line. Each item line is composed of four parts: The key is one of either: A) A caret: ^ (i.e. the ASCII value 0x5E), or B) One or more characters from the set: abcdefghijklmnopqrstuvwxyz0123456789:- In the case of B, keys cannot begin or end with a colon, nor can two or more colons be directly beside each other. The space is the ASCII value 0x20. The value is zero or more characters from the set of 0x20 to 0x7e (inclusive) in ASCII, i.e. all non-control-code characters from 7-bit ASCII. The line feed is the ASCII value 0x0A. Whitespace is strict - the space and line feed characters cannot be substituted with any other character. If a key is followed by two or more spaces, all but the first space becomes the initial part of the value. Carriage return characters (ASCII 0x0D) are invalid anywhere in an ESON document. There are no upper length restrictions on keys, values, or the number of lines. Unless there is a specific reason not to, item lines should be sorted in ascending order, without regard to the meaning of the characters (i.e. a binary comparison, e.g. case insensitive). The final, empty, line that signals the end of the ESON document is a single ASCII value 0x0A. For example, here's a complete document with a single item: And here's an ESON document with no items:

1.2 Interpretation

The abstract parsed representation ESON is a map of keys (which are strings), to lists of values (which are also strings). An item line with a non-caret key represents a new entry in the map, with the key pointing to a list with a single item - the value. An item line with a caret key represents an append of its value to the most- recent non-caret key's list. For example, the following ESON document: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ one two ^ three ^ four five six seven eight ^ nine ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Could be represented in JSON as: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ { "one": ["two", "three", "four"], "five": "six", "seven": ["eight", "nine"] } ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Note that in the above example, the list with a single item ["six"] was represented as a simpler scalar value of "six". This is a valid interpretation, but not mandatory - it's up to the language/library which is doing the parsing. An ESON document starting with an item line with a caret key is invalid. There is no upper limit to the number of items in a list. An item line with a zero-length value is not semantically equivalent to the absence of that line - a zero-length value represents an empty string. When a duplicate key is encountered during parsing, it is semantically equivalent to first discarding the complete list of values that the key previously mapped to. For example: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ a 1 ^ 2 b 3 a 4 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Is equivalent to: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ { "a": "4", "b": "3" } ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

1.3 ESON as used in Edsu

Any block which is interpreted as an ESON document in Edsu must have a payload which contains nothing except exactly one valid ESON document (i.e. no extraneous data before or after). Extraneous ESON keys which are not defined by this specification will be ignored, unless otherwise stated. All ESON keys starting with "edsu:" are reserved.

2 Value Types Used in Edsu

Below are the requirements for converting ESON values to more meaningful types and back again. These types will be referred to in the rest of this document. Unless otherwise noted, whitespace is significant and is never discarded, and case is significant.

2.1 bool

One of exactly two strings: "true" and "false".

2.2 u16, u32, u64

Textual representations of non-negative integers, base 10. Leading zero and characters outside of the set 0123456789 (e.g. decimals or commas) are not permitted. The 16, 32, and 64 refer to the number of bits that the resulting number must fit into.

2.3 value

No conversion should be done.

2.4 version

A u16 followed immediately by a period (ASCII 0x2E) followed immediately by another u16

2.5 versions

One or more version values, separated by single spaces (ASCII 0x20)

2.6 encoding

An encoding ID. As of this version there is only one: deflate Any future encodings will be composed entirely of the following characters: abcdefghijklmnopqrstuvwxyz0123456789- Note that unlike HTTP, deflate refers specifically to a raw RFC 1951 encoding, without any encapsulation by the zlib or gzip formats. Servers must ignore any encodings that they do not recognize. The encodings of gzip, zlib, and brotli are disallowed.

2.7 encodings

One or more encoding IDs, separated by single spaces (ASCII 0x20)

2.8 multihash

A multihash as defined by the IPFS project, encoded in the base58 format using the following alphabet: 123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz As of this version of the protocol, servers must produce multihashes with the hash function sha2-256 (0x12) with a digest length of 32 bytes (0x20). It is expected that other hash functions and digest lengths will be added to maintain cryptographic strength as technology improves.

2.9 username

An ID followed immediately by an @ (ASCII 0x40) followed immediately by a domain name. The domain name must be apex: i.e. composed of exactly two labels joined with a period, the latter being the TLD; i.e. subdomains are not permitted. The ID has identical restrictions to a domain name label: it must be between 1 and 63 characters from the set: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789- and not start or end with a dash. Usernames are case-insensitive on interpretation (i.e. a@B.net and A@b.NET are equivalent). Clients and servers are required to accept mixed or uppercase versions of usernames, but are required to only emit lowercase versions.

2.10 name

An Edsu name is nine or more characters from the set: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_. A name cannot start or end with a period, nor can two or more periods be directly beside each other. The first 8 characters of a name must be one of the following: pub.std. pub.app. pub.srv. grp.std. grp.app. grp.srv. prv.std. prv.app. prv.srv. The maximum length of a name is 255 characters. The groups of characters between periods in a name are referred to in this specification as segments. Names must have 5 or more segments. Names with the third segment of "edsu" are reserved. By convention, names with a second segment of "app" have no restrictions, and those with "std" should be used in accordance with some published specification. Names with the second segment of "srv" are used exclusively by the server, and in interactions with it. By convention, in names with a second segment of "app", the third segment represents the user or organization which created the app, with the fourth segment representing the name of the app. For example, the name starting with pub.app.Ada.BernoulliCalc would be read as an app named BernoulliCalc created by someone named "Ada". By convention, in names with a second segment of "std", the third segment represents a category of standards, with the fourth naming the standard itself. For example a name starting with pub.std.communications.email would be read as relating to a communications standard, specifically email.

2.11 name-pattern

A name pattern is as follow: Where name is exactly as defined above. ".*" is the ASCII characters 0x2E followed by 0x2A - it may be omitted, in which case that name pattern and a name are identical.

2.12 ESON-encoded UTF-8

This encoding takes any UTF-8 string and encodes it so it may be stored in an ESON value. Every byte which is valid in an ESON value, except for ~ (ASCII 0x7E), may be encoded as itself - all other bytes will be called "non-encodable" in this section. Every occurrence of one or more contiguous non-encodable bytes is replaced by: Where ~ and ^ are ASCII characters 0x7E and 0x5E respectively, and base58 is the non-encodable bytes base58-encoded as described elsewhere in this specification. It is permitted to treat any or all encodable bytes as non-encodable (i.e. include them in the base58 encoding), but it is not recommended due to increased size and lack of consistency when being hashed. Likewise it is recommended to group as many adjacent non-encodable bytes together as possible before base58 encoding (for the same reason as just mentioned), however the encoder is permitted to group in any way.

2.13 address

An address is composed of usernames, names, and multihashes in one of three forms: Where : is ASCII 0x3A, and # is 0x23.

2.14 length-or-stop

If this is correctly parses as a u16, it is done so, otherwise it is treated as a unmodified value.

2.15 oob-code

One of: advisory redirect authentication-error authentication-challenge dropped-subs not-found hash-mismatch timed-out permission-denied invalid-input invalid-content rate-limited just-sent server-error

2.16 chain

The chain format is detailed in the section of this specification devoted to chaining.

2.17 token

The tokens used in the hello message must be: * 64 characters in length or less * Meet the same requirements as a name segment

3 Edsu Wire Protocol

Edsu is a client/server, connection-based protocol. All connections are made, conceptually, between two Edsu users: the user-from and the user-to. The user- from is the client, with the server acting on behalf of the user-to. It's often the case that user-to and user-from are the same. The first step to initiate a connection is to convert the user-to's username into a fully qualified domain name (FQDN): this is done by replacing the "@" with the characters ".edsu.", e.g. hello@example.com becomes hello.edsu.example.com. Once the IP address is resolved from the FQDN, a TCP/IP connection is established by the client to one of port 3216 or 443 at that address. Both ports only accept connections which are encapsulated with TLS. The protocol spoken on ports 3216 and 443 are identical, except that on port 443 it is further encapsulated by the WebSocket protocol. Edsu WebSocket connections must be encapsulated by SSL, and use the path /edsu/ws on connection. Once the connection is established, communication consists of messages sent from client to server and vice versa. A message is composed of a header, which always a valid ESON document, followed by an optional payload. A payload is zero or more unrestricted bytes (i.e. from 0-0xFF), followed by a line feed character (0x0A). The maximum size of a payload (excluding the aforementioned terminating line feed character) is 64,512 bytes (i.e. 63kB). Before or after messages, the client and server may send zero or more line feed characters - these must be discarded. The first item line of a header ESON document must contain the key "edsu" with the value stating the message type. Both whitespace and letter case is always significant in both the keys and values of message headers. If the header is found to be invalid (i.e. not a valid ESON document), if the values are not correctly formatted for the message type (e.g. a malformed multihash), or if the terminating payload character is not a line feed, the server response must be an oob with the code invalid-input and close-connection set to "true". Every key across all Edsu messages maps to a scalar value - i.e. the list functionality of ESON is not used. The exhaustive list of messages that may be sent from the client: hello block-put block-get name-put name-get name-append sub-put sub-clear ping pong And the similarly exhaustive list of messages from the server: hello oob ok block name sub-notify authenticated ping pong Though some of them share the same name, the client and server messages are distinct, and messages defined as being from the server must not be sent by the client, and vice versa. Extraneous keys in the ESON header of messages are allowed, however they must contain at least one ":" character and cannot begin with "edsu:".

3.1 Order of Actions

Unless otherwise specified, the actions taken by the Edsu server must be performed in the order that the instigating messages arrive on the connection. The exceptions are: * Block-get messages and any resulting chains may be completed at any time after the message arrives. * Ping messages may be serviced at any time. Multiple connections to the same hosts are independant of each other from an order of actions perspective.

3.2 Channels

In every client-to-server message there is an optional key "channel", the value of which can be any valid ESON value. Similarly, for every server-to-client message there is a required channel key of the same type. If used in a client message, all server messages generated as a result must echo the channel unchanged. If unused, or in the case of a server message that is unrelated to any specific client message, the channel returned in the server message must be the default value of "0". In all message descriptions that follow, the channel key for both client and server messages is implied.

3.3 Encoding

The client message "block-put" and the server messages "block" and "oob" have a key called encoding. This states the encoding (presumably but not necessarily a compression algorithm) which has been used on the payload of that message. The decoding capabilities of both the client and server are exchanged during connection initiation. It is at their descrection which encoding to use (or none) on a per-message basis, provided that the encoding is reported to be decodable by the receiver.

3.4 Out Of Band Message

At any time the server may send an Out Of Band message (oob). This has the format: The oob-code key describes the reason for the message. The close-connection key, if present and "true", means that the server has requested that the client close the connection immediately. The server will stop processing further messages from the client and will close the connection on its side if the client does not do it quickly enough. The retry-delay-ms key is a recommendation that the request that caused this OOB message to be retried, unchanged, after a wait. That wait is specified in milliseconds. This is an advisory and it can be ignored by retrying at any time or not at all. The payload-length key specifies, in bytes, how long the payload of the message is (if there is one). Likewise, the encoding key specifies the payload's encoding (if any). The payload is dependant on the oob-code. If not detailed explicitly in this specification, the server may send a payload that is an ESON document, and it may include debug information using the key "debug", with the value being a natural language description of what went wrong, and any futher details.

3.5 Connection Initiation

The first messages sent on a newly established connection must be the client sending a hello. Once that is sent (i.e. before the response is received) it is then free to send any other valid client-to-server message, though it is disallowed (OOB code invalid-input) to send another hello message during the lifetime of the connection. The first message sent from the server to the client must be a hello in response to the client's. Like the client, that hello can be sent only once per connection. The client-to-server hello message is: The versions key states the versions of the Edsu protocol that the client is able to speak. If the server does not speak any of them it must send an oob with the code server-error, with close-connection set to "true". The encodings key states the payload encodings that the client is able to parse. If none of secret, token, or visitor-username is sent, the client is requesting an anonymous connection. Alternatively, they can request to authenticate themselves as A) the user-to via secret, B) the user-to via token, or C) a user- from via token and visitor-username. No matter the method of authentication, the immediate response from the server is: The version key states the version of the Edsu protocol that the rest of the connection will be using. The encodings key states the payload encodings that the server is able to parse.

3.5.1 Secret

The value of the secret key must be exactly 64 characters long. In the likely event that the actual secret is shorter than this, it must extended on the right with spaces (ASCII 0x20). To arrive at the secret to be compared, the server must first strip all trailing spaces, then converted to UTF-8 using the method described elsewhere in this specification, and then normalized using the NFKC form. The secret key is exclusive of the token and visitor-username keys. If the secret (which is established via some external mechanism) matches, the client is authenticated as the user-to; i.e. user-from and user-to are the same. A message is sent: This indicates to the client that the authentication is successful. That capability block must at least grant the required permissions to use the functionality at pub.srv.edsu.authentication.tokens.generate.*. If the password doesn't match, an Out Of Band message (oob) is sent with the oob-code of "authentication-error", and close-connection set to "true".

3.5.1.1 Authentication Challenge

On a successful match of the secret, instead of responding with an authenticated message, the server may send an OOB message with the code "authentication- challenge". The payload of this message will be an ESON document with the key "edsu:type". The value pointed to by this key must be: "question" (it is expected that in later versions of this specification there will be other possible values). In the case of a "question" challenge, the ESON will also contain a key "edsu:question", with an ESON-encoded UTF-8 value which should be presented verbatim to the user. The client must then collect an answer from the user, base58-encode it, then append that base-58 encoded value to the string "pub.srv.edsu.authentication.challenge.answer.", forming a name. This name should then be sent to the server in a name-get message. If the challenge has been successfully answered, the server will respond with a name message - the hash of which has no meaning and should be ignored - followed by an authenticated message identical to what is sent when there is no challenge and the secret match is successful. Alternately, if the challenge has not been met, an OOB message with the code "not-found" will be sent, followed by an OOB message with the code "authentication-error". This second OOB will have the key "close-connection" set to "true" and the connection will be closed shortly after.

3.5.2 Token, Without Visitor-Username

A token key without a visitor-username is interpreted as requesting authentication as the user-to based on that token. The content of the token is left up to the server, as long as it meets the requirements of the "token" type described elsewhere in this specification. The mechanism for validating the token is up to the server. The response to the client is identical to that of a secret-based authentication which generates no challenge.

3.5.3 Token, With Visitor-Username

A hello message with both token key and visitor-username present is interpreted as requesting authentication as the user stated in visitor-username; if authentication is successful, the user-from of the connection is set to the contents of visitor-username. To authenticate, the server which received the hello message (referred to in this section as the user-to server), opens a non-WebSocket, anonymous Edsu connection to the user-from's server, acting as a client. On connection, a name-get message is sent, with the name being: Where token is the value from the client's hello message. Authentication is considered successful if the user-from server responds with a multihash which resolves to a valid visitor capabilities block. These capabilities are then applied to the client's connection. The user-to server may optionally cache the results of this authentication for up to 4096 seconds, after which it must repeat the transaction before performing any further actions for the client. In the event of the capabilities being different from those obtained by a previous authentication, the connection must be amended to reflect this. The response to the client hello message from the user-to server is identical to that of a token authentication without a username. Before being used in the hello message, a visitor token must be specialized on the user-to. First, the visitor token (as returned from the token generating service) is divided into two parts: the prefix and the postfix, with the former being the first 8 characters, and the latter being the remaining character. An HMAC-SHA256 function is then applied, with its key being the postfix, and its message being the user-to username. The result of this function is base58-encoded, and then appended to the prefix to produce the specialized visitor token which can then be used in the hello message. Server implementations must take care that the postfix is still of sufficient length to provide adequate security properties.

3.6 General OOB Messages

3.6.1 At Any Time

The server may at any time send an oob with the codes "advisory", "server- error", "redirect", and "dropped-subs". Advisory oobs can always be safely ignored. As of this version of the specification, there are no advisories specified. Server-error oobs can be sent at any time (using channel "0"), or in response to a particular client message (in which case it will use that message's channel value, or "0" if none was given). It indicates that something has interfered with the normal operation on the server in such a way that the client should respond. The client response is dictated by the presence and values of the close-connection and retry-delay-ms keys. Redirect oobs can be sent at any time, and indicate that the client should drop the connection and reconnect to a different server. The close-connection key must be "true". The payload is a (poentially encoded, using the encoding specified in the message) ESON document with the following keys: Username is what should be used as the basis for the reconnection - i.e. it will be used to determine the FQDN of the server to connect to. A redirect with alias set to "false" indicates that the server has decided that another server on a different host is better equipped to handle the connection. Semantically it is completely transparent: the user-to is, other than from the perspective of the network protocols that underly Edsu, the same as before the redirect. This is used for load balancing. A redirect with alias set to "true" indicates the user-to has aliased the username to another. Semantically, the client is aware of the alias and can respond as it wants to this information. It is used in the event of an Edsu user changing their username, but wants addresses featuring their old username to resolve to their current account. Dropped-subs is addressed in the section of this specification which details subscriptions.

3.6.2 In Response to Client Messages

Not-found indicates that the name or block which has been requested does not exist on the server, or in some cases it exists but the connection is lacking the capabilities to know of its existance. Hash-mismatch is used to indicate when messages with existing-hash keys encounter a different multihash than what is specified. Timed-out indicates that the request took too long to complete and the server has given up on it. Note that it is possible that the request actually did complete, but the server was unaware of this at the time it generated the timed- out oob. This message is the only response to the originating client message - i.e. the server must not send a timed-out oob and then afterwards send a message indicating success or failure of the original request. Permission-denied indicates that the connection does not have sufficient capabilities to perform the action requested. Invalid-input indicates that the contents of the requesting message are non- sensical in some way. Invalid-content indicates that the data stored on the server does not make sense in the context of the request. E.g. if a request presumes that a certain block stored on the server is an appendable block, and it isn't, it will generate an invalid-content oob. Rate-limited indicates that either the connection, or all connections to a particular user-to have asked more of the server than it is willing to provide. This oob must specify a retry-delay-ms to advise when a reasonable time to retry (unchanged) the client message will be. However, this is not to be interpreted as a guarantee that waiting that amount of time will satisfy the rate limiter. Just-sent is detailed in the section of this specification devoted to chaining.

4 Blocks

Blocks are in one of two formats: text or binary. In either case, the upper size limit for the final encoded form is 64,512 bytes (i.e. 63kB).

4.1 Text Blocks

Text blocks consist of a header and a payload. The header is in the form: Where ~ and line feed are ASCII 0x7E and 0x0A respectively, and salt is zero or more characters in any valid UTF-8 encoding. The payload is zero or more characters in any valid UTF-8 encoding. Any Unicode byte order mark is prohibited anywhere in the block.

4.2 Binary Blocks

Binary blocks consist of a header and a payload. The header is in the form: Vers is a byte indicating the encoding format's version, which is 0.1. The four most significant bits are the major version (0), and the four least significant bytes are the minor version (1). So 0.1 is 0x01, 3.4 is 0x34, etc. Servers must not reject blocks with a vers greater than what they recognize, however, the format is forwards-compatible, so any block must be parsable as if it were version 0.1 Cloc-msb and cloc-lsb are the most and least signficant bytes (respectively) of the 16-bit location of the start byte of the contents. Hloc-msb and hloc-lsb are the most and least signficant bytes (respectively) of the 16-bit location of the start byte of the hashes. Salt is zero or more bytes of salt, each byte of which can be any value from 0x00-0xFF. Hashes is the concatenation of zero or more byte-encoded multihashes. These are not the same multihash format as used elsewhere in this specification - the multihash format is the same, but it is not Base58-encode. E.g. a SHA-256 multihash would always be represented as 34 bytes: two bytes of header followed by 32 bytes of digest. Contents is zero or more bytes, each byte of which can be any value from 0x00-0xFF.

4.3 Block Messages

4.3.1 Block Put

The client stores blocks on the server using the block-put message: Where payload-stop is either a u16 or an unparsed value. If it is a u16, the server will read that many bytes of payload, then expect a line feed. The line feed is discarded. If payload-stop is not a u16, the server will read payload bytes until that value is encountered, after which it will expect a line feed. Both the payload- stop value and the line feed are discarded. If the connection does not have the capability to upload blocks, the server will respond with an oob with the code "permission-denied" and stop. If the encoding key is present, the server will then decode the payload using the specified method. If the specified method is not supported, or the encoding is invalid, the server will respond with an oob using the code "invalid-input" and stop. The decoded payload is referred to as "the payload" for the remainder of this section. The payload must be a valid text or binary block. If not the server will respond with an oob using the code "invalid-input" and stop. The payload is now stored on the server. It may be associated with the user-to account, but that is not necessary (i.e. it is acceptable for the server to pool blocks between users). If successful (i.e. none of the above-mentioned oobs are returned, nor any of the more generic oobs such as "server-error" or "timed-out"), the following message is sent to the client: Where hash key must be present, and is the multihash generated from the bytes of the payload (i.e. the block).

4.3.2 Block Get

The client retrieves blocks from the server using the block-get message: Where hash is the multihash returned when the block was stored with block-put. The chain key is addressed in the Chaining section of this specification - however no matter its value, the behaviour described below is invariant. The server responds with one of: a generic oob, an oob with the code "not-found" (if the server does not have a block stored at that hash), or the following message: Hash is echoed unaltered from the same key in the client message. Payload- length is the length of the payload in bytes, and encoding is the optional encoding of the payload. The server is free to use any encoding that the client has reported it supports in its hello message, or none, at its discretion.

4.4 Name Messages

4.4.1 Name Put

In addition to blocks, Edsu stores name associations. A name association is a one-to-one mapping between a name and a block, as referenced by its hash. The block referenced by the name must exist on the server. The client changes the association of a name with a block using the name-put message: Name is the name association to be operated on. Hash is optional: if it is not present the assocation is deleted. If it exists the association is either created with the specified hash, or updated to it. Existing-hash is required for every operation which isn't a creation. If the association exists, it must match the hash that the association currently points to. If it doesn't match, the server must send an oob with the code "hash- mismatch" and stop. A non-existant hash compared to an existant one (e.g. existant-hash exists but the association doesn't) it is considered a mismatch. As a special case, if the hash in name-put already matches that name's existing association, existing-hash is disregarded and the server must act as if the name-put operation had occured and was successful. When hash is specified, if a corresponding block is found on the server the server will respond with an oob with the code "not-found" and stop. The corresponding block must be a text block, containing exactly one valid ESON document, with no extraneous data before or after it. If this requirement is not met, the server must respond with an oob with the code "invalid-content" and stop. If the name-put results in an updated hash, and that block's ESON has the key "edsu:previous", the list pointed to by that key must contain a single item which matches the existing-hash key in the client message. That is, if the name block has an "edsu:previous" key, it must be accurate. If not, the server must send an oob with the code "invalid-content" and stop. If the connection lacks the capabilities to perform the operation requested, the server must respond with an oob with the code "permission-denied" and stop. If the operation is successful the following message is sent: Where the hash key is identical to the same key from the client message (including its absence if it is absent from the client message).

4.4.2 Name Get

To retrieve a name association from the server, the client sends a name-get message: Where name is the association requested. If the connection lacks the required capabilities to retrieve the association, the server will send an oob with the code "permission-denied" and stop. If the association doesn't exist, or the connection lacks the required capabilities to know if it exists or not, the server will send an oob with the code "not-found" and stop. Otherwise (and unless a more generic oob is sent), the server will respond with: Name is an echo of the name from the client message. Hash is the multihash which is associated with that name.

4.4.3 Name Append

The message name-append appends a value to a named appendable block and changes the name association to point to the resultant block: Name is an association which points to an appendable block. If the block associated with the name isn't an appendable block, the server will send an oob with the code "invalid-content" and stop. If the block does not exist, the server will send an oob with the code "not-found" and stop. If the connection lacks the required capabilities to retrieve the association, or fails to meet the requirements stated in the appendable block, the server will send an oob with the code "permission-denied" and stop. If the rate conditions stated in the appendable block would be exceeded by the append, the server will send an oob with the code "rate-limited", along with an advisory "retry-delay-ms" which will suggest when to try again, and stop. If none of the above-mention disqualifiers exist, and barring other more generic oob responses, the server will ensure that the following append value is added to the appendable block: Where time is a u64 representing when the message was received, encoded as the number of seconds elapsed since the Unix epoch. Both spaces are ASCII 0x20. User-from is that of the connection. Hash is the hash contained in the message from the client - it does not need to reference a block which exists on the server. If the key "edsu:appended" does not exist in the appendable block, the key will be added, pointing to a list with a single item, the append value. If the key "edsu:appended" does exist, and the addition of the append value will not violate the size restriction specified in the appendable block, the append value is added to the top of the list (i.e. it will become its first item) of that key. If the key "edsu:appended" does exist, and the addition of the append value will violate the size restriction specified in the appendable block, the append value is added to the top of the list, and then the block is duplicated, with the top (most recent) half of the "edsu:appended" list being placed in one, and the bottom half being placed in the other. If the number of items is odd, the larger of the two lists will be that of the least recent block. The least recent block is stored in the user-to's storage, and its hash is placed in the most recent block under the key "edsu:append-previous" (replacing what came before if necessary). The amended appendable block (or the most recent one in the case of a split) is then stored in the user-to's account, and the name stated in the client message will have its association changed to the amended block's hash. Then the server sends the following message: Where hash is the multihash of the new name association. However, if the connection lacks the capabilities to see the name association, the hash key will not be present in the message. If more than one name-append message arrives at once or during the servicing of an already-begun name-append operation, the server is required to ensure that all of their resultant values are present in the appendable block, and that the times listed in the "edsu:appended" list monotonically decrease. Further details on appendable blocks can be found in the section devoted to them.

4.5 Subscription Messages

4.5.1 Subscription Put

Subscriptions are added or amended via the following message: Name is the name association to be monitored; it does not need to exist. When the message is received, the value of existing-hash is compared against the hash currently associated with the name given, and if they differ a notification is sent - note that the absence of this key matches the absence of the name assocation, and if one exists without the other this is considered a mismatch. Once being set to "true" causes the subscription to be deleted once it is responsible for a notification being sent to the client. If expires exist, once the server clock (as measure in seconds since the Unix epoch) matches its value, the subscription is deleted. If chain exists, that chain will be executed every time a sub-notify message is generated as a result of this subscription. Note that permissions checks are performed when notifications are generated, not when subscriptions are being created, so sub-put will never fail due to insufficient permissions. Each connection has a set between 0 and not more than 256 subscriptions. If a sub-put message is received which would result in the creation of a 257th subscription, the server will send an oob with the code "invalid-input" and stop. If no generic oobs are sent, the server will send the following message: The hash key will not be present.

4.5.2 Subscription Clear

The client can delete all subscriptions attached to its connection by sending the following message: The server will respond with: The hash key will not be present.

4.5.3 Subscription Notify

At any time the server may send a sub-notify to indicate the current state of a subscribed name: Name is the subscribed name. Hash is the associated hash - if the name association does not exist, the hash key will not be present. The server must send a sub-notify message for every change in any subscribed name association - i.e. it is created, deleted, or becomes associated with a different hash. However, there are no guarentees about not sending duplicate or spurious notifications, other than the hash key in each sub-notify message must always be accurate at the time of message being sent. If there are too many subscription notifications to be sent (as determined by the server), the server may choose to not send them. If this choice is made, the server must also delete all subscriptions on the connection (i.e. the same as if the client had sent a sub-clear message), and send an oob with the code "dropped-subs".

4.6 Ping Messages

Both the client and server can send ping messages at any time: The receiver is obligated to send a pong message as soon as possible:

5 Special Blocks

Certain blocks will cause the behaviour of the Edsu server to change, or create an expectation on the part of those using its data. These are called special blocks. All of these blocks are required to be text blocks containing exactly one valid ESON document without any extraneous data outside of it document. A block is interpreted as one or more types of special blocks dependent on the context in which it is used. Any keys that are not recognized within the context that the block is being interpreted are ignored.

5.1 Group Block

A group block contains the key "edsu:users", and with pointing to a list of one or more of the items, as follows. A block that does not contain the key "edsu:users" is interpreted as an empty group. Note that membership in a group is a set union function - if multiple items indicate that a user is part of the group, that is semantically equivalent to them being included once. These are the possible items:

5.1.1 Uername

A user whose username appears in the list is part of the group.

5.1.2 Wildcard

An * (ASCII 0x2A) indicates that all Edsu users are part of this group

5.1.3 Domain Wildcard

Where * and @ are ASCII 0x2A and 0x40 respectively and the domain is a valid domain name part of a username. Any Edsu user whose username contains the same domain name is part of this group.

5.1.4 Block Include

The "#" is ASCII 0x23; hash is a multihash which points to another group block which is stored on the server. This group block is interpreted to be a union set with the referencing group block.

5.1.5 Name Include

The ":" is ASCII 0x3A; name is a name association in the user-to's account which is associated with a group block. This group block is interpreted to be a union set with the referencing group block. If the association is non-existant, or if it's associated block is not a valid group block, this item is interpreted as not existing.

5.2 Name Block

When a name association is queried to find out its hash, the ESON document contained in the block which is referenced by that hash is interpreted.

5.2.1 Previous

As mentioned in the Name Messages section, "edsu:previous" must be accurate in that its value must equal the hash of the association that immediately preceeded the present one.

5.2.2 Once

If the name block contains the key "edsu:once" and it is set to "true", any action which sends any client the associated hash will then immediately delete the name association; e.g. name-get will trigger the deletion, as well as sub- notify. The server must guarantee that the deletion is atomic - i.e. the hash is seen exactly at most once.

5.2.3 Expires

If the name block contains the key "edsu:expires", and its value is a u64, the name association will be (to all outside appearences) deleted when the server time (number of seconds elapsed since the Unix epoch) equals the value of the key.

5.2.4 Deny and Allow

The values of both "edsu:deny" and "edsu:allow" are interpreted identically to the "edsu:users" key in a group block. If the connection's user-from belongs to the group specified by "edsu:deny", any attempt to determine the association of the name pointing to the name block will fail with either an "permission-denied" or "not-found" oob. If the connection's user-from belongs to the group specified by "edsu:allow", and does not match the group specified by "edsu:deny", this is factored into the permissions required to determine the of the name pointing to the name block, depending on the context. See the section on permissions for more details.

5.2.5 Description

A name block may optionally include the key "edsu:description", which is an ESON-encoded UTF-8 string which describes what the block contains. This value is meant to be read by humans.

5.3 Metablock

A metablock contains keys that refer to other blocks. The key "edsu:metablocks" should contain a list of one or more hashes of other blocks that are to be interpreted as metablocks. The key "edsu:blocks" should contain a list of one or more hashes of blocks that are not metablocks. By convention, when "resolving" a metablock into a single sequence of bytes, the steps are: 1. Start with an empty buffer for bytes 2. Concatenate onto the buffer all contents of blocks listed in "edsu:blocks", in the order in which they appear 3. For each block in "edsu:metablocks", in the order in which they appear, repeat steps 2 and 3 I.e. metablock trees are resolved depth-first, with the contents of the nodes being resolved before their branches. Any metablock may contain one or both of the keys "edsu:contents-size" and "edsu:size". The former is the total, in bytes, of the block contents referred to in that block's "edsu:blocks" key, and recursively the same for all metablocks in its "edsu:metablocks" key. I.e. the size of the resulting concatenation of block contents produced by the above procedure for resolving a metablock. The latter ("edsu:size") is the same, except instead of the contents of the blocks, what is totalled is the entire block size of each (i.e. including the header). Note that this total specifically does not include the size of the metablocks themselve, only the blocks which appear in "edsu:blocks" lists. For both "edsu:contents-size" and "edsu:size", blocks which appear more than once contribute to the total as many times as they appear.

5.4 Appendable Block

5.4.1 Deny and Allow

The values of both "edsu:append-deny" and "edsu:append-allow" are interpreted identically to the "edsu:users" key in a group block. If the connection's user-from is part of the "edsu:append-deny" group, an attempt to perform a name-append on that name block results in an oob with the code of "permission-denied". If the connection's user-from is not part of the "edsu:append-allow", then the result is the same. A block with does not contain the "edsu:append-allow" key is not considered an appendable block. An attempt to perform a name-append on that name block results in an oob with the code of "invalid-content".

5.4.2 Appended

All items appended appear in the "edsu:appended" key. If this key does not exist it will be created on the first successful append.

5.4.3 Division

If the result of an append is a block whose size is larger than the u16 specified in "edsu:append-bytes-max", the server will divide the block in two, as described in the section detailing the name-append message. The key "edsu:append-previous" will be set so that the divisions form a linked list from most to least recent. If the "edsu:append-bytes-max" key is absent, the default of 63,488 (62kB) is used.

5.4.4 Rate Limiting

If the key "edsu:append-rate-limit" exists, appends will be rate-limited. The format of the value is a single space (ASCII 0x20) separated string of sub- items. Sub-items may be in any order. The format for a sub-item is: In this case, the sub-values are all u64 values. The sub-keys are "seconds", "all-users", and "per-user". "Seconds" is required, as is at least one of "all- users", and "per-user". "Seconds" must be greater than 0 and less than 86401 (one day). Rate limiting is based on a sliding window which is "seconds" long. "All- users", and "per-user" are the number of appends allowed within the time window. For example, a value of: "seconds:3600 all-users:10 per-user:1" would be interpreted as: no more than 10 appends per hour, with no more than 1 append per hour by any one user". Of the two rate limits (per-user and all-user), whichever one is the most restrictive for that append is applied.

5.4.5 Time to Live

If the key "edsu:appended-ttl" is present in the head of the appendable block linked list (i.e. the one pointed to by a name), the server will periodically remove items from the "edsu:appended" list. An item will be removed if it is older than time to live specified by the u64 value of "edsu:appended-ttl". Note that if any item is removed, all the hashes up the linked list of appendable blocks will change, including an updated name association with the head. There are no guarantees as to when or how often this will take place, but if the server is being run by a commercial entity that charges based on amount of storage used, garbage collection should be performed before those charges are evaluated.

5.5 Capabilities Block

Capabilities blocks are detailed in the pemissions section of this specification.

6 Chaining

When a block is sent to the client as result of a client message that contains a chain key, for the next one second the server may send additional blocks specified by the value of that key. If during that one second the client sends a block-get requesting a block that was sent as a result of the chain that began the one second countdown, the server may, instead of a block message, send an oob with a code "just-sent" and "retry-delay-ms" set to the milliseconds remaining in the one second period. A chain consists of zero or more chain items, separated by single spaces (ASCII 0x20), in the order of when the client would like them to be sent. Each item is one of the following: Where key has the same definition as an ESON key, *, $, >, and @ are ASCII 0x2A, 0x24, 0x3E, and 0x40 respectively, key is identical to the ESON key format, and id is one or more characters of the set: abcdefghijklmnopqrstuvwxyz0123456789- An ID represents a block. The ID "0" is implicit, and represents the block requested in a block-get message, or the block associated with the name in a name-get message or a subscription notification. The characters *, $, and @ are selectors. They cause the block identified by the ID to be parsed as ESON, and then have the selector operation performed on it, as described below. If the block does not contain a valid ESON document, it is treated as an ESON document with zero items. The * symbol indicates that any valid multihashes found in the list pointed to by the key in the chain item are requested blocks. The $ symbol indicates that the first item found in the list pointed to by the key in the chain item represents a requested block, if it is a valid multihash. If the item contains both a $ and > symbol, then the block requested by the $ operation thereafter becomes identifiable by the ID which comes after the > symbol, to be used in following chain items. The @ symbol indicates that the value pointed to by the key in the chain item (provided that it is in a valid chain format) should be substituted inside current chain, replacing that chain item. The sequence of ID -> selector -> ID, etc. represents a chain, each starting from the "0" ID. The server may send zero or more blocks from any of these chains, but it must not "skip" sending blocks within those chains. I.e. at any time during the one second, the client must have the data on hand to verify that the server has executed the chain faithfully. In the case of a name-get or subscription notification any chain value, including one with zero items, implies the request for the name-associated block to be sent.

7 Garbage Collection

Blocks may be deleted by the server when they are no longer referenced, ultimately, by a name association. The process of identifying and deleting these unreferenced blocks is called garbage collection. References are recursive; e.g. if block A is considered referenced by a name association and within it there is a multihash for block B (i.e. a reference to block B), block B is also considered referenced by a name association. What is considered a reference within a block is detailed below. The server may garbage collect when and as often as it chooses, however if that server is being run by a commercial entity that charges based on amount of storage used, garbage collection should be performed before those charges are evaluated.

7.1 Binary Block References

The header of a binary block contains zero or more multihashes, as detailed elswhere in this specification; from the perspective of the garbage collector these are references to other blocks. No other part of the block is examined for references.

7.2 Text Block References

The payload of text blocks are examined for likely multihashes; any that it finds is considered a reference. A likely multihash is in the format: Where non-alphanum is any UTF-8 character which is not in the following set: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 Alternately, the beginning or ending points of the block payload qualifies as a substitution for non-alphanum; e.g. a payload consisting entirely of a single multihash is interpreted as having one likely multihash. B58s is defined as more than 31 and less than 191 contiguous characters which are part of the set of valid base58 characters as defined by the multihash format.

7.3 Recently Put Blocks

Any block stored (or would have been stored had it not existed already) by any operation, including those done by the server (e.g. a name-append block split), is considered to be referenced by a name association for the 256 seconds following that storage event.

8 Permissions

Edsu permissions are layered, though not all layers are involved for each action. Before a requested action is performed, each relevant layer is evaluated, and the permissions are logically ANDed together; i.e. permission must be given at all levels before the action is permitted.

8.1 Name Prefix Layer

The first four characters of any name must be one of: pub. grp. prv. Actions involving a name which starts with pub. will always receive permission from this layer. Actions involving a name which starts with grp. will recieve permission from this layer if the connection has a user-from associated with it; i.e. some form of authentication has succeeded, and therefore it is not an anonymous connection. Actions involving a name which starts with prv. will recieve permission from this layer only if the connection's user-to and user-from are equal.

8.2 Block Layer

Special blocks can have keys which enact this layer when actions involving them are requested.

8.2.1 Name Block

The keys "edsu:deny" and "edsu:allow", if present in a name block, are enacted for any action which would reveal that the hash of that name block is associated with any given name (e.g. name-get, sub-notify, and the name listing service). If the key "edsu:deny" exists, and the connection's user-from is part of the group it defines, permission is denied at this layer. If the key "edsu:allow" exists, and the connection's user-from is not part of the group it defines, permission is denied at this layer. The above two permissions are logically ANDed together.

8.2.2 Appendable Block

The keys "edsu:append-deny", "edsu:append-allow", and "edsu:append-allow- anonymous", when present, are enacted when the name-append operation is attempted. They are orthogonal to the name block permission keys. If the key "edsu:append-deny" exists, and the connection's user-from is part of the group it defines, permission is denied at this layer. If the key "edsu:append-allow" exists, and the connection's user-from is not part of the group it defines, permission is denied at this layer. If the key "edsu:append-allow-anonymous" exists, its value is "true", and the connection is anonymous (i.e. no authentication was attempted or has succeeded), permission is granted. The permission at this layer is in the form of ((deny AND allow) OR allow- anonymous), where deny, allow, and allow-anonymous are the permission outcomes of the previous three paragraphs respectively.

8.3 Capabilities Layer

When a connection is authenticated, a capabilities block, chosen by the server as part of its authentication scheme, is interpreted and its capabilities are assigned to the connection for its duration. The capabilities block is a special block, with the following definition: Where "edsu:version" must be "0.1", list-of-name-patterns is an ESON list of name patterns, and name-pattern-and-put-opts consists of a list of values consisting of a name pattern followed by zero, one, or both of the following (if both, the order is undefined): Where space is ASCII 0x20, and "destructive" and "visitable" are the ASCII representations of those words. These are called options. In the context of capabilities blocks, a name pattern is considered to match a name if the two become identical when the final segment (and only the final segment) of the name is replaced by a "*". Each complete sequence of characters representing a name pattern (as distinct from its semantic meaning) must appear no more than once in each list, otherwise the capabilities block must be rejected as invalid.

8.3.1 Edsu:name-get

Any action which would reveal the hash associated with a name (or the existance of that association) only receives permission from this layer if the name matches at least one of the name patterns found in this list.

8.3.2 Edsu:name-append

A name-append message only receives permission from this layer if the name in the message matches at least one of the name patterns found in this list.

8.3.3 Edsu:name-put

To receive permission from this layer, a name-put message must match one or more of the values in this list. To match a value, the name in the message must match thats value's name pattern, as well its options. If the name-put operation is destructive, the "destructive" option must not be present for the value to match. A name-put operation is considered destructive if it will potentially expose blocks to deletion by the garbage collector which otherwise might not be (either because the hash key is omitted from the name-put message, or the introduction or amendment of "edsu:once", "edsu:expires", or "edsu:appended-ttl" keys), and if the proposed new name block does not contain a valid "edsu:previous" key. If the name-put operation has an impact on the effective permissions of a visitor in relation to the name, the "visitable" option must be present. Impact on the effective permissions means that the new name block must have identical non-existance or values for "edsu:deny", "edsu:allow", "edsu:append- deny", "edsu:append-allow", "edsu:append-rate-limit", and "edsu:append-bytes- max" as the name block the action would replace (or in the case of a new name, must not contain any of these keys).

8.3.4 Edsu:visiting-name-get

This is identical in function to edsu:name-get, except that edsu:name-get applies only to connections authenticated as owner, and edsu:visiting-name-get applies only to connections authenticated as visitor.

8.3.5 Edsu:visiting-name-append

This is identical in function to edsu:name-append, except that edsu:name-append applies only to connections authenticated as owner, and edsu:visiting-name- append applies only to connections authenticated as visitor.

8.3.6 Edsu:omni

The "edsu:omni" key being "true" will cause all actions to be evaluated as permitted at this layer.

8.3.7 Derived Permissions

If the lists for the keys "edsu:name-append" or "edsu:name-put" are not empty, or if "edsu:omni" is true, the connection is permitted to put blocks.

8.4 Services layer

Services, denoted by a second segment of "srv" in names used to interact with them, may institute their own permissions layer, as detailed in their specifications.

9 Block Gets

If the hash is possessed by the client, regardless of the authentication method or capabilities of their connection, they can request the matching block (e.g. via the block-get message or as the result of a chain).

10 Services

Unless otherwise noted, services accessed using a name-put will generate a new ESON block containing the detailed keys, and create or update a name association. This will be referred to as the "return block" and "return name" in this section. The return block may contain the "edsu:once" key with a value of "true" and an "edsu:expires" key with a value of not less than 256 seconds after the block is created. The return name will be the request name with an additional segment, that segment being an echo of the "hash" key in the initiating name-put message.

10.1 prv.srv.edsu.time.unix

A name-get for this name will result in the hash which resolves to a name block as follows: Where edsu:time is the server time in number of seconds elapsed since the Unix epoch.

10.2 pub.srv.edsu.authentication.visitors.capabilities.*

This service is described in the connection initiation section of this specification.

10.3 pub.srv.edsu.listings.blocks

A name-put to this address will cause the server to interpret the block referenced in that message as a metablock. The server generates a new metablock containing the hashes of all blocks referenced in the client's metablock which the server does not have stored - this is the return block. This service is the semantic equivalent of the client sending a block-get for each hash and compiling a list of hashes for which the server responds with a "not-found" oob.

10.4 pub.srv.edsu.listings.names

A name-put to this address will cause the server to interpret the block referenced in the message as follows: The server will create a list of all name associations contained in the user- to's account and sort them in ascending order according to the numeric ASCII values of the characters in the name. Further, for a name to appear in this listing, it must be named (either directly or using a wildcard) in the "edsu:name-get" value list of the capabilities block for the connection. I.e. the permission rules for listing and name- getting are similar, but not identical; they differ in that the rule stating that anything in the pub.* namespace is always readable does not apply in the case of listing. Otherwise they are the same. If the key "edsu:after" is present, all names from the beginning of the list up to and including the name specified will be discarded. If the key "edsu:prefix" is present, all names which do not have an identical initial segments as the specified name will be discarded. A name matching the specified name exactly is interpreted as having the same initial segments. A return block with the key "edsu:names" pointing to the remaining list of names will be created by the server. Alternatively the server may (because it would exceed the maximum size of a valid Edsu block or for any other reason) include only an initial portion of that list, in which case it must also include a key "edsu:truncated" with the value "true".

10.5 pub.srv.edsu.host.contact

This retrieves the contact information of whoever is hosting the Edsu account. There are currently two optional keys, each pointing to a single value: edsu:email, and edsu:www. These point to the email address and web site of the host, respectively, and are formatted according to their respective requirements.

10.6 prv.srv.edsu.authentication.tokens.generate.*

This service generates either an owner or a visitor token. To use it, a name is first generated by appending either "owner" or "visitor" to "prv.srv.edsu.authentication.tokens.generate.". This name is then used in a name-put operation, using the hash of an ESON block with the following structure: Where "edsu:capabilities" is the hash of a previously-stored capabilities block; "edsu:label" is an ESON-encoded UTF-8, human-readable string which can help the client identify the purpose of the token later (e.g. for removal); "edsu:ttl" (Time To Live), when present, sets the token to expire a given number of seconds after it is created (or never, if omitted); "edsu:once", if set to "true", will cause the token to be single-use; and "edsu:request-id" is machine-readble ID which can be used to group related tokens together - this can be any value, but must be unique to the request (e.g. a large randomized value). This will generate either a token, or produce an OOB error. In the former case, the response can be read by appending the multihash of the above ESON block as a segment to the name used in the request, and then using that in a name-get. This will reference a block in the form: Where "edsu:ok" will be "true", and where the token value can be used in the hello operation as described in that section.

10.7 prv.srv.edsu.authentication.tokens.delete.*

This service deletes either an owner or a visitor token. To use it, a name is first generated by appending either "owner" or "visitor" to "prv.srv.edsu.authentication.tokens.delete.", and then further appending the token itself as another segment. In the case of a visitor token, note that the token to be used is the original, i.e. what was returned from the token generate service. This name is then used in a name-get operation, which will either produce an OOB error, or delete the token and return the hash of the following block: Where the value of "edsu:ok" is "true". This result happens whether the token in question exists or not - semantically its meaning is that the token no longer exists.

10.8 *.srv.ext.*

Servers may introduce services not listed in this specification, but all names must fit the pattern of one of: *.srv.ext.app.* *.srv.ext.std.* Where in each pattern the initial asterisk is replaced by one of pub, grp, and prv, and the trailing asterisk is replaced by one or more segments of the server's choosing. By convention, the first, "app", pattern's use is unrestricted, while the second, "std", pattern should be used according to some published specification.