Endpoint Registry Service - Version 1.0-rc2 πŸ”—

Abstract πŸ”—

This specification defines an endpoint registry extension to the xRegistry document format and API specification. An endpoint registry allows for publishing and discovery of asynchronous event sources, sinks, and subscription points in the scope of a system, along with important configuration parameters.

Table of Contents πŸ”—

Overview πŸ”—

This specification defines a registry of metadata definitions for abstract and concrete network endpoints to which messages can be produced, from which messages can be consumed, or which make messages available for subscriptions.

The metadata model defined in this specification is specifically focused on describing unidirectional endpoints for asynchronous information flows like discrete events, event streams, queueing, and publish/subscribe patterns.

Endpoint information is provided through the registry for discovery of endpoints, dynamic configuration of clients, and code generation.

The model allows for a loose correlation of endpoints through a channel concept, to designate, for instance, the input and output ends of a queue, but it intentionally avoids being specific about the shape of messaging and eventing entities. The correlation of endpoints for the purposes of realizing specific message exchange patterns like request/response or scatter/gather is also intentionally out of scope for this metadata model, because modeling such correlation contracts with sufficient depth is a whole additional definition layer above what this model aims to achieve.

The goal of this endpoint metadata model is to provide metadata structure to asynchronous topics and streams and queues in a way that is similar to how table and column definitions provide structure to databases. Database schemas structure data that you have. Endpoint definitions with their referenced or embedded message definitions with referenced or embedded schema definitions structure data that you will yet receive.

Continuing that analogy, the endpoint scope corresponds to the database, the message groups are schema scopes, and the message definitions correspond to tables. The schema associated with a message definition determines the column layout of the table. As event streams often end up landing in databases for long-term archival and analysis, this structural alignment is very helpful.

Endpoints πŸ”—

In a distributed system, the networked input sinks and output sources of an application, or an application infrastructure component like an event broker, are commonly called β€œendpoints”. Those endpoints serve as communication conduits to make information available to the application, or for the application to make information available to others.

In this specification we distinguish three distinct usage roles for endpoints:

Each endpoint definition defines a protocol selector by which the specific network application protocol is chosen to communicate with the endpoint. If a networked entity supports multiple protocols, each protocol endpoint MUST be declared separately, even if those are multiplexed over the same port. For instance, if you have an MQTT broker that can dynamically select between MQTT 3.1.1 and MQTT 5.0, those endpoints are to be declared separately as the capabilities differ substantially.

The protocol selector value determines which protocoloptions become available to define for the endpoint. The protocol-specific options are enumerated and explained in protocol options.

Each endpoint MAY also define an envelope selector which allows for defining particular envelopeoptions at the endpoint level. For CloudEvents, this permits definition the serialization mode (binary or structured) and the event format for structured mode.

Message Groups πŸ”—

The Endpoint is an xRegistry group-level construct that is conceptually an extension of the message definition group, meaning there are no β€œgroups of endpoints”. An endpoint MAY contain directly embedded message definitions. In the simplest case, an endpoint MAY embed its message definitions locally which in turn MAY embed their data schema definitions locally, all in one compact construct.

More commonly, Endpoints will reference one or more message definition groups that are defined externally in a message registry, and which have the advantage of being shareable across multiple Endpoints.

For describing a message broker queue, the best approach is to define a message group that defines the messages that are be allowed to flow on the queue and for that message group to then be referenced from the producer and the consumer endpoints of the queue, whereby both of those endpoints share the same channel identifier.

In the endpoint context, message groups have a function similar to interfaces in programming languages. They define related sets of messages that can be associated with an endpoint all at once.

Scenarios πŸ”—

Endpoint definitions can be abstract or concrete, distinguished by the deployed flag. If the deployed flag is set to true, one can expect the endpoint to be reachable on the network with the given parameters, assuming the client is within the same network scope. If the flag is set to false, the endpoint definition is to be treated like a template where configuration elements like the endpoint URI will have to be supplied by external configuration for the client to become functional.

A possible application of the latter are Endpoint definitions that define endpoint patterns as they are common for applications of the MQTT protocol. For instance, the Eclipse Sparkplug protocol is a convention that defines endpoint roles and messages and schemas for MQTT, and the SparkPlugB example that illustrates how Registry Endpoints can model the convention in formal terms.

Notations and Terminology πŸ”—

Notational Conventions πŸ”—

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

For clarity, OPTIONAL attributes (specification-defined and extensions) are OPTIONAL for clients to use, but the servers' responsibility will vary. Server-unknown extension attributes MUST be silently stored in the backing datastore. Specification-defined, and server-known extension, attributes MUST generate an error if corresponding feature is not supported or enabled. However, as with all attributes, if accepting the attribute would result in a bad state (such as exceeding a size limit, or results in a security issue), then the server MAY choose to reject the request.

In the pseudo JSON format snippets ? means the preceding attribute is OPTIONAL, * means the preceding attribute MAY appear zero or more times, and + means the preceding attribute MUST appear at least once. The presence of the # character means the remaining portion of the line is a comment. Whitespace characters in the JSON snippets are used for readability and are not normative.

Terminology πŸ”—

This specification defines the following terms:

Endpoint πŸ”—

An "endpoint" is a logical or physical network location to which messages can be produced, from which messages can be consumed, or which makes messages available via subscription for delivery to a consumer-designated endpoint.

Endpoint Registry Model πŸ”—

The Endpoint Registry is a registry of metadata definitions for abstract and concrete network endpoints to which messages can be produced, from which messages can be consumed, or which makes messages available via subscription and delivery to a consumer-designated endpoint.

As discussed in the CloudEvents Registry overview, endpoints are supersets of message definition groups and MAY contain inlined messages. Therefore, the Resources in the meta-model for the Endpoint Registry are likewise messages as defined in the message catalog specification.

The formal xRegistry extension model of the Endpoints Registry resides in the model.json file.

For easy reference, the JSON serialization of an Endpoint Registry adheres to this form:

{
  "specversion": "<STRING>",
  "registryid": "<STRING>",
  "self": "<URL>",
  "xid": "<XID>",
  "epoch": <UINTEGER>,
  "name": "<STRING>", ?
  "description": "<STRING>", ?
  "documentation": "<URL>", ?
  "labels": {
    "<STRING>": "<STRING>" *
  }, ?
  "createdat": "<TIMESTAMP>",
  "modifiedat": "<TIMESTAMP>",

  "model": { ... }, ?

  "endpointsurl": "<URL>",
  "endpointscount": <UINTEGER>,
  "endpoints": {
    "<KEY>": {
      "endpointid": "<STRING>",                   # xRegistry core attributes
      "self": "<URL>",
      "xid": "<XID>",
      "epoch": <UINTEGER>,
      "name": "<STRING>", ?
      "description": "<STRING>", ?
      "documentation": "<URL>", ?
      "labels": { "<STRING>": "<STRING>" * }, ?
      "createdat": "<TIMESTAMP>",
      "modifiedat": "<TIMESTAMP>",
      "deprecated": { ... }, ?

      "usage": [ "<STRING>" + ] ,                 # subscriber, consumer, producer
      "channel": "<STRING>", ?

      # Start of Endpoint extension attributes
      "envelope": "<STRING>", ?                   # e.g. CloudEvents/1.0
      "envelopeoptions": {
        "<STRING>": <JSON-VALUE> *

        # CloudEvents/1.0 options
        "mode": "<STRING>", ?                     # binary, structured
        "format": "<STRING>" ?                    # e.g. application/json
      },

      "protocol": "<STRING>", ?                   # e.g. HTTP/1.1
      "protocoloptions": {
        "<STRING>": <JSON-VALUE> *

        # Common protocol options
        "endpoints": [
          {
            "url": "<URL>"                        # plus endpoint extensions
          } *
        ], ?
        "authorization": {
          "type": "<STRING>", ?
          "resourceuri": "URI", ?
          "authorityuri": "URI", ?
          "grant_types": [ "<STRING>" * ] ?
        }, ?
        "deployed": <BOOLEAN>, ?

        # "HTTP" protocol options
        "method": "<STRING>", ?                          # Default: POST
        "headers": [ { "name": "<STRING>", "value": "<STRING>" } * ], ?
        "query": { "<STRING>": "<STRING>" * } ?

        # "AMQP/1.0" protocol options
        "node": "<STRING>", ?
        "durable": <BOOLEAN>, ?                          # Default: false
        "linkproperties": { "<STRING>": "<STRING>" * }, ?
        "connectionproperties": { "<STRING>": "<STRING>" * }, ?
        "distributionmode": "move" | "copy" ?          # Default: move

        # "MQTT/3.1.1" protocol options
        "topic": "<STRING>", ?
        "qos": <UINTEGER>, ?                             # Default: 0
        "retain": <BOOLEAN>, ?                           # Default: false
        "cleansession": <BOOLEAN>, ?                     # Default: true
        "willtopic": "<STRING>", ?
        "willmessage": "<XID>" ?

        # "MQTT/5.0" protocol options
        "topic": "<STRING>", ?
        "qos": <UINTEGER>, ?                             # Default: 0
        "retain": <BOOLEAN>, ?                           # Default: false
        "cleansession": <BOOLEAN>, ?                     # Default: true
        "willtopic": "<STRING>", ?
        "willmessage": "<XID>" ?

        # "KAFKA" protocol options
        "topic": "<STRING>", ?
        "acks": <INTEGER>, ?                             # Default: 1
        "key": "<STRING>", ?
        "partition": <INTEGER>, ?
        "consumergroup": "<STRING>", ?
        "headers": { "<STRING>": "<STRING>" * } ?

        # "NATS" protocol options
        "subject": "<STRING>" ?
      }, ?

      "messagegroups": [ XID * ], ?
      # End of Endpoint extensions

      "messagesurl": "<URL>", ?
      "messagescount": <UINTEGER>, ?
      "messages": {
        "<KEY>": {                                # messageid
          # See Message Definition spec for details
        } *
      } ?
    } *
  } ?
}

Endpoints Groups πŸ”—

The Group plural name (<GROUPS>) is endpoints, and the Group singular name (<GROUP>) is endpoint.

The following attributes are defined for the endpoint object in addition to the xRegistry-defined core attributes:

usage πŸ”—

channel πŸ”—

envelope πŸ”—

envelopeoptions πŸ”—

This specification defines the following envelope options for the indicated envelope values:

CloudEvents/1.0 πŸ”—

protocol πŸ”—

protocoloptions πŸ”—

protocoloptions.endpoints πŸ”—
protocoloptions.authorization πŸ”—
protocoloptions.authorization.type πŸ”—
protocoloptions.authorization.resourceuri πŸ”—
protocoloptions.authorization.authorityuri πŸ”—
protocoloptions.authorization.grant_types πŸ”—
protocoloptions.deployed πŸ”—
protocoloptions.options πŸ”—

messagegroups πŸ”—

The messagegroups attribute is an array of XID-references to message definition groups. The messagegroups attribute is used to reference message definition groups that are not inlined in the endpoint definition.

Example:

{
  "protocol": "HTTP/1.1",
  "protocoloptions": {
    "method": "POST"
  },
  "messagegroups": [
    "/messagegroups/mygroup"
  ]
}

messages πŸ”—

Endpoints are supersets of message definition groups and MAY contain inlined messages. See Message Definitions.

Example:

{
  "protocol": "HTTP/1.1",
  "protocoloptions": {
    "method": "POST"
  },

  "messagesurl": "...",
  "messagescount": 1,
  "messages": {
    "myevent": {
      "envelope": "CloudEvents/1.0",
      "envelopemetadata": {
        "attributes": {
          "type": {
            "value": "myevent"
          }
        }
      }
    }
  }
}

When this specification, and the message specification, are used with specifications such as CloudEvents, where a semantically unique identifier is used in a runtime message (e.g. CloudEvent's type attribute), it is STRONGLY RECOMMENDED that the messageid values of the message definitions for an Endpoint match that unique identifier and therefore be unique across all messages within the messages collection and the messages referenced by the messagegroups attribute. This will allow for an easy "lookup" from an incoming runtime message to its related message definition.

However, there are times when this is not possible. For example, take the case where an Endpoint might have the same semantic message defined twice, once for a JSON serialization and once for an XML serialization. Using the same messageid value is not possible (even though the CloudEvent type attribute would be the same for both runtime messages), so one (or both) message definition's messageid values might not match the runtime message's type value. In those cases, finding the appropriate message definition will need to be done via examination of some other metadata - such as the message's envelopemetadata.type value along with its envelopeoptions.format value. These details are out of scope for this specification to define and are left as an implementation detail.

Implementations MAY choose to generate an error if it detects duplicate messageid values across the messages collection message definitions and the messagegroups referenced message definitions, if that is the desired constraint for their users.

Protocol Options πŸ”—

The following protocol options (protocoloptions.options) are defined for the respective protocols. All of these are OPTIONAL.

HTTP options πŸ”—

The endpoint URIs for "HTTP" endpoints MUST be valid HTTP URIs using the "http" or "https" scheme.

The following options are defined for HTTP:

The values of all query and headers MAY contain placeholders using the RFC6570 Level 1 URI Template syntax. When the same placeholder is used in multiple properties, the value of the placeholder is assumed to be identical.

Example:

{
  "protocol": "HTTP/1.1",
  "protocoloptions": {
    "method": "POST",
    "headers": [
      {
        "name": "Content-Type",
        "value": "application/json"
      }
    ],
    "query": {
      "operation": "send"
    }
  }
}
AMQP options πŸ”—

The endpoint URIs for "AMQP" endpoints MUST be valid AMQP URIs using the "amqp" or "amqps" scheme. If the path portion of the URI is present, it MUST be a valid AMQP node name.

The following options are defined for AMQP endpoints.

The values of all linkproperties and connection-properties MAY contain placeholders using the RFC6570 Level 1 URI Template syntax. When the same placeholder is used in multiple properties, the value of the placeholder is assumed to be identical.

Example:

{
  "usage": [ "producer" ],
  "protocol": "AMQP/1.0",
  "protocoloptions": {
    "node": "myqueue",
    "durable": true,
    "linkproperties": {
      "mylinkproperty": "mylinkpropertyvalue"
    },
    "connection-properties": {
      "my-connection-property": "my-connection-property-value"
    },
    "distributionmode": "move"
  }
}
MQTT options πŸ”—

The endpoint URIs for "MQTT" endpoints MUST be valid MQTT URIs using the (informal) "mqtt" or "mqtts" scheme. If the path portion of the URI is present, it MUST be a valid MQTT topic name. The informal schemes "tcp" (plain TCP/1883), "ssl" (TLS TCP/8883), and "wss" (Websockets/443) MAY also be used, but MUST NOT have a path.

The following options are defined for MQTT endpoints.

Example:

{
  "usage": [ "producer" ],
  "protocol": "MQTT/5.0",
  "protocoloptions": {
    "topic": "mytopic",
    "qos": 1,
    "retain": false,
    "cleansession": false,
    "willtopic": "mytopic",
    "willmessage": "/messagegroups/mygroup/messages/mywillmessage"
  }
}
KAFKA options πŸ”—

The endpoint URIs for "Kafka" endpoints MUST be valid Kafka bootstrap server addresses. The scheme follows Kafka configuration usage, e.g. SSL://<HOST>:<PORT> or PLAINTEXT://<HOST>:<PORT>.

The following options are defined for Kafka endpoints.

Example:

{
  "usage": [ "producer" ],
  "protocol": "Kafka/2.0",
  "protocoloptions": {
    "topic": "mytopic",
    "acks": 1,
    "key": "mykey",
  }
}
NATS options πŸ”—

The endpoint URIs for "NATS" endpoints MUST be valid NATS URIs. The scheme MUST be "nats" or "tls" or "ws" and the URI MUST include a port number, e.g. nats://<HOST>:<PORT> or tls://<HOST>:<PORT>.

The following options are defined for NATS endpoints.

Example:

{
  "usage": [ "producer" ],
  "protocol": "NATS/1.0.0",
  "protocoloptions": {
    "subject": "mysubject"
  }
}