NOC Nucleus

# Nucleus / Backbone

The concept of the backbone is to enable platform agnostic communication between different and in some cases proprietary platforms.

The backbone routes traffic between a number of known clients using publish/subscribe or broadcast mechanisms.

Messages are exchanged using an agreed schema so that all partners can understand the messages sent over the backbone.

Each client implements an adapter layer which handles the transformation between this common format and the local representation of those messages.

Adapters can be written in any language. We have published adapters in python and javascript. The adapter code provides stubs which can be sub-classed and overridden to handle the required processing within any given client application.

# How it fits together

The communication is handled by queues with publish/subscribe and broadcast flows linking the queues together.

On top of RabbitMQ is a python layer called the soar_bus which handles the creation of queues and exchanges and the consumers and publishers.

Then there is a python flask API which handles authentication and the sending and receiving of messages to and from the bus.

Client applications register with the API and receive a config file containing their credentials.

The applications connect to the API, do a client credentials grant to get a token and then requests to send and receive messages are made with the bearer token.

# Messages and schemas

The backbone doesn't care what you send through it. You send a message and tell it to publish it to a topic or broadcast it to all clients. Messages are delivered to each matching subscriber's inbox and the backbone will return all messages from the inbox to the client.

The adapters do care about messages. When you create an instance of an adapter you give it an instance of a protocol and when you create the instance of the protocol you give it a schema.

The schema is defined in JSONSchema and the protocol tells the adapter how to validate messages using that schema.

We have defined a top level message which can include different message types in a payload. This means that everything received from the backbone validates against the same top-level message definition.

# The SoAR protocol

The SoAR protocol (opens new window) defines the messages we will be using in the SoAR project trials.

When building an adapter for SoAR you should be planning to send and receive messages conforming to this schema.

# Topics

When sending messages you should publish them using a topic matching this definition (opens new window).

# The test schema

For the adapter test suite a simpler test schema (opens new window) is defined.

Example messages matching this schema are also provided in the test suite fixtures (opens new window).

The tests and the example client implementations use this test schema.

If you want to configure the example clients to use the real schema you will need to create a new protocol sub-class passing in the real schema and change the validation to point to the top-level MESSAGE schema.

# Other protocols

You could write a protocol implementation to send and receive messages in any format provided you write a protocol handler that understands how to parse and validate the messages.

All partners sharing a backbone need to share a common protocol so in implementing a different protocol you may need to think about whether parsers and validators are available in the languages used by all clients.

# How to run your own instance of the comms backbone

To set up an end-to-end working system you need:

  • A RabbitMQ container
  • A running soar_bus
  • A running API
  • A client configured with a .json config file

The config file can be named and stored anywhere provided the config settings are passed to the adapter when it is instantiated.

This could be a secret manager or gitcrypt or an env var or CI pipeline var but should not be stored in a public repository.

We have created example implementations of each adapter to test sending and receiving messages through the backbone.

These are just examples. Each client is different so how the adapter should be implemented is a decision that depends on the architecture of the client application the adapter will be running inside.

For this reason the adapter functionality has been left quite open.

# Running the backbone

The instructions for running the backbone include running RabbitMQ, running the bus and the API and creating a client.

The simplest way to run an instance of the backbone is using docker-compose

Running via docker-compose (opens new window)

If you want more control over the individual components you can choose to run each component manually.

Running outside docker (opens new window)

Note: Both cases assume running RabbitMQ in a docker container.

# Creating client config

Instructions for creating clients (opens new window).

At present when you create a client by POSTing to the API /client endpoint it gives back a file containing everything you need for your client config except for the root endpoint for the API which needs to be added.

The credentials returned from the API can be stored anywhere but the simplest option is to store them locally as a .json file.

When implementing your own client you can store this however you like.

It should be parsed and passed to the Adapter constructor at runtime.

An example config file (opens new window).

# Running an example client

There is an example client in the backbone-adapter-python/example (opens new window) repository.

To run the example client you need to create a client and store the config in the example folder as soar-config.json. You will need to add the root url for the api into the config settings.

For testing the easiest thing to do is set the client subscription to soar.# meaning that anything you publish you should receive back.

# Sending and receiving test messages

There are a number of ways to send test messages through the backbone. You can do this using the example python adapter (opens new window) by putting messages into the transmit directory.

Alternatively you can use the client_send.py and client_read.py sample scripts (opens new window) (It should be noted that these bypass authentication and write directly to the rabbitmq queues)

Finally you can use postman to send and receive via the API. For this you will need to use your client_id and secret to get a token from GET /token and then pass your token as an Authorisation: Bearer [token] header to make subsequent requests.