Overview

Message Exchange engine handles incoming messages in different format, their temporary storage before delivery, and routing based on configuration and rules.

Overview Diagram

Diagram is work in progress. Later should include communication with OCM and other mxengine deployments.

Inbound Message Lifecycle

The following diagram describes the processing steps which takes place when a message is accepted for delivery.

After a message enter the inbound handlers, a small envelope record describing the message metadata is constructed and stored in relational database. The raw message payload is stored in S3-compatible storage (bucket).

Envelope Data Model

Below is described the relational model of message envelope and its delivery status. Some fields like timestamps are left out to avoid clutter and focus on the most important attributes, but all records have created_at and updated_at values. You can see the complete description of the data model in the db migrations files /db/migrations.

Service Configuration

Configuration is passed to the service as ENV variables. Please refer to the /internal/config/config.go file inside the project directory, for a complete list of all supported configuration variables and their description.

Inbound SMTP with TLS

The inbound SMTP server supports TLS. It is disabled by default, but can be enabled with the following configuration variables:

INBOUND_SMTP_TLS="true"
INBOUND_SMTP_TLS_PUBLIC_KEY="/etc/certs/mxengine/localhost.pem"
INBOUND_SMTP_TLS_PRIVATE_KEY="/etc/certs/mxengine/localhost-key.pem"

• INBOUND_SMTP_TLS_PUBLIC_KEY points to a (mounted) file containing the server's public certificate encoded in PEM format.
• INBOUND_SMTP_TLS_PRIVATE_KEY points to a (mounted) file containing the server's private key encoded in PEM format.

To test locally with enabled TLS, you can install the mkcert tool. It can also create a certificate on your local machine. This would allow you to send an email successfully from the sendmail.go local program and it will recognize the server certificate as valid, as installing mkcert adds it as Certificate Authority on your local machine.

# first install mkcert, e.g. by using brew or downloading a binary
brew install mkcert

# next install it as CA
mkcert -install

# if you want/need, create a new certificate by specifying ServerName
cd $GOPATH/src/code.vereign.com/svdh/workspace/certs/mxengine
mkcert localhost

Delivery Configuration Rules

The REST API allows for managing delivery configurations (rules). They are prioritized and executed for each message in a user-defined order. The rules are used to determine what outbound path a message should take.

The service must be initialized with at least one delivery configuration rule before it can deliver any messages. Please, see the /swagger/ endpoint for Delivery Configuration API details or check the Configuration Examples for how to create/update delivery configurations.

Storage

The service uses PostgreSQL for relational storage and S3-compatible buckets for raw message payload.

Data Archiving and long-term Storage

The service stores raw message payload in S3/Blob storage and message metadata and message status records in relational DB tables.

1. Raw payload is deleted from S3 storage after the message is delivered, or after the message cannot be delivered for a configured number of retries.

2. Message metadata is deleted from DB at the same time as (1)

3. Message status records are stored for much longer periods of time for administrative, support and informational purposes. After some period of time has passed, message status records are archived (moved) to a separate DB table for long-term storage (10 years by default).

Read here for more details and archive configuration.

DB migrations

SQL migration files are created in /db/migrations directory by using the Golang migrate tool. Example command to make a new migration file:

migrate create -ext sql -dir ./db/migrations messages_table

Metrics

The service exposes the following Prometheus metrics.

Metric Name	Type
http_requests_total	Counter with labels [operation, code]
http_request_duration_seconds	Histogram with labels [operation, code]
smtp_incoming_total	Counter
smtp_incoming_stored	Counter
smtp_incoming_duration_seconds	Histogram
delivered_messages_total	Counter with labels [strategy]
delivered_messages_total_size_bytes	Counter with labels [strategy]
failed_messages_total	Counter
bounce_replies_total	Counter

Dependencies and Vendor

The project uses Go modules for managing dependencies, and we commit the vendor directory. When you add/change dependencies, be sure to clean and update vendor before submitting your Merge Request for review.

go mod tidy
go mod vendor

Tests and Linters

To execute the units tests for the service go to the root project directory and run:

go test -race ./...

To run the linters go to the root project directory and run:

golangci-lint run

Configuration Examples

SMTP to Seal:

curl --location 'http://localhost:8100/v1/config' \
--header 'Content-Type: application/json' \
--data '{
    "name": "smtpToSeal",
    "priority": 99,
    "configuration": {
        "params": {
            "channel": "smtp"
        },
        "outbound": [
            "seal"
        ]
    }
}'

JSON to SMTP:

curl --location 'http://localhost:8100/v1/config' \
--header 'Content-Type: application/json' \
--data '{
  "name": "jsonToSMTP",
  "priority": 98,
  "configuration": {
    "params": {
        "channel": "json"
    },
    "outbound": [
        "smtp"
    ]
  }
}'

Reply - json, sender to SMTP:

curl --location 'http://localhost:8100/v1/config' \
--header 'Content-Type: application/json' \
--data-raw '{
    "name": "sealReply",
    "priority": 9,
    "configuration": {
        "params": {
            "channel": "json",
            "sender": "no-reply@hin.ch"
        },
        "outbound": [
            "smtp"
        ]
    }
}'