Data Retention
mxengine is processing messages by storing the complete message (raw payload) in S3 compatible storage and some metadata attributes in relational DB tables.
There are three DB tables storing metadata: messages, message_status and message_status_archive.
To see the definition of the tables columns you can see the /db/migrations
directory inside the project.
messages table description
| Column name | Description |
|---|---|
| message_id | Taken from Message-ID header if present, otherwise generated by mxengine |
| message_type | Message type (i.e. email) |
| sender | Message sender (i.e. email address) |
| recipients | Message recipients as comma separated string |
| subject | Message subject |
| channel | Inbound message channel (i.e. smtp or json) |
| status | Message status used during internal processing delivery stages |
| return_path | Taken from Return-Path header if present, otherwise sender address is used |
| datafile | S3 filename pointing to the raw message payload |
| strategy | Optional delivery strategy explicitly requested when message is submitted |
| created_at | Unix timestamp when record is created |
| updated_at | Unix timestamp when record is last updated |
message_status table description
| Column name | Description |
|---|---|
| message_id | Unique message identifier |
| subject | Message subject |
| sender | Message sender (i.e. email address) |
| recipient | Message recipient (i.e. email address) |
| delivery_strategy | Strategy that has been used to deliver the message |
| status | Delivery status (i.e. delivered, failed) |
| created_at | Unix timestamp when record is created |
| updated_at | Unix timestamp when record is last updated |
message_status_archive must have the same schema as message_status.
Data retention and removal
The service stores raw message payload in S3/Blob storage and message metadata and message status records in relational DB tables.
-
Raw payload is deleted from S3 storage after the message is delivered, or after the message cannot be delivered for a configured number of retries.
-
Message metadata is deleted from DB at the same time as (1)
-
Message status records are stored for much longer periods of time for administrative, support and informational purposes. After some period of time has passed, message status records are archived (moved) to a separate DB table for long-term storage (10 years by default).
Configuration
ARCHIVE_INTERVAL specifies how often the archival process wakes up to archive records
and clean up expired archive records. The value is 30 minutes by default.
// Interval specifies how often archiving will be performed.
Interval time.Duration `envconfig:"ARCHIVE_INTERVAL" default:"30m"`
ARCHIVE_RETENTION_PERIOD specifies how long archived records are kept before being deleted.
The value is 10 years by default.
// RetentionPeriod specifies how long archived records are kept before being deleted.
// Default value is specified in hours: 87672h = 3650 days + 3 days leap = 10 years
RetentionPeriod time.Duration `envconfig:"ARCHIVE_RETENTION_PERIOD" default:"87672h"`
ARCHIVE_BATCH_SIZE specifies how many records to archive on a given run. This value can
be fine-tuned if it turns out to be too small, but a balance must be found for DB load and
efficiency. The value is 1000 by default.
// BatchSize specifies how many records to archive on a given run.
BatchSize int `envconfig:"ARCHIVE_BATCH_SIZE" default:"1000"`
WAIT_BEFORE_ARCHIVE specifies how much time to wait before a record is considered for archiving.
The value is 90 days by default.
// WaitBeforeArchive specifies how much time to wait before a record is going to be archived.
WaitBeforeArchive time.Duration `envconfig:"WAIT_BEFORE_ARCHIVE" default:"2160h"` // 2160h = 90d