Persisted email queue

[bythewiseman]

We run Workflow in an Azure environment. We use Azure Web App slots to auto deploy to, to allow us to check things over and then we swap to production.

This works beautifully for our sites as the down time of production is minimal to nonexistent sometimes. But it seems to cause an issue with Workflow sending emails out. And I get it, you don't want to duplicate the emails.

But because of our swap slots pointing to the same database as the live site in order to be able to test against current content, Umbraco registers a new server and makes it the active one, which we have found for the most part not an issue as at some point the slot goes to sleep and Umbraco re-registers the server. It does mean our UmbracoServer table is quite big sometimes. But no issues and our log files for all of sites don't show an issue.

That is until Workflow. It seems that emails get missed when this happens, the registering and deregistering of servers. They get missed in that if they were in memory in the ConcurrentQueue code, the could be dequeued but not requeued.

Is that a possible outcome?

So a feature request could be to move your email queue from ConcurrentQueue to an actual table umbracoWorkflowQueue with a flag saying "sent" and that way any registered server could send an email? Or something to that affect. This might be a rubbishy solution, but we've had quite a few months of working out why emails weren't sending with a lot of blame going between us, the client's email server and MailJet (and MJ was to blame a couple of times 😄 ).

Or is there a way we can write our own handler for the queue and using DI have the IEmailQueue implement it instead?

I guess for the vast majority of users, this isn't a problem, but for that niche set of users in a non load balanced multi slot multi environment testing work, this does cause issues.

Hope you can help,

Colin

---

This item has been added to our backlog AB#52895

[nathanwoulfe]

Hey @bythewiseman, I think you're right - the email queue is held in memory, so would be lost on a slot swap. Will need to investigate the best way to manage this, but I'm thinking of shifting the email generation out of the workflow process, and into the emailer, and persisting the workflow process ids to the DB. Then, when the email sender task runs, it can generate the message body.

That would shift a bit of work out of the main workflow process, which is a nice bonus too.

Like I said, needs some thought but sounds like a reasonable request.

You could replace the default IEmailQueue implementation if you really wanted to, replacing the ConcurrentQueue implementation with a database integration

[bythewiseman]

Thanks Nathan. I did a quick implementation of the IEmailQueue to test how it would work persisting to the database.

public EmailQueueItem? DequeueEmail()
{
    EmailQueueItem? email = null;
    using (var scope = _scopeProvider.CreateScope())
    {
        var items = scope.Database.Fetch<umbracoWorkflowEmails>("DELETE TOP(1) umbracoWorkflowEmails WITH (READPAST) OUTPUT DELETED.* WHERE Id = (SELECT TOP 1 Id FROM umbracoWorkflowEmails ORDER BY Timestamp ASC)");
        if (items.Any())
        {
            email = Newtonsoft.Json.JsonConvert.DeserializeObject<EmailQueueItem>(items.FirstOrDefault()?.Json);
        }
        scope.Complete();
    }

    return email;
}

public bool Peek()
{
    int count = 0;
    using (var scope = _scopeProvider.CreateScope())
    {
        count = scope.Database.ExecuteScalar<int>("SELECT COUNT(*) FROM UmbracoWorkflowEmails");
        scope.Complete();
    }

    return count > 0;
}

public void QueueEmail(EmailQueueItem item)
{
    var json = Newtonsoft.Json.JsonConvert.SerializeObject(item);
    using (var scope = _scopeProvider.CreateScope())
    {
        scope.Database.Execute("INSERT INTO UmbracoWorkflowEmails (Json) VALUES (@0)", json);
        scope.Complete();
    }
}

And this worked really well. Convert your email object into json saved me having to map columns...lazy but good.

However, what didn't work well was the case of when the email failed to send for whatever reason. Workflow doesn't requeue the email in the event that something is broken - which in my test case was that I had changed MailJet's host to an unknown one to test if Workflow did requeue.

If Workflow requeued in the even of a failure, no emails would be lost.

If you do go the route of a database table, propagating the values of what would be in the email to a table means it wouldn't matter about replica servers. The code just runs to send an email and doesn't care where it is, imho of course 😄

[bythewiseman]

I could probably change this

var items = scope.Database.Fetch

to this

var items = scope.Database.Single

for a bit of optimisation...