Building a Scalable Newsletter System with Background Jobs

The Challenge with Bulk Emails

Sending newsletters to thousands of subscribers in a single HTTP request isn't just slow—it's impossible. Let's build a system that scales.

Architecture Overview

// 1. Queue newsletter send
app.post('/send-newsletter', async (req, res) => {
  const { subject, content, subscriberList } = req.body;

  const apiKey = process.env.DISPATCHED_API_KEY;

  const response = await fetch('https://dispatched.dev/api/jobs/dispatch', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${apiKey}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      payload: {
        task: 'send_newsletter',
        subject,
        content,
        subscriberList
      }
    })
  });

  const { jobId } = await response.json();
  res.json({ jobId });
})

// 2. Process newsletter in batches
app.post('/webhooks/dispatched', express.json(), (req, res) => {
  const webhookSecret = process.env.DISPATCHED_WEBHOOK_SECRET;

  if (req.headers.authorization !== `Bearer ${webhookSecret}`) {
    return res.status(401).json({ error: 'Unauthorized' });
  }

  const { payload } = req.body;
  const BATCH_SIZE = 100;

  try {
    // Split subscribers into batches
    const batches = chunk(payload.subscriberList, BATCH_SIZE);

    for (const batch of batches) {
      await sendEmailBatch({
        to: batch,
        subject: payload.subject,
        content: payload.content
      });

      // Rate limiting
      await sleep(1000);
    }

    res.status(200).json({ status: 'success' });
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

Key Features

• Batch processing prevents timeouts
• Rate limiting respects email provider limits
• Automatic retries on failure
• Progress tracking per batch

Handling Edge Cases

function chunk(array, size) {
  return Array.from({ length: Math.ceil(array.length / size) }, (_, i) =>
    array.slice(i * size, i * size + size)
  );
}

async function sendEmailBatch({ to, subject, content }) {
  const failedEmails = [];

  for (const email of to) {
    try {
      await emailProvider.send({
        to: email,
        subject,
        content
      });
    } catch (error) {
      failedEmails.push({ email, error: error.message });
    }
  }

  return failedEmails;
}

Scaling Up

This setup can handle:

• 100k+ subscribers
• Multiple newsletters per day
• Various email providers
• Failed delivery tracking

Next Steps

1. Create your API key
2. Set up your webhook endpoint
3. Start sending newsletters at scale

Ready to handle massive email campaigns? Get started with Dispatched.