Processing User-Generated Content at Scale

The UGC Challenge

Processing user-generated content is tricky. Images need resizing, videos need transcoding, and content needs moderation. Do this in a single request, and you'll hit timeouts. Let's fix that.

Image Processing Pipeline

app.post('/upload-image', async (req, res) => {
  const image = req.files.image;

  // 1. Upload original to storage
  const imageUrl = await uploadToStorage(image);

  const apiKey = process.env.DISPATCHED_API_KEY;

  // 2. Queue processing job
  const response = await fetch('https://dispatched.dev/api/jobs/dispatch', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${apiKey}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      payload: {
        task: 'process_image',
        imageUrl,
        sizes: [
          { width: 800, height: 600 },
          { width: 400, height: 300 },
          { width: 200, height: 150 }
        ]
      }
    })
  });

  const { jobId } = await response.json();
  res.json({ jobId, originalUrl: imageUrl });
})

// Process images in background
app.post('/webhooks/dispatched', express.json(), async (req, res) => {
  const { payload } = req.body;

  try {
    // Download original
    const image = await downloadFromStorage(payload.imageUrl);

    // Generate variants
    const variants = await Promise.all(
      payload.sizes.map(async size => {
        const resized = await sharp(image)
          .resize(size.width, size.height)
          .jpeg({ quality: 80 })
          .toBuffer();

        const url = await uploadToStorage(
          resized,
          `${payload.imageUrl}-${size.width}x${size.height}.jpg`
        );

        return { size, url };
      })
    );

    res.json({ status: 'success', variants });
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

Video Processing Example

app.post('/upload-video', async (req, res) => {
  const video = req.files.video;
  const videoUrl = await uploadToStorage(video);

  const apiKey = process.env.DISPATCHED_API_KEY;

  await fetch('https://dispatched.dev/api/jobs/dispatch', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${apiKey}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      payload: {
        task: 'process_video',
        videoUrl,
        formats: ['mp4', 'webm'],
        generateThumbnails: true
      }
    })
  });

  res.json({ status: 'processing', videoUrl });
})

Content Moderation

async function moderateContent(payload) {
  const content = await downloadFromStorage(payload.contentUrl);

  // Run moderation checks
  const [toxicityScore, nsfw] = await Promise.all([
    checkToxicity(content),
    checkNSFW(content)
  ]);

  if (toxicityScore > 0.8 || nsfw) {
    await markContentAsFlagged(payload.contentUrl);
    return false;
  }

  return true;
}

Best Practices

1. Upload files to storage first
2. Process in background via webhooks
3. Generate multiple formats/sizes
4. Store results back in storage
5. Update database with new URLs

Performance Stats

• Average processing time:
• Images: 2-3 seconds
• Videos: 1-5 minutes
• Supports files up to 5GB
• Parallel processing of variants

Start processing content at scale. Try Dispatched today.