Migrate send_old_cases_reminder to Celery Task

[ad-m-ss]

Migrate send_old_cases_reminder to Celery Task

Overview

Migrate the send_old_cases_reminder management command to a Celery task with enhanced scheduling and notification management. This task sends monthly reminders about old cases that may need deletion or archival.

Current Implementation Analysis

Command Location

• File: poradnia/cases/management/commands/send_old_cases_reminder.py
• Current Schedule: Monthly on 2nd at 06:00 ("0 6 2 * *")
• Execution: Via cron with file locking (.contrib/docker/cron/run_locked.sh)

Task Complexity

• Purpose: Send notifications about old cases requiring attention
• Operations: Case analysis, age calculation, notification composition and sending
• Dependencies: Case models, User models, notification system
• Runtime: Medium (depends on case volume and notification method)
• Failure Points: Database queries, notification delivery, case filtering logic

Implementation Tasks

1. Create Celery Task

• Create or update poradnia/cases/tasks.py
• Convert management command logic to @shared_task decorated function
• Maintain existing case analysis and filtering logic
• Preserve notification content and recipients

2. Error Handling & Reliability

• Implement retry logic for notification failures (max 3 retries)
• Add specific exception handling for:
  • Database query timeouts
  • Notification delivery failures
  • Case filtering errors
  • Template rendering issues
• Graceful handling of large case datasets
• Individual notification failure handling

3. Logging & Monitoring

• Structured logging with case processing progress
• Track case analysis results (counts by age categories)
• Log notification delivery status
• Performance metrics (cases analyzed, notifications sent)
• Monthly execution tracking and history

4. Performance Optimization

• Efficient database queries for case age analysis
• Batch processing for large case volumes
• Memory-efficient case iteration
• Progress tracking for monthly operations

5. Scheduling Configuration

• Configure Celery beat periodic task (monthly on 2nd at 06:00)
• Use database-backed scheduling (django-celery-beat)
• Allow runtime schedule modifications through Django admin
• Maintain current monthly timing

Files to Modify/Create

New/Updated Files

• poradnia/cases/tasks.py - Celery task implementation (create or update)

Modified Files

• poradnia/settings/base.py - Add old cases reminder to Celery beat schedule
• poradnia/cases/management/commands/send_old_cases_reminder.py - Update or deprecate
• docs/celery.rst – Documentation updates

Configuration

# poradnia/settings/base.py - Add to CELERY_BEAT_SCHEDULE
'send-old-cases-reminder': {
    'task': 'poradnia.cases.tasks.send_old_cases_reminder',
    'schedule': crontab(hour=6, minute=0, day_of_month=2),  # Monthly on 2nd at 06:00
},

Dependencies

• BLOCKED BY: Phase 1: Celery Infrastructure Setup #1828 - Phase 1: Celery Infrastructure Setup
• REQUIRES: Redis service, Celery worker service, notification system
• PARENT: Phase 2: Background Task Migration to Celery (Umbrella Issue) #1829 - Phase 2: Background Task Migration to Celery (Umbrella Issue)

This issue cannot begin until the Celery infrastructure from #1828 is fully operational.

Related Issues

• Infrastructure: Phase 1: Celery Infrastructure Setup #1828 - Phase 1: Celery Infrastructure Setup (must be completed first)
• Umbrella: Phase 2: Background Task Migration to Celery (Umbrella Issue) #1829 - Phase 2: Background Task Migration to Celery (tracks overall progress)
• Parallel Tasks:
  • Migrate run_court_session_parser to Celery Task #1830 - Migrate run_court_session_parser to Celery Task
  • Migrate send_event_reminders to Celery Task #1831 - Migrate send_event_reminders to Celery Task
  • Migrate Django clearsessions to Celery Task #1833 - Migrate Django clearsessions to Celery Task

Can be developed in parallel with other task migrations once infrastructure is ready.

Testing Requirements

Unit Tests

• Test task execution with mock cases data
• Test case age calculation logic
• Test notification generation and delivery
• Test error handling for various failure scenarios
• Test edge cases (no old cases, system cases, etc.)

Integration Tests

• Test full Celery task execution
• Test monthly scheduling via Celery beat
• Test notification delivery with real notification backend
• Test task result storage and monthly tracking

Data Testing

• Test with various case age distributions
• Test performance with large case datasets
• Test filtering logic accuracy
• Test notification content generation

Implementation Example Structure

# poradnia/cases/tasks.py
from celery import shared_task
from celery.utils.log import get_task_logger
from datetime import datetime, timedelta
from django.utils import timezone

logger = get_task_logger(__name__)

@shared_task(bind=True, autoretry_for=(Exception,), retry_kwargs={'max_retries': 3, 'countdown': 1800})
def send_old_cases_reminder(self):
    """
    Send monthly reminder about old cases requiring attention.
    Migrated from management command with enhanced reliability.
    """
    try:
        logger.info("Starting old cases reminder task")
        
        # Calculate cutoff dates for old cases
        now = timezone.now()
        old_threshold = now - timedelta(days=365)  # Cases older than 1 year
        
        # Query old cases
        old_cases = get_old_cases(old_threshold)
        
        if not old_cases.exists():
            logger.info("No old cases found requiring attention")
            return {"status": "completed", "old_cases_count": 0}
        
        # Generate and send notifications
        notification_count = 0
        failed_count = 0
        
        # Group cases by responsible users/admins
        case_groups = group_cases_by_responsible(old_cases)
        
        for recipient, cases in case_groups.items():
            try:
                send_old_cases_notification(recipient, cases)
                notification_count += 1
                logger.info(f"Sent old cases reminder to {recipient}")
            except Exception as notify_exc:
                failed_count += 1
                logger.error(f"Failed to send reminder to {recipient}: {notify_exc}")
        
        result = {
            "status": "completed",
            "old_cases_count": old_cases.count(),
            "notifications_sent": notification_count,
            "notifications_failed": failed_count,
            "execution_date": now.isoformat()
        }
        
        logger.info(f"Old cases reminder completed: {result}")
        return result
        
    except Exception as exc:
        logger.error(f"Old cases reminder task failed: {exc}")
        raise self.retry(exc=exc)

Acceptance Criteria

• Celery task successfully analyzes and reports old cases
• Task runs on the same monthly schedule (2nd at 06:00)
• All current case filtering and analysis logic is preserved
• Notification content and recipients remain unchanged
• Error handling improves reliability over cron system
• Task execution can be monitored through Celery
• Monthly execution history is tracked
• Individual notification failures don't crash entire task

Monthly Task Considerations

• Handle month-specific scheduling edge cases
• Account for months with fewer than 31 days
• Proper timezone handling for monthly execution
• Historical execution tracking for compliance/audit
• Handle daylight saving time transitions

Rollback Plan

• Keep original management command during transition
• Maintain cron job as backup for initial months
• Monitor monthly execution reliability
• Document rollback procedure for monthly tasks

Success Metrics

• Reliability: 100% monthly execution with retry handling
• Accuracy: Correct identification and categorization of old cases
• Notification Delivery: 95%+ notification success rate
• Monitoring: Full visibility into monthly task execution and results

References

• Current management command: poradnia/cases/management/commands/send_old_cases_reminder.py
• Case models: poradnia/cases/models.py
• Current cron schedule: .contrib/docker/cron/set_crontab.sh
• Related notification systems in the application