Phase 1: Celery Infrastructure Setup

[ad-m-ss]

Phase 1: Celery Infrastructure Setup

Overview

Set up the complete Celery infrastructure with RabbitMQ backend to replace the current cron-based background job system. This provides the foundation for migrating all existing management commands to robust, scalable background tasks.

Scope

This issue covers the basic Celery infrastructure setup without any task migration. Task migration will be handled in subsequent issues.

Tasks

1.1 Docker Services Configuration

• Add RabbitMQ service to docker-compose.yml
• Add Celery worker service for background task processing
• Add Celery beat service for periodic task scheduling
• Configure proper networking and environment variables

1.2 Dependencies

• Add Celery dependencies to requirements/base.txt:
  • celery[rabbimtq]>=5.3.0
  • django-celery-beat>=2.5.0 (database-backed periodic tasks)
  • django-celery-results>=2.5.0 (task result storage)

1.3 Django Configuration

• Create poradnia/celery.py with Celery app configuration
• Update poradnia/__init__.py to import Celery app
• Configure Django settings for Celery broker, backend, and task routing
• Set up task autodiscovery for all Django apps
• Add Celery-related settings to development and production configurations

Files to Modify

Docker Configuration

• docker-compose.yml - Add RabbitMQ, Celery worker, and Celery beat services
• .contrib/docker/ - Update Docker-related scripts if necessary

Requirements

• requirements/base.txt - Add Celery and RabbitMQ dependencies

Django Configuration

• poradnia/celery.py - NEW FILE - Main Celery configuration
• poradnia/__init__.py - Import Celery app
• poradnia/settings/base.py - Celery broker/backend configuration
• poradnia/settings/dev.py - Development-specific Celery settings
• poradnia/settings/production.py - Production Celery configuration

Acceptance Criteria

• RabbitMQ service runs successfully in Docker environment
• Celery worker service starts without errors
• Celery beat service starts without errors
• Django can successfully connect to RabbitMQ broker
• Celery can autodiscover tasks from Django apps
• All services integrate properly with existing development workflow
• make start command works with new services
• New Celery-specific make commands are available

Development Commands to Add

make celery-worker          # Start Celery worker
make celery-beat           # Start Celery beat scheduler  
make celery-status         # Check worker status

Dependencies

This issue has no dependencies and can be worked on immediately.

Related Issues

This issue enables the following task migration work:

• Blocks: Phase 2: Background Task Migration to Celery (Umbrella Issue) #1829 - Phase 2: Background Task Migration to Celery (Umbrella Issue)
• Blocks: Migrate run_court_session_parser to Celery Task #1830 - Migrate run_court_session_parser to Celery Task
• Blocks: Migrate send_event_reminders to Celery Task #1831 - Migrate send_event_reminders to Celery Task
• Blocks: Migrate send_old_cases_reminder to Celery Task #1832 - Migrate send_old_cases_reminder to Celery Task
• Blocks: Migrate Django clearsessions to Celery Task #1833 - Migrate Django clearsessions to Celery Task

All Phase 2 task migration issues depend on the completion of this infrastructure setup.

Testing

• Verify all Docker services start successfully
• Test RabbitMQ connectivity from Django
• Verify Celery worker can receive and process test tasks
• Confirm Celery beat can schedule test periodic tasks

Current System Context

The project currently uses cron-based background jobs managed in .contrib/docker/cron/. This infrastructure setup maintains the existing system while adding Celery capability. The cron system will be phased out in subsequent task migration issues.

References

• Current cron configuration: .contrib/docker/cron/set_crontab.sh
• Existing management commands to be migrated later:
  • poradnia/judgements/management/commands/run_court_session_parser.py
  • poradnia/events/management/commands/send_event_reminders.py
  • poradnia/cases/management/commands/send_old_cases_reminder.py

[ad-m-ss]

✅ Implementation Complete

Phase 1 Celery infrastructure has been successfully implemented and is ready for review.

🔗 Pull Request Created

Draft PR: #1849 - feat: Phase 1 Celery Infrastructure Setup

📋 Implementation Summary

All acceptance criteria from the original issue have been completed:

✅ Docker Services Configuration

• RabbitMQ service with management UI (ports 5672, 15672)
• Celery worker service for background task processing
• Celery beat service with database-backed scheduling
• Proper networking and environment variables

✅ Dependencies Added

• celery[rabbitmq]==5.4.0
• django-celery-beat==2.7.0
• django-celery-results==2.5.1

✅ Django Configuration

• config/celery.py - Main Celery app configuration with test tasks
• config/__init__.py - Django integration setup
• Comprehensive settings in common.py, local.py, production.py
• Task autodiscovery enabled for all Django apps

✅ Development Commands

make celery-worker    # Start Celery worker
make celery-beat      # Start Celery beat scheduler  
make celery-status    # Check worker status
make celery-all       # Start all services

🧪 Testing Results

• ✅ All Django configuration checks pass
• ✅ Dependencies properly installed and working
• ✅ Docker services start successfully
• ✅ RabbitMQ broker connectivity confirmed
• ✅ Database migrations applied (32 Celery tables)
• ✅ End-to-end task execution verified
• ✅ CELERY_BEAT_SCHEDULE automatically syncs to Django admin

📚 Documentation

Comprehensive Polish documentation added in docs/celery.rst covering:

• Complete setup and usage guide
• CELERY_BEAT_SCHEDULE configuration patterns
• Automatic synchronization with PeriodicTask model
• Production vs development environment management
• Phase 2 migration planning

🛠️ Infrastructure Features

• RabbitMQ Management UI: http://localhost:15672 (development)
• Django Admin Integration: Tasks visible in /admin/django_celery_beat/
• Automatic Sync: CELERY_BEAT_SCHEDULE entries create PeriodicTask records
• Environment Separation: Test tasks in dev, production-ready config
• Monitoring Ready: Task events, results storage, health checks

🚀 Next Steps

This infrastructure enables Phase 2 task migration work:

• Ready for issues Phase 2: Background Task Migration to Celery (Umbrella Issue) #1829-1833 (individual task migrations)
• No impact on existing cron system during transition
• Foundation prepared for issue Phase 3: Remove Legacy Cron-Based Background Job System #1834 (legacy cleanup)

The implementation provides a robust, production-ready Celery infrastructure that maintains backward compatibility while enabling the planned migration to modern background task processing.

Status: Ready for code review and testing
Branch: resolve-1828
PR: #1849 (draft)