Production Deployment

Best practices and checklists for deploying Easy AppServer to production.

Security Checklist

Secrets & Credentials

• Generate strong secrets (32+ characters):

  # Generate 32-character key
  openssl rand -hex 16 | cut -c1-32
  
  # Generate strong password
  openssl rand -base64 24

• KRATOS_SECRETS_COOKIE - Exactly 32 characters
• KRATOS_SECRETS_CIPHER - Exactly 32 characters
• KRATOS_WEBHOOK_SECRET - At least 32 characters
• HYDRA_SYSTEM_SECRET - Strong secret
• HYDRA_PAIRWISE_SALT - Strong salt
• APPSERVER_SETTINGS_ENCRYPTION_KEY - Exactly 32 characters
• POSTGRES_PASSWORD - Strong password
• RABBITMQ_PASSWORD - Change from default
• GRAFANA_ADMIN_PASSWORD - Change from default

TLS/SSL

• Enable gRPC TLS:

  APPSERVER_GRPC_TLS_ENABLED=true
  APPSERVER_GRPC_TLS_CERT_FILE=/etc/appserver/tls/server.crt
  APPSERVER_GRPC_TLS_KEY_FILE=/etc/appserver/tls/server.key

• Consider mTLS for app authentication:

  APPSERVER_GRPC_MTLS_ENABLED=true
  APPSERVER_GRPC_MTLS_CA_FILE=/etc/appserver/tls/ca.crt

• Configure SSL termination at load balancer
• Use valid certificates (not self-signed)
• Set up certificate renewal automation

Database Security

• Enable PostgreSQL SSL:

  APPSERVER_DB_SSLMODE=verify-full

• Use separate database users with minimal privileges
• Enable connection encryption
• Configure connection limits

Application Security

• Disable bootstrap registration:

  APPSERVER_AUTH_ALLOW_BOOTSTRAP_REGISTRATION=false

• Disable developer tools:

  APPSERVER_GRAPHQL_PLAYGROUND_ENABLED=false
  APPSERVER_GRPC_REFLECTION_ENABLED=false

• Set environment mode:

  APPSERVER_ENV=production

Network Security

• Configure firewall rules:
  • Only expose ports 80/443 (load balancer)
  • Internal services on private network only
  • Block direct database access
• Use private Docker network
• Restrict management UI access (RabbitMQ, Grafana, etc.)

Cookie Security

• Set secure cookie attributes (handled by Kratos):
  • Secure: true (HTTPS only)
  • HttpOnly: true (no JS access)
  • SameSite: Lax (CSRF protection)

---

Performance Tuning

Database Connection Pooling

PostgreSQL connection settings in AppServer:

# Managed by database driver
# Max connections: 25
# Max idle connections: 5

For high-traffic deployments:

-- Check current connections
SELECT count(*) FROM pg_stat_activity;

-- Recommend max_connections based on load
-- Rule of thumb: (2 * CPU cores) + effective_spindle_count

Redis Caching

# Local cache TTL (in-memory, per instance)
APPSERVER_CACHE_TTL=300

# Settings cache TTL
APPSERVER_SETTINGS_CACHE_TTL=60

Cache hierarchy:

1. Local in-memory cache (fastest)
2. Redis cache (shared across instances)
3. OpenFGA query (authoritative)

RabbitMQ Prefetch

# Balance between throughput and memory
APPSERVER_RABBITMQ_PREFETCH_COUNT=10

# For high-throughput scenarios
APPSERVER_RABBITMQ_PREFETCH_COUNT=50

HTTP Compression

# Asset compression settings
APPSERVER_UI_COMPRESSION_MIN_SIZE=1024    # 1KB minimum
APPSERVER_UI_COMPRESSION_LEVEL=6          # 1-9 (6=balanced)
APPSERVER_UI_CACHE_TTL=300                # 5 minutes
APPSERVER_UI_MAX_CACHE_SIZE=104857600     # 100MB

CDN Configuration

For static assets:

# Nginx CDN cache headers
location /apps/ {
    proxy_pass http://appserver:8080;
    proxy_cache_valid 200 1h;
    add_header Cache-Control "public, max-age=3600";
}

---

High Availability

Multiple AppServer Instances

Run multiple AppServer instances behind a load balancer:

# docker-compose.override.yml
services:
  appserver:
    deploy:
      replicas: 3

Load balancer configuration:

upstream appserver {
    least_conn;
    server appserver-1:8080;
    server appserver-2:8080;
    server appserver-3:8080;
}

Load Balancing

HTTP load balancing:

upstream shell {
    least_conn;
    server shell-1:3000;
    server shell-2:3000;
}

server {
    listen 443 ssl http2;

    location / {
        proxy_pass http://shell;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

gRPC load balancing:

upstream appserver_grpc {
    server appserver-1:9090;
    server appserver-2:9090;
}

server {
    listen 443 ssl http2;

    location /grpc {
        grpc_pass grpcs://appserver_grpc;
    }
}

Database Replication

PostgreSQL primary-replica setup:

postgres-primary:
  image: postgres:15
  environment:
    POSTGRES_REPLICATION_MODE: master

postgres-replica:
  image: postgres:15
  environment:
    POSTGRES_REPLICATION_MODE: slave
    POSTGRES_MASTER_HOST: postgres-primary

Redis Clustering

For session and cache redundancy:

redis:
  image: redis:8.2
  command: redis-server --cluster-enabled yes

Or use Redis Sentinel:

redis-sentinel:
  image: redis:8.2
  command: redis-sentinel /etc/redis/sentinel.conf

RabbitMQ Clustering

rabbitmq-1:
  image: rabbitmq:4.2-management
  environment:
    RABBITMQ_ERLANG_COOKIE: shared-secret

rabbitmq-2:
  image: rabbitmq:4.2-management
  environment:
    RABBITMQ_ERLANG_COOKIE: shared-secret
    RABBITMQ_NODE_NAME: rabbit@rabbitmq-2

---

Backup Strategy

Database Backups

Daily automated backups:

#!/bin/bash
# backup-database.sh
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR=/backups/postgres

# Full database dump
docker exec appserver-postgres \
    pg_dump -U partner partner > $BACKUP_DIR/partner_$DATE.sql

# Compress
gzip $BACKUP_DIR/partner_$DATE.sql

# Retain last 30 days
find $BACKUP_DIR -name "*.gz" -mtime +30 -delete

Point-in-time recovery (WAL archiving):

# postgresql.conf
archive_mode = on
archive_command = 'cp %p /backups/wal/%f'

Settings Encryption Key

Critical: Back up APPSERVER_SETTINGS_ENCRYPTION_KEY securely!

# Store in secure vault (HashiCorp Vault, AWS Secrets Manager, etc.)
vault kv put secret/appserver encryption_key=$APPSERVER_SETTINGS_ENCRYPTION_KEY

If lost, encrypted app settings cannot be recovered.

Certificate Backups

# Backup TLS certificates
tar -czf /backups/certs/tls_$(date +%Y%m%d).tar.gz \
    /etc/appserver/tls/

Volume Backups

# Backup Docker volumes
for volume in postgres-data redis-data rabbitmq-data; do
    docker run --rm \
        -v ${volume}:/data \
        -v /backups:/backup \
        alpine tar czf /backup/${volume}_$(date +%Y%m%d).tar.gz /data
done

---

Disaster Recovery

Recovery Time Objective (RTO)

Component	Target RTO	Strategy
Database	< 1 hour	Replica promotion or restore
Cache	< 5 min	Redis restart (data loss acceptable)
AppServer	< 5 min	Container restart or failover
Auth Services	< 15 min	Restore from backup

Recovery Point Objective (RPO)

Data Type	Target RPO	Strategy
User data	< 1 hour	Hourly backups + WAL
App settings	< 24 hours	Daily encrypted backups
Sessions	0 (acceptable loss)	In-memory, re-authenticate
Metrics	< 15 days	Prometheus retention

Recovery Procedures

Database Recovery:

# 1. Stop dependent services
docker-compose stop appserver shell

# 2. Restore database
gunzip -c /backups/postgres/partner_20240101.sql.gz | \
    docker exec -i appserver-postgres psql -U partner

# 3. Run migrations if needed
docker-compose run --rm db-migrate

# 4. Restart services
docker-compose up -d appserver shell

Full Stack Recovery:

# 1. Restore volumes from backup
docker-compose down -v

for volume in postgres-data redis-data rabbitmq-data; do
    docker volume create $volume
    docker run --rm \
        -v ${volume}:/data \
        -v /backups:/backup \
        alpine tar xzf /backup/${volume}_latest.tar.gz -C /
done

# 2. Start services
docker-compose up -d

# 3. Verify health
docker-compose ps

---

Scaling

Horizontal Scaling

AppServer instances:

services:
  appserver:
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '2'
          memory: 4G

Shell instances:

services:
  shell:
    deploy:
      replicas: 2
      resources:
        limits:
          cpus: '1'
          memory: 2G

Vertical Scaling

Recommended minimums:

Service	CPU	Memory	Storage
PostgreSQL	2 cores	4GB	50GB SSD
Redis	1 core	2GB	10GB SSD
RabbitMQ	1 core	2GB	10GB SSD
AppServer	2 cores	4GB	-
Shell	1 core	2GB	-
Kratos	1 core	1GB	-
OpenFGA	1 core	1GB	-

Production recommendations:

Service	CPU	Memory	Storage
PostgreSQL	4+ cores	16GB+	200GB+ SSD
Redis	2 cores	8GB	20GB SSD
RabbitMQ	2 cores	4GB	20GB SSD
AppServer	4 cores	8GB	-
Shell	2 cores	4GB	-

Auto-scaling

Kubernetes HPA example:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: appserver
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: appserver
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Resource Limits

Docker Compose resource limits:

services:
  appserver:
    deploy:
      resources:
        limits:
          cpus: '4'
          memory: 8G
        reservations:
          cpus: '2'
          memory: 4G

---

Monitoring Checklist

• Enable telemetry:

  APPSERVER_METRICS_ENABLED=true
  APPSERVER_TRACING_ENABLED=true

• Configure Grafana dashboards
• Set up alerts for:
  • High error rate (> 1%)
  • High latency (p95 > 1s)
  • Database connection saturation
  • Disk usage > 80%
  • Memory usage > 90%
• Configure log aggregation (Loki)
• Set up on-call rotation

---

Deployment Checklist

Pre-deployment

• All secrets generated and stored securely
• TLS certificates provisioned
• DNS records configured
• Firewall rules configured
• Backup strategy tested
• Load balancer configured
• Health checks verified

Deployment

• Database migrations run successfully
• OpenFGA store/model created
• All services healthy
• Smoke tests pass
• Monitoring dashboards working
• Alerts configured

Post-deployment

• Monitor error rates for 24 hours
• Verify backup jobs running
• Document any configuration changes
• Update runbook if needed

---

Related Topics

• Docker Infrastructure - Service setup
• Environment Reference - Configuration options
• Observability Stack - Monitoring setup
• AppServer Guide - Backend deployment
• Shell Guide - Frontend deployment