Platform Architecture

Understanding the Easy AppServer architecture and how components work together.

Overview

Easy AppServer v2 is a complete rewrite featuring a modern microservices architecture with comprehensive identity management, fine-grained authorization, and full observability.

High-Level Architecture

┌──────────────────────────────────────────────────────────────┐
│                       Easy AppServer                          │
├──────────────────────────────────────────────────────────────┤
│   gRPC API (9090)  │  HTTP/GraphQL (8080)  │  WebSocket      │
├──────────────────────────────────────────────────────────────┤
│                    Application Services                       │
│  Marketplace │ Hooks │ Activity │ Settings │ UI │ Proxy      │
├──────────────────────────────────────────────────────────────┤
│                   Infrastructure Layer                        │
│  EventBus (RabbitMQ) │ Cache (Redis) │ Auth (OpenFGA)        │
├──────────────────────────────────────────────────────────────┤
│                    Data Persistence                           │
│          PostgreSQL (Apps, Assets, State, Sessions)           │
└──────────────────────────────────────────────────────────────┘

Core Components

1. AppServer (Go)

The central orchestration platform that manages the entire application lifecycle.

Key Responsibilities:

Application registration and lifecycle management
Asset serving with SRI (Subresource Integrity)
Intelligent HTTP proxying with permission checks
Event-driven architecture coordination
Real-time updates via GraphQL subscriptions

Ports:

8080 - HTTP/REST API and GraphQL endpoint
9090 - gRPC API for app communication

Code Location: cmd/appserver-v2/main.go

2. Identity & Authorization Stack

Ory Kratos - Identity Management

Handles user identity, registration, login, and profile management.

Features:

Self-service registration and login
Account recovery and verification
Profile management
Session management
Multi-factor authentication support

Ports:

4433 - Public API (user-facing)
4434 - Admin API (backend-facing)

Use Cases:

User registration and login
Password recovery
Email verification
Session validation

Ory Hydra - OAuth2/OIDC Provider

Provides OAuth2 and OpenID Connect functionality for third-party integrations.

Features:

OAuth2 authorization server
OpenID Connect provider
Consent management
Token issuance and validation

Ports:

4444 - Public endpoint (authorization, token)
4445 - Admin API (client management)

Use Cases:

Third-party app authorization
Single sign-on (SSO)
API access tokens
Service-to-service authentication

OpenFGA - Fine-Grained Authorization

Relation-based access control system inspired by Google Zanzibar.

Features:

Relationship-based access control
Role hierarchy (easy, admin, advertiser, publisher)
Resource-level permissions
Real-time permission checks

Ports:

8090 - HTTP API
8091 - gRPC API

Authorization Model:

user → role → permissions → resources

Roles:
- easy (god mode - internal employees)
- admin (platform administrator)
- advertiser (advertiser access)
- publisher (publisher access)

Use Cases:

Check if user can access an app
Verify permissions before API calls
Role-based UI rendering
Multi-tenant access control

Playground: http://localhost:8090/playground

Ory Oathkeeper - Identity & Access Proxy

Reverse proxy that enforces authentication and authorization policies.

Features:

Request authentication
Token validation
Access rule evaluation
Header enrichment

Ports:

4455 - Proxy endpoint
4456 - API for configuration

Use Cases:

Protect backend services
Enforce authentication on routes
Add identity information to requests

3. Data Layer

PostgreSQL

Shared relational database with multiple schemas for different services.

Schemas (6 total):

kratos - Identity data
hydra - OAuth2 clients and tokens
openfga - Authorization tuples
appserver - Apps, hooks, activities
shell - Shell application data
partner - Partner/tenant data

Port: 5432

Features:

ACID transactions
Full-text search
JSON/JSONB support
Connection pooling (25 max open, 5 max idle)

Redis

High-performance cache and session store.

Port: 6379

Use Cases:

Distributed caching (app metadata, settings)
Local in-memory cache (TTL: 300s)
Session storage (fallback to DB)
Rate limiting counters

Configuration:

Default TTL: 300 seconds
Max cache size: 100 MB
Automatic invalidation on updates

4. Message Broker

RabbitMQ

Event bus for asynchronous communication and event-driven workflows.

Ports:

5672 - AMQP protocol
15672 - Management UI (guest/guest)

Features:

Pub/sub messaging
Durable queues
Message acknowledgment
Dead-letter queues
Retry logic (max 3 retries)

Event Types:

app.installed - App installation completed
app.uninstalled - App removed
app.state_changed - App state transition
hook.triggered - Hook event fired
activity.executed - Activity completed

Use Cases:

App lifecycle notifications
Hook delivery between apps
Activity execution triggers
Audit logging
Async task processing

5. Observability Stack

Complete telemetry infrastructure for monitoring and debugging.

OpenTelemetry Collector

Central hub for collecting and processing telemetry data.

Ports:

4317 - OTLP gRPC (recommended)
4318 - OTLP HTTP

Features:

Traces, metrics, and logs collection
Data transformation and filtering
Multi-backend export
Buffering and retry

Prometheus - Metrics

Time-series database for metrics storage.

Port: 9090

Features:

Metrics scraping (15s interval)
15-day retention (configurable)
PromQL query language
Alerting rules

Key Metrics:

HTTP request rate and latency
gRPC call metrics
Database connection pool usage
Cache hit/miss rates
Event bus queue depths

Loki - Logs

Log aggregation system optimized for Kubernetes/Docker.

Port: 3100

Features:

Label-based indexing
LogQL query language
30-day retention (configurable)
Integration with traces

Use Cases:

Structured log queries
Error log aggregation
Trace correlation
Debugging production issues

Tempo - Distributed Tracing

Tracing backend for tracking requests across services.

Port: 3200

Features:

Distributed trace storage
TraceQL query language
Service dependency graphs
RED metrics from traces

Use Cases:

Request flow visualization
Performance bottleneck identification
Service dependency mapping
Latency analysis

Grafana - Visualization

Unified dashboard for all observability data.

Port: 3000 (admin/admin)

Features:

Pre-configured datasources
Custom dashboards
Unified alerting
Trace ↔ Log correlation
Service graphs

Pre-configured Datasources:

Prometheus (metrics)
Loki (logs)
Tempo (traces)

Application Services

Marketplace Service (gRPC)

Manages application lifecycle and registration.

Capabilities:

App registration with manifest
Installation and uninstallation
State management (registered → installed → running → uninstalled)
Dependency resolution (HARD/SOFT dependencies)
Health monitoring with auto-recovery

App States:

Registered → Installing → Installed → Running → Uninstalling → Uninstalled
                ↓                         ↓
              Failed                   Unhealthy

Proto: easy.proto/v2/protos/services.proto (Marketplace service)

Hooks Service (gRPC)

Event-driven communication between applications.

Features:

Hook registration and discovery
Event publishing and subscription
Async delivery via RabbitMQ
Event filtering and routing
Delivery guarantees (at-least-once)

Use Cases:

App-to-app communication
Workflow automation
Event notifications
Data synchronization

Proto: easy.proto/v2/protos/services.proto (Hooks service)

Activity Service (gRPC)

Background task execution and activity management.

Features:

Activity registration
Execution scheduling
Progress tracking
Result storage

Use Cases:

Long-running tasks
Background jobs
Scheduled operations
Batch processing

Proto: easy.proto/v2/protos/services.proto (Activity service)

Settings Service (gRPC)

Encrypted settings storage for applications.

Features:

Key-value storage
AES-256-GCM encryption for sensitive values
Schema validation
Versioning

Security:

Encryption key: 32-byte (required via APPSERVER_SETTINGS_ENCRYPTION_KEY)
Encryption at rest
Automatic decryption on retrieval

Proto: easy.proto/v2/protos/services.proto (Settings service)

UI Service (HTTP)

Microfrontend asset serving with security and performance optimizations.

Features:

Asset serving with SRI headers
Automatic compression (gzip/brotli)
Immutable caching (max-age=31536000)
ETag support
Module Federation support
Web Components support
ESM modules support

Asset Types:

JavaScript (ES modules, UMD)
CSS stylesheets
Images and fonts
JSON manifests

Example:

GET /apps/de.easy-m.hello-world/assets/main.js

Response Headers:
X-Subresource-Integrity: sha256-3x8DE279hr8o/Aq0dEdH4WApIwn5rbRKhugPzn6Bofw=
Cache-Control: public, max-age=31536000, immutable
Content-Encoding: brotli

HTTP Proxy Service

Intelligent proxy to application backends with permission enforcement.

Features:

Routes: /api/apps/{appName}/** → app upstream
Permission verification before forwarding
Rate limiting per route
Circuit breaking for failing backends
Request/response logging

Security:

Validate session (Kratos)
Check permissions (OpenFGA)
Forward to app backend
Return response

API Types

1. gRPC API (Port 9090)

Bidirectional streaming API for apps to communicate with the platform.

Services:

Marketplace - App lifecycle
Hooks - Event communication
Activity - Task execution
Settings - Configuration storage

Features:

Protobuf serialization
Bidirectional streaming
Automatic retry
Load balancing

Documentation:

Proto definitions: easy.proto/v2/protos/
gRPC reflection: Enable with APPSERVER_GRPC_REFLECTION_ENABLED=true

Tools:

# List services
grpcurl -plaintext localhost:9090 list

# Call method
grpcurl -plaintext -d '{"name": "de.easy-m.test"}' \
  localhost:9090 easy.marketplace.v2.Marketplace/GetApp

2. GraphQL API (Port 8080/graphql)

Query and mutation API with real-time subscriptions.

Features:

Type-safe queries
Real-time subscriptions (WebSocket)
Cursor-based pagination
Field-level permissions
Introspection

Query Examples:

# Query all apps
query {
  apps(first: 10) {
    edges {
      node {
        id
        name
        state
        version
      }
    }
  }
}

# Subscribe to app state changes
subscription {
  appStateChanged {
    appId
    state
    timestamp
  }
}

# Subscribe to hook events
subscription {
  hookEvent(appId: "de.easy-m.test") {
    hookId
    payload
    timestamp
  }
}

Playground:

Enable: APPSERVER_GRAPHQL_PLAYGROUND_ENABLED=true (dev only)
URL: http://localhost:8080/graphql

3. HTTP REST API (Port 8080)

RESTful endpoints for asset serving and health checks.

Endpoints:

GET /health - Health check
GET /ready - Readiness check
GET /apps/{appName}/assets/{assetName} - Serve assets
/api/apps/{appName}/** - Proxy to app upstream

Example:

# Health check
curl http://localhost:8080/health

# Serve asset
curl http://localhost:8080/apps/de.easy-m.hello/assets/main.js

Data Flow

App Installation Flow

1. App Registration (gRPC)
   ↓
2. Validate Manifest
   ↓
3. Check Dependencies
   ↓
4. Store in PostgreSQL
   ↓
5. Publish "app.registered" event (RabbitMQ)
   ↓
6. Trigger Installation
   ↓
7. Execute onCreate Hook
   ↓
8. Update State: Installed
   ↓
9. Publish "app.installed" event
   ↓
10. GraphQL Subscription Notification

Hook Delivery Flow

1. App Publishes Hook (gRPC)
   ↓
2. Validate Hook Registration
   ↓
3. Find Subscribers (PostgreSQL)
   ↓
4. Publish to Event Bus (RabbitMQ)
   ↓
5. Consumer Receives Event
   ↓
6. Check Permissions (OpenFGA)
   ↓
7. Deliver to Subscriber App (gRPC)
   ↓
8. Ack/Nack Response
   ↓
9. Retry on Failure (max 3 times)

Asset Serving Flow

1. HTTP Request: GET /apps/{appName}/assets/main.js
   ↓
2. Check Cache (Redis)
   ↓ (miss)
3. Load from Database (PostgreSQL)
   ↓
4. Compute SRI Hash
   ↓
5. Compress (gzip/brotli)
   ↓
6. Cache in Redis
   ↓
7. Return with Headers:
   - X-Subresource-Integrity
   - Cache-Control
   - Content-Encoding
   - ETag

Permission Check Flow

1. HTTP Request → Proxy Endpoint
   ↓
2. Extract Session Cookie
   ↓
3. Validate Session (Kratos)
   ↓
4. Get User Identity
   ↓
5. Check Permission (OpenFGA)
   Query: user:{id}#viewer@app:{appName}
   ↓
6. Allow/Deny
   ↓
7. Forward to App Backend (if allowed)

Security Model

Authentication

Session-based (Kratos):
- Cookie: ory_kratos_session
- Validation on each request
- Automatic renewal
Certificate-based (Apps):
- X.509 client certificates
- Mutual TLS for app-to-appserver
- Signature verification with replay protection

Authorization

OpenFGA Relation Model:

User → Role → Permissions → Resource

Example:
- User "alice" is assignee of role "admin"
- Role "admin" has viewer relation to app "de.easy-m.test"
- Therefore, alice can view the app

Permission Hierarchy:

owner > editor > viewer

Encryption

In Transit:
- TLS/SSL for all external connections
- mTLS for service-to-service
At Rest:
- Settings encryption (AES-256-GCM)
- Sensitive fields in database
- Optional: PostgreSQL encryption

Secrets Management

Required Secrets:

KRATOS_SECRETS_COOKIE - Exactly 32 characters
KRATOS_SECRETS_CIPHER - Exactly 32 characters
HYDRA_SYSTEM_SECRET - Strong random secret
POSTGRES_PASSWORD - Database password
APPSERVER_SETTINGS_ENCRYPTION_KEY - 32-byte encryption key

Best Practices:

Use environment variables
Never commit secrets to Git
Rotate regularly
Use secret management tools (Vault, AWS Secrets Manager)

Scalability & Performance

Horizontal Scaling

Stateless Components (can scale horizontally):

AppServer instances (behind load balancer)
Oathkeeper proxies
OpenTelemetry Collectors

Stateful Components (require special handling):

PostgreSQL (read replicas, partitioning)
Redis (cluster mode, sentinel)
RabbitMQ (clustering)

Caching Strategy

Multi-Level Cache:

In-Memory (local to each AppServer instance)
- TTL: 300 seconds
- Size: Limited per instance
- Use: Hot data, session info
Redis (distributed)
- TTL: Configurable per key
- Use: Shared cache across instances
Database (PostgreSQL)
- Permanent storage
- Source of truth

Cache Invalidation:

On update: Invalidate local + Redis
On delete: Purge all levels
TTL expiry: Automatic

Connection Pooling

PostgreSQL:

Max open connections: 25
Max idle connections: 5
Connection lifetime: 5 minutes

Redis:

Pool size: 10
Idle timeout: 5 minutes

Deployment Topology

Development

All services on single machine via Docker Compose
- Easy setup
- Fast iteration
- Complete feature set

Production

┌─────────────────────────────────────────┐
│          Load Balancer (HTTPS)          │
└─────────────────────────────────────────┘
           │                │
    ┌──────┴──────┐   ┌────┴─────┐
    │ AppServer 1 │   │ AppServer 2 │
    └─────────────┘   └──────────────┘
           │                │
    ┌──────┴────────────────┴──────┐
    │   PostgreSQL (Primary +       │
    │   Read Replicas)              │
    └───────────────────────────────┘
    ┌───────────────────────────────┐
    │   Redis Cluster               │
    └───────────────────────────────┘
    ┌───────────────────────────────┐
    │   RabbitMQ Cluster            │
    └───────────────────────────────┘

Recommendations:

Use managed services (RDS, ElastiCache, etc.)
Enable SSL/TLS everywhere
Implement rate limiting
Set up monitoring and alerting
Use blue-green deployment
Implement circuit breakers

Monitoring & Health

Health Check Endpoints

AppServer:

/health - Basic health (always returns 200 if running)
/ready - Readiness (checks DB, Redis, RabbitMQ)

Dependencies:

Kratos: http://localhost:4433/health/ready
Hydra: http://localhost:4444/health/ready
OpenFGA: http://localhost:8090/healthz

Key Metrics

Application Metrics:

Request rate (requests/sec)
Response time (p50, p95, p99)
Error rate (5xx responses)
Active connections

Infrastructure Metrics:

Database connection pool usage
Redis hit/miss rate
RabbitMQ queue depth
Memory and CPU usage

Distributed Tracing

Trace Propagation:

W3C Trace Context headers
Automatic context propagation
Service-to-service correlation

Span Attributes:

Service name
Operation name
HTTP method/status
Database queries
Cache operations

Next Steps

Quick Start - Get the platform running
Installation Guide - Detailed setup instructions
Development Environment - Set up for local development
Building Apps - Create your first app

Overview​

High-Level Architecture​

Core Components​

1. AppServer (Go)​

2. Identity & Authorization Stack​

Ory Kratos - Identity Management​

Ory Hydra - OAuth2/OIDC Provider​

OpenFGA - Fine-Grained Authorization​

Ory Oathkeeper - Identity & Access Proxy​

3. Data Layer​

PostgreSQL​

Redis​

4. Message Broker​

RabbitMQ​

5. Observability Stack​

OpenTelemetry Collector​

Prometheus - Metrics​

Loki - Logs​

Tempo - Distributed Tracing​

Grafana - Visualization​

Application Services​

Marketplace Service (gRPC)​

Hooks Service (gRPC)​

Activity Service (gRPC)​

Settings Service (gRPC)​

UI Service (HTTP)​

HTTP Proxy Service​

API Types​

1. gRPC API (Port 9090)​

2. GraphQL API (Port 8080/graphql)​

3. HTTP REST API (Port 8080)​

Data Flow​

App Installation Flow​

Hook Delivery Flow​

Asset Serving Flow​

Permission Check Flow​

Security Model​

Authentication​

Authorization​

Encryption​

Secrets Management​

Scalability & Performance​

Horizontal Scaling​

Caching Strategy​

Connection Pooling​

Deployment Topology​

Development​

Production​

Monitoring & Health​

Health Check Endpoints​

Key Metrics​

Distributed Tracing​

Next Steps​

Overview

High-Level Architecture

Core Components

1. AppServer (Go)

2. Identity & Authorization Stack

Ory Kratos - Identity Management

Ory Hydra - OAuth2/OIDC Provider

OpenFGA - Fine-Grained Authorization

Ory Oathkeeper - Identity & Access Proxy

3. Data Layer

PostgreSQL

Redis

4. Message Broker

RabbitMQ

5. Observability Stack

OpenTelemetry Collector

Prometheus - Metrics

Loki - Logs

Tempo - Distributed Tracing

Grafana - Visualization

Application Services

Marketplace Service (gRPC)

Hooks Service (gRPC)

Activity Service (gRPC)

Settings Service (gRPC)

UI Service (HTTP)

HTTP Proxy Service

API Types

1. gRPC API (Port 9090)

2. GraphQL API (Port 8080/graphql)

3. HTTP REST API (Port 8080)

Data Flow

App Installation Flow

Hook Delivery Flow

Asset Serving Flow

Permission Check Flow

Security Model

Authentication

Authorization

Encryption

Secrets Management

Scalability & Performance

Horizontal Scaling

Caching Strategy

Connection Pooling

Deployment Topology

Development

Production

Monitoring & Health

Health Check Endpoints

Key Metrics

Distributed Tracing

Next Steps