EFP Allocations

Overview

The EFP Allocations system provides a standardized interface for Hosting Entities (HEs) to receive, process, and manage EuroHPC resource allocations while maintaining local autonomy. Built on the Waldur marketplace platform, it integrates with the broader EuroHPC Federation Platform to enable seamless resource allocation, user management, and usage reporting across diverse HPC infrastructures.

EFP Allocations System Architecture

Core Components

The EFP Allocations system is built on three main layers: the Waldur core services, integration options for different technical requirements, and local HE infrastructure.

graph TB
    subgraph "EFP Allocations Core (Waldur)"
        MARKETPLACE[Service Catalog<br/>Services & Orders]
        PROJECTS[Project Management<br/>Allocation Tracking]
        USERS[User Management<br/>RBAC & Permissions]
        BILLING[Usage<br/>Consumption Tracking]
        API[REST API<br/>Integration Interface]
    end

    subgraph "Integration Layer"
        SDK[Waldur SDK<br/>Python/JS Libraries]
        AGENT[Site Agent<br/>Reference Implementation]
        DIRECT[Direct API<br/>Custom Integration]
        WEBUI[Web UI<br/>Manual Operations]
    end

    subgraph "Hosting Entity Systems"
        SCHEDULER[Batch Schedulers<br/>Slurm, PBS, LSF]
        STORAGE[Storage Systems<br/>Lustre, GPFS, S3]
        IDENTITY[Identity Management<br/>LDAP, AD, Local]
        MONITORING[Usage Monitoring<br/>Accounting Systems]
    end

    MARKETPLACE --> API
    PROJECTS --> API
    USERS --> API
    BILLING --> API

    API --> SDK
    API --> AGENT
    API --> DIRECT
    API --> WEBUI

    SDK --> SCHEDULER
    AGENT --> SCHEDULER
    DIRECT --> SCHEDULER

    SCHEDULER --> STORAGE
    SCHEDULER --> IDENTITY
    SCHEDULER --> MONITORING

Data Flow Architecture

The complete allocation lifecycle involves four key phases: initial project setup from peer review approval, user registration and provisioning, ongoing resource usage tracking, and allocation management for changes and extensions.

sequenceDiagram
    participant PRP as Peer Review Platform
    participant EFP as EFP Allocations
    participant HE as Hosting Entity
    participant USER as End User

    Note over PRP,USER: 1. Project Approval & Allocation Creation
    PRP->>EFP: Project approval notification
    EFP->>EFP: Create allocation in marketplace
    EFP->>HE: Allocation creation request
    HE->>HE: Create local allocation (Slurm account, quotas)
    HE->>EFP: Confirm allocation created

    Note over PRP,USER: 2. User Onboarding
    USER->>EFP: User registration via MyAccessID
    EFP->>EFP: User vetting & verification
    EFP->>HE: User provisioning request
    HE->>HE: Create local user account
    HE->>EFP: Confirm user provisioned

    Note over PRP,USER: 3. Resource Usage
    USER->>HE: Access resources (SSH, jobs, storage)
    HE->>HE: Track resource consumption
    HE->>EFP: Regular usage reports
    EFP->>EFP: Update billing & allocation status

    Note over PRP,USER: 4. Allocation Management
    EFP->>HE: Allocation changes (quota, extension)
    HE->>HE: Update local allocation settings
    HE->>EFP: Confirm changes applied

Integration Patterns

Hosting Entities can choose from four integration approaches based on their technical requirements and operational preferences. Each option provides different levels of control, development effort, and maintenance responsibility.

Option A: Direct API Integration

graph LR
    HE[HE Local Systems] --> CUSTOM[Custom Integration Code]
    CUSTOM --> EFP[EFP Allocations API]

• Best for: HEs with strong development capabilities
• Advantages: Full control, custom logic, optimal performance
• Requirements: Development resources, API expertise

Option B: SDK Integration

graph LR
    HE[HE Local Systems] --> SDK[Waldur Python/JS SDK]
    SDK --> EFP[EFP Allocations API]

• Best for: HEs preferring pre-built integration libraries
• Advantages: Reduced development effort, maintained by EFP team
• Requirements: Python/JavaScript development skills

SDKs are available in 3 languages:

• Python: https://github.com/waldur/py-client
• JavaScript: https://github.com/waldur/js-client
• Go: https://github.com/waldur/go-client

Option C: Site Agent Integration

graph LR
    HE[HE Local Systems] --> AGENT[EFP Site Agent]
    AGENT --> EFP[EFP Allocations API]

• Best for: HEs wanting minimal local development
• Advantages: Standardized approach, EFP-maintained
• Requirements: Container deployment capability

Option D: Web UI Integration

graph LR
    STAFF[HE Staff] --> EFP[EFP Allocations Web GUI]

• Best for: Initial integration, debugging, and low-volume operations
• Advantages: No programming required, immediate availability for testing
• Requirements: Staff training, manual processes
• Note: This option is primarily intended as a helper for initial integration and debugging. HEs are expected to transition to fully automated options (A, B, or C) for production operations.

EFP Allocations Integration Model

The EFP Allocations system leverages Waldur's marketplace architecture to provide standardized resource management. This model maps hosting entity partitions to marketplace offerings, allocation requests to orders, and tracks resource consumption through dedicated usage components.

graph TB
    subgraph "Service Catalog Layer"
        SP[Service Provider<br/>Hosting Entity]
        OFF[Offering<br/>HE Partition]
        COMP[Components<br/>Node-hours, GPU-hours]
    end

    subgraph "Order Processing Layer"
        ORDER[Order<br/>Allocation Request]
        PROC[Order Processor<br/>Provisioning Logic]
        RES[Resource<br/>Active Allocation]
    end

    subgraph "Accounting Layer"
        USAGE[Component Usage<br/>Consumption Tracking]
    end

    SP --> OFF
    OFF --> COMP

    ORDER --> PROC
    PROC --> RES
    OFF --> ORDER

    COMP --> USAGE
    RES --> USAGE

Key Concepts

• Service Provider: Each HE is registered as a service provider in the marketplace
• Offerings: Define available partitions as used by JU Peer Review Platform
• Components: Individual billable items (Node hours, GPU hours, etc.)
• Orders: Allocation requests that trigger resource provisioning
• Resources: Active allocations with lifecycle management

Technical Architecture Details

Authentication & Authorization Flow

Users authenticate once via MyAccessID federation, then EFP Allocations coordinates with hosting entities to provision local accounts with proper access mapping.

sequenceDiagram
    participant USER as End User
    participant AAI as EFP AAI (MyAccessID)
    participant ALLOC as EFP Allocations
    participant HE as Hosting Entity

    USER->>AAI: Login via MyAccessID
    AAI->>AAI: Identity verification & MFA
    AAI->>ALLOC: User authentication token
    ALLOC->>ALLOC: Check user permissions
    ALLOC->>HE: User provisioning request
    HE->>HE: Create local account mapping
    HE->>ALLOC: Confirm user provisioned

Resource Lifecycle Management

Allocations follow standard Waldur marketplace resource states: initial provisioning (CREATING), active operation (OK), modification (UPDATING), deletion (TERMINATING), and error recovery (ERRED).

stateDiagram-v2
    [*] --> CREATING : Order approved & executing
    CREATING --> OK : Provisioning success
    CREATING --> ERRED : Provisioning failed
    OK --> UPDATING : Update requested (quota/extension)
    OK --> TERMINATING : End date reached or deletion requested
    UPDATING --> OK : Update success
    UPDATING --> ERRED : Update failed
    TERMINATING --> TERMINATED : Deletion success
    TERMINATING --> ERRED : Deletion failed
    ERRED --> OK : Error resolved
    ERRED --> UPDATING : Retry update
    ERRED --> TERMINATING : Force deletion
    TERMINATED --> [*]