Resource Quotas
Admin-level compute resource quota management to prevent resource contention
Resource Quotas
The resource quota system allows administrators to set upper limits on CPU, memory, GPU, and container instance usage at both the platform and project levels, preventing any single project or user from monopolizing resources.
Overview
Resource quotas operate at two levels:
- Global quota: Caps the total resource usage across all projects
- Project quota: Caps per-project resource usage (defaults are derived from the global quota)
When a project attempts to enable or start a container, the system checks quota availability. If resources are insufficient, the request is rejected with a clear bottleneck message.
Admin Configuration
Entry: Admin → Resource Quotas (/workspace/admin/resource-quotas)
Settings Tab
| Setting | Description |
|---|---|
| Quota status | Enable or disable quota enforcement |
| Max CPU cores | Total CPU cores allowed across all containers |
| Max Memory (MB) | Total memory allowed across all containers |
| Max Persistent Containers | Maximum number of concurrently running persistent containers |
| Max Ephemeral Containers | Maximum number of concurrently running ephemeral Agent containers |
| Max GPU Count | Total GPU devices allowed across all containers |
Default values are auto-calculated from host hardware (typically 80% of host CPUs and 75% of host RAM).
Running Containers Tab
A paginated table showing all currently running containers:
- Project name
- Container image
- CPU, memory, and GPU resource allocation
- Last active time
Rejected Requests Tab
Rejected quota events help admins understand bottlenecks:
- Rejection time and reason
- The bottleneck resource type
- The affected project
Resource Lease Model
The system uses a lease-based model for resource management:
- Containers acquire leases for their respective resources when enabled/started
- Leases are released when containers are stopped/disabled
- Leases are tracked per resource type (CPU, memory, GPU, container instances)
- Lease operations include concurrency locking to prevent race conditions
Waiting for Resources and Auto-Wake
When a container cannot start due to quota limits:
- The container enters a
waiting_for_resourcesstate - The workspace header status indicator shows an amber alert with bottleneck details
- When other containers stop or quotas are increased, the system automatically retries waiting containers (resource wake)
- Waiting Agent tickets are also automatically woken when resources become available
State Sync After Host Restart
If the host machine restarts:
- The system automatically detects the actual status of containers across all container-enabled projects
- Containers that are still running have their resource leases re-acquired
- Lost containers are marked as stopped
- Container states and resource leases remain consistent
State sync also applies to Kubernetes backends:
- Missing persistent-container Pods / Jobs are marked stopped or errored
- Stopped, disabled, or completed runs release their leases
- The status controller checks abnormal states on a configured interval to reduce manual cleanup
Project-Level Quotas
Project settings' container panel also allows viewing and adjusting project-level resource limits:
- Managers and owners can tune per-project CPU, memory, and GPU limits
- Project quotas cannot exceed global quotas
- Useful for allocating proportional resources across multiple projects
Difference from Storage Quotas
This page manages runtime compute resources: CPU, memory, GPU, persistent container slots, and ephemeral Agent container slots.
Project file capacity is a storage quota. In deployments with PROJECT_STORAGE_QUOTA=juicefs, JuiceFS manages those limits:
- Project lists and project settings can show used storage and quota limit
- Local workspace uploads that exceed storage quota return a project-storage-quota exceeded message
- Raising storage quota does not increase the number of containers that can run; running containers are still controlled by the resource quotas on this page
Typical Scenarios
Case 1: Shared Resources Across Multiple Projects
- Admin sets global quotas based on server capacity
- Higher-priority projects receive larger project quotas
- Lower-priority projects are capped, ensuring core projects always have resources
Case 2: Preventing Single-Project Monopolization
- Set reasonable per-project quota limits
- No single project can start containers beyond its quota
- Other projects always have resources available