SchedulingConfigTemplate

SchedulingConfigTemplate is the Schema for the schedulingconfigtemplates API.

Kubernetes Resource Information

Field	Value
API Version	tensor-fusion.ai/v1
Kind	SchedulingConfigTemplate
Scope	Cluster

Table of Contents

• API Information
• Spec
• Status

Spec

Place the workload to right nodes and scale smart.

Property	Type	Constraints	Description
autoScaling ↓	object		scale the workload based on the usage and traffic
hypervisor ↓	object		single GPU device multi-process queuing and fair scheduling with QoS constraint
placement ↓	object		place the client or worker to best matched nodes
reBalancer ↓	object		avoid hot GPU devices and continuously balance the workload implemented by trigger a simulation scheduling and advise better GPU nodes for scheduler

autoScaling

scale the workload based on the usage and traffic

Properties

Property	Type	Constraints	Description
autoSetLimits ↓	object		layer 1 vertical auto-scaling, turbo burst to existing GPU cards quickly VPA-like, aggregate metrics data <1m
autoSetReplicas ↓	object		layer 2 horizontal auto-scaling, scale up to more GPU cards if max limits threshold hit HPA-like, aggregate metrics data 1m-1h (when tf-worker scaled-up, should also trigger client pod's owner[Deployment etc.]'s replica increasing, check if KNative works)
autoSetRequests ↓	object		layer 3 adjusting, to match the actual usage in the long run, only for N:M remote vGPU mode, not impl yet Adjust baseline requests to match the actual usage in longer period, such as 1day - 2weeks

autoSetLimits

layer 1 vertical auto-scaling, turbo burst to existing GPU cards quickly
VPA-like, aggregate metrics data <1m

Properties

Property	Type	Constraints	Description
enable	boolean
evaluationPeriod	string
extraTFlopsBufferRatio	string
ignoredDeltaRange	string
maxRatioToRequests	string		the multiplier of requests, to avoid limit set too high, like 5.0
prediction ↓	object
scaleUpStep	string
targetResource	string		target resource to scale limits, such as "tflops", "vram", or "all" by default

prediction

Properties

Property	Type	Constraints	Description
enable	boolean
historyDataPeriod	string
model	string
predictionPeriod	string

autoSetReplicas

layer 2 horizontal auto-scaling, scale up to more GPU cards if max limits threshold hit
HPA-like, aggregate metrics data 1m-1h (when tf-worker scaled-up, should also trigger client pod's owner[Deployment etc.]'s replica increasing, check if KNative works)

Properties

Property	Type	Constraints	Description
enable	boolean
evaluationPeriod	string
scaleDownCoolDownTime	string
scaleDownStep	string
scaleUpCoolDownTime	string
scaleUpStep	string
targetTFlopsOfLimits	string

autoSetRequests

layer 3 adjusting, to match the actual usage in the long run, only for N:M remote vGPU mode, not impl yet
Adjust baseline requests to match the actual usage in longer period, such as 1day - 2weeks

Properties

Property	Type	Constraints	Description
aggregationPeriod	string
enable	boolean
evaluationPeriod	string
extraBufferRatio	string		the request buffer ratio, for example actual usage is 1.0, 10% buffer will be 1.1 as final preferred requests
percentileForAutoRequests	string
prediction ↓	object
targetResource	string		target resource to scale requests, such as "tflops", "vram", or "all" by default

prediction

Properties

Property	Type	Constraints	Description
enable	boolean
historyDataPeriod	string
model	string
predictionPeriod	string

hypervisor

single GPU device multi-process queuing and fair scheduling with QoS constraint

Properties

Property	Type	Constraints	Description
autoFreezeAndResume ↓	object		additional layer to save VRAM, auto-freeze memory and cool down to RAM and Disk Hypervisor will monitor and trigger freeze of inactive workers, Operator should mark them as scaled-to-zero and release the GPU pool resources, don't scale down CPU client part, so that they can continue to serve the traffic or scale down by other auto-scaling solutions like KEDA/KNative
multiProcessQueuing ↓	object		Hypervisor will move low priority jobs to pending queue if GPU is full This config can adjust hypervisor's queueing behavior to balance the co-scheduling CUDA calls

autoFreezeAndResume

additional layer to save VRAM, auto-freeze memory and cool down to RAM and Disk
Hypervisor will monitor and trigger freeze of inactive workers, Operator should mark them as scaled-to-zero and release the GPU pool resources, don't scale down CPU client part, so that they can continue to serve the traffic or scale down by other auto-scaling solutions like KEDA/KNative

Properties

Property	Type	Constraints	Description
autoFreeze ↓	array
intelligenceWarmup ↓	object

autoFreeze (items)

Properties

Property	Type	Constraints	Description
enable	boolean
freezeToDiskTTL	string
freezeToMemTTL	string
qos	string	low medium high critical

intelligenceWarmup

Properties

Property	Type	Constraints	Description
enable	boolean
historyDataPeriod	string
model	string
predictionPeriod	string

multiProcessQueuing

Hypervisor will move low priority jobs to pending queue if GPU is full
This config can adjust hypervisor's queueing behavior to balance the co-scheduling CUDA calls

Properties

Property	Type	Constraints	Description
enable	boolean
interval	string
queueLevelTimeSlices	array

placement

place the client or worker to best matched nodes

Properties

Property	Type	Constraints	Description
allowUsingLocalGPU	boolean		Default: true
gpuFilters ↓	array
mode	string	CompactFirst LowLoadFirst	Default: CompactFirst

gpuFilters (items)

Properties

Property	Type	Constraints	Description
params	object
type	string

reBalancer

avoid hot GPU devices and continuously balance the workload
implemented by trigger a simulation scheduling and advise better GPU nodes for scheduler

Properties

Property	Type	Constraints	Description
enable	boolean
interval	string
reBalanceCoolDownTime	string
threshold ↓	object

threshold

Properties

Property	Type	Constraints	Description
matchAny	object

Status

SchedulingConfigTemplateStatus defines the observed state of SchedulingConfigTemplate.