binacs.space - Blogs

godel_collection-of-public-articles-related-to-godel.md

Collection of Public Articles Related to Gödel

Gödel System

About Colocation

The left figure below describes the resource allocation of online and offline tasks in a cluster over a period of time. In the initial stage, online tasks do not consume many resources, and a large amount of computing resources are allocated to offline tasks with lower priority. When the resource demand of online tasks surges due to a special event (emergency, hot search, etc.), Gödel immediately allocates resources to online tasks, and the resource allocation of offline tasks decreases rapidly. After the peak, online tasks begin to reduce resource requests, and the scheduler shifts resources to offline tasks again. By combining offline pools and dynamic resource transfer, ByteDance can always maintain a high resource utilization rate. During the evening peak hours, the average resource rate of the cluster reaches more than 60%, and it can also be maintained at around 40% during the daytime trough stage.

colocation

ML on Gödel

BigData on Gödel

Paper link: Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance

Paper link: ResLake: Towards Minimum Job Latency and Balanced Resource Utilization in Geo-distributed Job Scheduling

High-level Evolution

Hybrid Deployment (Colocation)