Shepherd: Serving DNNs in the Wild

Abstract

Model-serving systems must scale, guarantee high goodput, and maximize compute utilization despite tight latency constraints and unpredictable request streams. Shepherd decouples serving into planning and online-serving modules. The planner aggregates request streams into moderately sized groups whose demand is more predictable, enabling scalable utilization; the online algorithm uses preemption and model-specific batching to guarantee goodput under uncertainty. On production workloads, Shepherd delivers up to 18.1 times higher goodput and 1.8 times better utilization than prior systems while scaling to hundreds of workers.

BibTeX

@inproceedings{shepherd,
  author = {Zhang, Hong and Tang, Yupeng and Khandelwal, Anurag and Stoica, Ion},
  title = {{Shepherd: Serving DNNs in the Wild}},
  booktitle = {USENIX NSDI},
  month = apr,
  year = {2023}
}