Inference System

System for machine learning inference.

Benchmark

• Wanling Gao, Fei Tang, Jianfeng Zhan, et al. "AIBench: A Datacenter AI Benchmark Suite, BenchCouncil".(https://arxiv.org/pdf/2005.03459.pdf)(https://www.benchcouncil.org/AIBench/index.html)
• BaiduBench: Benchmarking Deep Learning operations on different hardware.(https://github.com/baidu-research/DeepBench#inference-benchmark)
• Reddi, Vijay Janapa, et al. "Mlperf inference benchmark." arXiv preprint arXiv:1911.02549 (2019).(https://arxiv.org/pdf/1911.02549.pdf)(https://github.com/mlperf/inference)
• Bianco, Simone, et al. "Benchmark analysis of representative deep neural network architectures." IEEE Access 6 (2018): 64270-64277.(https://arxiv.org/abs/1810.00736)
• Almeida, Mario, et al. "EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices." The 3rd International Workshop on Deep Learning for Mobile Systems and Applications. 2019.(https://arxiv.org/pdf/1905.07346.pdf)

Model Management

• Model Card Toolkit. The Model Card Toolkit (MCT) streamlines and automates generation of Model Cards [1], machine learning documents that provide context and transparency into a model's development and performance.(https://arxiv.org/pdf/1810.03993.pdf)(https://github.com/tensorflow/model-card-toolkit)
• DLHub: Model and data serving for science.(https://arxiv.org/pdf/1811.11213.pdf)
  • Chard, R., Li, Z., Chard, K., Ward, L., Babuji, Y., Woodard, A., Tuecke, S., Blaiszik, B., Franklin, M. and Foster, I., 2019, May.
  • In 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS) (pp. 283-292). IEEE.
• Publishing and Serving Machine Learning Models with DLHub.(https://dl.acm.org/doi/10.1145/3332186.3332246)
• TRAINS - Auto-Magical Experiment Manager & Version Control for AI(https://github.com/allegroai/trains)
• ModelDB: A system to manage ML models(https://github.com/mitdbg/modeldb)(https://mitdbg.github.io/modeldb/papers/hilda_modeldb.pdf)
• iterative/dvc: Data & models versioning for ML projects, make them shareable and reproducible(https://github.com/iterative/dvc)

Model Serving

• Announcing RedisAI 1.0: AI Serving Engine for Real-Time Applications(https://redislabs.com/blog/redisai-ai-serving-engine-for-real-time-applications/)
• Cloudburst: Stateful Functions-as-a-Service. [Paper] [GitHub]
  • Vikram Sreekanti, Chenggang Wu, Xiayue Charles Lin, Johann Schleier-Smith, Joseph E. Gonzalez, Joseph M. Hellerstein, Alexey Tumanov
  • VLDB 2020
  • A stateful FaaS platform. (1) feasibility of general-purpose stateful serverless computing. (2) Autoscaling via logical disaggregation of storage and compute, state management via physical colocation of caches with compute services. (3) LDPC design pattern
• Optimizing Prediction Serving on Low-Latency Serverless Dataflow(https://arxiv.org/pdf/2007.05832.pdf)
  • Sreekanti, Vikram, Harikaran Subbaraj, Chenggang Wu, Joseph E. Gonzalez, and Joseph M. Hellerstein.
  • arXiv preprint arXiv:2007.05832 (2020).
• Serving DNNs like Clockwork: Performance Predictability from the Bottom Up.(https://arxiv.org/pdf/2006.02464.pdf)
  • Gujarati, A., Karimi, R., Alzayat, S., Kaufmann, A., Vigfusson, Y. and Mace, J., 2020.
  • OSDI 2020
• Swayam: distributed autoscaling to meet SLAs of machine learning inference services with resource efficiency(https://www.microsoft.com/en-us/research/uploads/prod/2018/01/2017.Middleware.Swayam.TailLatencyInAzureML.pdf)
  • Gujarati, Arpan, Sameh Elnikety, Yuxiong He, Kathryn S. McKinley, and Björn B. Brandenburg.
  • In Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference, pp. 109-120. 2017.
  • Summary: a cloud autoscaler. (1) model-based autoscaling that takes into account SLAs and ML inference workload characteristics, (2) a distributed protocol that uses partial load information and prediction at frontends to provi- sion new service instances, and (3) a backend self-decommissioning protocol for service instances
• Swift machine learning model serving scheduling: a region based reinforcement learning approach.(https://dl.acm.org/doi/10.1145/3295500.3356164)(https://github.com/SC-RRL/RRL)
  • Qin, Heyang, Syed Zawad, Yanqi Zhou, Lei Yang, Dongfang Zhao, and Feng Yan.
  • In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-23. 2019.
  • Summary: The system performances under different similar con- figurations in a region can be accurately estimated by using the system performance under one of these configurations, due to their similarity. Region based DRL is designed for parallelism selection.
• TorchServe is a flexible and easy to use tool for serving PyTorch models.(https://github.com/pytorch/serve)
• Seldon Core: Blazing Fast, Industry-Ready ML. An open source platform to deploy your machine learning models on Kubernetes at massive scale.(https://github.com/SeldonIO/seldon-core)
• MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving(https://www.usenix.org/system/files/atc19-zhang-chengliang.pdf)(https://github.com/marcoszh/MArk-Project)
  • Zhang, C., Yu, M., Wang, W. and Yan, F., 2019.
  • In 2019 {USENIX} Annual Technical Conference ({USENIX}{ATC} 19) (pp. 1049-1062).
  • Summary: address the scalability and cost minimization issues for model serving on the public cloud.
• Parity Models: Erasure-Coded Resilience for Prediction Serving Systems(SOSP2019)(http://www.cs.cmu.edu/~rvinayak/papers/sosp2019parity-models.pdf)(https://github.com/Thesys-lab/parity-models)
• Nexus: Nexus is a scalable and efficient serving system for DNN applications on GPU cluster (SOSP2019)(https://pdfs.semanticscholar.org/0c0f/353dbac84311ea4f1485d4a8ac0b0459be8c.pdf)(https://github.com/uwsampl/nexus)
• Deep Learning Inference Service at Microsoft(https://www.usenix.org/system/files/opml19papers-soifer.pdf)
  • J Soifer, et al. (OptML2019)
• {PRETZEL}: Opening the Black Box of Machine Learning Prediction Serving Systems.(https://www.usenix.org/system/files/osdi18-lee.pdf)
  • Lee, Y., Scolari, A., Chun, B.G., Santambrogio, M.D., Weimer, M. and Interlandi, M., 2018. (OSDI 2018)
• Brusta: PyTorch model serving project(https://github.com/hyoungseok/brusta)
• Model Server for Apache MXNet: Model Server for Apache MXNet is a tool for serving neural net models for inference(https://github.com/awslabs/mxnet-model-server)
• TFX: A TensorFlow-Based Production-Scale Machine Learning Platform(http://stevenwhang.com/tfx_paper.pdf)(https://www.tensorflow.org/tfx)(https://github.com/tensorflow/tfx)
  • Baylor, Denis, et al. (KDD 2017)
• Tensorflow-serving: Flexible, high-performance ml serving(https://arxiv.org/pdf/1712.06139)(https://github.com/tensorflow/serving)
  • Olston, Christopher, et al.
• IntelAI/OpenVINO-model-server: Inference model server implementation with gRPC interface, compatible with TensorFlow serving API and OpenVINO™ as the execution backend.(https://github.com/IntelAI/OpenVINO-model-server)
• Clipper: A Low-Latency Online Prediction Serving System(https://www.usenix.org/system/files/conference/nsdi17/nsdi17-crankshaw.pdf)(https://github.com/ucbrise/clipper)
  • Crankshaw, Daniel, et al. (NSDI 2017)
  • Summary: Adaptive batch
• InferLine: ML Inference Pipeline Composition Framework(https://arxiv.org/pdf/1812.01776.pdf)(https://github.com/simon-mo/inferline-models)
  • Crankshaw, Daniel, et al. (SoCC 2020)
  • Summary: update version of Clipper
• TrIMS: Transparent and Isolated Model Sharing for Low Latency Deep LearningInference in Function as a Service Environments(https://arxiv.org/pdf/1811.09732.pdf)
  • Dakkak, Abdul, et al (Preprint)
  • Summary: model cold start problem
• Rafiki: machine learning as an analytics service system(http://www.vldb.org/pvldb/vol12/p128-wang.pdf)(https://github.com/nginyc/rafiki)
  • Wang, Wei, Jinyang Gao, Meihui Zhang, Sheng Wang, Gang Chen, Teck Khim Ng, Beng Chin Ooi, Jie Shao, and Moaz Reyad.
  • Summary: Contain both training and inference. Auto-Hype-Parameter search for training. Ensemble models for inference. Using DRL to balance trade-off between accuracy and latency.
• GraphPipe: Machine Learning Model Deployment Made Simple(https://github.com/oracle/graphpipe)
• Orkhon: ML Inference Framework and Server Runtime(https://github.com/vertexclique/orkhon)
• NVIDIA/tensorrt-inference-server: The TensorRT Inference Server provides a cloud inferencing solution optimized for NVIDIA GPUs.(https://github.com/NVIDIA/tensorrt-inference-server)(https://on-demand.gputechconf.com/gtc-cn/2019/pdf/CN9506/presentation.pdf)
• torchpipe: Ensemble Pipeline Serving with Pytorch Frontend. Boosting DL Service Throughput 1.5-4x by Ensemble Pipeline Serving with Concurrent CUDA Streams for PyTorch/LibTorch Frontend and TensorRT/CVCUDA, etc., Backends.(https://github.com/torchpipe/torchpipe)
• INFaaS: Automated Model-less Inference Serving(https://github.com/stanford-mast/INFaaS),(https://www.usenix.org/conference/atc21/presentation/romero)
  • Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis (ATC 2021)
• Llama: A Heterogeneous & Serverless Framework for Auto-Tuning Video Analytics Pipelines
  • Francisco Romero, Mark Zhao, Neeraja J. Yadwadkar, and Christos Kozyrakis (SoCC 2021)
• Scrooge: A Cost-Effective Deep Learning Inference System(https://dl.acm.org/doi/pdf/10.1145/3472883.3486993)
  • Yitao Hu, Rajrup Ghosh, Ramesh Govindan
• Apache PredictionIO® is an open source Machine Learning Server built on top of a state-of-the-art open source stack for developers and data scientists to create predictive engines for any machine learning task(http://predictionio.apache.org/)

Cache for Inference

• Kumar, Adarsh, et al. "Accelerating deep learning inference via freezing." 11th {USENIX} Workshop on Hot Topics in Cloud Computing (HotCloud 19). 2019.(http://shivaram.org/publications/freeze-hotcloud19.pdf)
• Xu, Mengwei, et al. "DeepCache: Principled cache for mobile deep vision." Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. 2018.(https://arxiv.org/pdf/1712.01670.pdf)
• Park, Keunyoung, and Doo-Hyun Kim. "Accelerating image classification using feature map similarity in convolutional neural networks." Applied Sciences 9.1 (2019): 108.(https://www.mdpi.com/2076-3417/9/1/108/htm)
• Cavigelli, Lukas, and Luca Benini. "CBinfer: Exploiting frame-to-frame locality for faster convolutional network inference on video streams." IEEE Transactions on Circuits and Systems for Video Technology (2019).(https://arxiv.org/pdf/1808.05488)

Inference Optimization

• Jointly Optimizing Preprocessing and Inference for DNN-based Visual Analytics(https://arxiv.org/pdf/2007.13005.pdf)
  • Daniel Kang, Ankit Mathur, Teja Veeramacheneni, Peter Bailis, Matei Zaharia
  • VLDB 2021
• Willump: A Statistically-Aware End-to-end Optimizer for Machine Learning Inference.(https://arxiv.org/pdf/1906.01974.pdf)(https://github.com/stanford-futuredata/Willump)
  • Peter Kraft, Daniel Kang, Deepak Narayanan, Shoumik Palkar, Peter Bailis, Matei Zaharia.
  • arXiv Preprint. 2019.
• TensorRT is a C++ library that facilitates high performance inference on NVIDIA GPUs and deep learning accelerators.(https://github.com/NVIDIA/TensorRT)
• Dynamic Space-Time Scheduling for GPU Inference(http://learningsys.org/nips18/assets/papers/102CameraReadySubmissionGPU_Virtualization%20(8).pdf)(https://github.com/ucbrise/caravel)
  • Jain, Paras, et al. (NIPS 18, System for ML)
  • Summary: optimization for GPU Multi-tenancy
• Dynamic Scheduling For Dynamic Control Flow in Deep Learning Systems(http://www.cs.cmu.edu/~jinlianw/papers/dynamic_scheduling_nips18_sysml.pdf)
  • Wei, Jinliang, Garth Gibson, Vijay Vasudevan, and Eric Xing. (On going)
• Accelerating Deep Learning Workloads through Efficient Multi-Model Execution.(https://cs.stanford.edu/~matei/papers/2018/mlsys_hivemind.pdf)
  • D. Narayanan, K. Santhanam, A. Phanishayee and M. Zaharia. (NeurIPS Systems for ML Workshop 2018)
  • Summary: They assume that their system, HiveMind, is given as input models grouped into model batches that are amenable to co-optimization and co-execution. a compiler, and a runtime.
• DeepCPU: Serving RNN-based Deep Learning Models 10x Faster(https://www.usenix.org/system/files/conference/atc18/atc18-zhang-minjia.pdf)
  • Minjia Zhang, Samyam Rajbhandari, Wenhan Wang, and Yuxiong He, Microsoft AI and Research (ATC 2018)

Cluster Management for Inference (now only contain multi-tenant)

• Ease. ml: Towards multi-tenant resource sharing for machine learning workloads(http://www.vldb.org/pvldb/vol11/p607-li.pdf)(https://github.com/DS3Lab/easeml)(http://www.vldb.org/pvldb/vol11/p2054-karlas.pdf)
  • Li, Tian, et al
  • Proceedings of the VLDB Endowment 11.5 (2018): 607-620.
• Perseus: Characterizing Performance and Cost of Multi-Tenant Serving for CNN Models(https://arxiv.org/pdf/1912.02322.pdf)
  • LeMay, Matthew, Shijian Li, and Tian Guo.
  • arXiv preprint arXiv:1912.02322 (2019).

Machine Learning Compiler

• Hummingbird: Hummingbird is a library for compiling trained traditional ML models into tensor computations. Hummingbird allows users to seamlessly leverage neural network frameworks (such as PyTorch) to accelerate traditional ML models.(https://github.com/microsoft/hummingbird)
• {TVM}: An Automated End-to-End Optimizing Compiler for Deep Learning(https://www.usenix.org/system/files/osdi18-chen.pdf)(https://www.youtube.com/watch?v=I1APhlSjVjs)(https://tvm.ai/)
  • Chen, Tianqi, et al. (OSDI 2018)
  • Summary: Automated optimization is very impressive: cost model (rank objective function) + schedule explorer (parallel simulated annealing)
• Facebook TC: Tensor Comprehensions (TC) is a fully-functional C++ library to automatically synthesize high-performance machine learning kernels using Halide, ISL and NVRTC or LLVM.(https://github.com/facebookresearch/TensorComprehensions)
• Tensorflow/mlir: "Multi-Level Intermediate Representation" Compiler Infrastructure(https://github.com/tensorflow/mlir)(https://www.youtube.com/watch?v=qzljG6DKgic)
• PyTorch/glow: Compiler for Neural Network hardware accelerators(https://github.com/pytorch/glow)
• TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions(https://cs.stanford.edu/~matei/papers/2019/sosp_taso.pdf)(https://github.com/jiazhihao/TASO)
  • Jia, Zhihao, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken. (SOSP 2019)
  • Experiments tested on TVM and XLA
• SGLAng: Manage KV cache through radix attention(https://arxiv.org/pdf/2312.07104.pdf)(https://github.com/sgl-project/sglang)
  • Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng