加载中...

SUPPORT

觉得内容有帮助？请站长喝杯咖啡 ☕

网站所有内容免费，但运维不免费 😭 AI、服务器、域名……你的支持将用来维持网站正常运行，让这个小站还能继续写下去~

微信赞助收款二维码

打开微信扫一扫 · 金额随意

首页知识库教程工具博客资讯术语表面试题库关于?个人中心

← 返回课程

Machine Learning Engineering Open Book

47 / 52·markdown

文档

01Machine Learning Engineering Open Book
42Contributors
44Machine Learning Engineering
52TODO

insights

02The AI Battlefield Engineering - What You Need To Know
03How to Choose a Cloud Provider
04When Is It Worth Upgrading GPUs?

compute

05Compute
06Accelerators
07Accelerator Benchmarks
08Troubleshooting NVIDIA GPUs
09Troubleshooting AMD GPUs
10AMD GPUs Performance
11CPU
12CPU Memory

storage

13Storage: File Systems and IO

network

14Inter-Node and Intra-Node Networking Hardware
15Communication Patterns
16Network Debug
17Networking Benchmarks

orchestration

18Orchestration
19Working in SLURM Environment
20SLURM Administration
21SLURM for Users
22SLURM Performance
23Single and Multi-Node Launchers with SLURM
24Kubernetes

training

25Training
26Model Parallelism
27Software Tune Up For The Best Performance
28Fault Tolerance
29Reproducibility
30Avoiding, Recovering From and Understanding Instabilities
31Understanding Training Loss Patterns
32Checkpoints
33Selecting Training Hyper-Parameters And Model Initializations
34Tensor Precision / Data Types
35Emulate a Multi-Node Setup Using Just a Single Node
36Re-Train HF Hub Models From Scratch Using Finetuning Examples

inference

37Inference

debug

38Debugging and Troubleshooting
39Debugging PyTorch Programs
40A Backup of Scripts
45Faster Debug and Development with Tiny Models, Tokenizers and Datasets
46NCCL: Debug and Performance
47Diagnosing Hangings and Deadlocks in Multi-Node Multi-GPU Python Programs
48Underflow and Overflow Detection

testing

41Writing and Running Tests

courses

43Lessons Learned from Training LLMs (a course)

model parallelism

49Model Parallelism

stabs

50Stabs
51Things to add / integrate

Chapter 47

Diagnosing Hangings and Deadlocks in Multi-Node Multi-GPU Python Programs

Diagnosing Hangings and Deadlocks in Multi-Node Multi-GPU Python Programs

Moved to Debugging PyTorch Programs.

Helper tools

Moved to Debugging PyTorch Programs.

Approaches to diagnosing multi-gpu hanging / deadlocks

Moved to Debugging PyTorch Programs.

py-spy

Moved to Debugging PyTorch Programs.

multi-process py-spy

Moved to Debugging PyTorch Programs.

multi-node py-spy via srun

Moved to Debugging PyTorch Programs.

multi-node py-spy via pdsh

Moved to Debugging PyTorch Programs.

multi-node py-spy via ds_ssh

Moved to Debugging PyTorch Programs.

Network-level hanging

Moved to Debugging PyTorch Programs.

Isolate problematic GPUs

Moved to Debugging PyTorch Programs.

python `trace`

Moved to Debugging PyTorch Programs.

NicerTrace

Moved to Debugging PyTorch Programs.

Working with generated trace files

Moved to Debugging PyTorch Programs.

good old `print`

Moved to Debugging PyTorch Programs.

core files

Moved to Debugging PyTorch Programs.

Code loops

Moved to Debugging PyTorch Programs.

Hardware-specific issues

Moved to Debugging PyTorch Programs.

AMD/ROCm hangs or slow with IOMMU enabled

Moved to Debugging PyTorch Programs.

上一章

NCCL: Debug and Performance

下一章

Underflow and Overflow Detection

本页镜像自上游仓库，固定版本 7bc7c305。许可证：需人工确认。原文只读，笔记与划线保存在你的助记词身份下。

阅读进度

0%

整课进度0/52 章 · 0%

快捷键：←/→ 换章 · C 完成 · F 专注 · Esc 退出

阅读设置

字号

行宽

行距

本章笔记

选中正文可划线或写笔记。随助记词身份同步，仅自己可见。

暂无笔记。选中一段文字试试。

本章导航

Helper tools
Approaches to diagnosing multi-gpu hanging / deadlocks
py-spy
multi-process py-spy
multi-node py-spy via srun
multi-node py-spy via pdsh
multi-node py-spy via ds_ssh
Network-level hanging
Isolate problematic GPUs
python `trace`
NicerTrace
Working with generated trace files
good old `print`
core files
Code loops
Hardware-specific issues
AMD/ROCm hangs or slow with IOMMU enabled

中文 AI 学习与实践平台。知识库、教程、工具与资讯集中一站，帮你把人工智能真正学到手。

联系support.aimaster.cc@gmail.com

意见反馈 · 内容纠错 · 需求交流

探索

知识库
教程
工具
博客
资讯
术语表
面试题库
关于我们
隐私政策
服务条款

专题

精选教程
机器学习
深度学习
自然语言处理
计算机视觉

© 2026 AI Master. All rights reserved.

隐私政策·服务条款·关于我们·精通人工智能 · ai-master.cc