大规模并行处理器程序设计（英文版·原书第4版） / 经典原版书库
定价：￥129.00
作者： [美]胡文美（Wen-mei W. Hwu） [美]大卫·B. 柯克（David B. Kirk） [黎巴嫩]伊扎特·埃尔·哈吉（Izzat El Hajj）
出版时间：2025-04-01
出版社：机械工业出版社
- 机械工业出版社
- 9787111774716
- 1-1
- 2025-04-01
- 979
内容简介
本书内容简洁、直观、实用，强调计算思维能力和并行编程技巧。本书主要分为四个部分：第 一部分介绍异构并行计算编程的基础概念，包括数据并行化、GPU架构、CUDA编程及程序性能优化方法等内容；第二部分介绍并行模式，包括卷积、模板、并行直方图、归约、前缀和、归并等内容；第三部分介绍高级模式及应用，包括排序、稀疏矩阵计算、图遍历、深度学习、迭代式磁共振成像重建、静电势能图和计算思维等内容；第四部分介绍高级编程实践，包括异构计算集群编程、CUDA动态并行化等内容。本书不仅适合高等院校计算机相关专业的学生学习，也适合并行计算领域的技术人员参考。
目录
Contents
Foreword
Preface
Acknowledgments
CHAPTER 1 Introduction 1
1.1 Heterogeneous parallel computing 3
1.2 Why more speed or parallelism? 7
1.3 Speeding up real applications 9
1.4 Challenges in parallel programming 11
1.5 Related parallel programming interfaces 13
1.6 Overarching goals 14
1.7 Organization of the book 15
References 19
Part I Fundamental Concepts
CHAPTER 2 Heterogeneous data parallel computing 23
With special contribution from David Luebke
2.1 Data parallelism 23
2.2 CUDA C program structure 27
2.3 A vector addition kernel 28
2.4 Device global memory and data transfer 31
2.5 Kernel functions and threading 35
2.6 Calling kernel functions 40
2.7 Compilation 42
2.8 Summary 43
Exercises 44
References 46
CHAPTER 3 Multidimensional grids and data 47
3.1 Multidimensional grid organization 47
3.2 Mapping threads to multidimensional data 51
3.3 Image blur: a more complex kernel 58
3.4 Matrix multiplication 62
3.5 Summary 66
Exercises 67
CHAPTER 4 Compute architecture and scheduling 69
4.1 Architecture of a modern GPU 70
4.2 Block scheduling 70
4.3 Synchronization and transparent scalability 71
4.4 Warps and SIMD hardware 74
4.5 Control divergence 79
4.6 Warp scheduling and latency tolerance 83
4.7 Resource partitioning and occupancy 85
4.8 Querying device properties 87
4.9 Summary 90
Exercises 90
References 92
CHAPTER 5 Memory architecture and data locality 93
5.1 Importance of memory access efficiency 94
5.2 CUDA memory types 96
5.3 Tiling for reduced memory traffic 103
5.4 A tiled matrix multiplication kernel 107
5.5 Boundary checks 112
5.6 Impact of memory usage on occupancy 115
5.7 Summary 118
Exercises 119
CHAPTER 6 Performance considerations 123
6.1 Memory coalescing 124
6.2 Hiding memory latency 133
6.3 Thread coarsening 138
6.4 A checklist of optimizations 141
6.5 Knowing your computation’s bottleneck 145
6.6 Summary 146
Exercises 146
References 147
Part II Parallel Patterns
CHAPTER 7 Convolution
An introduction to constant memory and caching 151
7.1 Background 152
7.2 Parallel convolution: a basic algorithm 156
7.3 Constant memory and caching 159
7.4 Tiled convolution with halo cells 163
7.5 Tiled convolution using caches for halo cells 168
7.6 Summary 170
Exercises 171
CHAPTER 8 Stencil 173
8.1 Background 174
8.2 Parallel stencil: a basic algorithm 178
8.3 Shared memory tiling for stencil sweep 179
8.4 Thread coarsening 183
8.5 Register tiling 186
8.6 Summary 188
Exercises 188
CHAPTER 9 Parallel histogram 191
9.1 Background 192
9.2 Atomic operations and a basic histogram kernel 194
9.3 Latency and throughput of atomic operations 198
9.4 Privatization 200
9.5 Coarsening 203
9.6 Aggregation 206
9.7 Summary 208
Exercises 209
References 210
CHAPTER 10 Reduction
And minimizing divergence 211
10.1 Background 211
10.2 Reduction trees 213
10.3 A simple reduction kernel 217
10.4 Minimizing control divergence 219
10.5 Minimizing memory divergence 223
10.6 Minimizing global memory accesses 225
10.7 Hierarchical reduction for arbitrary input length 226
10.8 Thread coarsening for reduced overhead 228
10.9 Summary 231
Exercises 232
CHAPTER 11 Prefix sum (scan)
An introduction to work efficiency in parallel algorithms 235
With special contributions from Li-Wen Chang, Juan Go′mez-Luna and John Owens
11.1 Background 236
11.2 Parallel scan with the Kogge-Stone algorithm 238
11.3 Speed and work efficiency consideration 244
11.4 Parallel scan with the Brent-Kung algorithm 246
11.5 Coarsening for even more work efficiency 251
11.6 Segmented parallel scan for arbitrary-length inputs 253
11.7 Single-pass scan for memory access efficiency 256
11.8 Summary 259
Exercises 260
References 261
CHAPTER 12 Merge
An introduction to dynamic input data identification 263
With special contributions from Li-Wen Chang and Jie Lv
12.1 Background 263
12.2 A sequential merge algorithm 265
12.3 A parallelization approach 266
12.4 Co-rank function implementation 268
12.5 A basic parallel merge kernel 273
12.6 A tiled merge kernel to improve coalescing 275
12.7 A circular buffer merge kernel 282
12.8 Thread coarsening for merge 288
12.9 Summary 288
Exercises 289
References 289
Part III Advanced Patterns and Applications
CHAPTER 13 Sorting 293
With special contributions from Michael Garland
13.1 Background 294
13.2 Radix sort 295
13.3 Parallel radix sort 296
13.4 Optimizing for memory coalescing 300
13.5 Choice of radix value 302
13.6 Thread coarsening to improve coalescing 305
13.7 Parallel merge sort 306
13.8 Other parallel sort methods 308
13.9 Summary 309
Exercises 310
References 310
CHAPTER 14 Sparse matrix computation 311
14.1 Background 312
14.2 A simple SpMV kernel with the COO format 314
14.3 Grouping row nonzeros with the CSR format 317
14.4 Improving memory coalescing with the ELL format 320
14.5 Regulating padding with the hybrid ELL-COO format 324
14.6 Reducing control divergence with the JDS format 325
14.7 Summary 328
Exercises 329
References 329
CHAPTER 15 Graph traversal 331
With special contributions from John Owens and Juan Go′mez-Luna
15.1 Background 332
15.2 Breadth-first search 335
15.3 Vertex-centric parallelization of breadth-first search 338
15.4 Edge-centric parallelization of breadth-first search 343
15.5 Improving efficiency with frontiers.345
15.6 Reducing contention with privatization 348
15.7 Other optimizations 350
15.8 Summary 352
Exercises 353
References 354
CHAPTER 16 Deep learning 355
With special contributions from Carl Pearson and Boris Ginsburg
16.1 Background 356
16.2 Convolutional neural networks 366
16.3 Convolutional layer: a CUDA inference kernel 376
16.4 Formulating a convolutional layer as GEMM 379
16.5 CUDNN library 385
16.6 Summary 387
Exercises 388
References 388
CHAPTER 17 Iterative magnetic resonance imaging reconstruction 391
17.1 Background 391
17.2 Iterative reconstruction 394
17.3 Computing FHD 396
17.4 Summary 412
Exercises 413
References 414
CHAPTER 18 Electrostatic potential map 415
With special contributions from John Stone
18.1 Background 415
18.2 Scatter versus gather in kernel design 417
18.3 Thread coarsening 422
18.4 Memory coalescing 424
18.5 Cutoff binning for data size scalability 425
18.6 Summary.430
Exercises 431
References 431
CHAPTER 19 Parallel programming and computational thinking 433
19.1 Goals of parallel computing 433
19.2 Algorithm selection 436
19.3 Problem decomposition 440
19.4 Computational thinking 444
19.5 Summary 446
References 446
Part IV Advanced Practices
CHAPTER 20 Programming a heterogeneous computing cluster
An introduction to CUDA streams 449
With special contributions from Isaac Gelado and Javier Cabezas
20.1 Background 449
20.2 A running example 450
20.3 Message passing interface basics 452
20.4 Message passing interface point-to-point communication 455
20.5 Overlapping computation and communication 462
20.6 Message passing interface collective communication 470
20.7 CUDA aware message passing interface 471
20.8 Summary 472
Exercises 472
References 473
CHAPTER 21 CUDA dynamic parallelism 475
With special contributions from Juan Go′mez-Luna
21.1 Background 476
21.2 Dynamic parallelism overview 478
21.3 An example: Bezier curves 481
21.4 A recursive example: quadtrees 484
21.5 Important considerations 490
21.6 Summary 492
Exercises 493
A21.1 Support code for quadtree example 495
References 497
CHAPTER 22 Advanced practices and future evolution 499
With special contributions from Isaac Gelado and Mark Harris
22.1 Model of host/device interaction 500
22.2 Kernel execution control 505
22.3 Memory bandwidth and compute throughput 508
22.4 Programming environment 510
22.5 Future outlook 513
References 513
CHAPTER 23 Conclusion and outlook 515
23.1 Goals revisited 515
23.2 Future outlook 516
Appendix A: Numerical considerations 519
Index 537
Foreword
Preface
Acknowledgments
CHAPTER 1 Introduction 1
1.1 Heterogeneous parallel computing 3
1.2 Why more speed or parallelism? 7
1.3 Speeding up real applications 9
1.4 Challenges in parallel programming 11
1.5 Related parallel programming interfaces 13
1.6 Overarching goals 14
1.7 Organization of the book 15
References 19
Part I Fundamental Concepts
CHAPTER 2 Heterogeneous data parallel computing 23
With special contribution from David Luebke
2.1 Data parallelism 23
2.2 CUDA C program structure 27
2.3 A vector addition kernel 28
2.4 Device global memory and data transfer 31
2.5 Kernel functions and threading 35
2.6 Calling kernel functions 40
2.7 Compilation 42
2.8 Summary 43
Exercises 44
References 46
CHAPTER 3 Multidimensional grids and data 47
3.1 Multidimensional grid organization 47
3.2 Mapping threads to multidimensional data 51
3.3 Image blur: a more complex kernel 58
3.4 Matrix multiplication 62
3.5 Summary 66
Exercises 67
CHAPTER 4 Compute architecture and scheduling 69
4.1 Architecture of a modern GPU 70
4.2 Block scheduling 70
4.3 Synchronization and transparent scalability 71
4.4 Warps and SIMD hardware 74
4.5 Control divergence 79
4.6 Warp scheduling and latency tolerance 83
4.7 Resource partitioning and occupancy 85
4.8 Querying device properties 87
4.9 Summary 90
Exercises 90
References 92
CHAPTER 5 Memory architecture and data locality 93
5.1 Importance of memory access efficiency 94
5.2 CUDA memory types 96
5.3 Tiling for reduced memory traffic 103
5.4 A tiled matrix multiplication kernel 107
5.5 Boundary checks 112
5.6 Impact of memory usage on occupancy 115
5.7 Summary 118
Exercises 119
CHAPTER 6 Performance considerations 123
6.1 Memory coalescing 124
6.2 Hiding memory latency 133
6.3 Thread coarsening 138
6.4 A checklist of optimizations 141
6.5 Knowing your computation’s bottleneck 145
6.6 Summary 146
Exercises 146
References 147
Part II Parallel Patterns
CHAPTER 7 Convolution
An introduction to constant memory and caching 151
7.1 Background 152
7.2 Parallel convolution: a basic algorithm 156
7.3 Constant memory and caching 159
7.4 Tiled convolution with halo cells 163
7.5 Tiled convolution using caches for halo cells 168
7.6 Summary 170
Exercises 171
CHAPTER 8 Stencil 173
8.1 Background 174
8.2 Parallel stencil: a basic algorithm 178
8.3 Shared memory tiling for stencil sweep 179
8.4 Thread coarsening 183
8.5 Register tiling 186
8.6 Summary 188
Exercises 188
CHAPTER 9 Parallel histogram 191
9.1 Background 192
9.2 Atomic operations and a basic histogram kernel 194
9.3 Latency and throughput of atomic operations 198
9.4 Privatization 200
9.5 Coarsening 203
9.6 Aggregation 206
9.7 Summary 208
Exercises 209
References 210
CHAPTER 10 Reduction
And minimizing divergence 211
10.1 Background 211
10.2 Reduction trees 213
10.3 A simple reduction kernel 217
10.4 Minimizing control divergence 219
10.5 Minimizing memory divergence 223
10.6 Minimizing global memory accesses 225
10.7 Hierarchical reduction for arbitrary input length 226
10.8 Thread coarsening for reduced overhead 228
10.9 Summary 231
Exercises 232
CHAPTER 11 Prefix sum (scan)
An introduction to work efficiency in parallel algorithms 235
With special contributions from Li-Wen Chang, Juan Go′mez-Luna and John Owens
11.1 Background 236
11.2 Parallel scan with the Kogge-Stone algorithm 238
11.3 Speed and work efficiency consideration 244
11.4 Parallel scan with the Brent-Kung algorithm 246
11.5 Coarsening for even more work efficiency 251
11.6 Segmented parallel scan for arbitrary-length inputs 253
11.7 Single-pass scan for memory access efficiency 256
11.8 Summary 259
Exercises 260
References 261
CHAPTER 12 Merge
An introduction to dynamic input data identification 263
With special contributions from Li-Wen Chang and Jie Lv
12.1 Background 263
12.2 A sequential merge algorithm 265
12.3 A parallelization approach 266
12.4 Co-rank function implementation 268
12.5 A basic parallel merge kernel 273
12.6 A tiled merge kernel to improve coalescing 275
12.7 A circular buffer merge kernel 282
12.8 Thread coarsening for merge 288
12.9 Summary 288
Exercises 289
References 289
Part III Advanced Patterns and Applications
CHAPTER 13 Sorting 293
With special contributions from Michael Garland
13.1 Background 294
13.2 Radix sort 295
13.3 Parallel radix sort 296
13.4 Optimizing for memory coalescing 300
13.5 Choice of radix value 302
13.6 Thread coarsening to improve coalescing 305
13.7 Parallel merge sort 306
13.8 Other parallel sort methods 308
13.9 Summary 309
Exercises 310
References 310
CHAPTER 14 Sparse matrix computation 311
14.1 Background 312
14.2 A simple SpMV kernel with the COO format 314
14.3 Grouping row nonzeros with the CSR format 317
14.4 Improving memory coalescing with the ELL format 320
14.5 Regulating padding with the hybrid ELL-COO format 324
14.6 Reducing control divergence with the JDS format 325
14.7 Summary 328
Exercises 329
References 329
CHAPTER 15 Graph traversal 331
With special contributions from John Owens and Juan Go′mez-Luna
15.1 Background 332
15.2 Breadth-first search 335
15.3 Vertex-centric parallelization of breadth-first search 338
15.4 Edge-centric parallelization of breadth-first search 343
15.5 Improving efficiency with frontiers.345
15.6 Reducing contention with privatization 348
15.7 Other optimizations 350
15.8 Summary 352
Exercises 353
References 354
CHAPTER 16 Deep learning 355
With special contributions from Carl Pearson and Boris Ginsburg
16.1 Background 356
16.2 Convolutional neural networks 366
16.3 Convolutional layer: a CUDA inference kernel 376
16.4 Formulating a convolutional layer as GEMM 379
16.5 CUDNN library 385
16.6 Summary 387
Exercises 388
References 388
CHAPTER 17 Iterative magnetic resonance imaging reconstruction 391
17.1 Background 391
17.2 Iterative reconstruction 394
17.3 Computing FHD 396
17.4 Summary 412
Exercises 413
References 414
CHAPTER 18 Electrostatic potential map 415
With special contributions from John Stone
18.1 Background 415
18.2 Scatter versus gather in kernel design 417
18.3 Thread coarsening 422
18.4 Memory coalescing 424
18.5 Cutoff binning for data size scalability 425
18.6 Summary.430
Exercises 431
References 431
CHAPTER 19 Parallel programming and computational thinking 433
19.1 Goals of parallel computing 433
19.2 Algorithm selection 436
19.3 Problem decomposition 440
19.4 Computational thinking 444
19.5 Summary 446
References 446
Part IV Advanced Practices
CHAPTER 20 Programming a heterogeneous computing cluster
An introduction to CUDA streams 449
With special contributions from Isaac Gelado and Javier Cabezas
20.1 Background 449
20.2 A running example 450
20.3 Message passing interface basics 452
20.4 Message passing interface point-to-point communication 455
20.5 Overlapping computation and communication 462
20.6 Message passing interface collective communication 470
20.7 CUDA aware message passing interface 471
20.8 Summary 472
Exercises 472
References 473
CHAPTER 21 CUDA dynamic parallelism 475
With special contributions from Juan Go′mez-Luna
21.1 Background 476
21.2 Dynamic parallelism overview 478
21.3 An example: Bezier curves 481
21.4 A recursive example: quadtrees 484
21.5 Important considerations 490
21.6 Summary 492
Exercises 493
A21.1 Support code for quadtree example 495
References 497
CHAPTER 22 Advanced practices and future evolution 499
With special contributions from Isaac Gelado and Mark Harris
22.1 Model of host/device interaction 500
22.2 Kernel execution control 505
22.3 Memory bandwidth and compute throughput 508
22.4 Programming environment 510
22.5 Future outlook 513
References 513
CHAPTER 23 Conclusion and outlook 515
23.1 Goals revisited 515
23.2 Future outlook 516
Appendix A: Numerical considerations 519
Index 537
