Contents

Background and Preview 1

1. The Filtering Problem 1

2. Linear Optimum Filters 4

3. Adaptive Filters 4

4. Linear Filter Structures 6

5. Approaches to the Development of Linear Adaptive Filters 12

6. Adaptive Beamforming 13

7. Four Classes of Applications 17

8. Historical Notes 20

Chapter 1 Stochastic Processes and Models 30

1.1 Partial Characterization of a Discrete-Time Stochastic Process 30

1.2 Mean Ergodic Theorem 32

1.3 Correlation Matrix 34

1.4 Correlation Matrix of Sine Wave Plus Noise 39

1.5 Stochastic Models 40

1.6 Wold Decomposition 46

1.7 Asymptotic Stationarity of an Autoregressive Process 49

1.8 Yule–Walker Equations 51

1.9 Computer Experiment: Autoregressive Process of Order Two 52

1.10 Selecting the Model Order 60

1.11 Complex Gaussian Processes 63

1.12 Power Spectral Density 65

1.13 Properties of Power Spectral Density 67

1.14 Transmission of a Stationary Process Through a Linear Filter 69

1.15 Cramér Spectral Representation for a Stationary Process 72

1.16 Power Spectrum Estimation 74

1.17 Other Statistical Characteristics of a Stochastic Process 77

1.18 Polyspectra 78

1.19 Spectral-Correlation Density 81

1.20 Summary and Discussion 84

Problems 85

Chapter 2 Wiener Filters 90

2.1 Linear Optimum Filtering: Statement of the Problem 90

2.2 Principle of Orthogonality 92

2.3 Minimum Mean-Square Error 96

2.4 Wiener–Hopf Equations 98

2.5 Error-Performance Surface 100

2.6 Multiple Linear Regression Model 104

2.7 Example 106

2.8 Linearly Constrained Minimum-Variance Filter 111

2.9 Generalized Sidelobe Cancellers 116

2.10 Summary and Discussion 122

Problems 124

Chapter 3 Linear Prediction 132

3.1 Forward Linear Prediction 132

3.2 Backward Linear Prediction 139

3.3 Levinson–Durbin Algorithm 144

3.4 Properties of Prediction-Error Filters 153

3.5 Schur–Cohn Test 162

3.6 Autoregressive Modeling of a Stationary Stochastic Process 164

3.7 Cholesky Factorization 167

3.8 Lattice Predictors 170

3.9 All-Pole, All-Pass Lattice Filter 175

3.10 Joint-Process Estimation 177

3.11 Predictive Modeling of Speech 181

3.12 Summary and Discussion 188

Problems 189

Chapter 4 Method of Steepest Descent 199

4.1 Basic Idea of the Steepest-Descent Algorithm 199

4.2 The Steepest-Descent Algorithm Applied to the Wiener Filter 200

4.3 Stability of the Steepest-Descent Algorithm 204

4.4 Example 209

4.5 The Steepest-Descent Algorithm Viewed as a Deterministic Search Method 221

4.6 Virtue and Limitation of the Steepest-Descent Algorithm 222

4.7 Summary and Discussion 223

Problems 224

Chapter 5 Method of Stochastic Gradient Descent 228

5.1 Principles of Stochastic Gradient Descent 228

5.2 Application 1: Least-Mean-Square (LMS) Algorithm 230

5.3 Application 2: Gradient-Adaptive Lattice Filtering Algorithm 237

5.4 Other Applications of Stochastic Gradient Descent 244

5.5 Summary and Discussion 245

Problems 246

Chapter 6 The Least-Mean-Square (LMS) Algorithm 248

6.1 Signal-Flow Graph 248

6.2 Optimality Considerations 250

6.3 Applications 252

6.4 Statistical Learning Theory 272

6.5 Transient Behavior and Convergence Considerations 283

6.6 Efficiency 286

6.7 Computer Experiment on Adaptive Prediction 288

6.8 Computer Experiment on Adaptive Equalization 293

6.9 Computer Experiment on a Minimum-Variance Distortionless-Response

Beamformer

302

6.10 Summary and Discussion 306

Problems 308

Chapter 7 Normalized Least-Mean-Square (LMS) Algorithm and Its

Generalization 315

7.1 Normalized LMS Algorithm: The Solution to a Constrained Optimization Problem 315

7.2 Stability of the Normalized LMS Algorithm 319

7.3 Step-Size Control for Acoustic Echo Cancellation 322

7.4 Geometric Considerations Pertaining to the Convergence Process for Real-Valued

Data 327

7.5 Affine Projection Adaptive Filters 330

7.6 Summary and Discussion 334

Problems 335

Chapter 8 Block-Adaptive Filters 339

8.1 Block-Adaptive Filters: Basic Ideas 340

8.2 Fast Block LMS Algorithm 344

8.3 Unconstrained Frequency-Domain Adaptive Filters 350

8.4 Self-Orthogonalizing Adaptive Filters 351

8.5 Computer Experiment on Adaptive Equalization 361

8.6 Subband Adaptive Filters 367

8.7 Summary and Discus