Notation

Throughout this book, we adhere to the following notational conventions. Note that some of these symbols are placeholders, while others refer to specific objects. As a general rule of thumb, the indefinite article "a" often indicates that the symbol is a placeholder and that similarly formatted symbols can denote other objects of the same type. For example, "$x$: a scalar" means that lowercased letters generally represent scalar values, but "$\mathbb{Z}$: the set of integers" refers specifically to the symbol $\mathbb{Z}$.

Numerical Objects

• $x$: a scalar
• $\mathbf{x}$: a vector
• $\mathbf{X}$: a matrix
• $\mathsf{X}$: a general tensor
• $\mathbf{I}$: the identity matrix (of some given dimension), i.e., a square matrix with $1$ on all diagonal entries and $0$ on all off-diagonals
• $x_i$, $[\mathbf{x}]_i$: the $i^\textrm{th}$ element of vector $\mathbf{x}$
• $x_{ij}$, $x_{i,j}$,$[\mathbf{X}]_{ij}$, $[\mathbf{X}]_{i,j}$: the element of matrix $\mathbf{X}$ at row $i$ and column $j$.

Set Theory

• $\mathcal{X}$: a set
• $\mathbb{Z}$: the set of integers
• $\mathbb{Z}^+$: the set of positive integers
• $\mathbb{R}$: the set of real numbers
• $\mathbb{R}^n$: the set of $n$-dimensional vectors of real numbers
• $\mathbb{R}^{a\times b}$: The set of matrices of real numbers with $a$ rows and $b$ columns
• $|\mathcal{X}|$: cardinality (number of elements) of set $\mathcal{X}$
• $\mathcal{A}\cup\mathcal{B}$: union of sets $\mathcal{A}$ and $\mathcal{B}$
• $\mathcal{A}\cap\mathcal{B}$: intersection of sets $\mathcal{A}$ and $\mathcal{B}$
• $\mathcal{A}\setminus\mathcal{B}$: set subtraction of $\mathcal{B}$ from $\mathcal{A}$ (contains only those elements of $\mathcal{A}$ that do not belong to $\mathcal{B}$)

Functions and Operators

• $f(\cdot)$: a function
• $\log(\cdot)$: the natural logarithm (base $e$)
• $\log_2(\cdot)$: logarithm to base $2$
• $\exp(\cdot)$: the exponential function
• $\mathbf{1}(\cdot)$: the indicator function; evaluates to $1$ if the boolean argument is true, and $0$ otherwise
• $\mathbf{1}_{\mathcal{X}}(z)$: the set-membership indicator function; evaluates to $1$ if the element $z$ belongs to the set $\mathcal{X}$ and $0$ otherwise
• $\mathbf{(\cdot)}^\top$: transpose of a vector or a matrix
• $\mathbf{X}^{-1}$: inverse of matrix $\mathbf{X}$
• $\odot$: Hadamard (elementwise) product
• $[\cdot, \cdot]$: concatenation
• $\|\cdot\|_p$: $\ell_p$ norm
• $\|\cdot\|$: $\ell_2$ norm
• $\langle \mathbf{x}, \mathbf{y} \rangle$: inner (dot) product of vectors $\mathbf{x}$ and $\mathbf{y}$
• $\sum$: summation over a collection of elements
• $\prod$: product over a collection of elements
• $\stackrel{\textrm{def}}{=}$: an equality asserted as a definition of the symbol on the left-hand side

Calculus

• $\frac{dy}{dx}$: derivative of $y$ with respect to $x$
• $\frac{\partial y}{\partial x}$: partial derivative of $y$ with respect to $x$
• $\nabla_{\mathbf{x}} y$: gradient of $y$ with respect to $\mathbf{x}$
• $\int_a^b f(x) \;dx$: definite integral of $f$ from $a$ to $b$ with respect to $x$
• $\int f(x) \;dx$: indefinite integral of $f$ with respect to $x$

Probability and Information Theory

• $X$: a random variable
• $P$: a probability distribution
• $X \sim P$: the random variable $X$ follows distribution $P$
• $P(X=x)$: the probability assigned to the event where random variable $X$ takes value $x$
• $P(X \mid Y)$: the conditional probability distribution of $X$ given $Y$
• $p(\cdot)$: a probability density function (PDF) associated with distribution $P$
• ${E}[X]$: expectation of a random variable $X$
• $X \perp Y$: random variables $X$ and $Y$ are independent
• $X \perp Y \mid Z$: random variables $X$ and $Y$ are conditionally independent given $Z$
• $\sigma_X$: standard deviation of random variable $X$
• $\textrm{Var}(X)$: variance of random variable $X$, equal to $\sigma^2_X$
• $\textrm{Cov}(X, Y)$: covariance of random variables $X$ and $Y$
• $\rho(X, Y)$: the Pearson correlation coefficient between $X$ and $Y$, equals $\frac{\textrm{Cov}(X, Y)}{\sigma_X \sigma_Y}$
• $H(X)$: entropy of random variable $X$
• $D_{\textrm{KL}}(P\|Q)$: the KL-divergence (or relative entropy) from distribution $Q$ to distribution $P$

Discussions