Intuitively, the matrix 2-norm measures the greatest action achievable by applying A to x. A = UΣV*, where U and V are orthogonal matrices and Σ is a non-negative diagonal matrix. This is the full singular value decomposition of A. If A is hermitian, then U = V. Basically, A is equivalent to an orthonormal basis change (U) followed by scaling (Σ) and another orthonormal basis change (V). The L^2 norm of a vector is equivalent to the "energy" of a discrete signal in a signal processing sense up to a power:
L^2 norm of vector: (xTx)1/2
Energy of discrete signal: xTx
Orthogonal transforms, like rotations and reflections, preserve norms by Parseval's theorem:
(Qx)*(Qx)=x*Q*Qx = x*x
This is why the induced matrix 2-norm of matrix A is equal to its biggest singular value. The matrix 2-norm measures the greatest action achievable by applying A:
||A||2 = supx∈Cn (||Ax||2)/(||x||2)
Basically, the matrix 2-norm measures the greatest action achievable by applying A. Because U and V preserve the L^2 norm, the supremum is achieved, when all of a discrete signal/ vector's energy is scaled by the largest singular value.
||A||2 = supx∈Cn, ||x||2=1 (||Ax||2)
||A||2 = supx∈Cn, ||x||2=1 (||Σx||2)
The unit vector that achieves this largest scaling is the right singular vector v1, or the first column of V.
Notice, that V*v1 = e1, and Σe1 = σmax(A), which is the largest singular value of A.
This is relevant to lipschitz continuous functions: functions f : Rn to Rm where the following holds for some K ≥ 0 for all x1, x2 in Rn:
||f(x1) - f(x2)||2 / ||x1 - x2||2 ≤ K
If f is linear, then f(x1) - f(x2) becomes f(x1 - x2) = A(x1 - x2) for some A. So, in this case, K, the lipschitz constant is simply the largest singular value of that matrix A. This would mean that Ax would always have less signal energy than x.
With a little more work, this observation can be extended to nonlinear functions via their linearization (jacobian matrix). I won't explain it full here, but this line of thinking is very useful for convergence proofs of dynamical systems and indirect/iterative solvers.