Part IV

(14) Advanced Calculus

This topic is also part of calculus but
concentrates on surfaces, volumes, and
maybe the math of flows of heat, water,
etc.

Commonly this part of calculus makes heavy
use of vectors and matrices.

Also covered can be Fourier series and
integrals.

A pure tone in music, say, from an organ,
has a fundamental frequency, say, 440
cycles per second and also overtones at
natural number multiples of that
fundamental frequency. Each of the
overtone frequenciess works like an axis
of orthogonal coordinates and is a
terrific way to analyze or synthesize such
a tone.

The Fourier transform is closely related
and has an orthogonal coordinate axis for
each real number.

Engineering is awash in linear systems
that do not change their properties over
time -- time invariant linear systems.
Such systems merely adjust some Fourier
coefficients. This is a powerful result.

(15) Measure Theory

This topic develops integral calculus with
a lot more generality.

(16) Probability Theory

In probability we imagine that we do
some experimental trials and observe the
results. We have a sample space, our
set of trials.

We regard the set of all trials like a
region with area, and we say that the area
is 1.

Maybe we flip a coin and get Heads. The
set of all trials that yield Heads is one
event Heads.

The event Heads has area, probability

P(Heads)

With a fair coin we have

P(Heads) = 1/2

A random variable X is some outcome of a
trial. Usually X is a number. In this
case we can write

P(X >= 2)

for the probability that random variable X
is >= 2.

Events A and B are independent provided

P(A and B) = P(A)P(B)

In probability we can have algebraic
expressions with random variables, have a
distance between random variables, have
a sequence of random variables converge
to a random variable, etc.

Given a real valued random variable X, for
real number x we can let

F_X(x) = P(X <= x)

Then F_X is the cumulative distribution
of X, and

f_X(x) = d/dx F_X(x)

is the probability density of X.

One of the major results is the central
limit theorem that shows a distribution
converging to the Gaussian bell curve.

Two more results are the weak and strong
laws of large numbers that show
convergence to average values (means,
expectations).

One more result, amazing, is the
martingale convergence theorem.

The Radon-Nikodym theorem of measure
theory provides an amazing general theory
of conditional expectation which is one
way to look at the value of information.

In statistics we are given the values of
some random variables and try to infer
the values of something related.

The more serious work in probability and
statistics is done on the foundation of
measure theory.

(17) Optimization

We have a furniture factory and a new
shipment of wood. We have some freedom in
what we do with this wood. An
optimization problem is how to use the
freedom to make the greatest revenue from
the wood.

More generally we have some work to do
with some freedom in how we do the work,
and an optimization problem is how best to
exploit the freedom to have the best
outcome, profit,
for the work.

If the profit is linear and if the work
and freedom are described by linear
equations, then we have the topic of
linear programming.

Linear programming has played a role in
the Nobel prizes in economics.

The main means of solving linear
programming problems is the simplex
algorithm which is a not very large
modification of Gauss elimination.

In practice the simplex algorithm is
shockingly fast; in theory its worst case
performance is exponential.

In the furniture making, we can't sell a
fraction, say, 1/3rd, of a dining room
chair. So, we want our solution to
consist of integers. The simplex
algorithm does not promise us integers.
So we have a problem in integer linear
programming.

In practice from the simplex algorithm we
may get to make 601 + 1/3rd dining room
chairs and round that to 601 or just 600.
No biggie.

If we insist on integers, then,
surprisingly, even shockingly, we have one
of the early examples of a problem in NP
complete.

In practice there are still ways to
proceed; often we can get the optimal
integer solutions we want easily;
sometimes such solutions are challenging;
sometimes we have to settle for
approximations; often the approximations
can be quite good, like forgetting about
1/3rd of a chair; in theory the ways have
worst case performance that is exponential
and nearly useless.

One integer linear programming problem I
attacked has 600,000 variables and 40,000
constraints. I wrote some software that
in 600 seconds on a slow computer found an
integer solution within 0.025% of
optimality. The solution made heavy use
of the simplex algorithm, non-linear
duality theory, and something called
Lagrangian relaxation.

In addition to optimization, the simplex
algorithm and related math make nice
contributions to geometry, e.g., prove
theorems of the alternative used in
proving the famous Kuhn-Tucker conditions
necessary for optimality.

Since it is possible that the work we are
trying to optimize is complicated, its
mathematical description may also be
complicated and the optimization effort,
difficult.

Optimization problems are common in animal
feeding, cracking crude oil,
transportation problems, and much more.

Game theory can be regarded as a special
case. One proof of the famous Nash result
(featured in the movie about Nash, A
Beautiful Mind) is via linear
programming.

Some problems are to find curves, say, how
to climb, cruise, and descend an airplane
to minimize fuel burned or the best curve
for a missile to attack an enemy fighter
jet. Newton found the brachistochrone
curve solution to the problem of the
frictionless curve that would let a bead
slide down in minimum time.

Relevant fields are calculus of variations
and deterministic optimal control. One
famous result is the Pontryagin maximum
principle.

Some problems have uncertainty; so the
optimization may be to find the solution
with the best expected outcome. For
such problems there is the field of
stochastic optimal control.