The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare continue offer
high quality educational resources for free.
To make a donation or to view additional materials from
hundreds of MIT courses, visit MIT OpenCourseWare at
ocw.mit.edu.
PROFESSOR: I just started talking a little bit last time
about viewing L2, namely this set of functions that are
square integrable as a vector space.
And I want to reveal a little bit about why
we want to do that.
Because after spending so long worrying about all these
questions of measurability and all that, you must wonder, why
do we want to look at it a different way now.
Well, a part of it is we want to be able to look at
orthogonal expansions geometrically.
In other words, we would like to be able to
draw pictures of them.
You can draw pictures of a function, but you're drawing a
picture of one function.
And you can't draw anything about the relationship between
different functions, except by drawing the
two different functions.
You get all the detail there, and you can't
abstract it at all.
So somehow we want to start to abstract some of this
information.
And we want to be able to draw pictures which look more like
vector pictures than like functions.
And you'll see why that becomes important in a while.
The other thing is, when you draw a function as a function
of time, the only thing you see it how it behaves in time.
When you take the Fourier transform and you draw it in
frequency, the only thing you see is how
it behaves in frequency.
And, again, you don't see what the relationship is between
different functions.
So you lose all of that.
When you take this signal space viewpoint, what you're
trying to do there is to not stress time or frequency so
much, but more to look at the relationship
between different functions.
Why do we want to do that?
Well, because as soon as we start looking at noise and
things like that, we want to be able to tell something
about how distinguishable different functions are from
each other.
So the critical question there you want to ask, when you ask
how different functions are, is how do you look at two
functions both at the same time.
So, again, that's --
all of this is coming back to the same thing.
So we want to be able to draw pictures of functions which
show how functions are related, rather than show all
the individual detail of function.
Finally, what we'll see at the end of today that it gives us
a lot of capability for understanding much, much
better what's going on when we look at conversions of
orthogonal series.
This is something we haven't had any way to look at before
we talked about the Fourier series, the discrete time
Fourier transform, and things like this.
But we haven't been able to really say something general.
Which, again, is what we want to do now.
I'm going to go very quickly through these axioms of a
vector space.
Most of you have seen them before.
Those of you who haven't seen axiomatic treatments of
various things in mathematics are going to be
puzzled by it anyway.
And you're going to have to sit at home or somewhere on
your own and puzzle this out.
But I just wanted to put them up so I could start to say
what it is that we're trying to do with these axioms.
What we're trying to do is to say everything we know about
n-tuples, which we've been using all of our lives.
All of these tricks that we use, most of them we can use
to deal with functions also.
The pictures, we can use.
All of the ideas about dimension, all of the ideas
about expansions.
All of this stuff becomes useful again.
But, there are a lot of things that are true about functions
that aren't true about vectors.
There are lots of things which are true about n-dimensional
vectors that aren't true about functions.
And you want to be able to go back to these axioms when you
have to, and say, well, is this property we're looking at
something which is a consequence of the axioms.
Namely, is this an inherent property of vector spaces, or
is it in fact something else which is just because of the
particular kind of thing we're looking at.
So, vector spaces.
Important thing is, there's an addition operation.
You can add any two vectors.
You can't multiply them, by the way.
You can only add them.
You can multiply by scalars, which we'll talk about in the
next slide.
But you can only add the vectors themselves.
And the addition is commutative, just like
ordinary addition of real numbers or complex numbers is.
It's associative, which says v, u plus w in parentheses.
Now, you see why we're doing this is, we've said by
definition that for any two vectors, u and w, there's
another vector, which is called u plus w.
In other words, this axiomatic system says, whenever you add
two vectors you have to get a vector.
There's no way around that.
So u plus w has to be a vector.
And that says that v plus u plus w has to be a vector.
Associativity says that you get the same vector if you
look at it the other way, if first adding v and u
and then adding w.
So, anything what you call a vector space has
to have this property.
Of course, the real numbers have this property.
The complex numbers have this property.
All the n-tuples you deal with all the
time have this property.
Functions have this property.
Sequences, infinite length sequences have this property,
so there's no big deal about it.
But that is one of the axioms.
There's a unique vector 0, such that when you add 0 to a
vector, you get the same vector again.
Now, this is not the 0 in the real number system.
It's not the 0 in a complex number system.
In terms of vectors, if you're thinking of n-tuples, this is
the n-tuple which is all 0's.
In terms of other vectors, spaces, it
might be other things.
Like, in terms of functions, 0 means the function which is 0
everywhere.
And finally, there's the unique vector minus v, which
has the property that v plus minus v -- in other words, v
minus v -- is equal to 0.
And in a sense that defines
subtraction as well as addition.
So we have addition and subtraction, but we don't have
multiplication.
And there's scalar multiplication.
Which means you can multiply a vector by a scalar.
And the vector spaces we're looking at here only have two
kinds of scalars.
One is real scalars and the other is complex scalars.
The two are quite different, as you know.
And so when we're talking about a vector space we have
to say what the scalar field is that we're talking about.
So, for every vector.
And also for every scalar, there's another vector which
is the scalar times the vector.
Which means, you have scalar multiplication.
You can multiply vectors by scalars in terms of n-tuples.
When you're multiplying a scalar by an n-tuple, you just
multiply every component by that scalar.
When you take the scalar 1, and in a field there's always
an element 1, there's always an element 0.
When you take the element 1, and you multiply it by a
vector, you got the same vector.
Which, of course, is what you would expect if you were
looking at functions.
You multiply a function.
For every t you multiply it by 1, and you get the same thing
back again.
For an n-tuple, you multiply every component by 1, you get
the same thing back again.
So that's --
well, it's just another axiom.
Then we have these distributive laws.
And I won't read them out to you, they're
just what they say.
I want to talk about them in a second a little bit.
And as we said, the simplistic example of a vector space, and
the one you've been using for years, partly because it saves
you a lot of notation.
Incidentally, one of the reasons I didn't list for
talking about L2 as a vector space is, it saves you a lot
of writing.
Also, just like when you're talking about n-tuples it
saves you writing.
It's a nice, convenient notational device.
I don't think any part of mathematics becomes popular
with everyone, unless it saves notation as well as well as
simplifying things.
So we have these n-tuples, which is what we mean by r sub
n or c sub n.
In other words, when I talk about r sub n, I don't mean
just a vector space of dimension n.
I haven't even defined dimension yet.
And when you talk in this generality, you have to think
a little bit about what it means.
So we're just talking about n-tuples now. r sub n is
n-tuples of real numbers. c sub n is
n-tuples of complex numbers.
Addition and scalar multiplication are component
by component, which we just said.
In other words, w equals u plus v means this, for all i.
Scalar multiplication means this.
The unit vector, e sub i, is a 1 in the i'th position, a 0
everywhere else.
And what that means is that any vector v which is an
n-tuple, n rn or n cn, can also be expressed as the
equals summation of these coefficients,
times these unit factors.
That looks like we're giving up the simple notation we have
and going to more complex notation.
This is a very useful idea here.
That you can take even something as simple as
n-tuples and express every vector as a sum of these
simple vectors.
And when you start drawing pictures, you start to see why
this makes sense.
I'm going to do this on the next slide.
So let's do it.
And on the slide here, what I've done is to draw a diagram
which is one you've seen many times, I'm sure.
Except for the particular things on it.
Of two-dimensional space, where you take a 2-tuple and
you think of the first element in the 2-tuple, v1, a being
the horizontal axis.
The second element, v2, as being the vertical axis.
And then you can draw any 2-tuple by going over v1 and
then up v2, which is what you've done all your lives.
The reason I'm drawing this is to show you that you can take
any two vectors.
First vector, v. Second vector, u.
One thing that --
I'm sure you all traditionally do is you view a
vector in two ways.
One, you view it as a point in two-dimensional space.
And the other is you view it as a line from 0 up to v. And
when you put a little arrow on it, it looks like a vector,
and we call it a vector.
So it must be a vector, OK?
So either the point or the line represents a vector.
Here's another vector, u.
I can take the difference between two vectors, u minus
v, just by drawing a line from v up to u.
And thinking of that as a vector also.
So this vector really means a vector starting here, going
parallel to this, up to this point, which is what w is.
But it's very convenient to draw it this way, which lets
you know what's going on.
Namely, you represent u as the sum of v plus w.
Or you represent w as a difference between u and v.
And all of this is trivial.
I just want to say it explicitly so you start to see
what the connection is between these axioms, which if I don't
talk about them a little bit, I'm sure you're just going to
blow them off and decide, eh, I know all of that, I can look
at things in two dimensions and I don't have to think
about them at all.
And then, in a few days, when we're doing the same thing for
functions, you're suddenly going to become very confused.
So you ought to think about it in these simple terms.
Now, when I take a scalar multiple of the vector, in
terms of these diagrams, what we're doing is taking
something on this same line here.
I can take scalar multiples which go all the way up here,
all the way down there.
I can take scalar multiples of u which go up here.
And down here.
The interesting thing here is when I scale v down by alpha
and I scale u down by alpha, I can also scale w down by alpha
and I connect it just like that, from alpha v to alpha u.
Which axiom is that?
I mean, this is the canonic example of
one of those axioms.
AUDIENCE: [UNINTELLIGIBLE]
PROFESSOR: What?
AUDIENCE: Distributed.
PROFESSOR: Distributed, good.
This is saying that alpha times the quantity u minus v
is equal to alpha u minus alpha v. That's so trivial
that it's hard to see it, but that's what it's saying.
So that the distributive law really says the triangles
maintain their shape.
Maybe it's easier to just say distributive
law, I don't know.
So, for the space of L2 complex functions, we're going
to define u plus v as w, where w of t equals u
of t plus v of t.
Now, I've done a bunch of things here all at once.
One of them is to say what we used to call a function, u of
t, we're now referring to with a single letter, boldface u.
What's the advantage of that?
Well, one advantage of it is when you talk about a function
as u of t, you're really talking about
two different things.
One, you're talking about the value of the function at a
particular argument, t.
And the other is, you're talking
about the whole function.
I mean, sometimes you want to say a
function has some property.
A function is L2.
Well, u of t at a particular t is just a number.
It's not a function.
So this gives us a nice way of distinguishing between
functions and between the value of functions for
particular arguments.
So that's one more notational advantage you get by talking
about vectors here.
We're going to define the sum of two functions just as the
point y sum.
In other words, these two functions are
defined at each t.
w defined at a t is the sum of this and that.
The scalar multiplication is just defined by, at every t, u
of t is equal to alpha times v of t.
Just what you'd expect.
There's nothing strange here.
I just want to be explicit in saying how everything follows
from these axioms here.
And I won't say all of it because
there's too much of it.
With this addition and scalar multiplication, L2, the set of
finite energy measurable functions is in fact the
complex vector space.
And it's trivial to go back and check through all the
axioms with what we said here, and I'm not
going to do it now.
There's only one of those axioms which is a bit
questionable.
And that is, when you add up two functions which are square
integrable, is the sum going to be square integrable also.
Well, you nod and say yes, but it's worthwhile proving it
once in your lives.
So the question is, is this function here less than
infinity if the integral of u of t squared and the integral
of v of t squared are both integrable.
Well, it's a useful inequality, which looks like a
very weak inequality but it, in fact, is not weak.
It says that u of t plus v of t.
This is just at a particular value of t.
So this is just an inequality for real
numbers and complex numbers.
It says that this u of t plus v of t, quantity squared,
magnitude squared, is less than or equal to 2 times u of
t squared plus 2 times v of t squared.
You wonder, at first, what's the 2 doing in there.
But then think of making v of t equal to u of t.
You have 2 times u of t squared, which is 4
times u of t squared.
Well, that's what you need here to make this true.
So, in that example this inequality is
satisfied with equality.
To verify the inequality in general, you just
multiply this out.
It's u of t squared plus v of t squared plus two cross-terms
and it all works.
This vector space here is not quite the vector space we want
to talk about.
But let's put off that question for a while and we'll
come to it later.
We will come up with a vector space which is just slightly
different than that.
The main thing you can do with vector spaces is talk about
their dimension.
Well, there are a lot of other things you can do, but this is
one of the main things we can do.
And it's an important thing which we have to talk about
before going into inner product spaces, which is what
we're really interested in.
So we need a bunch of definitions here.
All of this, I'm sure is familiar to most of you.
For those of you that it's not familiar to, you just have to
spend a little longer with it.
There's not a whole lot involved here.
If you have a set of vectors, which are in a vector space,
you say that they span the vector space if in fact every
vector in this vector space is a linear
combination of those vectors.
In other words, any vector, u, can be made up as some sum of
alpha i times v sub i.
Now, notice we've gone a long way here beyond those axioms.
Because we're talking about scalar multiplications.
And then we're talking about a sum of a lot of scalar
multiplications.
Each scalar multiplication is a vector.
The sum of a bunch of vectors, by the associative law is, in
fact, another vector.
So every one of these sums is a vector.
And by definition, a set of vectors spans a vector space
if every vector can be represented as some linear
combination of them.
In
other words, there isn't something outside of here
sitting there waiting to be discovered later.
You really understand everything that's there.
And we say that v is finite dimensional if it is spanned
by a finite set of vectors.
So that's another definition.
That's what you mean by finite dimensional.
You have to be able to find a finite set of vectors such
that linear combinations of those vectors gives you
everything.
It doesn't mean you have a finite set of vectors.
You have an infinite set of vectors because you have an
infinite set of scalars.
In fact, you'd have an uncountably infinite set of
vectors because of these scalars.
Second definition.
The vector v1 to vn are linearly independent -- and
this is a mouthful -- if u equals the sum of alpha sub i
v sub i equals 0, only for alpha sub i equal
to 0 for each i.
In other words, you can't take any linear combination of the
v sub i's and get 0 unless that linear combination is
using all 0's.
You can't, in any non-trivial way, add up a bunch of
vectors and get 0.
To put it another way, none of these basis vectors is a
linear combination of the others.
That's usually a more convenient way to put it.
Although it takes more writing.
Now, we say that a set of vectors are a basis for v if
they're both linearly independent and if they span
v. When we first talked about spanning, we didn't say
anything about these vectors being linearly independent, so
we might have had many more of them than we needed.
Now, when we're talking about a basis, we restrict ourselves
to just the set we need to span the space.
And then the theorem, which I'm not going to prove, but
it's standard Theorem One of any linear algebra book --
well, it's probably Theorem One, Two, Three and Four all
put together -- but anyway, if it says if v1 and v sub n span
v, then a subset of these vectors is the basis of b.
In other words, if you have a set of vectors which span a
space, you can find the basis by systematically eliminating
vectors which are linear combinations of the others,
until you get to a point where you can't do that any further.
So this theorem has an algorithm tied into it.
Given any set of vectors which span a space, you can find the
basis from it by perhaps throwing out
some of those vectors.
The next part of it is, if v is a finite dimensional space,
then every basis has the same size.
This, in fact, is a thing which takes a little bit of
work proving it.
And, also, any linearly independent set, v1 to v sub
n, is part of the basis.
In other words, here's another algorithm you can use.
You have this big, finite dimensional space.
You have a few vectors which are linearly independent.
You can build a basis starting with these, and you just
experiment.
You experiment to find new vectors, which are not linear
combinations of that set.
As soon as you find one, you add it to the basis.
You keep on going.
And the theorem says, by time you get to n of
them, you're done.
So that, in a sense, spanning sets are too big.
Linearly independent sets are too small.
And what you want to do is add the linearly independent sets,
shrink the spanning sets, and come up with a bases.
And all bases have the same number of vectors.
There many different bases you come up with, but they all
have the same number of vectors.
Not going to talk at all about infinite dimensional spaces
until the last slide today.
Because the only way I know to understand infinite
dimensional vector spaces is by thinking of them, in some
sort of limiting way, as finite dimensional spaces.
And I think that's the only way you can do it.
A vector space in itself has no sense of distance
or angles in it.
When I drew that picture before for you -- let me put
it up again --
it almost looked like there was some sense
of distance in here.
Because when we took scalar multiples, we scaled these
things down.
When I took these triangles, we scaled down the triangle,
and the triangle was maintained.
But, in fact, I could do this just as well if I took v and
moved it down here almost down on this axis, and if I took u
and moved that almost down on the axis.
I have the same picture, the same kind of distributive law.
And everything else.
You can't --
I mean, one of the troubles with n-dimensional space, to
study what a vector space is about, is it's very hard to
think of n-tuples without thinking of distance.
And without thinking of things being orthogonal to each
other, and of all of these things.
None of that is talked about at all in any of these axioms.
And, in fact, you need some new axioms to be able to talk
about ideas of distance, or angle, or any of these things.
And that's what we want to add here.
So we need some new axioms.
And what we need is a new operation on the vector space,
before the only -- the only operations we have are
addition and scalar multiplication.
So that vector spaces are really
incredibly simple animals.
There's very little you can do with them.
And this added thing is called an inner product.
An inner product is a scalar valued
function of two vectors.
And it's represented by these little brackets.
And the axioms that these inner products have to satisfy
is, if you're dealing with a complex vector space.
In other words, where the scalars are complex numbers,
then this inner product, when you switch it around, you have
Hermitian symmetry.
Which means that this inner product is equal to u v
complex conjugate.
We've already seen that sort of thing in
taking Fourier series.
And the fact that when you're dealing with complex numbers,
symmetry doesn't usually hold, and you usually need some kind
of Hermitian symmetry in most of the things you do.
The next axiom is something called bilinearity.
It says that if you take a vector which is alpha times a
vector v, plus beta times a vector u, take the inner
product of that with w, it splits up as alpha times v w
plus beta times u w.
How about if I do it the other way?
See if you understand what I'm talking about at all here.
Suppose we take w alpha u plus beta v. What's that equal to?
Well, it's equal to alpha something w u plus beta w v.
Except that's wrong, It's right for real vector spaces,
it's wrong for complex vector spaces.
What am I missing here?
I need complex conjugates here and complex conjugates here.
I wanted to talk about that, because when you're dealing
with inner products, I don't know whether you're like me,
but every time I start dealing with inner products and I get
in a hurry writing things down, I forgot to put those
damn complex conjugates in them.
And, just be careful.
Because you need them.
At least go back after you're all done and put the complex
conjugates in.
If you're dealing with real vectors, of course you don't
need to worry about any of that.
And all the pictures you draw are always of real vectors.
Think of trying to draw this picture.
This is the simplest picture you can draw.
Of two-dimensional vectors.
This is really a picture of a vector space of dimension two,
for real vectors.
What happens if you try to draw a picture of complex
two-dimensional vector space?
Well, it becomes very difficult to do.
Because you're really talking about the real part of, if
you're dealing with a basis which consists of two complex
vectors, then instead of v1, you need a real part of v1 and
imaginary part of v1.
Instead of v2, you need real part of v2 and
imaginary part of v2.
And you can always draw this in four dimensions.
And I even have trouble drawing in three dimensions,
because somehow my pen doesn't make marks in three
dimensions.
And in four dimensions, I'm a blinking 12.
And have no idea of what to do.
So you have to remember this.
If you're dealing with rn or cn, almost always you define
the inner product of v and u as the sum of the components
with the second component complex conjugated.
This is just a standard thing that we do all the time.
When we do this, and we use unit vectors, the inner
product of v with the i'th unit vector is just v sub i.
That's what this formula says.
Because e sub i is this vector u, in which there's a 1 only
in the i'th position, and a 0 everywhere else.
So v e i is always the v i, and e i v is always v i
complex conjugate.
Again, this Hermitian nonsense that comes up to
plague us all the time.
And from that, if you make v equal to e sub j or e sub i,
you get the inner product of two of these basis vectors is
equal to 0 for i unequal to j.
In other words, the standard way of drawing pictures when
you make it into an inner product space, those unit
vectors become orthonormal.
Because that's the way you like to draw things.
You like to draw one here and one there.
And that's what we mean by perpendicular, which the
two-dimensional or three-dimensional word for
orthogonal.
So we have a couple of definitions.
The inner product of v with itself is called inner product
v squared, which is called the squared norm of the vector.
The squared norm has to be non-negative, by axiom.
It has to be greater than 0, unless this vector, v, is in
fact a 0 vector.
And the length is just the square
root of the norm squared.
In other words, the length and the norm are the same thing.
The norm of a vector is the length of the vector.
I've always called it the length, but a lot of people
like to call it the norm.
v and u are orthogonal if the inner product of v and u is
equal to 0.
How did I get that?
I defined it that why.
Everybody defines it that way.
That's what you mean by orthogonality.
Now we have to go back and see if it makes any sense in terms
of these nice simple diagrams.
But first I'm going to do something called the
one-dimensional projection theorem.
Which is something you all know but you probably have
never thought about.
And what it says is, if you have to vectors, v and u, you
can always break v up into two parts.
One of which is on the same line with u.
In other words, is colinear with u.
I'm drawing a picture here for real spaces, but when I say
colinear, when I'm dealing with complex spaces, I mean
it's u times some scalar, which could be complex.
So it's somewhere on this line.
And the other part is perpendicular to this line.
And this theorem says in any old inner product space at
all, no matter how many dimensions you have, infinite
dimensional, finite dimensional, anything, if it's
an inner product space on either the scalars r or the
scalars c, you can always take any old two vectors at all.
And you can break one vector up into a part that's colinear
with the other, and another part which is orthogonal.
And you can always draw a picture of it.
If you don't mind just drawing the picture for real vectors
instead of complex vectors.
This is an important idea.
Because what we're going to use it for in a while, is to
be able to talk about functions which are these
incredibly complicated objects.
And we're going to talk about two different functions.
And we want to be able to draw a picture in which those two
functions are represented just as points in a
two-dimensional picture.
And we're going to do that by saying, OK, I take one of
those functions.
And I can represent it as partly being colinear with
this other function.
And partly being orthogonal to the other function.
Which says, you can forget about all of these functions
which extend to infinity, in time extend to infinity, in
frequency and everything else.
And, so long as you're only interested in some small set
of functions, you can just deal with them as a finite
dimensional vector space.
You can get rid of all the mess, and just think of them
in this very simple sense.
That's really why this vector space idea, which is called
signal space, is so popular among engineers.
It lets than get rid of all the mess and think in terms of
very, very simple things.
So let's see why this complicated
theorem is in fact true.
Let me state the theorem first.
It says for any inner products space, v. And any vectors, u
and v in v, with u unequal to 0 --
I hope I said that in the notes -- the vector v can be
broken up into a colinear term plus an orthogonal term, where
the colinear term is equal to a scalar times u.
That's what we mean by colinear.
That's just the definition of colinear.
And the other vector is orthogonal to u.
And alpha is uniquely given by the inner product v u divided
by the norm squared of u.
Now, there's one thing ugly about this.
You see that norm squared, you say, what is that doing there.
It just looks like it's making the formula complicated.
Forgot about that for the time being.
We will get into it in a minute and explain why we have
that problem.
But, for the moment, let's just prove this
and see what it says.
So what we're going to do is say, well, if I don't look at
what the theorem is saying, what I'd like to do is look at
some generic element which is a scalar multiple times u.
So I'll say, OK let v parallel to u be alpha times u.
I don't know what alpha is yet, but alpha's going to be
whatever it has to be.
We're going to choose alpha so that this other vector, v
minus v u, is a thing we're calling v perp.
So that that, the inner product of that and u, is
equal to 0.
So what I'm trying to do is to find a vector --
strategy here, is to take any old vector along this line and
try to choose the scalar alpha in such a way that the
difference between this point and this point is orthogonal
to this line.
That's why I started out with alpha unknown.
Alpha unknown just says we have any
point along this line.
Now I'm going to find out what alpha has to be.
I hope I will find out that it has to be only one thing, and
it's uniquely chosen.
And that's what we're going to find.
So v minus this projection term, this is called a
projection of v on u, is equal to v u minus a projection
inner product with u.
So it's equal to this difference here.
This is equal to the inner product.
Since we have chosen this term to be alpha times u, we can
bring the alpha out.
So it's alpha times the inner product of u with itself.
That's the norm squared of alpha.
So this is inner product of v and u minus alpha times the
norm squared.
This is 0 if and only if you set this equal to 0.
And the only value alpha can have to make this 0 is the
inner product of v and u divided by the norm squared.
So I would think that if I ask you to prove this without
knowing the projection theorem, I would hope that if
you weren't afraid of it or something, and you just sat
down and tried to do it, you would all do it in
about half an hour.
It would probably take most of you longer than that, because
everybody gets screwed up in the notation when they first
try to do this.
But, in fact, this is not a complicated thing.
This is not rocket science.
Now.
What is this norm squared doing here?
The thing we have just proven is that with any two vectors v
and u, the projection of v on u, namely that vector, there
which has the property that v minus v u is perpendicular to
u, we showed, is this inner product divided by the norm
squared times u.
Now, let me break up this norm squared.
Which is just some positive number.
It's a positive real number.
Into the length times the length.
Namely, the norm u is just some real number.
And write it this way, as the inner product of v with u
divided by the length of u times u divided by
the length of u.
Now, what is the vector u divided by the length of u?
AUDIENCE: [UNINTELLIGIBLE]
PROFESSOR: What?
AUDIENCE: [UNINTELLIGIBLE]
PROFESSOR: It's the same direction of u, but
it has length 1.
In other words, this is the normalized form of u.
And I have the normalized form of u in here also.
So what this is saying is that this projection is also equal
to the projection of v on the normalized form of u, times
the normalized form of u.
Which says that it doesn't make any difference what the
length of u is.
This projection is a function only of the direction of u.
I mean, this is obvious from the picture, isn't it?
But again, since we can't draw pictures for complex valued
things, it's nice to be able to see it analytically.
If I shorten u, or lengthen u, this projection is still going
to be exactly the same thing.
And that's what the norm squared of u is doing here.
The norm squared of u is simply sitting there so it
does this normalization function for us.
It makes this projection equal to the inner product of v with
the normalized form of u, times the
normalized vector for u.
The Pythagorean theorem, which doesn't follow from this, it's
something else, but it's simple -- in fact, we can do
it right away -- it says if v and u are orthogonal, then the
norm squared of u plus v is equal to u
squared plus v squared.
I mean, this is something you use
geometrically all the time.
And you're familiar with this.
And the argument is, you just break this up into the norm
squared of u plus the norm squared of v, plus the two
cross-products, the inner product of u times the inner
product of u with v, which is 0, and the inner product of v
with u, which is 0.
So the two cross-terms go away and you're
left with just this.
And for any v and u, the Schwarz inequality says that
the inner product, the magnitude of the inner product
of v and u, is less than or equal to the length of v times
the length of u.
The Schwarz inequality is probably -- well, I'm not sure
that it's probably, but it's perhaps the most used
inequality in mathematics.
Any time you use vectors, you use this all the time.
And it's extremely useful.
I'm not going to prove it here because it's in the notes.
It's a two-step proof from what we've done.
And the trouble is watching two-step proofs in class.
At a certain point you saturate on them.
And I have more important things I want to do later
today, so I don't want you to saturate on this.
You can read this at your leisure and
see why this is true.
I did want to say something about it, though.
If you divide the left side by the right side, you can write
this as the magnitude of the inner product of the
normalized form of v with the normalized form of u.
If we're talking about real vector space, this in fact is
the cosine of the angle between v and u.
It's less than or equal to 1.
So for real two-dimensional vectors, the fact that the
cosine is less than or equal to 1 is really equivalent to
the Schwarz inequality.
And this is the appropriate generalization for any old
vectors at all.
And the notes would say more about that if that went a
little too quickly.
OK, the inner product space of interest to us is this thing
we've called signal space.
Namely, it's a space of functions which are measurable
n square integrals.
In other words, finite value when you take the square and
integrate it.
And we want to be able to talk about the set of either real
or complex L2 functions.
So, either one of them, we're going to define the inner
product in the same way for each.
It really is the only natural way to define an
inner product here.
And you'll see this more later as we start doing other things
with these inner products.
But just like what when you're dealing with n-tuples, there's
only one sensible way to define an inner product.
You can define inner products in other ways.
But it's just a little bizarre to do so.
And here it's a little bizarre also.
There's a big technical problem here.
And if you look at, it can anybody spot what it -- no,
no, of course you can't spot what it is unless you've read
the notes and you know what it is.
One of the axioms of an inner product space is that the only
vector in the space which has an inner product with itself,
a squared norm equal to zero is the zero vector.
Now, we have all these crazy vector we've been talking
about, which are zero, except on a set of measures zero.
In other words, vectors' functions which have zero
energy but just pop up at various isolated points and
have values there.
Which we really can't get rid of if we view a function as
something which is defined at every value of t.
You have to accept those things as part of what you're
dealing with.
As soon as you started integrating things, those
things all disappear.
But the trouble is, those functions, which are zero
almost everywhere, are not zero.
They're only zero almost everywhere.
They're zero for all engineering purposes.
But they're not zero, they're only zero almost everywhere.
Well, if you define the inner product in this way and you
want to satisfy the axioms of an inner product space, you're
out of luck.
There's no way you can do it, because this
axiom just gets violated.
So what do you do?
Well, you do what we've been doing all along.
We've been sort of squinting a little bit and saying, well,
really, these functions of measure 0 are really 0 for all
practical purposes.
Mathematically, what we have to say is, we want to talk
about an equivalence class of functions.
And two functions are in the same equivalence class if
their difference has 0 energy.
Which is equivalent to saying their difference is zero
almost everywhere.
It's one of these bizarre functions which just jumps up
at isolated points and doesn't do anything else.
Not impulses at isolated points, just non-zero at
isolated points.
Impulses are not really functions at all.
So we're talking about things that are functions, but
they're these bizarre functions which we talked
about and we've said they're unimportant.
So, but they are there.
So the solution is to associate vectors with
equivalence classes.
And d of t and u of t are equivalent if the v of t minus
u of t is zero almost everywhere.
In other words, when we talk about a vector, u, what we're
talking about is an
equivalence class of functions.
It's the equivalence class of functions for which two
functions are in the same equivalence class if they
differ only on a set of measure zero.
In other words, these are the things that gave us trouble
when we were talking about Fourier transforms.
These are the things that gave us trouble when we were
talking about Fourier series.
When you take anything in the same equivalence class,
time-limited functions, and you form a Fourier series,
what happens?
All of the things in the same equivalence class have the
same Fourier coefficients.
But when you go back from the Fourier series coefficients
back to the function, then you might go back in a bunch of
different ways.
So, we started using this limit in the mean notation and
all of that stuff.
And what we're doing here now is, for these vectors, we're
just saying, let's represent a vector as this whole
equivalence class.
While we're talking about vectors, we're almost always
interested in orthogonal expansions.
And when we're interested in orthogonal expansions, the
coefficients in the orthogonal expansions are found as
integrals with the function.
And the integrals with different functions in the
same equivalence class are identical.
In other words, any two functions in the same
equivalence class have the same coefficients in any
orthogonal expansion.
So if you talk only about the orthogonal expansion, and
leave out these detailed notions of what the function
is doing at individual times, then you don't have to worry
about equivalence classes.
In other words, when you --
I'm going to say this again more carefully later, but let
me try to say it now a little bit crudely.
One of the things we're interested in doing this
taking this class of functions, mapping each
function into a set of coefficients, where the set of
coefficients are the coefficients in some
particular orthogonal expansion that we're using.
Namely, that's the whole way that we're using to get from
waveforms to sequences.
It's the whole --
it's the entire thing we're doing when we start out on a
channel with a sequence of binary digits and then a
sequence of symbols.
And we modulate it into a waveform.
Again, it's the mapping from sequence to waveform.
Now, the important thing here about these equivalence
classes is, you can't tell any of the members of the
equivalence class apart within the sequence that we're
dealing with.
Everything we're interested in has to
do with these sequences.
I mean, if we could we'd just ignore the waveforms
altogether.
Because all the processing that we do is with sequences.
So the only reason we have these equivalence classes is
because we need them to really define what the functions are.
So, we will come back to that later.
Boy, I think I'm going to get done today.
That's surprising.
The next idea that we want to talk
about is vector subspaces.
Again, that's an idea you've probably heard of,
for the most part.
A subspace of a vector space is a subset of the vector
space such that that subspace is a vector
space in its own right.
An equivalent definition is, for all vectors u and v, in
the subspace alpha times u plus beta
times v is in s also.
In other words, a subspace is something which you can't get
out of by linear combinations.
If I take one of these diagrams back here that I keep
looking at --
I'll use this one.
If I want to form a subspace of this subspace, if I want to
form a subspace of this two-dimensional vector space
here, one of the subspaces includes
u, but not v, perhaps.
Now, if I want to make a one-dimensional subspace
including u, what is that subspace?
It's just all the scalars times u.
In other words, it's this line that goes through the origin
and through that vector, u.
And a subspace has to include the whole line.
That's what we're saying.
It's all scalar multiples of u.
If I want a subspace that includes u and v, where u and
v are just arbitrary vectors I've chosen out of my hat,
then I have this two-dimensional subspace,
which is what I've drawn here.
In other words, this idea is something
we've been using already.
It's just that we didn't need to be explicit about it.
The subspace which includes u and v --
well, a subspace which includes u and v, is the
subspace of all linear combinations of u and v. And
nothing else.
So, it's all vectors along here.
It's all vectors along here.
And you fill it all in with anything here added to
anything here.
So you get this two-dimensional space.
Is 0 always in a subspace?
Of course it is.
I mean, you multiply any vector by 0 and
you get the 0 vector.
So you sort of get it as a linear --
what more can I say?
If we have a vector space which is an inner product
space; in other words, if we add this inner product
definition to our vector space, and I take a subspace
of it, that can be defined as an inner product space also,
with the same definition of inner
product that I had before.
Because I can't get out of it by linear combinations, and
the inner product is defined for every pair of vectors in
that space.
So we still have a nice, well-defined vector space,
which is an inner product space, and which is a subspace
of the space we started with.
Everything I do from now on, I'm going to assume that v is
an inner product space.
And want to look at how we normalize vectors.
We've already talked about that.
If I have a vector in this vector space that's
normalized, if its norm equals 1.
We already decided how to normalize a vector.
We took an arbitrary vector, u, divided by its norm.
And as soon as we divide by its norm, that vector, v,
divided by the norm of v, the norm of that is just 1.
So the projection, what the projection theorem says, and
all I need here is a one-dimensional projection
theorem, it says that v, in the direction of this vector
phi, is equal to the inner product of u
with phi times phi.
That's what we said.
As soon as we normalized these vectors, that ugly denominator
here disappears.
Because the norm of phi is equal to --
because the norm is equal to 1.
So, an orthonormal set of vectors is a set such that
each pair of vectors is orthogonal to each other, and
where each vector is normalized.
In other words, it has norm squared equal to 1.
So, the inner product of these vectors is just delta sub j k.
If I have an orthogonal set, v sub j, say, then phi sub j is
an orthonormal set just by taking each of these vectors
and normalizing it.
In other words, it's no big deal to take an orthogonal set
of functions and turn them into an
orthonormal set of functions.
The Fourier series was natural to define that, and most
people define it, in such a way that it's not orthonormal.
Because we're defining it over some interval of time, t, that
has a norm squared of t and, therefore, you have to divide
by square root of t to normalize it.
If you want to put everything in a common framework, it's
nice to deal with orthonormal series.
And therefore, that's what we're going to be stressing
from now on.
So I want to go on so the real projection theorem.
Actually, there are three projection theorems.
There's the one-dimensional projection theorem.
There's the n-dimensional projection theorem, which is
what this is.
And then there's an infinite-dimensional
projection theorem, which is not general for all inner
product spaces, but is certainly general for L2.
So I'm going to assume that phi 1 to phi sub n is an
orthonormal basis for an n-dimensional subspace, s,
which is a subspace of v. How do I know there is such an
orthonormal basis?
Well, I don't know that yet, and I'm going to come back to
that later.
But for the time being, I'm just going to assume that as
part of the theorem.
Assume I have some particular subspace which has the
property that it has an orthonormal basis.
So this is an orthonormal basis for this n-dimensional
subspace, s and v. For each vector in v, s now is some
small subspace.
v is a big subspace out around s.
What I want to do now is, I want to take some vector in
the big subspace.
I want to project it onto the subspace.
By projecting it onto the subspace, what I mean is, I
want to find some vector in the subspace such that the
difference between v and that point in the subspace is
orthogonal to the subspace itself.
In other words, it's the same idea as we used before for
this over-used picture.
Which I keep looking at.
Here, the subspace that I'm looking at is just the
subspace of vectors colinear with u.
And what I'm trying to do here is to find, from v I'm trying
to drop a perpendicular to this subspace, which is just a
straight line.
In general, what I'm trying to do is, I have an
n-dimensional subspace.
We can sort of visualize a two-dimensional subspace if
you think of this in three dimensions.
And think of replacing u with some space, some
two-dimensional space, which is going through 0.
And now I have this vector, v, which is
outside of that plane.
And what I'm trying to do here is to drop a perpendicular
from v onto that subspace.
The projection is where that perpendicular lands.
So, in other words, v, minus the projection, is this v perp
we're talking about.
And what I want to do is exactly the same thing that we
did before with the
one-dimensional projection theorem.
And that's what we're going to do.
And it works.
And it isn't really any more
complicated, except for notation.
So the theorem says, assume you have this orthonormal
basis for this subspace.
And then you take any old v in the entire vector space.
This can be an infinite dimensional vector space or
anything else.
And what we really want it to be is some element in L2,
which is some infinite-dimensional element.
It says there's a unique projection in the subspace s.
And it's given by the inner product of v with each of
those basis vectors.
That inner product, times v sub j.
For the case of a one-dimensional vector, you
take the sum out and that was exactly the
projection we had before.
Now we just have the multi-dimensional projection.
And it has a property that v is equal to this projection
plus the orthogonal thing.
And the orthogonal thing, the inner product of that with s
equal to 0 for all s in the subspace.
In other words, it's just what we got in this picture we were
just trying to construct, of a two-dimensional plane.
You drop a perpendicular two-dimensional plane.
And when I drop a perpendicular to a
two-dimensional plane, and I take any old vector in this
two-dimensional plane, we still have the
perpendicularity.
In other words, you have the notion of this vector being
perpendicular to a plane if it's perpendicular whichever
way you look at it.
Let me outline the proof of this.
Actually, it's pretty much a complete proof.
But with a couple small details left out.
We're going to start out the same way I did before.
Namely, I don't know how to choose this vector.
But I know I want to choose it to be in this subspace.
And any element in this subspace is a linear
combination of these p sub i's.
So this is just a generic element in s.
And I want to find out what element I have to use.
I want to find the conditions on these coefficients here
such that v minus the projection -- in other words,
this v perp, as we've been calling it -- is orthogonal to
each phi sub i.
Now, if it's orthogonal to each phi sub i, it's also
orthogonal to each linear combination
of the phi sub i's.
So in fact, that solved our problem for us.
So what I want to do is, I want to set 0 equal to v minus
this projection.
Where I don't yet know how to make the projection, because I
don't know what the alpha sub i's are.
But I'm trying to choose these so that this, minus the
projection, the inner product of that with phi j is equal to
0 for every j.
This inner product here is equal to the inner product of
v with phi sub j.
I have this difference here, so the inner product is the
inner product of v with phi sub j minus the inner product
of this with phi sub j.
Let me write that here.
v minus sum alpha i phi sub i comma phi sub j is equal to v
phi sub j minus summation of i alpha i t sub i.
phi sub j.
Which is equal to this.
All of these terms are 0 except where j is equal to i.
Where i is equal to j.
So, alpha sub j has to be equal to the inner product of
v with this basis vector here.
And, therefore, this projection is equal, which we
said was sum of alpha i phi sub i, that's really the sum
of v phi sub j, this inner product, times p sub j,
Now, if you really use your imagination and you really
think hard about the formula we were using for the Fourier
series coefficients, was really the same formula
without the normalization in it.
It's simplified by being already normalized for us.
We don't have that 1 over t in here, which we had in the
Fourier series because now we've gone
to orthonormal functions.
So, in fact that sort of proves the theorem.
If we express v as some linear combination of these
orthonormal vectors, then if I take the norm squared of v,
this is something we've done many times already.
I just express the norm squared, just by expanding
this the sum of -- well, here I've done it this way, so
let's do it this way again.
When I take the inner product of v with all of these terms
here, I get the sum of alpha sub j complex conjugated,
times the inner product of v with phi sub j.
But the inner product of v with phi sub j is just alpha
sub j times 1.
So it's a sum of alpha sub j squared.
OK, this is this energy relationship we've
been using all along.
We've been using it for the Fourier series.
We've been using it for everything we've been doing.
It's just a special case of this relationship here, in
this n-dimensional projection, except that there we were
dealing with infinite dimensions and here we're
dealing with finite dimensions.
But it's the same formula, and you'll see how it generalizes
in a little bit.
We still have the Pythagorean theorem, which in this case
says that the norm squared of vector v is equal to the norm
squared of a projection, plus the norm squared of the
perpendicular part.
In other words, when I start to represent this vector
outside of the space by a vector inside the space, by
this projection, I wind up with two things.
I wind up both with the part that's outside of the space
entirely, and is orthogonal to the space, plus the part which
is in the space.
And each of those has a certain amount of energy.
When I expand this by this relationship here --
I'm not doing that yet -- what I'm doing here is what's
called a norm bound.
Which says both of these terms are non-negative.
This term is non-negative in particular, and therefore the
difference between this and this is always positive, or
non-negative.
Which says that 0 has to be less than or equal to this,
because it's non-negative.
And this has to be less than or equal to the norm of v. In
other words, the projection always has less energy then
the vector itself.
Which is not very surprising.
So the norm bound is no big deal.
When I substitute this for the actual value, what I get is
the sum j equals 1 to n of the norm of the inner product of
v, with each one of these basis vectors, magnitude
squared, that's less than or equal to the energy in v. In
other words, if we start to expand, as n gets bigger and
bigger, and we look at these terms, we take these inner
products, square them.
No matter how many terms I take here, the sum is always
less than or equal to the energy in v. That's called
Bessel's inequality, and it's a nice, straightforward thing.
And finally, the last of these things --
well, I'll use the other one if I need it.
The last is this thing called the mean
square error property.
It says that if you take the difference between the vector
and its projection onto this space, this is less than or
equal to the difference between the vector and the
other s in the space.
Any other --
I can always represent v as being equal to this plus the
orthogonal component.
So I wind up with a sum here of two terms.
One is the difference between -- well, it's the --
it's the length squared of the projection.
Write it out.
v minus v s -- let me write this term out. v minus s is
equal to v, the projection, plus v perpendicular to the
subspace s minus this vector s.
This is perpendicular to this and this, so -- ah,
to hell with it.
Excuse my language.
I mean, this is proven in the notes.
I'm not going to go through it now because I want to finish
these other things.
I don't want to play around with it.
We left something out of the
n-dimensional projection theorem.
How do you find an orthonormal basis to start with?
And there's this neat thing called Gram-Schmidt, which I
suspect most of you have seen before also.
Which is pretty simple now in terms of
the projection theorem.
Gram-Schmidt is really a bootstrap operation starting
with the one-dimensional projection theorem, working
your way up to larger and larger dimensions.
And each case winding up with an orthonormal basis for what
you started with.
Let's see how that happens.
I start out with a basis for an inner product subspace.
So, s1 up to s sub n as a basis for this
inner product space.
First thing I do is, I start out with s1.
I find the normalized version of s1.
I call that phi 1.
So phi 1 is now an orthonormal basis for the subspace whose
basis is just phi 1 itself.
phi 1 is the basis for the subspace of all linear
combinations of s1.
So it's just a straight line in space.
The next thing I do is, I take s2.
I find the projection of s2 on this subspace s1.
I can do that.
So I find a part which is colinear with s1.
I find the part which is orthogonal.
I take the orthogonal part, and that's
orthogonal, to phi 1.
And I normalize it.
So I then have two vectors, phi 1 and phi 2, which span
the space of functions of linear
combinations of s1 and s2.
And I call that subspace S2, capital S2.
And then I go on.
So, given any orthonormal basis, phi 1 up to phi sub k
of the subspace s k generated by s1 to s k, I'm going to
project s k plus 1 onto this subspace s sub k, and then I'm
going to normalize it.
And by going through this procedure, I can in fact find
an orthonormal basis to any set of vectors that I want to,
to any subspace that I want to.
Why is this important, is this something you want to do?
Well, it's something you can program a computer to do
almost trivially.
But that's not why we want it here.
The thing we want it here is to say that there's no reason
to deal with bases other than orthonormal bases.
We can generate orthonormal bases easily.
The projection theorem now is valid for any n-dimensional
space because for any n-dimensional space we can
form this basis that we want.
Let me just go on and finish this, so we can start dealing
with channels next time.
So far, the projection theorem is just for
finite dimensional vectors.
We want to now extend it to infinite dimensional vectors.
To accountably infinite set of vectors.
So I'm given any orthogonal set of functions, status sub
i, we can first generate orthonormal functions as phi
sub i, which are normalized.
And that's old stuff.
I can now think of doing the same thing that we did before.
Namely, starting out, taking any old vector I want to, and
projecting it first on to the subspace with
only phi 1 in it.
Then the subspace generated by phi 1 and phi 2, then the
subspace generated by phi 1, phi 2 and phi 3, and so forth.
When I do that successively, which is successive
approximations in a Fourier expansion, or in any
orthonormal expansion, what I'm going to wind up with is
the following theorem that says, let phi sub m be a set
of orthonormal functions.
Let v be any L2 vector.
Then there exists an L2 vector, u, such that v minus u
is orthogonal to each phi sub n.
In other words, this is the projection theorem, carried on
as n goes to infinity.
But I can't quite state it in the way I did before.
I need a limit in here.
Which says the limit as n goes to infinity of u, namely what
is now going to be this projection, minus this term
here, which is the term in the subspace of
these orthonormal functions.
This difference goes to zero.
What does this say?
It doesn't say that I can take any function, v, and expand it
in an orthonormal expansion.
I couldn't say that, because I have nothing to know whether
this arbitrary orthonormal expansion I started with
actually spans L2 or not.
And without knowing that, I can't state a
theorem like this.
But what the theorem does say is, you take any orthonormal
expansion you want to.
Like the Fourier series, which only spans functions which are
time limited.
I take an arbitrary function.
I expand them in this Fourier series.
And, bingo, what I get is a function u, which is the part
of v, which is within these time limits and what's left
over, which is orthogonal, is the stuff outside of those
time limits.
So this is the theorem that says that in fact you can --
AUDIENCE: [UNINTELLIGIBLE]
PROFESSOR: What?
AUDIENCE: [UNINTELLIGIBLE]
PROFESSOR: It's similar to Plancherel.
Plancherel is done for the Fourier integral, and in
Plancherel you need this limit in the mean on both sides.
Here we're just dealing with a series.
I mean, we still have a limit in the mean sort of thing, but
we have a weaker theorem in the sense that we're not
asserting that this orthonormal series actually
spans all of L2.
I mean, we found a couple of orthonormal expansions
that do span L2.
So it's lacking in that.
OK, going to stop there.