The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare continue to
offer high quality educational resources for free.
To make a donation or view additional materials from
hundreds of MIT courses, visit MIT OpenCourseWare at
ocw.mit.edu.
PROFESSOR: So this is the outline of
the lecture for today.
So we just need to talk about the conditional densities for
Poisson process.
So these are the things that we've seen before in previous
sections, so I'm just going to give a simple proof for them
and some intuition.
So if you remember the Poisson process, this was the famous
figure that we had.
So our time is at zero and afterwards zero and then we
had some arrivals.
So our time is then t.
We had N of t arrivals, which is equal to n.
This is Sn.
This is the figure that we had.
So for the first equation, we're looking at--
We want to find the conditional distribution of
interval times condition and last arrival times.
So first of all, we want to find the joint distribution of
the interval arrival times and the last
arrival, this equation.
So you know that interval arrival times are independent
in the Poisson process, so just looking at this thing,
you know that these things are independent of each other so
we will have lambda to the power of Ne to
the power minus lambda.
And then, this last one corresponds to the fact that
xn plus 1 is equal to n plus 1 minus summation of x.
These two events are equivalent to each other, so
this is going to correspond to lambda e to the
power of minus lambda.
So we just did it the other way, so
it's going to be like--
So you can just get rid of these terms and it's going to
be that equation.
So it's very simple, it's just the independence of
inter-arrival times.
So you look at the last term as the inter-arrival time for
the last arrival, and then they're going to be
independent, and the terms kind of cancel out and you
have this equation.
So this conditional density will be the joint
distribution--
or the distribution of Sn plus 1, which is like these.
So the joint decision will be like this.
So what does the intuition behind this mean?
This means that--
you've seen this term before.
This means that we have uniform distribution over the
interval of zero to one, condition on something.
So previously, what we had was the conditional distribution
of arrival times condition on the last arrival, so F of s1
and condition on sn plus 1.
But here we have distribution of x1 to x and
condition on sn plus 1.
And previously, the constraint that we had was like we should
have ordering over the arrival times.
So because of this constraint we had this n factorial, if
you remember, that order of statistics things that you
talked about.
Now here, what we have is something like this.
So all the arrival times should be positive, and
summation of them should be less than t.
Because well, the summation of them to n plus 1 is
equal to sn plus 1.
And the last term is positive, so this thing should
be less than t.
So these two constraints are sort of dual to each other.
So because of this constraint, I had some n factorial over
the conditional distribution of arrival times, condition on
the last arrival time.
Now, I have inter-arrival conditional distribution of
inter-arrival time condition on last arrival time and we
should have this n factorial here.
The other interesting thing is that if I condition it under n
of t, so the number of arrivals at time [INAUDIBLE]
t.
The time is going to be very similar to the previous one.
Just to prove this thing, it's very simple again.
I just wanted to show you--
So what does this thing mean?
This thing means that these are the distributions of the
inter-arrival times and the last event corresponds to the
fact that xn plus 1 is bigger than t minus summation of xi.
Because we have an arrival time at [? snn, ?]
and then after that there's no arrival.
Because at time nt we have n arrivals.
So this corresponds to do is thing.
So again, we'll have something like lambda n, e
to the power of--
This is for the first event and you see that these are
independent because of the Poisson properties of the
Poisson process.
And the last term will be--
property of the Poisson and a variable bigger
than something is--
Yes, this cancel out.
And then we'll have this term, this first n lambda to the
power of n, just forget about n plus 1.
And property distribution of n of t equal to n is this term
without n plus 1.
So lambda n to the 12, and e to the power minus lambda t
over n factorial.
So these terms cancel out and we'll have this thing.
So again, we should have these constraints that summation of
xk, k equal to 1, n should be less than t.
No matter if they are conditioning of n of t or sn
plus 1, the last arrival time or the number of arrivals at
some time instant, the condition of distribution is
uniform subject to some constraints.
Is there any questions?
So if you look at the limit, and so again, looking at this
figure, you see that this corresponds to sn plus 1 and
you see that sn plus 1 is bigger than t.
So if you take the limit and take the t1 going to t, these
two distributions are going to be the same.
So t1 corresponds to sn plus 1.
t1.
So basically what's it's saying is that conditional
distribution of arrival times are uniform, independent of
what is going to happen in future.
So in future, there might be some, I mean, the knowledge
that we have about future could be the fact that knowing
what happens in this interval or the next arrival is going
to happen at some point, I mean, condition on these
things the arrival times are going to be uniform.
The other fact is that this distribution that I showed
you, they are all symmetric with respect to interval
arrival times.
So it doesn't matter, I mean, no arrival has more priority
to any other one or they're not different to each other.
So if you look at the permutation of them, they're
going to be similar.
So now, I want to look at the distribution of x1 or s1, they
are equal to each other.
I can easily calculate the probability of x1 bigger than
some tau condition, for example, n of t is equal to n.
So what is this conditional distribution?
So condition on n of t equal to n, I know that in this
interval I have n arrivals and n inter-arrival times, and I
want to know if x1 is in this interval, or s1.
s1 is equal to x1.
And you know that si's are also uniform.
So you know that these are independent of each other, and
so you can say that the probability of this thing is
like this one.
So all of them should be bigger than tau.
And since this is symmetric--
so this corresponds to x1.
Since this was--
everything was symmetric for all inter-arrival times.
This is also the complimentary distribution function for xk.
Is there any questions?
So the other think that are not in the slides that I want
to tell you is the conditional distribution of
si given n of t.
So I know that there n arrivals in this interval, and
I want to see if the i arrival is something like this.
So the probability that--
so this even corresponds to the fact that there is one
arrival in this interval and i minus 1 arrival was here and n
minus i minus 1 arrival was here.
Oh, no, sorry, n minus i arrival is here.
So the probability of having one arrival
here corresponds to--
since arrival times our uniform, and I want to have
one arrival in this interval like this, and I want to have
i minus arrivals here i minus one arrivals here, and n minus
i arrivals here.
So it's like aq binomial probability distribution so
out of the n arrival-- n minus 1 remaining arrivals, I want
to have some of them in this interval and
some of them here.
So this is the distribution of this thing.
Yeah, so just as a check we know that the last
inter-arrival--
last arrival time, the distribution
of that thing is--
Here, you just cancel these things out, too--
is--
We need to get back to this equation, but it's just the
same as this one but you let i equal to n.
This is OK?
Just to think i equal to n, you get this one.
AUDIENCE: Does this equation also apply to i equals
[INAUDIBLE]?
PROFESSOR: Yeah, sure.
Yeah, i equal to 1, we will have nt minus tau.
And if you look at this thing, so this is a complimentary
distribution function.
You just take to zero till you get that.
This is just a check of this formulation.
Yeah.
So is there any other questions?
Did anybody get why we have this kind of formulation?
Is was very simple, it's just the combination of a uniform
distribution and a binomial distribution.
AUDIENCE: What is tau?
PROFESSOR: Tau is this thing.
Tau is the si.
Change of notation.
AUDIENCE: [INAUDIBLE]?
PROFESSOR: I said that I want one arrival in this interval
and i minus 1 arrival was here and n minus i arrival is here.
So I define it, the Bernoulli process.
So priority of falling in this interval is tau
over t minus tau.
And I say that--
AUDIENCE: [INAUDIBLE]?
PROFESSOR: OK, so falling in this interval in a uniform
distribution corresponds to success.
I want, among n minus 1 arrivals, i minus 1 of them to
be successful and the rest of them to be [INAUDIBLE].
[INAUDIBLE]
that one of them should fall here and the rest of them
should fall here.
Richard?
AUDIENCE: So that's the [INAUDIBLE]?
PROFESSOR: Yeah, you just get rid of this dt, that's it.
[? AUDIENCE: Why do ?] those stay, n over t.
PROFESSOR: Which one?
AUDIENCE: Can you explain again the first term?
The first term there, the nd over t?
PROFESSOR: Oh, OK, so I want to have one interval
definitely here but among n arrivals I want
them to fall here.
So any one of them should cancel.
But then i minus 1 of them should fall here and n minus i
intervals should fall here.
Any other questions?
It's beautiful, isn't it?
There's a problem.
I studied x1 to xn, but what is the distribution of this
remaining part?
So if I condition on n, nt or condition on sn plus 1, what
is the distribution of this one or this one or doing
distribution off all these?
So this is what we're going to look.
So first of all, I do the conditioning over sn plus 1,
and I find the distribution of--
the distribution of x1 to xn condition on sn plus 1, is
there any randomness in distribution of xn plus 1?
So this is going to be sx plus 1.
So this is going to be xn plus 1.
So if I find a distribution of x1 to xn condition on this
condition is sn plus 1, it's very easy to show that xn plus
1 is equal to sn plus 1.
So there's no randomness here.
But looking at the figure, you see that--
and the distributions, you see that everything is symmetric.
So I've found the distribution x1 to xn and I can find xn
plus 1 easily from them.
But what if I find the distribution of x2 to xn plus
1, condition on this thing, and then find x1 from them?
So you can say that x1 is equal to xn minus
1 2 to n plus 1xi.
And then we have this solution.
So it seems that there's no difference between them and
actually, there's not.
So condition on sn plus 1, we have the same distribution for
x1 to xn or x2 to xn plus 1.
But there are only n free parameters because the n plus
1's can be calculated in your equations.
If you take any n random variables out of this n plus 1
random variables, we have the same story.
So it's going to be uniform, our 0 to t for this interval.
So it's symmetric.
Now, the very nice thing is about n of t, equal t--
[? so ?] n must be equal to n.
So this is easy because we had these equations, but what if I
call this thing xn star of n plus 1 and I do the
conditioning over n of t?
So I don't know the arrival time for n plus 1's arrival
but I do know the number of arrivals at time [INAUDIBLE]
t.
Condition on that, what is the distribution for example, for
xn star n plus 1 or x1 to xn?
Again, easily we can say that it's the same
story, so it's uniform.
So you can take out of x1 to xn and xn star n plus 1, you
can take n off them and find the distribution of that.
OK, so the other way to look at this is that if I find the
distribution of x1 to xn condition on n of t, I will
have the expect that xs star n plus 1 is equal to t minus sn,
which is t minus summation of x to i.
Can anyone of you see this?
Hope so.
Oh, OK.
So again, if I find the distribution of x1 to xn, I
can find sx star n plus 1 deterministically.
Just as a signage check, you can find that f of the star of
x-- oh, sorry, f of x star n plus 1 condition on--
So looking at this formulation and just replacing everything,
we can't find this thing, and you can see that this is
similar to this one again.
So this is the distribution for--
well, this is the distribution for x1.
So you can find the density and this
is going to be density.
So what I'm saying is that x1 has the same marginal
distribution as xn plus 1 star.
So you can get it with this kind of reasoning, saying that
the marginal distribution should be the same or looking
at this kind of formulation.
When you have this distribution you can find the
distribution of summation of xi, i equal to 1 to n.
And then you can find--
So t is determined, so you can just replace
everything and find it.
And you can see that these two are the same.
AUDIENCE: [INAUDIBLE]
the same distribution that is one?
PROFESSOR: Yeah.
Condition on nt [INAUDIBLE].
So everything is symmetric and uniform.
But you can only choose n for your parameters because the n
plus 1 is a combination of the--
Right?
So this is the end of the chapter [INAUDIBLE]
for Poisson process, is there any problems or any
questions about it?
So let's start the Markov chain.
So in this chapter we only talk about the finite-state of
Markov chains.
Markov chains are some processes that changing
integer-time sense.
So it's like we're quantizing time and any kind of change
can happen only in an integer-time sense.
And it's different in this way from Poisson processes because
we can have the definition of Poisson processes for any
continuous time, but Markov chains are only defined for
integer-times.
And finite-state Markov chains is a Markov chain where states
can be only in a finite set, so we usually call it 1 to M,
but you can name it in any way you like.
What we are looking for in finite-state Markov chains is
the probability distribution of the next state conditioned
on whatever [? history ?]
that I have.
So I know that at integer times
there can be some change.
So what is this change?
We model it with the probability distribution of
this change corresponding to the history.
So we can model any discrete integer time
processing this way.
Now, the nice thing about Markov chains or homogeneous
Markov chains is that this distribution is
equal to these things.
So you see i and j, so i is the previous state and j is
the state that I want to go in so the distribution of the
next state condition and all the history only depends on
the last step.
So even the previous state, the new state is independent
of all their earlier things that might happen.
So there's a very nice way to show it.
So in general, we say that xn plus 1 is
independent of xn plus 1.
So if I know the periods--
and these things are true for all the possible states.
So whatever the history in the earlier process that I had, I
only care about the state that I was in in previous time.
And this is going to show me what is the distribution of
next state.
So the two important things that you can see this in this
formulation is that, first of all, this kind of
independence, it is true for all i, j, k, m, and so on.
The other thing is that it's not time-dependent.
So if I'm in state, the probability distribution of my
next state is going to be the same if the same thing happens
in 100 years.
So it's not time-dependent.
That's why we call it homogeneous Markov chains.
And we can have non-homogeneous Markov chains,
but there are not many nice results about them.
I mean, we cannot find many good things.
And all the information that we know to characterize the
Markov chain is this kind of distribution.
So we call these things transition probabilities.
And something else which is called initial probabilities,
which tell me that, at time instance 0, what is the
probability distribution of the states.
So knowing Markov chains, you can find that probability
distribution of x1 is equal to probability of x1 condition
and x0 [INAUDIBLE].
So knowing this thing and these transition
probabilities, I can kind the probability
distribution of x1.
Well, very easily, the probability
of xn is equal to--
sorry.
so you can do this thing iteratively.
And just knowing the transition probabilities and
initial probabilities, you can find the probability
distribution of the states at any time instant to forever.
Do you see it's very easy?
You just do this thing iteratively.
So I talked about the independence and initial.
Yeah, so what we do is, we characterize the Markov chains
with these set of transition probabilities
and the initial state.
And you see that these transition probabilities, we
have, well, m times n minus 1, free parameters in them,
because the summation of each of them should be equal to 1.
So for each initial state, I can have a distribution over
the next step.
So I'm going to talk about it in the matrix form.
And what we're doing in practice usually, we assume
the initial state to be a constant state.
So we usually just define a state, call it initial state.
Usually, we call it x0.
I mean, it's what's you're going to usually see if you
want to study the behavior of Markov chains.
So these are the two ways that we can visualize the
transition probabilities.
So matrix form is very easy.
So you just look at the n probability
distributions that you have.
And with Markov chain with n number of states, you will
have n distributions, each of them corresponding to the
conditional distribution of next step, condition of the
present step equal to i.
So i can be anything.
So this was the thing I was saying, the number of free
parameters.
So I have n distribution, and each distribution has n minus
1 free parameters, because the summation of them should be
equal to 1.
So this is the number of free parameters that I have.
So is there any problem with the matrix form?
I'm assuming that you've all been seeing
this sometime before.
Fine?
So the nice thing about matrix form is that you can do many
nice stuff with them.
We can look at the notes for them, and we will see these
kinds of properties later.
But you can imagine that just looking at the matrix form and
doing some algebra over them, we can get very nice results
about the Markov chains.
The other representation of the Markov chains is using a
graph, a directed graph in which each node corresponds to
a state and each arc, or each edge, corresponds to a
transition probability.
And the very important thing about graphical model is that
there's a very clear difference between 0
transition probabilities and non-0 transition
probabilities.
So if there's any possibility of going from one state to
another state, we have an edge or an arc here.
So probability of 10 to the power of minus 5 is
different from 0.
We have an arc in one case and no arc in another case,
because there's a chance of going from this state to the
next state.
And even then, the probability is 10 to the power of minus 5.
And OK, so we can do a lot of inference by just looking at
the graphical model.
And it's easier to see some properties of the Markov chain
by looking at the graphical models.
We will see these properties that we can find by looking at
the graphical model.
So this is not really
classification of states, actually.
So there are some definitions, very
intuitive definitions here.
So there's something called a walk, which says that this is
an ordered string of the nodes, where the probability
of going from each node to the next one is non-0.
So for example, looking at this walk, you can have the
probability of going from 4 to 4 is positive, 4 to 1, 1 to 2,
2 to 3, and 3, to 2.
So there is no kind of constraints in the definition
of the walk.
There should be just positive probabilities in going from
one state to the next state.
So we can have repetition.
We can have whatever we like.
And the number of-- well, we can find the number
of states for sure.
So what is the maximum number of steps in a walk?
Or minimum number?
Minimum number of states is two--
[? one. ?]
So what is the maximum number of steps?
Well, there's no constraint, so it can be anything.
So the next thing that we look is a path.
A path is a walk where there is no repeated nodes.
So I never go through one node twice.
So for example, I have this path, 4, 1, 2, 3.
We can see here, 4, 1, 2, 3.
And now we can say, what is the maximum number
of steps in a path?
AUDIENCE: [INAUDIBLE].
PROFESSOR: Steps, not states.
AUDIENCE: [INAUDIBLE].
PROFESSOR: n minus 1, yeah.
Because you cannot go through one state twice.
So you have a maximum number of steps in n minus 1.
And the cycle is a walk in which the first and last node
is repeated.
So first of all, there's not a really great difference
between these two cycles, so it doesn't
matter where I start.
And we don't care about the repetition and
definition of cycle.
Oh, OK.
Yeah, sorry.
No node is repeated.
So sorry, something was wrong.
Yeah, we shouldn't have any repetition except for the
first and last nodes.
So the first and last node should be the same, but there
shouldn't be any repetition.
So what is the maximum number of steps in this case?
AUDIENCE: [INAUDIBLE].
PROFESSOR: n, yeah, because we have an additional step.
Yeah?
AUDIENCE: You said in a path, the maximum number of
steps is n minus 1?
PROFESSOR: Yeah.
AUDIENCE: I mean, if you have n equals, like, 6, couldn't
there be 1, 2, 3, 4, 5, 6?
PROFESSOR: No, it's the steps, not the states.
So if I have path from 1 to 2, and this is the path.
So there's one step.
Just whenever you're confused, make a simple model.
It's going to make everything very, very easy.
Any other questions?
So this is another definition.
We say that a node j is accessible from i if there is
a walk from i to j.
And by just looking at the graphical model, you can
verify the existence of this walk very easily.
But the nice thing, again, about this thing is what I
emphasized in the graphical model.
It talks about if there's any positive probability of going
from i to j.
So if j is accessible from i and you start at node i,
there's a positive probability that you will end in node j
sometime in the future.
There's nothing said about the number of steps needed to get
there, or the probability of that, except that this
probability is non-0.
It's positive.
So for example, if there's a state like k, that we have p
of ik is positive and p of kj is positive, then we will have
p of ij [? too ?] is positive.
So we have this kind of notation.
I don't have a pointer.
Weird.
So yeah, we have this kind of my notation. pijn means that
the probability of going from state i to state j in n steps.
And this is exactly like n steps.
Actually, this value could be different for
different number of n.
For example, if we have a Markov chain like this, p of
131 is equal to 0, but p of 132 is non-0.
So node j is accessible from i if pijn is positive for any n.
So we say that j is not accessible from i if this is 0
for all possible n's.
Mercedes?
AUDIENCE: Are those actually greater than or equal to?
PROFESSOR: No, greater.
They are always greater than or equal to 0.
Probability is always non-negative.
So what want is positive probability, meaning that
there's a chance that I will get there.
I don't care how small this chance is, but it
shouldn't be 0.
AUDIENCE: [INAUDIBLE].
So p to ij, though, couldn't it be equal to pik, [? pkj? ?]
PROFESSOR: Yeah, exactly.
So in this case, p132 is equal to p12, p23,
AUDIENCE: So I guess I'm asking if p2ij should be
greater than or equal to pfk.
PROFESSOR: Oh, OK.
No, actually.
You know why?
Because there can exist some other state like here.
AUDIENCE: Right.
But if that doesn't exist and there's only one path.
PROFESSOR: Yeah, sure.
But there's no guarantee.
What I'm saying is that Pik is positive, Pkj is positive.
So this thing is positive, but it can be bigger.
In this case, it can be bigger.
I don't care about the quantity, about the amount of
the probability.
I care about its positiveness, it's greater than 0.
I just want it to be non-0.
The other thing is that when we look at pijn, I don't care
what kind of walk I have between i and j.
It's a walk, so it can have repetition, it can have
cycles, or it can have anything.
I don't care.
So if you really want to calculate pij for n equal to
[INAUDIBLE]
1,000, you should really find all possible walks from i to j
and add the probabilities to find this value.
And all the walks within steps.
Not here.
Let's look at some examples.
So is node 3 accessible from node 1?
So you see that there's a walk like this, 1, 2, 3.
So there's a positive probability of going to state
3 from node 1.
So node 3 is accessible from 1.
But if you really want to calculate the probability, you
should also look at the fact that we can have cycles.
So 1, 2, 3 is a walk.
But 1, 1, 2, 3 is also a walk.
1, 2 3, 2, 3 is also a walk.
You see?
So you have to count all these things to find the probability
of p13n for any n.
What about state 5?
So is the node 3 accessible from node 5?
You see that it's not.
Actually, if you're going to state 5, you never go out.
With [INAUDIBLE]
probability of 1, you stay there forever.
So actually, no state except state 5 is accessible from 5.
So is node 2 accessible from itself?
So accessible means that I should have a walk from 2 to 2
in some number of steps.
So you can see that we can have 2, 3, 2,
or many other walks.
So it's accessible.
But as you see, node 6 is not accessible from itself.
If you are in node 6, you always go out.
So it's not accessible.
So let's go back to these definitions.
Yeah, this is what I said, and I'm emphasizing again.
If you want to say that j is not accessible from i, I
should have a pijn equal to 0 for all n.
The other thing is that, if there's a walk from i to j and
from j to k, we can prove easily that there's a walk
from i to k.
So having a walk from i to j means that for some n, this
thing is positive.
So this is i to j.
From j to k, this means that p of jk, for
some n, this is positive.
So looking at i two k, I can say that p of i to k, m plus
n, is greater than.
And I have this thing here because of the reason that I
explained right now, because there might be other paths
from i to k, except the paths going through j, or except the
walks that have n steps to get to j and n steps to get to k.
So we know that, well, we can concatenate the walks from--
so if there's a walk from i to j and j to k, we have a walk
from i to k.
So we say that states i and j communicate if there's a walk
from i to j and from j to i.
So it can go back and forth.
So it means that there's a cycle from from i
two i or j two j.
This is the implication.
So it's, again, very simple to prove that if i communicates
with j and j communicates with k, then i communicates with k.
In order to prove that, I should assume that i
communicates to j, I need to prove that there is a walk
from i to k, and I need to prove that there is a walk
from k to i.
So this means that i communicates with k.
So these two things can be proved easily from the
concatenation of the walks and the fact that i and j
communicate and j and k communicate.
Now, what I can define is something
called a class of states.
So a class is called a non-empty set of states, where
all the pairs of states in a class communicated with each
other, and none of them communicate with any other
state in the Markov chain.
So this is the definition of class.
So I just group the pairs that can communicate with the state
and just get rid of all those who do not communicate to the
[INAUDIBLE].
So for defining a class or for finding a class, or for naming
a class, we can have a representative state.
So I want to find all the states that communicate with
each other in a class.
So I can just pick one of the states in this class and find
all the states that communicate with this single
state, because if two states communicate with one state,
then these two states communicate with each other.
And if there's a state that doesn't communicate with me,
it doesn't communicate with anybody else whom I'm
communicating with.
I'm going to prove it now, just in a few moments.
Just, I want to look at this figure again.
So first I take state 2 and find a class that has this
state in itself.
So you see that in this class, we have state 2 on 3, because
there's only a state 3 that communicates with 2.
And correspondingly, we can have class
having state 4 and 5.
And you see that state 1 is communicating with itself, so
it's a class by itself.
But it doesn't communicate with anyone else.
So we have this class also.
So next question, why do we call C4 a class?
So it doesn't communicate with itself.
If you're in state 6, you go out of it with probability of
1, eventually.
So why do we call it a class?
Actually, so this is the definition of the classes.
But we want to have some very nice property of other classes
which says that we can partition the states in a
Markov chain by using the classes.
So if I don't count this case as a class, I cannot partition
it, because partitioning means that I should cover the whole
states in the classes.
What I to do is to do some kind of partitioning in a
Markov chain that I have, by using the classes, so that I
can have a representative state for each class.
And this is one way to partition the Markov chains.
And why do I say it's partitioning?
Well, it's covering the whole finite space of the states.
But I need to prove that there's no intersection
between classes also.
Why is it like that?
So meaning that I cannot have two classes where there is an
intersection between them, because if there's an
intersection--
for example, i belongs to C1 and i belongs to C2--
it means that i communicates with all the states in C1 and
i communicates with all states in C2.
So you can say that all states in C1 communicate with all
states in C2.
So actually, they should be the same.
And there's only these states that
communicate with each other.
We have this exclusivity that's conditional.
So we can have this kind of partitioning.
Is there any question?
Everybody's fine?
So another definition which is going to be very, very
important in the future for us is the recurrency.
And it's actually very simple.
So a state is called recurrent if for all the states that we
have j is accessible to i, we also have i is
accessible to j.
So if from some state i, in some number of states, I can
go to some state j, I can get back to i from
this state for sure.
And this should be true for all the
states in a Markov chain.
So if there's a walk from i to j, there should be a walk from
j to i, too.
If this is true, then i is recurrent.
This is the definition of recurrence.
Is it OK?
And if it's not recurrent, we call it transient.
And why do we call transient?
I mean, the intuition is very nice.
So what is a transient state?
A transient state says that there might be some kind of
walk from i to some k.
And then I can't go back.
So there's a positive probability that I go out of
state i in some way, in some walk, and never
come back to it.
So it's kind of transitional behavior.
AUDIENCE: [INAUDIBLE]?
PROFESSOR: No, no, with some probability.
So there exists some probability.
There is a positive probability that I go out of
it and never come back.
It's enough for definition of transience.
So you know why?
Because I have the definition of recurrence for all the j's
that are accessible from i.
So with probability 1--
oh, OK, so I cannot say with probability 1.
Yeah, I cannot say probability of 1 for recurrency.
But for transient behavior, there exists some probability.
There's a positive probability that I go out
and never come back.
State 1 is transient.
And state 3, is it recurrent or transient?
Some idea?
AUDIENCE: [INAUDIBLE].
PROFESSOR: Transient?
What about state 2?
It's wrong, because there should be something going on
[INAUDIBLE].
So now here, it's recurrent.
Good?
Oh yeah, we have examples here.
So states 2 and 3 are recurrent, because the only
state that I can go out from this state is to themselves.
So they are only accessible from themselves.
And there's a positive probability of going from one
to the other.
So they're recurrent.
But states 4 and 5 are transient.
And the reason is here, because there is a positive
probability that I go out of state 4 in that direction and
never come back.
And there's no way to come back.
And state 6 and 1 are also transient.
State 6 is very, very transient because, well, I go
out of it in one step and never come back.
And state 1 is also transient, because I can go out of it in
this direction and never be able to come back.
Do you see?
So this is a very, very, very important theorem, which says
that if we have some classes, the states in the class are
all recurrent or all transient.
This is what we are going to use a lot in the future, so
you should remember it.
And the proof is very easy.
So let's assume that state i is recurrent.
And let's define Si, all the states that communicate with
i, that are accessible from i.
And you know that since i is recurrent, being accessible
from i means that i is accessible from them.
So if j is accessible from i, i is also accessible from j.
So we know that i and j communicate with each other if
and only if j is in this set.
So this is a class.
This is the class that contains i.
So this is like the class that I told you, that we can have a
class where this state i is representative.
And actually, any state in a class can be representative.
It doesn't matter.
So by just looking at state i, I can define a class.
The states that are accessible from i-- and actually, this is
the class that contains i.
So let's assume that there's a state called k which is
accessible from some j which is in this set.
So k is accessible from j, and j is accessible from i.
So k is accessible from i.
But k accessible from i implies that i is also
accessible from k, because i is recurrent.
So you think i is accessible from k?
And you know that j is also accessible from i because,
well, Si was also a class, recurrent class.
So you see that j is also recurrent.
So if k is accessible from j, then j is also accessible from
k for any k.
So this is the definition of recurrency.
So what I want to say is that if i is recurrent and it's in
the same class as j, then j is recurrent for sure.
And we didn't prove it here, but if one of them is
transient, them all of them are transient too.
So the proof is very simple.
I proved that if there is any state like k that's is
accessible from j, I need to prove that j is also
accessible from k.
And I proved it like this.
It's easy.
So the next definition that we have is the definition of the
periodic states and classes.
So I told you that the number of steps in a walk-- so when I
say that there's a walk from some state to the other state,
I didn't specify the number of steps needed to get from the
state to another state.
So assuming that there is a walk from state i to i,
meaning that i is accessible from i, it's not always true.
You might go out of i and never come back to it.
So assuming that there's a positive probability that we
can go from state i to i, you can look at the number of
steps needed for this walk.
And we can find the greatest common divisor of
this number of steps.
And it's called the period of i.
And if this number is greater than 1, then i is
called to be periodic.
And if it's not, it's aperiodic.
So the very simple example of this thing
is this Markov chain.
So probability of 1,1 in step is 0.
probability of 1, 1 in two steps is positive.
And actually, probability of 1, 1 for all even number of
steps is positive.
But for all odd number of steps, it's 0.
And actually, you can prove that 2 is the
greatest common divisor.
So this is the very easy example to
show that it's periodic.
So there is a very simple thing to check if
something is aperiodic.
But it doesn't tell me--
I mean, if it doesn't exist, we don't know its periodicity.
So if there is a walk from state 1 or state i to itself,
and in this walk, we go from a state called, like, j, and
there is a loop here--
so Pjj is positive.
What can we say about the periodicity of i?
No, the period is 1.
Yeah, exactly.
So in this case, state i is always aperiodic.
So if there's a walk, and in this walk, there is a loop, we
always say that.
But if it doesn't exist, can we say anything about the
periodicity?
No.
It might be periodic.
It might be aperiodic.
It's just a check.
So whenever you see a loop, it's aperiodic.
Whenever you see a loop in the walk from i to i, if there's a
loop in the other side of the Markov chain, we don't care.
So the definition is fine.
So just looking at this example, so if we're going
from state 4 to 4, the number of states that we need is,
like, 4, 6, 8.
So 4, 1, 2, 3, 4 is a cycle, or is a walk from 4, to 4, and
the number of steps is 4.
4, 5, 6, 7, 8 9, 4 is another walk.
So this corresponds to n equal to 6.
And 4, 1, 2 3, 4, 1, 2, 3, 4 is another walk which
corresponds to n equal to 8.
So you see that we can go like this or this or this.
So these are different n's.
But we see that the greatest common divisor is 2.
So the period of state 4 is 2.
For state 7, we have this thing.
So the minimum number of steps to get from 7 to itself is 6,
and then you can get from 7 to 7 in 10 steps.
And I hope you see it.
So it's going to be like this.
And so again, the greatest common divisor is 2.
So we proved that if one state in a class is recurrent, and
then all the states are recurrent.
And we said that recurrency corresponds to having a cycle,
having a walk from each state to itself.
So this is the result, very similar to that one, which
says that all the states in the same class
have the same period.
And in this example, you see it too.
The proof is not very complicated, but it takes time
to do that.
So I'm not going to do it.
But it's all nice.
So it's good if you look at the text, you'll see that.
AUDIENCE: [INAUDIBLE]
only for recurrent states?
PROFESSOR: Yeah.
For non-recurrent ones, it's--
AUDIENCE: [INAUDIBLE].
PROFESSOR: It's 1.
It's aperiodic.
OK, yeah.
Periodicity is only defined for recurrent states, yeah.
We have another example here.
I just want to show you.
So I have two recurrent classes in this example.
Actually, three.
So one of them is this class.
What is the period for a class corresponding to state 1?
It's 1, because I have a loop.
It's very simple.
For any n, there is a positive probability.
What is the period for this class, states--
containing states 4 and 5?
Look at it.
There is definitely a loop in this class.
So it's 1.
No, I said that whenever there is a loop, the greatest common
divisor is 1.
So for going from a state 5 to 5, we can go it in one step,
two step, three step, four step, five.
So the greatest common divisor is 1.
So here, I showed you that if there is a loop, then it's
definitely aperiodic, meaning that the greatest common
divisor is 1.
AUDIENCE: [INAUDIBLE] have self-transitions [INAUDIBLE]
2?
PROFESSOR: No, I'm talking about 4 and 5.
In what?
AUDIENCE: If they didn't have self-transitions?
[INAUDIBLE].
PROFESSOR: Oh yeah.
It would be like this one?
AUDIENCE: Yeah, [INAUDIBLE].
PROFESSOR: Yeah, definitely.
So the class containing states 2 and 3, they are periodic.
AUDIENCE: Oh, wait.
You just said [INAUDIBLE].
PROFESSOR: Oh, OK.
AUDIENCE: [INAUDIBLE].
PROFESSOR: Yeah, actually, we can define
for transient states.
Yeah, for transient classes, we can define periodicity,
like this case.
Yeah, why not?
This is another very important thing that we can
do in Markov chains.
So I have a class of states in a Markov chain,
and they are periodic.
So the period of each state in a class is greater than 1.
And you know that all the states in a class have the
same period.
So it means that I can partition the class into these
subclasses, where there is only--
OK, so--
do I have that?
I don't know.
So let's assume that d is equal to 3.
And the whole team is the class of states that I'm
talking about.
I can partition into three classes, in which I only have
transitions from one of these to the other one.
So there's no transition in this subclass to itself.
And the only transitions are from one
class to the next one.
So this is sort of intuitive.
So just looking at three states, if the period is 3, I
can partition it in this way.
Or I can have it like two of these in this case, where I
have like this.
You see?
So there's a transition from this to these two states, and
from these two states to here, and from here to here.
But I cannot have any transition from here to
itself, or from here to here, or here to here.
Just look at the text for the proof and illustration.
But there exists this kind of partitioning.
You should know that.
Yeah, so if I am in a class, in a subclass, the next state
will be in the next subclass, for sure.
So I know this subclass in [INAUDIBLE].
And in two steps, I will be in the other [INAUDIBLE].
So you know that the set of classes that I can be in
state, nd plus m.
So the other definition that you can say is that, again,
you can choose one of the states in a class that I
talked about.
So let's say that I choose a state 1.
And I define Sm--
Sm corresponds to that subclass--
are all the j's where S1j.
So I told you that I will have d classes, I can define each
class like this.
So this is all the possible states that I can be in nd
plus m step.
So starting from state 1, in nd plus m step, I can be in
set of classes, for some m.
So m can we big.
But anyway, I call this the subclass number m.
So let's just talk about something.
So I said that in order to characterize the Markov
chains, I need to show you the initial state or initial state
distribution.
So it can be deterministic, like I start from some
specific state all the time, or there can be some
distribution, like px0, meaning that this is the
distribution that I would have at my initial state.
I don't have it here, but using, I think, chain rules,
we can easily find a distribution of states at each
time instant by having the transitional probabilities and
the initial state distribution.
I just wrote it for you.
So for characterizing the Markov chain, I just need to
tell you the transition probabilities
and the initial state.
So a very good question is that, is there any kind of
stable behavior when time goes, like in very far future?
So I had an example here, just looking at here.
So this is a very simple thing that I can say about this
Markov chain.
I know that in the very far future, there is a 0
probability that I'm in state 6.
And you know it why?
Because any time that I am in state 6, I will go out of it
with probability 1.
So there's no chance that I will be there.
Or I can say that, if I start from state 4, in very, very
far future, I will not be in state 1, 4, or 5, for sure.
You know why?
Because there is a positive probability of going out of
these threes states, like here.
And then if I go out of it, I can never come back.
So there's a chance of going from state 4 to state 2.
And if I ever go there, I will never come back.
So these are the statements that I can say about the state
of behavior or the very, very future behavior
of the Markov chain.
So the question is that, can we always say these kind of
statements?
And what kind of statements, actually, we can have?
And so you see that, for example, for a state six
[INAUDIBLE]
example, I could say that the probability of being in state
6 as n goes to infinity is 0.
So can I always have some kind of probability distribution
over the states in the future as n goes to infinity?
This is a very good question.
And actually, it's related to a lot of applications that we
have for Markov chains, like the queuing theory.
And you can have queues for almost anything.
So one of the most fundamental and interesting classes of
states are the states that are called ergodic, meaning that
they are recurrent and aperiodic.
So if I have a Markov chain that has only one class, and
this class is recurrent and aperiodic, then we call it a
ergodic Markov chain.
So we had two theorems saying that if a state in a class is
recurrent, then all the states are recurrent.
And if a state in a class is aperiodic, then all the states
are aperiodic.
So we can say that some classes are ergodic and some
are not ergodic.
So if the Markov chain has only one class, and this class
is aperiodic and recurrent, then we call it a ergodic
Markov chain.
And the very important and nice property of ergodic
Markov chains is that they lose memory as n goes to
infinity, meaning that whatever initial distribution
that I have for the initial state, I will
lose memory of that.
So whatever state that I start in, or whatever distribution
that I start in, after a while, for a large enough n,
the distribution of states does not depend on that.
So again, looking at that chain rule, I could say that I
can find the probability distribution of x by looking
at this thing.
And looking at these recursively, it all depends on
P of the initial distribution.
And the ergodic Markov chains have this property that, after
a while, this distribution doesn't depend on initial
distribution anymore.
So they lose memory.
And actually, usually we can calculate the stable
distribution.
So this thing goes to a limit which is called pj.
So the important thing is that it doesn't depend on i.
It doesn't depend where i starts.
Eventually, I will converge the distribution.
And then this distribution doesn't change.
So for a large enough n, I have this property for ergodic
Markov chains.
We will have a lot to do with these
properties in the future.
So I was saying that this pj, pi j, should be positive.
So being in each state has a non-0 probability.
The first thing that I need to prove is that pijn and is
nonzero for a large enough n for all j and all the initial
distributions.
And I want to prove that this is true for
ergodic Markov chains.
This is not true, generally.
Well, this is more a combinatorial issue, but there
is a theorem here which says that for an ergodic Markov
chain, for all n greater than this value, I have non-0
probability of going from i to j.
So the thing that you should be careful here is that, for
all n greater than this value.
So for going from a state 1 to 1, I can go it in six steps
and 12 steps and so on.
But I cannot go it in 24 steps, I think.
I cannot go from 1 to 1 in 25 steps.
But I can go from 1 to 1 in 26, 27, 28, 29, to 30.
So for n greater than n minus 1 squared plus 1, I can go
from any state to any state.
So I think this [? bond ?] is tied for state 4.
Sorry, maybe what I said is true for state 4.
Yeah.
So I'm not going to prove this theorem.
Well, actually you don't have the proof
in the notes, either.
But yeah, you can look at the example and the cases that are
discussed in the notes.
Is there any questions?
Fine?