The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or to view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
JEREMY KEPNER: All right.
I want to thank you all for coming
to the third lecture of the Signal Processing
on Databases class.
So we will go-- just to remind folks,
all the material is available in the distribution
of the software, which you all have
in your [INAUDIBLE] accounts.
Here, we're going to go to lecture two.
Courses.
Course, which brings it together.
Signal processing, which is based
on detection theory, which is based on linear algebra
with databases, which are based on strings and searching
and really bringing those two concepts together
in the course.
So that we can use the mathematics
we know from detection theory and apply it into new domains.
And I'm glad to see we have a full house again.
So far, I've failed in scaring off people.
So we'll see if we can do a better job.
I think this particular lecture should
do a very good job of that.
We're going to be getting into a little bit
some more of the mathematics that
underpins the associative array construct that is in the D4M
technology that we're using in the class
and gets into some group theory concepts that might be
a long time since any of you had any of that or even
recall what that is.
It's actually not that complicated.
Personally, this is my favorite lecture
in terms of it really gets into the deeper underlying elements.
I would fully expect it to make your head hurt a little bit.
We're going to do sort of a fairly rapid blast
through the mathematics here.
And you won't really have to know this really
to effectively use it.
Although, it's good to know that it exists
and that there might be pieces of it
that you'll want to drive into deeper.
And so really, the title of this talk
is called Spreadsheets, Big Tables, and the Algebra
Associative Arrays.
We've lectured on that topic in the past.
So with that, we will get right into it.
So here's the outline for the lecture.
And I should say, you know, there's
a balance between the lecture, and then
we'll have some examples.
This particular lecture will be a little bit more heavy
on the lecture side.
And the example piece is a little bit shorter.
And so a little bit more lecture and a little less example
on this particular class.
So we're going to talk about kind of a mathematical concept
for what are spreadsheets.
And in the introduction here, talk
about what our theoretical goals are,
really what associative arrays are.
Getting into the mathematical definitions,
what is the group theory?
And then leading into the sort of traditional linear algebra,
vector spaces and other types of things,
sort of getting you a sense of this
really is what connects those two things at the deeper
mathematical level.
So what are spreadsheets and big tables?
So spreadsheets-- obviously, Microsoft Excel
tends to dominate the spreadsheet world--
are arguably the most commonly used analytics structure
on Earth.
Probably 100 million people, if not a larger number,
use a spreadsheet every single day.
Perhaps a good the fraction of the planet
now has used a spreadsheet at some point or the other.
Big tables, which are these large triple store
databases that we have discussed in prior lectures,
are really what is used to store most of the analyzed data
on Earth.
So Google, Amazon, they all have these giant triple store
databases, which can be thought about as gigantic spreadsheets
really, rows and columns and values and stuff.
And the power is that they really can simultaneously
store diverse data.
You can hold in the same data structure strings,
and dates, and integers, and reels,
and all different types of data.
And you can treat them very differently.
As we've shown here, this is just one spreadsheet.
We can treat it as a matrix.
We can treat it as functions, hash table, little mini
databases.
All in the same spreadsheet, we can kind of
hold this information.
So it's a very powerful concept, the spreadsheet.
Yet, we lack any formal mathematical basis for this.
In fact, I looked up in the American Mathematical
Association in the SIAM databases the word spreadsheet.
And it did not appear in a single title
or an abstract other than, say, maybe with reference
to some software.
But this core thing that we use every day--
every single time we do a Google search we're using it,
every single time we do a spreadsheet,
we're using it-- has no formal mathematical structure.
And so that seems like a little bit of a problem.
Maybe we should have some formal mathematical structure.
And so this mathematical thing called an associative array
actually gives us a mathematical structure that
really encompasses this in a fairly nice way.
I'm sure there's cases that it doesn't.
But the core of what it is really, really works.
So I think that's a powerful concept.
So what is our overall goal here?
So we want to create a formal basis
for working with these types of data structures in a more
mathematically rigorous way.
This allows us to create better algorithms.
Because we can now apply the traditional tools
of linear algebra and detection theory to these types of data.
It saves time in terms of, generally,
if you have the mathematical background,
you can implement things using a lot less code, which
usually means a lot less lines.
And often, you can build on top of existing
optimized libraries, which give you better performance.
And we throw out a number of 50 x less effort.
For those with the required training, they can do this.
And that is something that we've observed a number of occasions.
And you know, it's good for managers, too.
I've tried to recruit group leaders at Lincoln Laboratory.
It's like, you really should use D4M.
I could connect it to SAP no problem.
Get them feeling like they're actually doing programming
again, you know?
I haven't had too many takers of that yet.
So hopefully, maybe as you folks start using it,
they'll see you using it.
Binding to the various enterprise
databases at the laboratory is very easy to do.
And using D4M to analyze it is fairly easy to do.
So I actually use it now instead of-- if I'm going to use Excel,
I'll often be like, no, I'll just write a CSV file
and just use D4M for manipulating basic Excel
types of operations.
Because it feels a lot more natural to me.
And so I certainly have done that.
So we're going to get into this associative array
concept, which we discussed in some of the earlier lectures.
We're going to get into it more deeply now.
But the real benefit of the associative array concept
is it naturally pulls together four different ways
of thinking about your data into one structure.
And you can apply the reasoning from any
of these four ways of thinking sort of interchangeably now.
So we can think about data as associative arrays,
as arrays that are indexed by words.
This is similar to what Perl does in hashes of hashes, where
we can have row keys that are words,
and column keys that are words, and then
values that are words or numbers.
This concept is one to one with our triple store databases.
So we can go naturally from triples to this
and inserting data into a database.
And if our data is an associative array,
we can very quickly say, please insert into a database.
And if we do a query, it can return an associative array
to us very nicely.
This obviously connects to graphs
if we want to store relationships and graphs
between vertices and entities.
So we have alice, bob, cited, which
is a duel to the linear algebraic matrix
formulation of graphs.
Row, Alice, column, Bob, their edge associated between them.
So we have these four very powerful ways
of looking at our data.
And they're now all sort of brought together
as one concept.
So I kind of argued it's one of the best deals in math
you'll get.
If you know one, you get the other three for free.
And that's really what the whole point of math is, right?
If you give me a whole bunch of information,
I can calculate something else that you didn't know.
And typically, in signal processing
that deal is more like, well, if give me
50 pieces of information, I can give you the 51st.
Here, if you give me one, I can give you three.
So it's a really good bargain.
As a little anecdote, I was working with my daughter
teaching her the parallel geometry lines problem.
You know, parallel lines.
I was telling her this is the best deal in math.
Because you know one angle, and you get the other seven free.
So we're not quite as good as that.
That's probably the best deal in math going.
And that's why they teach it so much.
If you know one angle, you get the other seven for free.
Here, if you know one of these concepts,
you essentially get the other for free.
So it's a very powerful concept.
And it's really kind of at the core of the benefit
of this technology.
As much as possible, we try and make it
that any operation on an associative array
returns another associative array.
So this gives you sort of the mathematical concept
of closure.
Linear algebra works because the vast majority
of operations on a matrix return another matrix.
And so you can compose these together mathematically
and create sequences of operation,
which are conceptually easy to think about.
So because of that, we can create operations
like adding associative arrays, subtracting associative arrays,
anding, [? warring ?] them, multiplying them together.
And the result of every one of these operations
will be another associative array.
And so that's a very powerful, powerful concept.
And likewise, we can do very easy query operations
if we want to get rows Alice and Bob, row Alice, rows
beginning with al, rows Alice to Bob, the first two rows,
everything equal to 47.
And every single one of these will return
another associative array.
So again, this compositional concept
is very, very important.
It's not to say we don't use the triples formulation, too.
And we have routines that very quickly bounce you
back and forth between them.
Because there are times you're just like,
no, I want to view this is as a set of triples.
And I want to do new operations that way as well.
So we support them both.
But this is really sort of the core composable aspect.
In fact, most people initially develop their codes
in this formalism, and then maybe
change some of their lines to working on native triples
if they need to get some improvement in performance,
memory handling, or the side like that.
And we certainly do that, too.
These associative arrays are actually
very easy to implement.
And the whole D4M library-- it's probably a little larger now,
but when I wrote this-- it was about 2,000 lines,
At which is very easy to do in programming environments that
have first class support for two dimensional arrays, operator
overloading.
We're overloading operators here like crazy
here, the plus, the minus, the and, parentheses,
all that kind of stuff.
And also, have first class support
for sparse linear algebra.
Internal, under the covers, we're
using sparse linear algebra like crazy to make all this work.
Of the languages on Earth that have those features,
MATLAB is by far the most popular one,
which is why we have chosen that language
to implement these features.
There are other languages that have these features.
And you can implement that.
But there are other languages that don't.
And if you want to implement D4M in those languages,
you certainly can.
You just have to write more code.
So this is sort of the language in which it can be done
in the minimum amount of code.
And again, we find that for complex analytics
that people typically can get their work done
with 50 times less code than equivalent Java and SQL.
And this naturally leads to high performance parallel
implementations, because these are all arrays and matrices.
And we have a very well-established literature
in the community about how to do parallel linear algebra,
how to make matrices work in parallel.
It's a very well-studied discipline.
And we have a lot of technology.
Just to remind people of how we actually
store the data in our databases and how the data will often
come out in our associate arrays,
we use this exploded schema.
So if we had data that looked like a standard table
here with some kind of key and three columns
here with different values associated
with every one of them, if we were to insert these
into a standard, say, SQL type table,
we would just simply create a table like this in SQL.
But if we wanted to look up anything quickly,
we would have to create ancillary tables that
would store indices to allow us to do things very quickly.
And if we wanted to add a totally new type of data,
we would have to rethink our schema.
So what we typically do is we create this exploded schema.
Because our triple stores are very comfortable
dynamically storing very large numbers of columns.
So we essentially typically take the type and append the value.
So we create a series of triples here,
usually with some minimal information
for the actual stored value, something
we would never want to look up.
Again, by itself this doesn't give you much.
Because most of these triple stores have an orientation,
typically row oriented.
So you can get any row quickly with relatively little effort.
And so we also store the transpose of that.
We now create a table pair, such that we can look up
any row or column very quickly.
And so essentially, we've indexed all the data.
And you can now do things very, very quickly.
So that's a very powerful, powerful concept.
And we exploit it all time.
And of course, this whole concept
looks very much like an associative array.
Mathematically, we can treat it as an associative array.
All right.
So that's sort of the easy part.
I think we've covered a lot of this
before in previous lectures.
Now, we're going to get into some
of the more mathematical elements, which
aren't necessarily hard.
They just impose notation that we don't necessarily
use in our regular day to day work lives.
So we're going to talk about what
are the values associations, keys, functions,
and getting into things like matrix multiply.
So mathematically, we treat an associative array
as sets of keys and values that are
drawn from infinite strict totally ordered set,
we'll call it S. You might be like, what is that?
What is an infinite strict totally ordered set?
It's just a set where there's an ordering, where
basically any two elements in the set,
there's a function that will tell you
one is greater than the other.
And they also have an equal operation, too.
So it will tell you that, you know, to give it any two
values, I can ask the question is one greater than the other
or are they equal?
So that's the strict part.
So that encompasses a lot.
Obviously, it encompasses the usual things like real numbers,
and integers, and things like that.
But it also encompasses strings if we
impose an ordering on them.
And we will choose lexicographic ordering almost always
to be our order.
In fact, in the implementation, we sort of impose that.
We always use lexicographic ordering.
We don't really have another ordering that is readily
at our disposal.
So given those keys and values, an associative array
is a partial function from a d dimensional set
of keys to a value.
So if we have the vector of key, so typically d will be 2.
And the first k1 will be the row key, and then a column key.
And it will map out its partial function
from the d keys to one value.
So it will look like this, an associative array
with a vector of keys, typically 2.
I will have a value vi and that it is empty otherwise.
So this is its function, because we only define it
where these keys exist.
So you can Imagine that every singe associative
array is a function over the entire space
of all possible keys.
But we're only giving you the ones that are defined.
And everywhere else, it's undefined.
OK.
This is actually a fairly large break from linear algebra,
where we can have functions-- a matrix is really
a function of the indices, the ij indices, which are integers.
Here, we allow them to be anything.
And we formally support the concept
of a completely empty row or column.
So associative arrays don't allow that.
They only store not empty information.
And in our implementation, we tend to store 0 as equivalent
to null in this space.
Because the underlying sparse matrix implementation
does that.
0 is essentially treated as the null character.
So that's a big difference.
Now, we can still use our linear algebraic intuition,
but that's the fundamental difference here.
So binary operations on any two associative
arrays-- so I have associative array
A1 and A1 with this binary operation--
are defined essentially by two functions.
The sort of upper function here says how
we're going to treat the keys.
So if we have two associate arrays,
their keys, some of them can overlap.
Some of them cannot overlap.
And our function will either choose
to look for the union or the intersection of those.
So for instance, if we do addition,
that's most associated with unioning the underlying keys.
If we do other operations, like and, that's
most consistent with intersecting
the two sets of keys.
So obviously, if it's union-like,
the result will always be non-empty.
If the function is intersection-like,
then there's a possibility you can do a binary operation.
And the result will be an empty associative array.
So we have that choice there.
And we choose union and intersection,
because they're the ones that are-- you
could do other functions.
But they're the ones that are formally covered in set theory
and keep us from having to relax this very
general condition here.
We can have equality.
So we can check for intersection.
And so we're OK there.
So if we have an associative array,
A1 with a set of keys here, ki, and a value, v1,
and another associative array with an intersecting
ki with v2, then A3 ki will be given by this function.
It's either going to be the keys overlap.
So if ki are the same, they'll overlap.
And so there you go.
And likewise, with this.
You have this choice.
It could either be a union or an intersection.
And then what we do with the values-- like,
we now have the collision is what we'll call these.
Two keys from two associative arrays, the row keys
and the column keys are the same.
Well, now, we want to do something.
We apply this function f, which we call the collision function.
And that will determine what the actual result is, what
the new value of A3 will be.
If one of these was empty, then obviously the union of this we
don't even have to worry about it.
That's kind of the way we approach this.
If there is no collision, so that
means you're applying your function with this,
then f is never called.
So we always just do that or that.
So if it's a union function, then you just
get v. If it's an intersection function,
then you would just get the empty set.
And the underlying function never is called.
And you might be like, well, why would I care about that?
Well, later on I'm going to get to deal with things like, well,
what is the zero in this math?
Or what is one in this math?
And this will allow me to sort of address some of those cases
by saying, look, if there's no key intersection,
we can do that.
So the high level usage of associative arrays
is essentially dictated by this mathematics.
And again, part of the reason we chose this very general
definition is that, by definition,
numbers, reals, and integers are all included here.
We have not thrown them out.
And we've also included strings.
I should say at any time we can say, look, our values are just
real numbers.
And then we essentially get the full power of linear algebra.
But that's true of this sort of card
that we always try and play last.
OK.
We try and keep our mathematics as broad as we can.
And if then there's particular instances like,
oh, well, I need the values to be real numbers or complex
numbers here, or integers, or whatever.
We can always say, all right, the values are that.
And now, we get a whole bunch of additional properties.
But we try and build up the mathematics
in this general way first, and then sort of play that later.
AUDIENCE: Question.
JEREMY KEPNER: Yes.
AUDIENCE: So the way you've-- the general definition of keys,
that's more general than you would need to analyze
spreadsheets.
Wouldn't it be sufficient for spreadsheets
to have your keys be integers [INAUDIBLE]?
JEREMY KEPNER: So I believe that actually spreadsheets
store things as R1, and R2, R3 internally and C1, that they
actually use a internal triple representation,
and then project that into this integer space.
So what the actual spreadsheet does, I think,
kind of depends on the specific spreadsheet.
So the question was-- they can't really hear you on this mic.
Is this more than you need to do a spreadsheet?
Could you get away with just integer indexing?
And I would say perhaps.
I don't really know.
And as I said, I think internally they
actually do something that's more akin to this.
Certainly when you do math in Microsoft Excel,
it has letters.
You give it A1, B1, those types of things.
AUDIENCE: But that's just because they're using letters
instead of a second integer.
JEREMY KEPNER: Yeah.
Yeah.
One could view it that way.
But people like it.
So we do what people like, right?
AUDIENCE: Yeah.
But you don't need an additional mathematical structure.
[INAUDIBLE]
You can just make it one to one [INAUDIBLE] letters to--
AUDIENCE: [INAUDIBLE] like in Excel, you
have different worksheets, too, and different columns.
So you have to reference both of those.
AUDIENCE: You have to reference row and column.
But you just need a pair of integers.
AUDIENCE: Well, it's a row and column.
But then it's also the worksheets that you're on.
AUDIENCE: OK.
So, three.
AUDIENCE: So there's three.
But they're not going to sequentially number all those.
Because it's a sparse space that you're using, right, probably?
So I'm sure they do some caching.
I don't think they store a number for every column
that you're not using.
AUDIENCE: Certainly, you can try that out
by creating an empty spreadsheet,
putting the number in one corner,
putting a number in another corner really far away
and see if it [INAUDIBLE] two [INAUDIBLE].
JEREMY KEPNER: It doesn't do that.
Obviously, it does compress.
AUDIENCE: [INAUDIBLE].
JEREMY KEPNER: Yeah.
So mathematically, it may be more than what is minimally
required to do a spreadsheet.
We will discover, though, it is very useful
and, in a certain sense, allows you to do things
with spreadsheets that are hard to do
in the existing technologies, which is why I use it.
There's operations that I can do with this
that are very natural.
And the spreadsheet data is there.
And I would like to be able to do it.
AUDIENCE: So Jeremy?
JEREMY KEPNER: Yes.
AUDIENCE: The actual binary operator
plus in associative array, is it union or intersection?
JEREMY KEPNER: So on the keys, it
will be union where the value is an intersection
with an empty set.
We don't care what the actual numerical thing is.
We'll just return the other thing.
Where there is a collision, and so now you have to resolve--
and that's kind of why we get a lot of mileage here.
Because most of the time, we are very few collisions.
And so it's not like we have to worry about huge things.
So where there are collisions, if the values are numbers,
then it'll just do the normal addition.
If the values are strings, then it's somewhat undefined
what we mean there.
And I forget if D4M throws an error in that situation
if the values are strings.
It might default to just doing something
like taking the max or the min and saying,
you tried to add two things for which the formal addition
collision function doesn't really make sense.
And I'm just going to put something in there.
We can test that out at the end of the class.
AUDIENCE: So a collision function
is a function of that type of the value?
JEREMY KEPNER: Yes.
AUDIENCE: OK.
JEREMY KEPNER: It has to be a function
of the type of the value.
Because if I'm going to add a real number with a string,
what do I do?
Because I can do that.
Yes, another question.
AUDIENCE: To follow up on the previous point,
it seems like in the case for addition, for example,
there might be cases where you'd actually want union, then add,
or intersection, then add.
It kind of depends on what the nulls in your array
actually represent.
JEREMY KEPNER: Oh, there are so many functions
where you can think, well, I'd kind of
like to have it mean this in this context.
AUDIENCE: Yeah.
So the question is can you pass in
as a parameter in D4M I want you to union, then operate,
I want you to intersect, then operate?
Or did they all just assume the most logical and do it?
So bigger lever question was sometimes you
maybe would like plus to mean some
of the other conceptual ways that we would mean it.
OK.
So in fact, we see that a lot.
There's a lot of things.
Now, in fact, we'll get into that space.
And there's hundreds, thousands, millions
of potential possible algebras that you can possibly want.
So what we have done is we think about the function.
And we think about in this mathematical context.
And we see what its mathematical implications
are in terms of is it consistent with the overall group
theory of the mathematics, in which case
we know that if someone uses it that the intuition will carry
through?
Or is it kind of forking them off
into it's something you might want to do,
but really the result is now taking you
into a sort of undefined space?
So that's kind of a thing we use for doing that.
We don't have the formal support in this of a multiply operation
that allows you to pass in an arbitrary
function for doing that.
I should say there are various tricks that
allow you to do that in just a couple lines or two.
Basically, what you can do is you
start with give me the intersection of the unions
first.
Now, I know I have only collisions.
Now, give me the values.
Apply whatever operator you want.
Stuff the values back in, and you're done.
And so you can kind of work your way through that.
And by doing that, you're doing it in a way such that you know,
yeah, you're outside of an algebra here.
You're just kind of doing what you're doing.
So encourage people to do that.
But I'm just telling you in terms
of how we choose what functions we pull and formally support.
And there's a few that I've been wrestling
with for a long time, for a very long time
and just haven't been able to decide what to do.
Maybe I should just make a decision.
Yeah.
AUDIENCE: Like a real simple example,
if I had an array, one, two, three, and ray
b was null on all three, there would only be one collision.
And if we were in add, it would only add them at the last--
JEREMY KEPNER: No.
If it was an addition, it was--
AUDIENCE: So one, two, three.
And then the other one was null, null, three.
JEREMY KEPNER: So but they have the same column or something?
These are vectors?
AUDIENCE: Yes.
JEREMY KEPNER: These are row vectors?
Yes.
So basically we have a one column vector with three,
another column with one.
And when we add them together, you would get a three.
Because it would be union.
It would union the keys.
And then the only addition operation
would be performed on the collision.
We did one or and an and operation, then the result
would just be one, an associative array
with one element in it.
AUDIENCE: And we can do either one depending on the context?
I kind of lost you there.
Because the question was is it union or intersection.
JEREMY KEPNER: It depends on the operation.
So plus formally is closer to set addition,
and so therefore, is a union operation.
And is formally closer to set intersection,
and so it's an intersection operation there.
Regardless, in both cases, the collision function
is only applied where there is a collision.
Yes.
AUDIENCE: I'm getting a little confused by [INAUDIBLE]
formally defined operations on associative arrays
in terms of values.
You've used a quality on keys at this point.
You haven't defined any other operations such as [INAUDIBLE]
intersection [INAUDIBLE].
JEREMY KEPNER: Well, so the keys are part of a set.
And so they get the union and intersection properties
of strict totally ordered set, which
is the intuitive use of union.
AUDIENCE: Right.
But they belong to the same set, [INAUDIBLE]
the same [INAUDIBLE] set.
Yet, you've only defined binary operations
on associative arrays.
You haven't defined them on the elements of the set.
JEREMY KEPNER: So we'll get into that a little bit later.
AUDIENCE: That's what I thought.
JEREMY KEPNER: But they essentially
have the binary operations of a strict totally ordered set,
which is equality and less than.
AUDIENCE: It seems to me the bulk of the questions
were about manipulations on keys we haven't even
talked about yet.
JEREMY KEPNER: Yes.
Yes.
Well, there's another 40 some slides.
But actually, you've gotten a lot of the good questions here.
So the thing though to recognize is that there's
a lot of choices here.
And the algebra you are in is determined by the function
that you choose.
And actually, you'll find yourself
switching between different algebras fairly frequently.
That's how we use spreadsheets.
Sometimes we do this operation, and now we're in this algebra.
And we do this operation, now we're in this algebra.
And so that's the big thing to kind of get away there.
So let's get into this a little bit more.
That's sort of the big overall.
So I think we've talked about this.
Let S is an infinitely strict totally ordered set.
Total order Is an implementation, not
a theoretical requirement.
So the fact that we were imposing this infinite strict--
that they're totally ordered, that we don't just
allow equality, is more of an implementation detail.
It is very useful for me to internally store
things in order so I could look things up quickly.
However, mathematically, strictly, we just
had a test of equality, all the math hangs together there.
All value's and keys are drawn from the set.
And the allowable operations on them, of two values or keys,
is less than, equal to, or greater than.
So those are the three functions that we essentially allow.
Strict totally ordered set only is
two, which is less than, less than or equal to.
But you get essentially the third one for free.
In addition, we have the concept of three special symbols
here, which we've talked about, which
is the null, the empty set, a least element in the set,
and a maximal element to the set.
So v less than or equal to plus infinity is always true.
And plus infinity is, we're saying,
the maximal element is a part of this set.
Greater than or equal to negative infinity
is always true.
In all set theory, the empty set is a formal member of all sets.
So now we'll talk about-- did that sort of get
to your question about the operations that
are defined on the keys?
AUDIENCE: Yeah.
Yeah.
You [INAUDIBLE]--
JEREMY KEPNER: Right.
AUDIENCE: [INAUDIBLE].
JEREMY KEPNER: Right.
So then we talked about the collision functions.
And there are two sort of contextual functions
here, union and intersection.
Because there's the two operations that makes sense.
And then three conditions, less than, equal to,
or greater than.
That is in order to preserve this strict totally ordered
set.
Once I go to values that are reals or integers,
I have more operations.
But if I'm just limiting myself to values
that are members of a strict totally ordered set,
this is all I have.
So that means that you have d plus 5 possible outcomes
to any collision function.
That is, you could have the collision function actually
choose to put out the value of its underlying key.
You could produce v1 or v2 or empty
or minus infinity or plus or sets of these.
So it's a fairly finite choice of results.
If we're going to stay in this sort of restricted thing,
we have a very limited number of things
that can actually happen.
When you're actually applying a collision function
on a particular where there's an intersection,
you have a fairly finite number of things.
However the total number of combinations of these functions
gives a very large number.
And their function pairs gives a very large number
of possible algebras.
I did a back [INAUDIBLE] calculation
said it was 10 to the 30th.
It might be even more that.
And in fact, it might be formally
infinite for all I know.
But that was an easy calculation.
There's a lot of possible algebras.
And this is a fairly impressive level
of functionality given our relatively small numbers
of assumptions that we have here.
But we are going to focus here on just the nice collection
functions, the ones that seem to feel like they give us
intuitively useful things.
So we are not going to use keys as outputs to our function
here.
OK.
We will in some contexts.
And we're going to say the results are always
single valued.
We're not going to deal with results,
expand our values to be sets of values.
Although, in certain contexts we will do that.
But mathematically, let's restrict ourselves to that.
We're going to do no tests on special symbols.
And we're just going to basically say our collision
function can be essentially-- if we
have v, less than v2, the answers can
be one of these five.
If it's equal to these, it can be one of these four.
If it's greater than these, it can be one of these five.
So it gives a fairly limited number of possible collision
functions.
And all these properties are consistent with strict
totally ordered set.
And generally, when we handle a value of this,
it's handled by the union and intersection function first.
It never actually gets passed through to this function.
All right.
So let's move on here.
And just as I said, well, what about concatenation?
In fact, there are contexts where
we will want to concatenate a couple of strings together
or something like that.
And you know, that's fairly supported.
I think the results can be the sets themselves.
You would have a new special symbol,
which is the set itself.
You have now different collision functions.
So you have union.
And then I want a union values as our intersect,
and then union values and union and intersection.
So those are your functions there.
And there's actually a few instances.
I think we've already showed one of them in one of the examples
where we actually do this.
So I'm just sort of throwing that in there
that concatenation is a concept that we do think about.
I don't think we've developed the formalism as richly
in a situation where the values are simply single values.
But we certainly can and do do that.
AUDIENCE: Jeremy.
JEREMY KEPNER: Yes?
AUDIENCE: Third example, was that [INAUDIBLE]
for identity instead of union?
[INAUDIBLE]
JEREMY KEPNER: This one here?
AUDIENCE: No, go up.
Get a little bit to the left.
The symbol-- [INAUDIBLE] was upside down.
AUDIENCE: I have a similar question.
Is the one above it [INAUDIBLE].
JEREMY KEPNER: Yeah.
I might have a typo here.
I should double check that.
Yeah.
So v union that.
All right, so that should be an intersection.
And that should be union.
And that should be intersection.
And that should be union.
So I just got them to flop there.
So thank you.
Full marks to you.
We will make that change.
All right.
So that one is intersection, correct?
And this
AUDIENCE: [INAUDIBLE].
JEREMY KEPNER: --three should be intersection.
All right.
Very good.
We'll check in the SVN.
In the next down time, it will be part of your [INAUDIBLE].
Moving on.
All right.
So one of the things we're eventually
going to work towards is trying to make matrix multiply work
in this context.
OK.
We've already talked about the duality
between the fundamental operation of grass,
which is breadth first search and vector matrix multiply.
We continue to want to be able to use that.
It's a very powerful feature.
And what we find is that in graph algorithms, where we're
dealing with things on strings or other types of operations,
that most graph algorithms can be reduced to operations,
what are called on semi-rings.
So those are generalizations of normal linear algebra, whereby
your two core operations here, what is called the element
wise multiply and the element addition operation,
are this has the property of being associative
and distribute over the plus operation.
And plus has the property of being
associative and commutative.
And examples of this include the traditional matrix multiply
plus and multiply, min.+, or max.+.
In fact, there's a whole sub branch
of algebra called max plus algebras, which do this.
Another one is or.and and other types of things.
So mathematically, semi-rings are certainly well-studied more
in the context of specific algebras, like min.+, min+,
or max+, or something like that.
Here, with the associative arrays,
we're dealing with you could be hopping
and popping back and forth within the same expression.
You could be moving back and forth
between these different algebras.
So I think that's a little bit different.
We're taking sort of a bigger view, a more data-centric view.
Our data can be of different kinds, and then impose upon it
different algebras as we move forward.
And the real theory questions that we're
trying to answer here is we have this concept
of associative arrays, which is kind of new.
We have the traditional linear algebra.
And there's going to be areas where they overlap.
Your ideas, your intuition from linear algebra,
and your intuition from associative arrays
will be very, very well-connected.
OK.
And there's going to places where associative arrays
give you new properties that you didn't have in linear algebra.
And those will allow you to do new things.
And there's going to be cases where your linear algebra
intuition is wrong.
That is, taking your linear algebra intuition
and applying it to associative arrays
will lead you into things that won't make sense.
And so this is kind of what we're
trying to give people a sense here, of where they overlap,
where you should watch out, and where there's new properties
that you didn't know about that you should maybe take advantage
of.
The biggest one being the universal conformance
of addition and multiplication of all associative arrays.
So any two associative arrays, regardless of their size,
can be multiplied and added to each other unlike traditional
linear algebra, where you have very strict constraints
about the dimensions of matrices in order for them to be added.
And we do use this all the time.
We use this all the time, essentially almost exactly
in the example that we talked about.
It was a question that was brought up earlier.
We can add a vector of three elements
and a vector of five elements and have
something that makes sense.
OK.
Likewise, we can multiply a 10 by 12 with a 6 by 14
and have something that actually makes sense.
So that's a very powerful concept.
And I'd say it's one of the most commonly used features.
All right.
So we've done the basic definition.
Now, we're going to kind of probably
move a little bit quicker here into the group theory.
So the group theory is when you pick various functions.
What kind of algebra's a result of that?
So we're going to talk about binary operators,
and then get into commutative monoids, semi-rings, and then
something we call the feld.
So our joke there is that so a field
is when you do the traditional mathematics of linear algebra.
You're doing them over the real field or the complex field
of numbers.
It's a set of numbers with certain defined properties.
OK.
When we impose this condition that we
are doing with strict totally ordered sets,
we don't have an additive inverse.
There's no way to formally subtract
two words from each other.
We can't have a string and a negative string.
So we don't get, like, a and minus a,
which is part of a field.
So we have is feld, which is a field without an inverse.
So mathematicians are very funny.
And when they do this, they often
will drop the i or something like that to make that concept.
So we will be doing math over a--
we can have much of the properties of linear algebra.
We just have to recognize that we don't have additive inverse.
So it's sort of like vector spaces over felds.
Or sometimes they're called semi-vectors spaces.
You can put semi in front of anything.
And it's like it's a little bit of this, but minus something.
So this is kind of our operator roadmap.
So you see here, we began with three definitions.
We will have narrowed it down to 200 operators of interest
of those nice collision functions and union functions.
There's about 200 combinations that work well.
We will find that 18 of those operators are associative.
So not associative in terms of associative array,
but associative in terms of the formal mathematical concept
of associativity.
And so those form what are called semigroups.
14 of them are commutative, which means
they are Abelian semigroups.
Commutativity, that just means that you can
switch the order of operations.
a plus b is the same as b plus a.
Not something that is a strict requirement.
But gosh, it's sure nice to have to not
have to be like oh, no, I switched the order.
And now the answer is different.
So traditional matrix multiply is like that, right?
a times b is not the same thing as b times a.
And we will have that as well.
But commutativity is really nice.
We then take these 14 operations and explore
all their possible pairs.
There's 196 of them.
We look at which one of them distribute.
There's 74 that distribute, so they form semirings.
We then look to see if we have an identity and an annihilator.
Some of our special symbols, can they
perform that role in which case they can form a feld?
And then we can create, essentially,
vector semi-spaces or vector spaces over these felds.
And we'll have about 18 of them when we're all done.
And the good thing is when you get
all done with this, all the ones that you like
are still in the game.
And a few of the ones you might like, some of my favorites,
they don't get as far down this path as you want.
But then you have these properties.
And you're like, OK, I have vector space properties
and other types of things.
And I'm sure some of you will find issues or comments
or criticisms that we haven't written this up formally
in journals.
I've been desperately trying to hire abstract algebraists
as summer students and have not been successful so far.
So if you know any abstract algebraists that
would like to work on this, we would
be happy to pay them to help us work on this.
If we extend our definition of values
to include sets, ie, to include concatenation,
this is essentially what it does.
It adds four functions here.
They actually make it through all these operations here.
And they basically keep on going.
But then they fall away here when we try and get
to this final feld step.
But you know, this is still a very useful space to be in.
All right.
So let me explain a little bit of how
we sort of navigate this space.
So if we can limit ourselves to special function combinations
that are associative and commutative-- so just remember,
associative just means you can group your parentheses however
you want.
I always have to look it up.
Never once been like associative--
I mean, I don't know.
It's never been intuitive to me.
But that's the grouping of parentheses.
Commutativity just means that you can flip.
So these are our functions.
So these are all the 18 here.
And then the grade-- one's grade out of the ones you lose,
because of the commutativity.
So basically this function, which
is just left, which just says return the left value
in all circumstances, well, obviously, that's
not commutative.
And quite frankly, left and right are
sort of silly functions.
You Sometimes it almost is essentially a no op.
If you knew you want the left value,
you just take the left value and move on.
So that doesn't really-- so then we're
left with a lot of ones we like, essentially union max.
So basically, take the max value, union min.
I call this the intersection delta function just return.
It only gives you an answer if the values are the same.
This is essentially the union.
It's sort of like the x or function.
If there's a collision, blow it away.
And then there are very sort of other kind
of more unusual combinations here dealing
with the special functions.
We tend to kind of really live mostly here.
These tend to be the ones that really
are the ones you use a lot.
I haven't really used these too much.
All right.
And so we're left with these, what are called,
Abelian semigroups are these 14 highlighted ones here.
AUDIENCE: So Jeremy, I feel I'm going to get lost here.
What does Abelian mean?
JEREMY KEPNER: So Abelian is a semigroup that is commutative.
That's all it means.
So basically, it means you have associativity.
And then it's Abelian if you add commutativity to it.
So any sets of numbers that obey grouping operations
and are commutative is Abelian.
I mean, I think Abel did a lot more
that he didn't need to get his name associated with that.
It could have just been called comm-- it's sometimes just
called a commutative semigroup.
So you know, it's lucky when you get
your name added to some really trivial operation.
I mean, he did a lot of things.
But they threw him on here.
And in fact, in group theory, they complain about this.
And there are numerous rants about how they've
named all their groups after people,
and they don't tell you anything.
As opposed to other branches of mathematics, who would simply
just call this a commutative semigroup or just a commutative
associative group.
That might be even more-- and of all the different properties,
most of them you can combine in whatever you want.
There's almost an infinite number of groups.
So there's certain ones that are most useful.
Let's see here.
So now, we start--
AUDIENCE: Along those lines, what's the--
and I'm sorry if you already said
this-- distinction between a semigroup and a group?
JEREMY KEPNER: A group and a semigroup.
I want to say-- oh, it's an Abelian group without inverses.
So a semigroup there.
So these are all Abelian semigroup or Abelian groups
without inverses or commutative associative groups
without inverses.
So you see the problem.
And so I guess, he studied them a lot.
So he got his name associated with them
and proved their properties and stuff.
So we have 14 of those that form 196 pairs.
So these will begin to-- so we want
to look at the ones that are distributive.
So that basically means we assign one of those operators
to be the addition operation, the other to be
the multiplication operation.
And we need to show that it is distributive.
And of those 196, 74 operator pairs are distributive.
These are called semirings.
Or they could be called rings.
So a semiring is a ring without an inverse, so rings
without inverses and without identity elements.
And if you look at the various definitions,
I mean, there's the Wikipedia definition,
which mathematicians will say is that's not really
a true definition.
There's a Wolfram math rule definition,
which is more rigorous.
And they often disagree on this stuff.
I tend to be like if Wikipedia says is true,
then most people on the planet Earth
believe that that is what is true.
And so therefore, you should be aware that is the truth
that most people will believe.
So I don't know which one we've chosen here, whether it's
the Wikipedia or the Wolfram.
There's also various encyclopedias
that define this stuff, too.
AUDIENCE: Jeremy?
JEREMY KEPNER: Yes.
AUDIENCE: Can you define ring?
JEREMY KEPNER: Define what?
AUDIENCE: Ring.
JEREMY KEPNER: Ring?
AUDIENCE: Yeah.
JEREMY KEPNER: Oh, uh, well, a ring would be this.
And it would have inverses and identity elements.
So it's basically something that satisfies
that's distributive and has inverses and identity elements.
But if you type ring in the Wiki-- and I should say,
the great thing about Wikipedia is like all the definitions
are linked.
So after seven or eight clicks, it kind of all holds together.
So it's pretty nice.
And Wolfram is the same way, just enough knowledge
to be dangerous.
The internet's your friend, right?
So moving on here.
And this is something that's less important, but kind
of for completeness if we're going to head towards vector
space, we need to address, which is the concept of identity
elements.
So zero is the additive identity.
When you think of normal math, you add zero to something,
it doesn't change it.
And the choices for additive identity elements,
we have three special symbols, and we could pick them.
We have the multiplicative identity, which is 1,
and the multiplicative annihilator.
So of the choices here, we have 12 semirings
with the appropriate zeros and ones.
We have four that actually have two combinations.
And we have 16 total operations-- of the 16,
there are 6 operators.
These are different operators.
And again, we call these felds without inverses.
I'll get into it a little bit more.
So for instance-- and we can skip that for now.
So just a better way to look at that is these
are our operator pairs.
OK.
And we wanted to see which one of the them
sort of form these felds.
So the ones that distribute are marked with a D here.
The ones that distribute and have
a 0, 1 operator pair that works, are shown in the square here.
And some of them have two.
So if I pair [INAUDIBLE] plus union min
and multiply a intersection min, and I define 0
to be the empty value, and 1 to be plus infinity,
then I can create a feld, essentially, out of that.
I had a lot of debate with someone about this one,
whether I can have plus infinity be the 0 element
and have it be less than the 1 element.
Was there a requirement that 0 actually be less than 1
in this definition?
When I talked to mathematicians, they're like,
eh, it's kind of what you want, then you make it that.
So this just shows the full space
of things that are possible.
If we go back to our concatenation operators here,
so these were our four concatenation operators
and our collision functions.
And this shows you what that set looks like.
So these are the four by four pairs here.
All operators, they all distribute.
And 16 of these form semirings.
Because you're able to construct these various zeroes.
And this isn't, like, rigorous.
I mean, I might have messed one up or a few up here or there
or something like that.
But this just kind of gives you a sense
of the space that we're working on.
AUDIENCE: You made some typos on that table [? you ?] fix later.
JEREMY KEPNER: Probably.
Probably.
Now, I'm going to kind of really move it forward here.
So we get into vector spaces.
So we can have associative array vector addition.
Again, all associative arrays are conformant.
We have the concept of scalar multiplication,
which is essentially applied to all values.
So one of the things I really struggle
with-- so scalar multiplication kind of makes sense, right?
If I have an associative array and I multiply it by a scalar,
I can imagine just applying that in an intersection
sense only to the keys are defined by one.
Scalar addition, though, is very difficult.
If I have a scalar plus an associative array,
does it only apply to the keys?
Or is a scalar really the associative
array that's defined everywhere over all things?
So it's infinite.
So that's something I struggle with.
So when you ask why is scalar addition not supported in D4M,
it's because I don't know what it's supposed to mean.
And you can easily just pop out the values,
add whatever you want, stuff them back in,
and can be on your way.
And then your safe.
So in the vector space that we form,
it meets the plus requirements.
It commutes.
It's associative.
We have an identity.
But we have no inverse.
So we have to be careful.
That's why we don't have add.
And a vector space, it meets the scalar requirements.
So all associative array operator
pairs that yield felds, also result in vector spaces
without inverse spaces.
Maybe we call these vectors semispaces, I don't know,
or vector spaces over a feld or something like that.
What kind of properties here?
Well, we have scale identities.
That's great.
You could create subspaces.
That makes sense, too.
The concept of a span, yes, you can definitely
do concepts of spans.
Does span equal a subspace?
So this is a big question in vector space
theory, spans on subspaces.
Not sure.
Linear dependence.
Is there a nontrivial linear combination
of vectors equal to the plus identity?
You really can't do this without an additive inverse.
And so that becomes a little bit--
so we really kind of need to redefine linear independence,
which we can do.
But there's a lot of the proofs of linear independence
and dependence that rely on the existence of inverses.
And you probably could circle your way around that.
AUDIENCE: Question.
Are you just missing the additive inverse
or both the additive and the [INAUDIBLE]?
JEREMY KEPNER: Both.
Both, yeah.
Yeah.
So one of the things is considering
a linear combination of two associative array vectors.
Under what conditions do they create a unique result?
So this really depends on what you choose.
So for instance, if we have a vector A1 and A2,
when we multiply it by coefficients A1 and A2,
and we use these as our plus and our multiply,
and this is our 0 and 1, where are A1 uniquely determined?
And so for instance, if I pick, in this case,
our canonical identity vectors, which is A1
is equal to infinity and minus infinity here,
then we find that we can cover the space very nicely.
If we do it the other, we can't.
A better way to view that is in the drawing here.
So here's my whole space of coefficients A1 and A2.
And we see that A1 and A2 uniquely define a result
with these basis vectors.
And A1 and A2 are completely degenerate with these basis
vectors.
So depending on the kinds of basis vectors you have,
you can create unique stuff or not unique stuff.
You can give yourselves actual values here.
Basically, if A1 is just equal to this value
and A2 is equal to this value, this
shows you what that looks like.
There's places where it's unique.
There's places where they're the same.
And there's places where one is unique,
but the other is not unique.
So again, you can construct using associative arrays
as basis vectors, a very rich set of things.
And the same thing goes with multivalued vectors.
Again, different types of spaces here.
We really need to kind of work this out.
If anybody's interested, we're very
interested in having people help us work this stuff out.
Which of these operations make sense?
Transpose makes total sense.
Transposing of associative arrays makes total sense.
And it's a very efficient operation, by the way, in D4M.
You can do transposes very efficiently.
It works out very nicely.
Special matrices, submatrices, zero matrices, square matrices,
diagonal matrices, yes.
Although, diagonal matrices are a little bit tricky.
Upper and lower triangular, yes, you can kind of do this.
Skew symmetric, no.
Hermitian, not really.
Elementary and row column, sort of.
Row column equivalence, sort of, under certain conditions.
These are all things you can do in linear algebra.
And sometimes you can do them with associative arrays
and sometimes you can't.
You have to think about them.
Matrix multiply is sort of our crown jewel.
Always conformant-- can multiply any sizes whenever you want.
There's two ways to think about this.
You can make your head hurt a little bit when you start
dealing with the no elements.
When does the union operator get applied?
And when the-- so when you do computation,
there's two ways to formulate a matrix multiply.
There's the inner product formulation,
which is typically what people use when they actually
program it up.
Because it tends to be more efficient.
That's basically you take each row and each column,
you do an inner product, and then do the result. OK.
Mathematically, from a theory perspective,
you get yourself in less trouble if you
think in terms of the outer product formulation, which
is basically you take each row and vector,
you do the outer product to form a matrix,
and then you take all of these, and then combine them
all together with the operation.
And that, theoretically, actually keeps you sane here.
And that's the way to think about it mathematically.
Variety of matrix multiplies examples,
I won't go into them here.
Obviously, they depend heavily on what our collision
function, g, is here.
It gives you different values and different behaviors.
The identity element, maybe a left identity, right identity.
In some instances, it seems to be OK.
But the identity [INAUDIBLE] is a little bit tricky.
Inverses, boy, is it hard to construct inverses when you
don't have underlying inverses.
I It's just really tricky.
And so, probably are not going to get anything
that looks like an inverse.
You can do Eigenvectors in certain restrictive cases
sort of.
And there are interesting papers written about this.
But it's only on very-- the row and column keys
need to be the same and stuff and so, you know.
One thing I really would like to explore is the pseudoinverse.
So pseudoinverse A plus satisfies these properties.
And I actually think that will are probably
be in pretty good shape for pseudoinverse.
And the pseudoinverse is what you
need to solve the least-squares problem.
And I think solving the least-squares problem is
actually something we might really be interested in doing
in some of our problems.
So I do need to work this out if people would
like to explore this with me.
We have a whole set of theorems that we'd like to prove.
Spanning theorems, linear dependence, identities,
inverses, determinants, pseudoinverses, Eigenvectors,
convolutions, for which of these do these apply?
A lot of good math that could be done here.
Call to arms for those people who are interested in this type
thing.
So just to summarize, you know, the algebra
of associative arrays provides us
this mathematical foundation.
I think I've have tried to show you the core parts that
are really well solid and expand that to the points here.
You can see where it is, while we don't really
know exactly what should be happening here,
and give you a little bit a logic behind how we do this.
A small number of assumption really
yields a rich mathematical environment.
And so I have a short code example.
It's not really teaching you anything new.
It's just to show you that I tested all these properties
using D4M, which was nice and really in a very kind
of spreadsheet kind of style.
And so I'm just going to show you that.
There's really just one example.
It takes a few seconds to run.
And then the assignments-- so these
are in this part in the directory--
And then the assignment, should you so do it--
if you didn't do the last assignment, well,
you're going to need to do that to now do this assignment.
So basically, for those of you who did the last assignment--
array of a drawing and looking at the edges
and stuff like that.
Now, I want you think about that associative array.
And think about which kind of these operations
would make sense if you were to try and add them or multiply
them or whatever.
Just explore that a little bit.
And just write up, OK, I think these kind of operations
would make sense here.
You know, addition would make sense if it's union
and the collision function is this or something like that.
You'll have to think about what your values are.
It could just be your values are just 1, something like that.
You might be like, oh, my values are 0,1 and I want 1 plus 1
to equal 0.
So you might have an "or" or "x or" operation or something
like that.
So just sort of think about what your example was.
And think about these ideas a little bit.
And just write a few sentences on kind of what that means.
The last slide here is-- we don't really get this question.
But if you want to compare what the difference
between associative rays are and the algebra defined
by Codd that sort of is the basis of SQL,
there's this little table that describes
some of the differences there.