The point of this class isn’t writing that code, but I will show it for the interested student!
Finding what’s next
I loop over all quotes.
I loop over all words in the quotes.
If the current word is “to”, I save the next word.
I look at all the next words.
next= []for quote in words:for loc, word inenumerate(quote):if word =="to":next.append(quote[loc +1])print(next)
['do', 'be', 'be', 'do', 'be', 'be']
Do this for all words
We aren’t restricted for doing this just for “to”
Do it for all the words!
We have to add one special case.
We check quote length to make sure there is a next word.
nexts = []for first in unique:next= []for quote in words:for loc, word inenumerate(quote):if word == first and loc +1<len(quote):next.append(quote[loc +1]) nexts.append(next)
We’ll use another coding thing that I should mention.
I think it will be important to AI in the future, not seeing much usage now.
Key-Value Storage
A dictionary
What we really need is something a lot like a dictionary.
Rather just have a list of lists of words, we want a list of words for each starting word.
Dictionaries also contain “lists of words” (definitions) for each word.
In computing, we term this “key-value storage”.
Keys are words
Values are definitions.
Colab has dictionaries
There so happens to be something called a dictionary (well, a dict) I can use.
d =dict()print(d)
{}
Adding keys
We add things to dictionary as key-value pairs
We take the name of the dictionary, like my_values
We add box brackets []
Within those brackets, we give the key (e.g. the word for which we are storing the definition
We use single-equals assignment to set the value of the key within the dictionary.
Like this:
d["to"] = ['do', 'be', 'be', 'do', 'be', 'be']
Seeing values
It is easy enough to see a value from here.
Same as setting a value, just without single-equals assignment.
We take the name of the dictionary, like my_values
We add box brackets []
Within those brackets, we give the key (e.g. the word for which we are storing the definition
print(d["to"])
['do', 'be', 'be', 'do', 'be', 'be']
Back To Work
Use A dict
Same as before, but we use a dictionary.
The key is “before” or “first” word.
The value is the “next” or “second” word.
nexts =dict()for first in unique:next= []for quote in words:for loc, word inenumerate(quote):if word == first and loc +1<len(quote):next.append(quote[loc +1]) nexts[first] =next
is ['to', 'to']
be ['is', 'or', 'do', 'do']
do ['is', 'be', 'be']
or ['no']
to ['do', 'be', 'be', 'do', 'be', 'be']
no ['to']
Takeaways
for first in nexts:print(first, nexts[first])
is ['to', 'to']
be ['is', 'or', 'do', 'do']
do ['is', 'be', 'be']
or ['no']
to ['do', 'be', 'be', 'do', 'be', 'be']
no ['to']
“no” only precedes “to”.
“or” only precedes “no”
“is” only precedes “to”
Others are more complex.
Early sketch
“no” only precedes “to”.
“or” only precedes “no”
“is” only precedes “to”
Meaning of weights
We have a problem now.
Historically, we have used determinism
Every single time we see a six-sided die, we say it represents a six.
Now we must use non-determinism
Sometimes when we see a “to”, it is followed by “be”
Sometimes it is followed by “do”.
How do we handle this?
Calculate Weights
Let’s take an edge weight to be the probability.
“to” will have an edge of weight \(\frac{2}{3}\) going to “be”.
Let’s look at “be”
The way I think of this is:
After a “be”, at 25% probability see an “is”
After a “be”, at 25% probability see an “or”
After a “be”, at 50% probability see an “do”
Generating Text
So, if we want to generate text.
We look at the current word.
We flip a coin or roll a die.
For “be” we use a four-sided die.
And two numbers correspond to “do”.
Rolling Die in Colab
We have already seen random numbers, when setting up our perceptron.
It is easy enough to.
Generate a random number between zero and one.
Multiply it by the number of “next words”
Round or truncate to a whole number
Look up the number in that position, which is now the next number.
In Colab
Here’s the code - don’t worry about understanding it unless you want.
nexts is the dictionary we made earlier.
def next_word(prev_word): die = np.random.rand() possible_words = nexts[prev_word] position = die *len(possible_words) position =int(position)return possible_words[position]
Try it out
We’ll ask for the next word several times and hope to get different answers.
next_word("be")
'or'
next_word("be")
'do'
next_word("be")
'is'
next_word("be")
'do'
next_word("be")
'do'
Generating text
We can now use a neural network to generate text!
We just provide a starting word.
We will also provide a length, though there’s ways around that!
word ="to"print(word)for _ inrange(4): word = next_word(word)print(word)
to
be
do
be
is
Clean it up
Rather than printing each word, we’ll add words together into one big thing to print.
We’ll also put all the code in a function.
We’ll allow the function to accept a starting word and a length.
def make_text(first_word, num_words): text = first_word word = first_wordfor _ inrange(num_words -1): word = next_word(word) text = text +" "+ wordprint(text)
Some examples
make_text("to", 5)
to be or no to
make_text("to", 5)
to be do be do
make_text("to", 5)
to do be is to
make_text("to", 5)
to do is to be
make_text("to", 5)
to be do is to
make_text("to", 5)
to be do be or
make_text("to", 5)
to do be is to
More examples
Longer.
make_text("to", 10)
to be do be is to do be or no
make_text("to", 10)
to be do be do is to be do is
Start with other words.
make_text("be", 5)
be is to be is
make_text("no", 5)
no to be do is
make_text("is", 5)
is to be do is
Bonuses
If Time
We can look at start and stop tokens.
We can discuss attention.
Summary
What we learned
You can generate text with neural networks.
We used a single layer, but of course…
It seems an awful lot like you can do anything by stacking them.
