See (https://reinforcementlearning4.fun/2019/06/16/gym-tutorial-frozen-lake/)
FrozenLake is an environment from the openai gym toolkit.
It may remind you of wumpus world.
The first step to create the game is to import the Gym library and create the environment. The code below shows how to do it:
# frozen-lake-ex1.py
import gym # loading the Gym library
env = gym.make("FrozenLake-v0")
env.reset()
env.render()
The first instruction imports Gym objects to our current namespace. The next line calls the method gym.make() to create the Frozen Lake environment and then we call the method env.reset() to put it on its initial state. Finally, we call the method env.render() to print its state.
Their meaning is as follows:
Red square: indicates the current position of the player Also, we can inspect the possible actions to perform in the environment, as well as the possible states of the game:
# frozen-lake-ex1.py
print("Action space: ", env.action_space)
print("Observation space: ", env.observation_space)
In the code above, we print on the console the field action_space and the field observation_space. The returned objects are of the type Discrete, which describes a discrete space of size n. For example, the action_space for the Frozen Lake environment is a discrete space of 4 values, which means that the possible values for this space are 0 (zero), 1, 2 and 3. Yet, the observation_space is a discrete space of 16 values, which goes from 0 to 15. Besides, these objects offer some utility methods, like the sample() method which returns a random value from the space. With this method, we can easily create a dummy agent that plays the game randomly:
# frozen-lake-ex2.py
import gym
MAX_ITERATIONS = 10
env = gym.make("FrozenLake-v0")
env.reset()
env.render()
for i in range(MAX_ITERATIONS):
random_action = env.action_space.sample()
new_state, reward, done, info = env.step(
random_action)
env.render()
if done:
break
Does anyone see a problem with the output? What is going on? (answer in zoom chat)
The code above executes the game for a maximum of 10 iterations using the method sample() from the action_space object to select a random action. Then the env.step() method takes the action as input, executes the action on the environment and returns a tuple of four values:
Finally, we use the method env.render() to print the grid on the console and use the returned “done” flag to break the loop. Notice that the selected action is printed together with the grid.
Note in the previous output the cases in which the player moves in a different direction than the one chosen by the agent. This behavior is completely normal in the Frozen Lake environment because it simulates a slippery surface. Also, this behavior represents an important characteristic of real-world environments: the transitions from one state to another, for a given action, are probabilistic. For example, if we shoot a bow and arrow there’s a chance to hit the target as well as to miss it. The distribution between these two possibilities will depend on our skill and other factors, like the direction of the wind, for example. Due to this probabilistic nature, the final result of a state transition does not depend entirely on the taken action.
By default, the Frozen Lake environment provided in Gym has probabilistic transitions between states. In other words, even when our agent chooses to move in one direction, the environment can execute a movement in another direction:
# frozen-lake-ex3.py
import gym
actions = {
'Left': 0,
'Down': 1,
'Right': 2,
'Up': 3
}
print('---- winning sequence ------ ')
winning_sequence = (2 * ['Right']) + (3 * ['Down']) + ['Right']
print(winning_sequence)
env = gym.make("FrozenLake-v0")
env.reset()
env.render()
for a in winning_sequence:
new_state, reward, done, info = env.step(actions[a])
print()
env.render()
print("Reward: {:.2f}".format(reward))
print(info)
if done:
break
print()
Executing the code above, we can observe different results and paths at each execution. Also, using the info object returned by the step method we can inspect the probability used by the environment to choose the executed movement
However, the Frozen Lake environment can also be used in deterministic mode. By setting the property is_slippery=False when creating the environment, the slippery surface is turned off and then the environment always executes the action chosen by the agent:
# frozen-lake-ex4.py
env = gym.make("FrozenLake-v0", is_slippery=False)
Observe that the probabilities returned in the info object is always equals to 1.0.
env.reset()
env.render()
for a in winning_sequence:
new_state, reward, done, info = env.step(actions[a])
print()
env.render()
print("Reward: {:.2f}".format(reward))
print(info)
if done:
break
print()
The default 4×4 map is not the only option to play the Frozen Lake game. Also, there’s an 8×8 version that we can create in two different ways. The first one is to use the specific environment id for the 8×8 map:
# frozen-lake-ex5.py
env = gym.make("FrozenLake8x8-v0")
env.reset()
env.render()
The second option is to call the make method passing the value “8×8” as an argument to the map_name parameter:
# frozen-lake-ex5.py
env = gym.make('FrozenLake-v0', map_name='8x8')
env.reset()
env.render()
And finally, we can create our custom map of the Frozen Lake game by passing an array of strings representing the map as an argument to the parameter desc:
custom_map = [
'SFFHF',
'HFHFF',
'HFFFH',
'HHHFH',
'HFFFG'
]
env = gym.make('FrozenLake-v0', desc=custom_map)
env.reset()
env.render()
Note: I believe that the lmbda parameter below is actually gamma, the discount factor that plaqued many of you in problem 17.5.
import numpy as np
def value_iteration(env, max_iterations=100000, lmbda=0.9):
stateValue = [0 for i in range(env.nS)]
newStateValue = stateValue.copy()
for i in range(max_iterations):
for state in range(env.nS):
action_values = []
for action in range(env.nA):
state_value = 0
for i in range(len(env.P[state][action])):
prob, next_state, reward, done = env.P[state][action][i]
state_action_value = prob * (reward + lmbda*stateValue[next_state])
state_value += state_action_value
action_values.append(state_value) #the value of each action
best_action = np.argmax(np.asarray(action_values)) # choose the action which gives the maximum value
newStateValue[state] = action_values[best_action] #update the value of the state
if i > 1000:
if sum(stateValue) - sum(newStateValue) < 1e-04: # if there is negligible difference break the loop
break
print(i)
else:
stateValue = newStateValue.copy()
return stateValue
env.reset()
statevalue = value_iteration(env)
env.nS and env.nA gives the total number of states and actions resp. But the most interesting is env.P ; env.P[0] outputs a dictionary like this. Here 0 in env.P[0] is the first state of the environment.
env.P[0]
Here as you can guess, the keys of the dictionary 0,1,2,3 are the actions we can take from state 0. And further each action contains a list, where each element of the list is a tuple showing the probability of transitioning into the state, next state, reward and if done=True done=False. (done=True if the next state is a Hole or the Goal). So env.P is a list containing all the states where each state contains a dictionary which maps all possible actions from that state to the next_state if we take that action, probability of going into that next state, reward and if the Game terminates there or not.
tuple: (probability of transition, next state, reward, done)
env.P
env.P[23]
Since this is not a 8x8 grid, there are not 64 cells. We are using the custom grid, which is 5x5. The goal cell is 24.
env.P[24]
So as per the Value Iteration formula, we iterate through all these actions and calculate the action-state value using the formula:
Prob (reward + discount_factor state_value of next state)
which are all provided in env.P . Then we update the value function of the state with the highest state-action value. We iterate through all the 25 states of the environment, till the difference between the new State Values and Old State Values after each iteration is negligibly small or if we have crossed the maximum number of iterations.
Now that we have the value function of all the states, our next step is to extract the policy from the Value Function.
We do this using a similar technique. For a particular state we calculate the state-action values of all the possible actions from that state and choose the action with the highest state-action value.
def get_policy(env,stateValue, lmbda=0.9):
policy = [0 for i in range(env.nS)]
for state in range(env.nS):
action_values = []
for action in range(env.nA):
action_value = 0
for i in range(len(env.P[state][action])):
prob, next_state, r, _ = env.P[state][action][i]
action_value += prob * (r + lmbda * stateValue[next_state])
action_values.append(action_value)
best_action = np.argmax(np.asarray(action_values))
policy[state] = best_action
return policy
policy = get_policy(env, statevalue)
Finally! Now that we have the policy we can follow that policy and see if our agent reaches the goal or falls in a hole.
def get_score(env, policy, episodes=1000):
misses = 0
steps_list = []
for episode in range(episodes):
observation = env.reset()
steps=0
while True:
action = policy[observation]
observation, reward, done, _ = env.step(action)
steps+=1
if done and reward == 1:
# print('You have got the fucking Frisbee after {} steps'.format(steps))
steps_list.append(steps)
break
elif done and reward == 0:
# print("You fell in a hole!")
misses += 1
break
print('----------------------------------------------')
print('You took an average of {:.0f} steps to get the frisbee'.format(np.mean(steps_list)))
print('And you fell in the hole {:.2f} % of the times'.format((misses/episodes) * 100))
print('----------------------------------------------')
We run the agent for 1000 episodes and calculate how many average steps it took to get to the Goal. We also calculate how many times it could not reach the goal and fell in a hole. Finally we get this answer after running the above function.
get_score(env, policy)
get_score(env, policy)
This is terrible! The example in the article was working on a larger grid, 8x8, rather than a smaller 5x5 grid. What's going on?
Answer in zoom chat.