Using Simulations and AI to Optimize Supply ChainS
Illustrated using the Beer Distribution Game
Berlin, 28.5.2020
Transforming Enterprises
Topics Today
Introduction to the Beer Distribution Game, a supply chain simulation.
Use computational modelling to understand supply chain dynamics and find better playing strategies.
Train autonomous agents to play the game using a machine learning / reinforcement learning approach
Watch A LIve Recording Of This Presentation
Download the Code and Computational Notebooks
Download all the code and
in-depth computational notebooks from GitHub.
All resources are available via www.transforming-enterprises.com
The Beer Distribution Game
An Introduction
- The Beer Game was developed in the 1960s at MIT to illustrate how difficult it is to manage dynamic systems – in this case a supply chain that delivers beer from a brewery to the end consumer
- The game became well known in the 1990s after Peter Senge's description of it in his world-wide best selling book The Fifth Discipline
- The game is great because even though it is about a very simple system and despite very simple rules, the resulting behaviour is quite complex.
A Brief Walkthrough OF the BEER Game
Try it yourself at beergame.transentis.com
The game is usually played with four players, but the single player version is also fun.
But let's look at the rules and some potential pitfalls first!
The situation
- You are part of a supply chain that delivers beer from a brewery to the end consumer
- try to meet the demand of your respective customer at all times, while keeping inventory low.
The Rules
The rules of the game are simple – in every round you perform the following steps:
- Check deliveries. Check how many units of beer are being delivery to you from your supplier in the supply chain.
- Check orders. Check how many units of beer your client in the supply chain has ordered.
- Deliver beer. Deliver as much beer as you can to satisfy demand (the game does this for you).
- Make an order decision. Decide how many units of beer you need from your supplier to keep your inventory stocked up.
1
2
3
4
Some Pitfalls to be aware of
Backorder and inventory costs
Delays
How Performance is Appraised
Individual Supply Chain Cost.
Your accumulated cost should remain below $8.300.
Overall Supply Chain Costs.
Total supply chain cost should remain below $29.300.
let's Give it A go
Order Behaviour in a typical Game
What the consumer orders
How the supply chain reacts
Change in order behaviour from 100 units to 400 units
Peak order of over 30,000 units!
Surplus in a typical Game
ThE Feedback Loop Governing The Supply Chain
ThE Feedback Loop Governing The Supply Chain
The problem: Each player can only control part of the control loop
ThE Error most people Make: Including the Back Order
Improvement Strategy 1: Ignore Back Orders
Outgoing Orders = Incoming Orders + Target Inventory - Inventory
Improvement Strategy 1: Ignore Back Orders
Improvement Strategy 1: Ignore Back Orders
Improvement Strategy 2: Remember opeN orders
Remember the orders that are in the supply line
Improvement Strategy 2: Remember opeN orders
Remember the orders that are in the supply line
The Target Supply Line Depends on the Delivery Delay
Target Supply Line = Delivery Delay * Incoming Orders
Orders = Incoming Orders + Target Inventory - Inventory + Target Supply Line - Supply Line
Order Behaviour with the new ordering policy
The Individual Cost Target is now reached
The whiplash effect
Even if every player behaves rationally, there will still be a "whiplash" effect – because the orders become successively larger along the supply chain
SuppLy chain costs are still way off target
Improvement Strategy 3: Increase Inventory adjustment time
The solution to dealing with the whiplash effect is to adjust inventory slowly!
ImproveD order Behavour
Target Supply chain costs are met
Target Retailer Costs are also Met
SuMMARY OF the computational modeling approach
Build a computational model to capture your understanding of the system
Use simulations to test the effect of different policies
Find policies that help you reach your targets
Controlling complex systems is hard and even harder when you only have partial control
We often don't behave rationally when overwhelmed with information and under pressure
Sometimes it pays to "have faith in the system"
Small changes can have large effects
Even if everybody optimises locally, this doesn't necessarily lead to a global optimum
Concrete learnings from the Beer Game
Training AI To Play the Beergame
An approach using reinforcement learning
The idea: use autonomous agents to play the game and train them using reinforcement learning, a machine learning technique.
Agents have information about their environment.
They perform actions and receive rewards (or punishments) for them.
They learn through trial and error.
Agents For the Beer game
The agents for the Beer Game are very simple
Q-Learning
A reinforcement learning technique
Each agent has a "Q-Table", which defines the expected total reward for each state and each possible action.
Agents start with an "empty" q-table and then fill it by learning through trial and error.
Always choose the action with the highest reward!
THe Key: Setting the right rewards
Much like with us human beings, we need to set the right rewards
A reward for reaching the cost targets (and some milestones along the way)
"Game over" as soon as cost targets are missed (to avoid wasting time and memory)
THe Results
Ten episodes
37,500 Episodes (almost there)
50,000 Episodes
The AI agents can now play the game
THe Winner Is ...
The agents outperform our initial ordering strategy!
But: the agents are optimised towards the concrete game situation - they would fail in a more dynamic stetting.
The ordering strategy we developed initially is robust in all ordering situations.
SUMMARY
Reinforcement learning algorithms are quite easy to implement.
Finding the right reward policies is difficult: it is hard to avoid setting rewards to narrow (much like in real life).
BUT: If you can define clear objectives and rewards, using reinforcement learning in combination with simulations can be very useful for automating control systems.
Next meetup After The Summer Break
Building An Adaptive Enterprise on 24.9.2020
What characteristics define an adaptive enterprise?
Which capabilities and buidling blocks do you need to create an adaptive enterprise?
How can you get there?
An adaptive enterprise is an enterprise that can flexibly adapt to a changing economic environment.
Save The Date!
"Virtual Coffee Lounge"
for post-event discussions
Dr. Oliver Grasl
Oliver co-founded transentis in 1997 and has been managing partner ever since.
After reading mathematics and theoretical physics at Cambridge University (MA Cantab) and the University of Innsbruck he later specialised in business engineering at the University of St. Gallen (Executive MBA, Dr. oec.).
Oliver’s personal mission is to help his clients to explore, re-design and transform their enterprises using his expertise in enterprise architecture and enterprise analytics.
oliver.grasl@transentis.com
+49 173 6546727
+49 30 800937050
