Automated Simulation Output Analysis
Professor Stewart Robinson, Dr Katy Hoad, Professor Ruth Davies
Figure 1: Overview of the Automated Analysis Procedure
Appropriate use of a simulation model requires accurate measures of model performance. This in turn, requires decisions concerning three key areas: warm-up, run-length and number of replications. Simulation software, however, gives little or no guidance in making these decisions. A three year project, "Automatic Simulation Output Analysis", sponsored by the EPSRC (Grant ref: EP/D033640/1) and SIMUL8 and undertaken by Warwick Business School, is investigating the development of a methodology for automatically advising a simulation user on these three decisions. This web site outlines the approach being taken, discusses progress to-date and provides links for research outputs.
Obtaining Accurate Simulation Results
Even with the best constructed simulation model if it is not used properly it is likely that the results will be inaccurate. If the results from a simulation of a system are not accurate they will not provide a good prediction of what would happen in the real life system even if the model itself is valid. The results could be biased and misleading, with the consequence that incorrect conclusions could be drawn.
There are many decisions that need to be made about how a simulation model is to be run. These include:
- Do I need to use a warm-up period; set initial conditions; or perhaps use a combination of both; or do I in fact want my simulation to start from empty ?
- If I do use a warm-up period how long should this period be ?
- Should I perform one long simulation run, or a number of independent runs (i.e. runs with different random streams, also called replications) ?
- How long should this one simulation run be ?
- How many replications is enough ?
In particular, the decisions on run-length, number of replications to perform in a trial and warm-up period require specific skills in statistics. Because of the variety and complexity of statistical methods used to make informed decisions on these matters, at the moment, most simulation software provide little or no guidance to users on making these important decisions. Because the setting of inappropriate run-lengths, run (replication) numbers and warm-up periods can lead to inaccurate and misleading results, it is considered important to deliver tools to simulation users that allow them to make informed decisions regarding these matters, whatever their statistical background.
It is with this in mind that Warwick Business School and SIMUL8 have collaborated on a project to create an automated tool that can analyse preliminary simulation output and hence provide recommendations on warm-up, run-length and number of replications.
Refs:
- “Automated Analysis Of Simulation Output Data”, Stewart Robinson (Warwick Business School, University of Warwick, Coventry, CV4 7AL, UK), Proceedings of the 2005 Winter Simulation Conference.
- “A Statistical Process Control Approach for Estimating the Warm-up Period”, Stewart Robinson (Warwick Business School, University of Warwick, Coventry, CV4 7AL, UK), Proceedings of the 2002 Winter Simulation Conference.
- “Simulation: The Practice of Model Development and Use”, Stewart Robinson (Warwick Business School, University of Warwick, Coventry, CV4 7AL, UK), 2004, Wiley, Chichester, UK.
- “The Problem of the Initial Transient: Techniques for Estimating the Warm-Up Period for Discrete-Event Simulation Models”, Stewart Robinson (Warwick Business School, University of Warwick, Coventry, CV4 7AL, UK) and Agis Ioannou (Process Solutions Centre, BT Exact, Orion Building, Adastral Park, Martlesham Heath, Ipswich, IP5 3RE, UK). (Not yet published)
- Full Project proposal
Project Personnel
Principal Web Site editor & Research Fellow: Dr Katy Hoad
Operational Research and Management Sciences Group
Warwick Business School
University of Warwick
Coventry
CV4 7AL
UK
Contact e-mail: kathryn dot hoad at wbs dot ac dot uk
Principal Investigator: Professor Stewart Robinson
Professor of Operational Research Associate Dean for Specialist Masters
Operational Research and Management Sciences Group
Warwick Business School
University of Warwick
Coventry
CV4 7AL
UK
e-mail: stewartrobinson at warwick dot ac dot uk
Co-Investigator: Professor Ruth Davies
Professor of Operational Research and Systems
Associate Dean for Academic Policy
Operational Research and Management Sciences Group
Warwick Business School
University of Warwick
Coventry
CV4 7AL
UK
Project Partner: SIMUL8 Corporation
Contact: Dr Mark Elder
141 St. James' Road
Glasgow
G4 0LT
Scotland