Skip to main content Skip to navigation

Online payments system - emails sent in error

18:00 GMT, Thursday 21st February 2019

An update to the Online Payments system on Thursday 21st February caused a large quantity of emails to be sent in error. These emails pertained to payments that were started, but not completed, but the system incorrectly sent emails saying that payment has been completed.

We would like to sincerely apologise, and reassure users that no money has been taken from their account. We have contacted everyone who received an email to let them know that this was a mistake on our part.

If you are concerned that money has been taken in error, please contact onlinepayment at warwick dot ac dot uk

Any further updates will appear here.

09:00 GMT, Friday 22nd February 2019

Our engineers are continuing to investigate the cause of the problems, which we believe to be an issue with our payment provider's systems and how we are interpreting data. We are working to check all of the data in our systems to ensure that everything is handled correctly.

12:00 GMT, Friday 22nd February 2019

We believe we have identified the fault that caused the problem, and are preparing to release a fix. This will correct any instances of data in the University's transaction tracking system (TTS), also known as ePayments, where the transaction history is now incorrectly showing a successful transaction.

We'd like to continue to reassure customers that no money was taken and no automated systems were updated with successful payments when they should not have been.

We will be in touch again with affected trading account owners this afternoon, once the data in the TTS has been updated, to give specific transaction IDs. We apologise again for the inconvenience caused.

16:00 GMT, Friday 22nd February 2019

We have now corrected and verified all of the data in the University's transaction tracking system and notified payment managers across the University about transactions that appeared to be successful for around a 24-hour period.

Timeline

14:20, 21/02: An update to the TTS is released which fixes a problem where transactions stay in the "Incomplete" state forever

15:00, 21/02: The TTS starts to process incomplete transactions from the previous 90 days. Our payment provider unfortunately returns "Successful" transaction responses for incomplete transactions that were started in a foreign currency. Nearly 9,000 incomplete transactions are incorrectly marked as successful, and emails are incorrectly sent to customers to inform them that their payment has gone through.

17:00, 21/02: After receiving reports from customers receiving incorrect emails, the change to the TTS is backed out and investigations by engineers begin.

17:30, 21/02: The IT Services Major Incident process is invoked to coordinate resources

18:45, 21/02: After identifying affected customers who received emails in error between 15:00 and 16:00, a further email is sent explaining the problem. An email is also sent to staff at the University who manage payments whose customers have been affected.

14:00, 22/02: A code change is released to the TTS that specifically targets the specific error case coming from our external payments partner and correctly treats those transactions as having "Timed out". The 8804 affected transactions are set back to "Incomplete" and the TTS processes them as normal, setting them all to Timed out.

15:00, 22/02: Trading account owners are notified of the list of transactions affected by the fault. The IT Services Major Incident process is stood down.