Those of you joining the Watch on Maundy Thursday may have noticed our stream abruptly cut out, and those of you hoping to watch the Liturgy of Good Friday will have noticed we didn’t stream it at all.
Read more: Oops: Streaming interruptions on Maundy Thursday and Good FridayWhat happened?
Partway through streaming the Watch on Maundy Thursday, we lost all internet connectivity to the church building, which meant we were unable to stream the final half of the Watch. This disruption continued through most of Good Friday, meaning we weren’t able to stream the Liturgy of Good Friday as planned.
Why did this happen?
Our internet connection in the church building is provided by a radio link to the neighbouring Community Centre. For some unknown reason, the side of the link on the Community Centre stopped responding to both its counterpart on the church, and all control signals. Recovering this required a complete restart of the link.
How are we making sure it doesn’t happen again?
The radio link is one of our oldest parts of network equipment, and is due for replacement soon. For that reason we’re not rushing to replace it out-of-schedule unless we start experiencing these failures more frequently. However, we’ve taken this opportunity to make sure all our firmware is up to date, to make sure our configuration is optimised, and to remove some barriers which made it harder to troubleshoot.
To help make this problem easier to solve in future if it reoccurs we’ve also updated our Handbook to include some error recovery steps in our streaming playbook, and we’ve added a new troubleshooting section to the physical operations manual which is kept on our tech desk.