Oops: Dial-A-Service outage

Over the past few days our Dial-A-Service phone number has been unavailable. This blog post takes a quick look at what happened, why it happened, how we fixed it, and how we’re stopping it from happening again.

What happened?

We store all of our service audio, along with a list of all our services and the times they happen, in a place called Github. We then use a platform called Twilio to do the actual heavy lifting of connecting to the telephone network.

Github isn’t really designed for serving up static content, so we use an intermediary layer called a CDN(1) which helps lighten the load by temporarily storing versions of our service audio.

On the internet, most things (including our CDN) use something called a certificate to identify who they are and encrypt traffic. Unfortunately, due to a misconfiguration, the CDN which we were using was no longer presenting a valid certificate. When our code on Twilio asked for the latest list of services, and the CDN responded with an invalid certificate, everything behaved exactly as it was supposed to and refused to accept the content.

Why did this happen?

We don’t know. We’re not responsible for maintaining this CDN, but we do know from experience that this problem can be caused by any one of many things.

How did we fix it?

We swapped to an alternative CDN provider with a valid certificate. This was a very small configuration change, with no impact on how the service itself works.

How are we making sure it doesn’t happen in future?

We’re doing a couple of things to help make sure this doesn’t happen again, or at least should it happen it’s better mitigated:

Better error handling

At the moment when things go wrong our code tends to fail entirely, which causes a very generic and very American-sounding message to be played back on the phone line. We’re going to tweak this behaviour so that instead people will get a more polite message letting them know that we’re experiencing some technical difficulties.

Seeing if we can move things to our website

Our own website already contains all of the data and audio files that are needed to put together Dial-A-Service, but this is a relatively new change. We’ll think about ways we might be able to remove a whole third-party dependency, as well as simplify our administration.

References

Weeknotes: Saturday 27 March 2021

It’s been a busy week as we get ready for Holy Week. Here’s what we’ve been up to.

The Passion

The tech team helped record and edit together the annual dramatic reading of the Passion, with the talents of Whitkirk Arts Guild.

A new clock

Although the tech desk at the back of the church has a radio-controlled clock, the vestry (where people like the clergy and servers get ready for the service) at the front of the church only had one clock. Matthew has spoken about this clock in one of our Midweek Musings, and although it’s a beautiful timepiece it’s tucked in a corner, hard to read at a glance, and doesn’t keep great time.

To fix this we swapped a thermometer that was hanging above the sink for a large digital clock. It’s radio-controlled (like the one on the tech desk) so we never have to worry about it being fast or slow, or adjusting it for daylight savings. It tells us the date, which is always useful when filling out paperwork. It even tells us the temperature so we don’t need to struggle to read a mercury thermometer any more.

Continue reading “Weeknotes: Saturday 27 March 2021”

Oops: Excessive buffering during the service

In this morning’s service a few people noticed that their connection was having to buffer more than usual, and that the quality of the video and audio kept fluctuating. This blog post takes a quick look at what happened, why it happened, and what we’re doing to fix it.

What happened?

The internet connection to the church wasn’t capable of sustaining the speeds needed for smooth streaming of video. This meant that people watching would see the video stutter or sometimes pause entirely.

Why did this happen?

We’re not sure. The path to get an internet connection into the church is a relatively complex one compared to what you might have at home, involving several points where things might slow down. We weren’t able to quickly identify what was wrong during the service.

What are we doing to fix it?

There are two problems we’re fixing:

Making sure we can stream services

We’re reducing the threshold at which we decide to use a backup mobile connection to stream services. It’s difficult to change our connection mid-service (although we can in an emergency), so for the time being we’ll be using our backup connection unless we’re absolutely certain our main one is behaving as expected.

Making sure the internet connection to the church is stable

We’ve put some extra monitoring in place to see if we can narrow down which bit of the chain is at fault and then investigate further, but since the problem doesn’t appear all the time it might take us a few weeks before we’re able to properly identify it.

Depending on where the problem lies the solution could be as simple as a quick configuration change, might need replacement hardware for our network, or might need us to involve our connection provider.

Weeknotes: Saturday 20 March 2021

Bits and pieces the technical team here at Whitkirk have been up to in the past week.

Cameras

The last bits of our cameras project were installed on Tuesday this week. Read all about it in our blog post.

A bit of title card fun

Closely related to the cameras project, we needed to create some title cards to display before and after a service. We also threw together a test card, inspired by the famous Test Card F and made possible by this little project.

An image of a television test card with a line drawing of St Mary's Church in the middle. The text reads "please stand by".
Continue reading “Weeknotes: Saturday 20 March 2021”

What the Cameras Project has brought us

Our cameras project has been the largest single investment in technology at St Mary’s for decades. Here’s a quick summary of what’s been added to the building – with pictures!

The cameras

We’ve fitted three permanent cameras to the church, which give us an enormous range of options when it comes to recording and streaming services.

The PTZ camera

The PTZ camera is mounted to the side of a pillar above the pews.

The PTZ camera – short for “Pan, Tilt, Zoom” – is our principal workhorse which will cover most of the time during a service. That’s because, as the name suggests, this camera can pan, tilt and zoom freely to capture different angles within the building. More importantly, we can remotely control this movement during services.

Continue reading “What the Cameras Project has brought us”

Weeknotes: Saturday 13 March 2021

What have the tech team been doing this week?

Fitting some monitors

We hooked up two new screens to help us see what we’re doing whilst streaming services and managing the technology within the church building.

Increasing the bus factor

Much of the technology in the church at the moment has been thrown together during lockdown by one person, which has left us with a very low bus factor (the number of people who need to suddenly become unavailable before a project became impossible to continue).

This isn’t a great position to be in, so we spent some time this week filling more people in on the intricacies of systems, and improving some of our documentation.

Continue reading “Weeknotes: Saturday 13 March 2021”

Now (nearly) showing on a screen near you.

The process of moving the technology in the church to one single place has been steady. It’s taken time to come up with a plan, build a desk, move the audio equipment (and its cables), move our video equipment (and its cables), install some new video cables and build a power distribution board.

Today we took the next step in the journey to completion – the addition of two new monitors. It might seem a bit over the top, but this is actually the bare minimum we need to have available for streaming our services once we finish the camera installation. One screen will show us the state of our various different cameras and our stream, and the other will be used for a supplementary PC responsible for mixing in graphics (like our title slides) and controlling more complex bits of the process.

A photograph of the technology desk at the back of the church building. It shows two monitors mounted on adjustable arms, a CD player and a sound mixing desk with a pair of headphones.
The technology desk, now with both of its monitors.
Continue reading “Now (nearly) showing on a screen near you.”

Weeknotes: Saturday 6 March 2021

Now that most of the physical work has been done for the cameras project and new desk, what have the tech team at Whitkirk been up to in the last week?

Less power!

We did some more maths on our tech desk’s total estimated power load and refined our numbers downwards. This meant we could swap some of the fuses for something even more conservative, bringing the already small risk of someone overloading the circuit in a dangerous way even lower.

“Il meglio è nemico del bene”

As Voltaire said(1), “perfect is the enemy of good”. The tech team always aim towards perfection, but we take a pragmatic view that most of the time “good enough” really is, and we can make improvements later.

Several years ago, when the Church website was redesigned into its current form, we made one of these “good enough” decisions about the look of our website header images. In doing so, we inadvertently left a bug lying around which would only manifest itself in a specific set of circumstances. Not a totally-breaking-things bug, but one which meant the website didn’t always look quite right. More specifically, if you uploaded an image that was narrower than a 9:2 ratio for the page header then it would be tiled incorrectly and look rubbish.

Continue reading “Weeknotes: Saturday 6 March 2021”

References

References
1 sort of, there was a bit of a translation error