Weeknotes: Sunday 11 April

After the busyness of Holy Week, what have the tech team been up to?

New PC for the tech desk

Since the beginning of the first lockdown, way back in March last year, every service we’ve streamed has involved some element of borrowed equipment.

This week we finally installed the last piece of the puzzle to allow services to be entirely streamed using our own gear – a small PC that lets the technician manage some of the more advanced features of our video mixer, start and stop the stream, keep an eye on our infrastructure, play audio files and more.

Because we love automation this PC has been hooked into our existing user and device management system, seamlessly setting up user accounts and configuring security policies.

Recording the services

We’ve attached a high-speed disk to the video mixer, so we can directly record services in high quality. In the past we’ve always had to include an extra step in our process to record services, so this not only simplifies our own content pathways for things we do with recordings (like pulling out service audio to deliver via phones and podcasts) and adds certainty that we get a good recording, but also gives us the possibility of offering video recordings of services such as weddings.

Improving the connectivity to the church

Following a particularly bad spell of connectivity which affected our streaming, we spent some time finding and fixing the problem.

Oops: Yet more excessive buffering during a service

Today’s service suffered from very poor connectivity throughout, leading to stuttering and buffering. This wasn’t up to the standard we aim to meet, so we spent a few hours getting to the bottom of exactly what was going on.

So, what was the problem?

After eliminating most possibilities we isolated the fault to the wireless link between the Community Centre and the church. The church building itself has no telephone line, so to provide internet we use a pair of high-power directional wifi devices. One of these is positioned on the outside of the Community Centre, but we’re not able to (and we have no real desire to) fasten one to the outside of our Grade I listed building and ruin the look of things.

Continue reading “Oops: Yet more excessive buffering during a service”

Weeknotes: Sunday 4 April

What have the technology team been up to in the last week?

Easter

Pretty much the entire week has been taken up with supporting our Easter worship, in a mix of live-streamed and pre-recorded services.

More interestingly, these services have included some new tricks:

Pre-recorded content

We’ve mixed pre-recorded content into a live stream before, but this week we saw the inclusion of pre-recorded content which was also watchable by people in the church via a temporary video screen. Although we don’t have any plans to do this kind of thing in future once we return to in-person worship, it’s been useful to prove we can use more flexible video paths.

‘Roving’ cameras

To add a bit of variation to the week, we made use of a portable camera rig to include a moving camera both inside and outside the building.

This suffered a few quality glitches we weren’t entirely happy about, but nonetheless added a new, more inclusive angle on what was happening, particularly the parts of services that would traditionally be a procession like Palm Sunday.

Fixing the phones

We were made aware of an outage to our Dial-A-Service phone system, which we investigated and fixed.

Oops: Dial-A-Service outage

Over the past few days our Dial-A-Service phone number has been unavailable. This blog post takes a quick look at what happened, why it happened, how we fixed it, and how we’re stopping it from happening again.

What happened?

We store all of our service audio, along with a list of all our services and the times they happen, in a place called Github. We then use a platform called Twilio to do the actual heavy lifting of connecting to the telephone network.

Github isn’t really designed for serving up static content, so we use an intermediary layer called a CDN(1) which helps lighten the load by temporarily storing versions of our service audio.

On the internet, most things (including our CDN) use something called a certificate to identify who they are and encrypt traffic. Unfortunately, due to a misconfiguration, the CDN which we were using was no longer presenting a valid certificate. When our code on Twilio asked for the latest list of services, and the CDN responded with an invalid certificate, everything behaved exactly as it was supposed to and refused to accept the content.

Why did this happen?

We don’t know. We’re not responsible for maintaining this CDN, but we do know from experience that this problem can be caused by any one of many things.

How did we fix it?

We swapped to an alternative CDN provider with a valid certificate. This was a very small configuration change, with no impact on how the service itself works.

How are we making sure it doesn’t happen in future?

We’re doing a couple of things to help make sure this doesn’t happen again, or at least should it happen it’s better mitigated:

Better error handling

At the moment when things go wrong our code tends to fail entirely, which causes a very generic and very American-sounding message to be played back on the phone line. We’re going to tweak this behaviour so that instead people will get a more polite message letting them know that we’re experiencing some technical difficulties.

Seeing if we can move things to our website

Our own website already contains all of the data and audio files that are needed to put together Dial-A-Service, but this is a relatively new change. We’ll think about ways we might be able to remove a whole third-party dependency, as well as simplify our administration.

References

Weeknotes: Saturday 27 March 2021

It’s been a busy week as we get ready for Holy Week. Here’s what we’ve been up to.

The Passion

The tech team helped record and edit together the annual dramatic reading of the Passion, with the talents of Whitkirk Arts Guild.

A new clock

Although the tech desk at the back of the church has a radio-controlled clock, the vestry (where people like the clergy and servers get ready for the service) at the front of the church only had one clock. Matthew has spoken about this clock in one of our Midweek Musings, and although it’s a beautiful timepiece it’s tucked in a corner, hard to read at a glance, and doesn’t keep great time.

To fix this we swapped a thermometer that was hanging above the sink for a large digital clock. It’s radio-controlled (like the one on the tech desk) so we never have to worry about it being fast or slow, or adjusting it for daylight savings. It tells us the date, which is always useful when filling out paperwork. It even tells us the temperature so we don’t need to struggle to read a mercury thermometer any more.

Continue reading “Weeknotes: Saturday 27 March 2021”

Oops: Excessive buffering during the service

In this morning’s service a few people noticed that their connection was having to buffer more than usual, and that the quality of the video and audio kept fluctuating. This blog post takes a quick look at what happened, why it happened, and what we’re doing to fix it.

What happened?

The internet connection to the church wasn’t capable of sustaining the speeds needed for smooth streaming of video. This meant that people watching would see the video stutter or sometimes pause entirely.

Why did this happen?

We’re not sure. The path to get an internet connection into the church is a relatively complex one compared to what you might have at home, involving several points where things might slow down. We weren’t able to quickly identify what was wrong during the service.

What are we doing to fix it?

There are two problems we’re fixing:

Making sure we can stream services

We’re reducing the threshold at which we decide to use a backup mobile connection to stream services. It’s difficult to change our connection mid-service (although we can in an emergency), so for the time being we’ll be using our backup connection unless we’re absolutely certain our main one is behaving as expected.

Making sure the internet connection to the church is stable

We’ve put some extra monitoring in place to see if we can narrow down which bit of the chain is at fault and then investigate further, but since the problem doesn’t appear all the time it might take us a few weeks before we’re able to properly identify it.

Depending on where the problem lies the solution could be as simple as a quick configuration change, might need replacement hardware for our network, or might need us to involve our connection provider.

Weeknotes: Saturday 20 March 2021

Bits and pieces the technical team here at Whitkirk have been up to in the past week.

Cameras

The last bits of our cameras project were installed on Tuesday this week. Read all about it in our blog post.

A bit of title card fun

Closely related to the cameras project, we needed to create some title cards to display before and after a service. We also threw together a test card, inspired by the famous Test Card F and made possible by this little project.

An image of a television test card with a line drawing of St Mary's Church in the middle. The text reads "please stand by".
Continue reading “Weeknotes: Saturday 20 March 2021”

What the Cameras Project has brought us

Our cameras project has been the largest single investment in technology at St Mary’s for decades. Here’s a quick summary of what’s been added to the building – with pictures!

The cameras

We’ve fitted three permanent cameras to the church, which give us an enormous range of options when it comes to recording and streaming services.

The PTZ camera

The PTZ camera is mounted to the side of a pillar above the pews.

The PTZ camera – short for “Pan, Tilt, Zoom” – is our principal workhorse which will cover most of the time during a service. That’s because, as the name suggests, this camera can pan, tilt and zoom freely to capture different angles within the building. More importantly, we can remotely control this movement during services.

Continue reading “What the Cameras Project has brought us”