Technology at Whitkirk

Weeknotes: Saturday 1 May 2021

Happy long weekend everyone!

The last week for the tech team here at Whitkirk has been all about configuring, optimising and planning. Here’s what we’ve been up to.

A bit of printer tweaking

We’ve changed how our printer is configured to connect to our network. This is a hangover from the days before we had a network spanning two buildings, and the printer was instead plugged directly into the office computer. The change makes it more reliable, more visible to our network management tools, and most crucially enables us to queue print jobs remotely using our VPN.

Optimise all the images

Following on from last week’s website tweaks, we’ve gone through our entire collection of uploaded images and run them through an optimiser, shaving an average of 60% off the size of each image we serve. These small changes add up, especially for people on lower bandwidth connections, making our site seems faster and more responsive.

Continue reading “Weeknotes: Saturday 1 May 2021”

Weeknotes: Saturday 24 April

Now we’re back in the church building we’ve had to make some tweaks to how we stream services, so here’s what happened last week!

Some new audio tweaks

We added two new bits of cabling to our audio setup – one which lets us bring audio from our new PC into the church’s sound system, and one which lets us bring a second output from our mixer into our video stream.

This second output gives us the ability to more cleanly separate the vocal microphones in the church – such as the lectern and altar microphone – from the organ microphone which picks up the music. By doing this we can reduce the amount of chatter which is picked up from the returning congregation at the end of the service, whilst still feeding our organ voluntaries out into the world.

Continue reading “Weeknotes: Saturday 24 April”

Weeknotes: Sunday 18 April

This has been a pretty uneventful week for the tech team, as we’ve focussed on building on last week’s fix for some connectivity problems.

A new router

We’ve switched our existing combination modem/router which connects the Community Centre to the internet into ‘bridge mode’, so it acts exclusively as a modem. To provide routing across our internal network, we’ve added a new, dedicated router.

This brings us a few extra goodies, such as better remote administration, better controls over how we handle data from various devices, and the ability for us to remotely connect to the Church’s network to perform maintenance and diagnose faults from anywhere.

Access point controller

To boost the reliability and flexibility of our wifi networks in the church we’ve added an access point controller, which lets us treat our entire wifi estate as one single managed entity. This helps give us a better image of how things are behaving, which is important as we encourage people to use their own devices to access the order of service.

Weeknotes: Sunday 11 April

After the busyness of Holy Week, what have the tech team been up to?

New PC for the tech desk

Since the beginning of the first lockdown, way back in March last year, every service we’ve streamed has involved some element of borrowed equipment.

This week we finally installed the last piece of the puzzle to allow services to be entirely streamed using our own gear – a small PC that lets the technician manage some of the more advanced features of our video mixer, start and stop the stream, keep an eye on our infrastructure, play audio files and more.

Because we love automation this PC has been hooked into our existing user and device management system, seamlessly setting up user accounts and configuring security policies.

Recording the services

We’ve attached a high-speed disk to the video mixer, so we can directly record services in high quality. In the past we’ve always had to include an extra step in our process to record services, so this not only simplifies our own content pathways for things we do with recordings (like pulling out service audio to deliver via phones and podcasts) and adds certainty that we get a good recording, but also gives us the possibility of offering video recordings of services such as weddings.

Improving the connectivity to the church

Following a particularly bad spell of connectivity which affected our streaming, we spent some time finding and fixing the problem.

Oops: Yet more excessive buffering during a service

Today’s service suffered from very poor connectivity throughout, leading to stuttering and buffering. This wasn’t up to the standard we aim to meet, so we spent a few hours getting to the bottom of exactly what was going on.

So, what was the problem?

After eliminating most possibilities we isolated the fault to the wireless link between the Community Centre and the church. The church building itself has no telephone line, so to provide internet we use a pair of high-power directional wifi devices. One of these is positioned on the outside of the Community Centre, but we’re not able to (and we have no real desire to) fasten one to the outside of our Grade I listed building and ruin the look of things.

Continue reading “Oops: Yet more excessive buffering during a service”

Weeknotes: Sunday 4 April

What have the technology team been up to in the last week?

Easter

Pretty much the entire week has been taken up with supporting our Easter worship, in a mix of live-streamed and pre-recorded services.

More interestingly, these services have included some new tricks:

Pre-recorded content

We’ve mixed pre-recorded content into a live stream before, but this week we saw the inclusion of pre-recorded content which was also watchable by people in the church via a temporary video screen. Although we don’t have any plans to do this kind of thing in future once we return to in-person worship, it’s been useful to prove we can use more flexible video paths.

‘Roving’ cameras

To add a bit of variation to the week, we made use of a portable camera rig to include a moving camera both inside and outside the building.

This suffered a few quality glitches we weren’t entirely happy about, but nonetheless added a new, more inclusive angle on what was happening, particularly the parts of services that would traditionally be a procession like Palm Sunday.

Fixing the phones

We were made aware of an outage to our Dial-A-Service phone system, which we investigated and fixed.

Oops: Dial-A-Service outage

Over the past few days our Dial-A-Service phone number has been unavailable. This blog post takes a quick look at what happened, why it happened, how we fixed it, and how we’re stopping it from happening again.

What happened?

We store all of our service audio, along with a list of all our services and the times they happen, in a place called Github. We then use a platform called Twilio to do the actual heavy lifting of connecting to the telephone network.

Github isn’t really designed for serving up static content, so we use an intermediary layer called a CDN(1) which helps lighten the load by temporarily storing versions of our service audio.

On the internet, most things (including our CDN) use something called a certificate to identify who they are and encrypt traffic. Unfortunately, due to a misconfiguration, the CDN which we were using was no longer presenting a valid certificate. When our code on Twilio asked for the latest list of services, and the CDN responded with an invalid certificate, everything behaved exactly as it was supposed to and refused to accept the content.

Why did this happen?

We don’t know. We’re not responsible for maintaining this CDN, but we do know from experience that this problem can be caused by any one of many things.

How did we fix it?

We swapped to an alternative CDN provider with a valid certificate. This was a very small configuration change, with no impact on how the service itself works.

How are we making sure it doesn’t happen in future?

We’re doing a couple of things to help make sure this doesn’t happen again, or at least should it happen it’s better mitigated:

Better error handling

At the moment when things go wrong our code tends to fail entirely, which causes a very generic and very American-sounding message to be played back on the phone line. We’re going to tweak this behaviour so that instead people will get a more polite message letting them know that we’re experiencing some technical difficulties.

Seeing if we can move things to our website

Our own website already contains all of the data and audio files that are needed to put together Dial-A-Service, but this is a relatively new change. We’ll think about ways we might be able to remove a whole third-party dependency, as well as simplify our administration.

References

Weeknotes: Saturday 27 March 2021

It’s been a busy week as we get ready for Holy Week. Here’s what we’ve been up to.

The Passion

The tech team helped record and edit together the annual dramatic reading of the Passion, with the talents of Whitkirk Arts Guild.

A new clock

Although the tech desk at the back of the church has a radio-controlled clock, the vestry (where people like the clergy and servers get ready for the service) at the front of the church only had one clock. Matthew has spoken about this clock in one of our Midweek Musings, and although it’s a beautiful timepiece it’s tucked in a corner, hard to read at a glance, and doesn’t keep great time.

To fix this we swapped a thermometer that was hanging above the sink for a large digital clock. It’s radio-controlled (like the one on the tech desk) so we never have to worry about it being fast or slow, or adjusting it for daylight savings. It tells us the date, which is always useful when filling out paperwork. It even tells us the temperature so we don’t need to struggle to read a mercury thermometer any more.

Continue reading “Weeknotes: Saturday 27 March 2021”