Oops: A trifecta of problems

Yesterday, we had three separate things go unexpectedly wrong during our services. Here are our explanations as to what happened, why it happened, and how we’re making sure it doesn’t happen again.


10.00 am Eucharist: No audio reinforcement in the church

What happened?

We didn’t switch on one of our equipment racks in the church before the start of the service, meaning that we had no radio microphones, in-church speakers or inductive hearing loop for the first few minutes.

Why did it happen?

We were distracted during our preflight checks by an unrelated problem with our video mixer, meaning we missed a check step for this equipment. Powering on this equipment rack is usually done by someone outside of the tech team, and we didn’t notice that it hadn’t happened.

How are we making sure it doesn’t happen again?

We’re moving the responsibility for this job to the technical team, and changing the wording in our pre-service checklist accordingly.


10.00 am Eucharist: No audio at the beginning of the video stream

What happened?

There was no audio on the live video stream at the start of the service.

Why did it happen?

We were distracted during our preflightchecks by an unrelated problem with our video mixer, meaning we didn’t properly switch on our audio mix at the start of the service.

How are we making sure it doesn’t happen again?

We’re moving towards always using a set of prerecorded, automated steps to begin a service, so we can’t forget to take this action again.


Choral Evensong: Low video quality at the beginning of the stream

What happened?

The beginning of our video stream for Choral Evensong was in lower quality than we usually stream.

Why did it happen?

As part of resolving the problem with our video mixer which led to issues in the morning service, we inadvertently reset some parts of our streaming configuration to their (low quality) default.

How are we making sure it doesn’t happen again?

We’re documenting this problem, so that anyone experiencing it in future has a single playbook they can use to fix it. This playbook includes making sure streaming settings are properly restored.


Bonus round: What went wrong with the video mixer?

We don’t know. But we did write about it publicly, and figured out how to fix it.

Post-mortem: Remembrance Sunday

We streamed Remembrance Sunday! This is a first for us, having pre-recorded the worship and act of remembrance last year, this year we wanted to make sure people could still take part even if they weren’t able or comfortable to join us in person.

We talked a bit about how in last week’s notes, but this is all about looking back at what happened; what went well, what didn’t go so well, and what we can do in the future.

Things that we liked

  • Feedback was almost immediate that it was widely appreciated, with people in the Community Centre who couldn’t make the procession to the War Memorial feeling like they were still part of the service
  • The audio system at the War Memorial – entirely battery powered – behaved exactly as we expected
  • The communication between different members of the Tech Team was useful in letting people know the current state of the broadcast

Things that we didn’t like

  • The radios we used to stay in touch caused some interference to our audio
  • One of our remote cameras failed to connect properly
    • Although this didn’t turn out to be a problem, because people then stood in front of it anyway
  • Mixing audio from remote camera sources didn’t work as smoothly as we would have liked
    • Although this didn’t turn out to be a problem, because we were down to one remote camera
  • We forgot to make sure the usual automated bells to signal the 10.00am service were switched off, so they started ringing 15 minutes into the Remembrance Sunday service.
  • Uniformed organisations weren’t in place in time, so we were left using less than ideal camera angles to cover the beginning of the service

Things we learned

  • We need to check that all our tripod mounts are compatible as part of our planning exercise
  • We need to have more planning for where people stand and when things happen, because mistakes are recorded for all time and are visible to the world
  • Where we have a roving camera who is focussing on action in front of them, they need a spotter if they’re moving around (especially if they aren’t moving forwards)
  • We need to make sure we have the necessary permissions in place for video work involving uniformed organisations further in advance

Things we’re going to do

  • Write a new checklist for hooking OBS into our usual stack ready for including external video sources
  • Write a new checklist for outside broadcast camera setup, to reduce unexpected configurations
  • Work out exactly how audio mixing should work, and make sure that’s part of the checklist
  • Start keeping common playbooks with churchwardens etc around these special services and events, so we have a shared understanding of what happens and when
  • Investigate ways to move the antenna for tech radios outside, to both improve reception and reduce interference

Weeknotes: Saturday 13 November

It’s been a busy week, with upgrades to our network and planning for our Remembrance Sunday service stream. here’s what we’ve been up to.

We upgraded our network edge equipment

Over the last couple of years our network has become slightly more important than it was before – instead of supporting an office PC, a couple of card payment devices and the occasional laptop it’s now also providing connectivity for weekly service streaming, and wireless networks in the church and Community Centre for both our own team and guests.

One of our guiding principles in the Tech Team is “do it right”, and we’ve been able to take advantage of a donation for technology in the church to replace some of the last pieces of borrowed equipment in our network. More specifically, we replaced our edge router (the bit which connects our internal network to the outside world) with a TP-Link ER605 Gateway, and we’ve replaced our temporary Omada controller on a Raspberry Pi with an OC200 Cloud Controller.

Together these pieces of equipment make our network faster and more secure, giving us improved reliability and more insight into what’s going on inside. They also give us a foundation for more improvements in the future.

We installed a network cabinet

Anyone who has ever worked with network equipment will know that the cables just love to tangle themselves at the first opportunity. No matter how neatly laid out they are, they’ll tie themselves in a knot and make future work far harder than it needs to be.

We’ve got a surprising amount of network equipment for a parish church, and with the new edge equipment above we wanted to make sure that it’s kept organised and protected.

So we cleared away our existing equipment in all its tangled glory, and in its place installed a 6U server rack. To this we added a proper surge-protected power supply, brush plate to help us keep our cables in order, and rack-mount shelving to keep our devices separated. When in future we install a new switch (as is part of our roadmap) we can pull out the shelf currently holding one, and rack the switch directly into the cabinet.

We planted some more trees

We know that technology – especially buying new technology – has a cost to the environment. To help negate this we carbon offset all our new technology purchases, as well as plant trees for the future.

To offset our new network equipment we’ve offset one tonne of CO2 emissions, and funded the planting of fifty trees in Mozambique. You can see all of St Mary’s offsetting and trees to date in our Ecologi forest.

We got ready for Remembrance Sunday

Remembrance Sunday is one of the key services we provide for the wider community in East Leeds every year. Once we have finished the eucharist service in the church, we proceed down Selby Road (ably assisted by West Yorkshire Police, who close the road for us) to the War Memorial where we lead the annual Act of Remembrance.

Last year we were unable to hold this service as we usually would, instead using a pre-recorded Act of Remembrance to give people a focus. This year, although we’re back to worshipping in person, we know that not everybody feels comfortable with crowds and might not be able to physically join us.

This left the Tech Team with a whole new challenge which we only briefly experimented with at Easter: can we get outside video reliably into our streaming services? The answer, after some planning and experimentation, is “yes”. Our aim is to move seamlessly from our usual eucharist service (starting earlier than usual at 9.30 am) into the procession, follow the procession down Selby Road to the memorial, and then pick up the Act of Remembrance.

To do this we’re using a combination of equipment, software and tools. The service is being mixed and streamed as usual via our ATEM mixer in the church, but we’ve also added a new laptop to the mix running OBS Studio. We’re taking the video mix from OBS and running it into the mixer in place of our font camera (which is very rarely used), so that we can pull in additional video from this external source.

The next piece of the equation is getting video into OBS from outside the church, and to do that we’re using a web-based tool called VDO.Ninja. By running this on a mobile phone hooked up to a regular mobile network, we can stream live video from the phone to our laptop running OBS over a technology called WebRTC.

We’re planning to use two phones for this – one will be mounted to a tripod at the War Memorial, and plugged into our sound reinforcement system so that we can get nice clear audio when people are speaking and when we play the Last Post. The second one will be strapped into a handheld mount along with an external microphone (with windshield), and will follow the procession down Selby Road. We’re using the mount so that our camera operator can use two hands to support the phone, which makes holding it for a period of time a lot easier and reduces shake. Once arriving at the Memorial, we’ll attach this phone to a second tripod to give us two camera angles, and be able to use the microphone from it for providing background noise.

We can use then OBS Studio to swap between these two external cameras and mix their audio outputs, and then feed this mix into ATEM where it can be encoded and sent on to the rest of the world.

Fingers crossed!

Weeknotes: Saturday 23 October

It’s been another week of technology here at Whitkirk. Here’s what we’ve been up to

Better order of service redirections

One thing we’ve been stuck doing manually each week is updating our “redirect to latest order of service” URL so that it actually went to the right place. We use this link for the QR Code in the church which points people to the order of service, so it’s important that it’s right.

We’ve now automated this process, so it’s one fewer thing to have to remember.

Buzz off

Our improved set of connections for in-house sound system.

Over the past few weeks an annoying buzzing sound has crept into the speakers in the Community Centre. We spent a bit of time fixing this (and tidying up some cables), but also took the opportunity to improve power management and connectivity.

To help save energy, our sound system is now on a timed power supply. A push-button switches the whole thing on for a few hours – plenty of time for bookings to make use of the speakers and hearing loop – and then it’ll turn itself off to help conserve energy.

As part of this we also added a new phono input and an additional 3.5mm jack input, giving more flexibility for our users.

Prepare for ludicrous speed!

via GIPHY

The lovely people at Openreach have finally switched on the ability for us to get fibre-optic cables run right into our Community Centre. This isn’t just about improving our internet speed (although we’ll certainly enjoy a speed boost), but also about making sure that we’re on more resilient digital infrastructure in the future.

We’re confident that we’ll be moving to this fibre-optic connection some time in the near future, but we’re still not sure exactly which provider and package we’ll be using. This last week we spent some time looking at options and putting together recommendations.

Weeknotes: Saturday 9 October

It’s been quiet on the tech front, but things are going to start getting busy again as we approach Advent and have to start thinking about things like carol concerts!

Fun on (and off) the stage

We do a lot of stuff in the church building, but we also take care of technology over in our Community Centre. Over the past week we’ve been thinking about work we need to do on stage to tidy things up and make it more usable for more people.

This is a big job which we’ll be breaking down and tackling in multiple stages, but we’re starting by rearranging some wiring to tidy things up. This is particularly a problem in the stage right corner where a tangle of cables is not only making it look a mess but is also making it unintuitive to use, hard to troubleshoot and is causing interference.

At the moment we’re still narrowing down exactly what we want to do on paper, thinking about our various Community Centre users to make sure we’re covering as many use cases as possible, and thinking about the future to make sure we don’t need to re-do this kind of work for a good few years.

Weeknotes: Saturday 18 September

More of a fortnightnotes than a weeknotes, but we’re here again!

The Temple Newsam Service

Tomorrow’s service at Temple Newsam House is an annual event in the calendar. Last year we stitched together a pre-recorded service to mark the occasion, but this year we’re venturing into uncharted territory with our first live outside broadcast.

All things going to plan, we’ll be leaving the relative safety of our tech desk and taking a pile of equipment with us; our trusty video mixer, a monitor, two cameras, a laptop and all of the associated cables. We’ve already done a couple of test runs, so we’re reasonably confident of things going to plan. In true Whitkirk Tech fashion, though, we’ve got a couple of contingency plans up our sleeves.

Weeknotes: Saturday 4 September

It’s been a few weeks since we last had an update from the tech team… because we’ve had a quiet few weeks! The period of Ordinary Time after Trinity has given us an opportunity to take a bit of a breather after the chaos of the last 18 months, but we’re starting to get back into things now.

The Heritage Open Days talk

As part of Heritage Open Days this year, we’ve got a talk! Although we’ve had plenty of talks given between the church and Community Centre over the years, this has been the first one since we’ve had our streaming equipment installed, so we’re sharing it with the world.

Since we’re not fans of doing things by halves, and since we’re always looking for opportunities to step up our game, we’ll be providing slides via projector and relay screen (to address some poor sightlines in the building), as well as interleaving them into the video stream using picture-in-picture features of our video mixer. We’ll also be providing a presenter display and clicker, for a little extra bit of quality of life.

The Temple Newsam Eucharist

Each year we hold a service at Temple Newsam House to celebrate the estate being acquired by the City of Leeds. This year, we’ve been asked to stream it live so that those who can’t attend in person can still join in.

Although we’re pretty confident that we can stream from outside the church building, we’ve been doing some planning and testing to make sure everything goes as planned.

More trees, less carbon

Both of the special events above have meant we’ve needed to get hold of some more equipment to help move video around, and as part of our ongoing commitment to reduce the harm we’re causing to the planet we’ve planted more trees and offset more carbon. Including our last two offsets, that’s a hundred trees and two full tonnes of carbon offset by the tech team.

Weeknotes: Saturday 14 August

Guest wifi in the Community Centre

We’ve finished up our resiliency and soft-launch testing, and we’re finally happy to roll out guest wifi in the Community Centre! If you’re visiting us, point your phone’s camera at one of our wifi QR Code posters to get connected, or manually enter the network details.

Getting technical with Raspberry Pi, Omada, SMTP and HTTPS

We did a load of stuff with our network controller, and we wrote a whole blog post about it.

Continue reading “Weeknotes: Saturday 14 August”