Technology at Whitkirk

Weeknotes: Saturday 12 February

It’s been a while since we had a weeknotes, hasn’t it? To cut a long story short, it’s been a busy winter for most of the tech team and we haven’t had as much time to do interesting stuff, let alone write about it. But we’re here now, and here’s a quick run-down of the past couple of months:

We did Christmas

There are three times in the year when we’re put under pressure to make sure things go seamlessly. Easter, Remembrance Sunday, and Christmas. This year for Christmas we went all-out and streamed every Christmas service, even streaming our crib service internally so we could use our Community Centre for socially distanced overflow if needed.

Most of these were streamed using our usual complement of equipment, but for the Festival of Lessons and Carols we built on things we had learned at Remembrance to add two whole new temporary camera angles. One of these gave us a closer view on the choir to help our viewers feel more connected to the music, and the other was an alternative angle on the lectern to give extra variety during readings. This went well for a couple of reasons; the extra angles make the process of streaming feel more creative and engaging both for the video operator and viewers, and it gives us a way to test out new angles when we’re thinking about the expansion of our permanent system before we commit to anything.

Our Omada write-up is being used

Way back last August we wrote a more in-depth dive into how we set up our Omada controller. Since then we’ve heard from a couple of people who followed the guide to get their own installation up and running. It’s always nice to know that we’re not just talking to the void!

We fixed some email configuration

As a sort of follow-up to the above Omada configuration, we identified a problem with our email settings where it wasn’t playing nice with Google’s two-factor authentication requirements. To resolve this we swapped our controller’s email configuration to use our Mailgun account instead.

We removed a load of equipment from the stage. Then we put it back.

Since our Community Centre was refurbished we’ve had a somewhat temporary kludge of stuff running the sound and stage lighting. During the pandemic there wasn’t much of drive to do anything about this, but now that we’re back up and running it was getting annoying.

To fix this we removed everything (including all our power and signal cables), came up with a tidier plan, and put it back again. We’ve got more work on this planned for the next couple of months to get things looking even smarter, keep them safer, and make them easier to maintain.

We started looking at better heating controls

Energy costs are going up, and our buildings are awkward and expensive places to heat. At the moment all our heating is controlled by fairly dumb thermostats with schedulers, and these need people to be physically stood next to them to update the schedule. We’re starting to look at connected solutions which not only let us update the schedule remotely, but which also support things like weather-aware pre-heating, and better zoning of our heat to get some potentially significant efficiency gains.

We built some tools to automate some workflows

One thing we dislike is having to repeat ourselves, which is why a lot of what we do relies on only having one ‘true’ source of information for each set of facts within the Church. For a lot of these things the source of truth is a thing called ChurchSuite – everything in our calendar comes straight from there, as does our list of what’s on (and we share the code we use for this). It knows about – at least in theory – all our upcoming services and events.

But quite a lot of our streaming services don’t yet rely on this single source of truth, instead relying on a series of checklists and our technical team manually stitching things together to make sure our orders of service and YouTube channel are in sync. This unfortunately involves quite a lot of repetition – we set things like dates and times in at least half a dozen places for every service, and write things like video titles in a very formulaic way.

Fortunately, computers are really good at copying things into lots of places in a formulaic way. So we built some bits of code which automate chunks of workflow for our Communications Team and Streaming Team. These pull events from ChurchSuite into another tool called Airtable, where these teams can then layer on top information specific to what they’re doing without needing to duplicate all the basics. And when something is updated in ChurchSuite, those replicated pieces of information are updated as well. Each team then uses Airtable to generate reports needed to do their job.

…and then we automated telling people the result

Once a week, another bit of tooling goes and grabs the information from Airtable, does some work to group it and highlight things of interest, and then emails the teams a summary. This improved visibility makes sure that our teams can more easily keep on top of what’s going on and what they need to do, stopping things from falling down the cracks and getting problems solved earlier.

…and then we automated some downstream processes

For the Streaming Team, though, we went a step further by automating to process of adding streaming services to YouTube. As soon as a service is marked as being streamed we now automatically create a new stream with the relevant details, ready to go.

Our plan for the next step once we’re confident in what we’ve built so far is to include the creation and interlinking of order of service entries as well, at which point we will have automated away the vast majority of weekly drudge work. This will free our time to worry about things that actually need a human’s attention, instead of copying and pasting.

We started working on an improved switcher for our stage equipment

As part of improving the house rig in the Community Centre, we’re building an improved mains switcher and interface so that Centre users can more intuitively manage the sound system and stage lighting. Our current plan is a pair of Raspberry Pis, one with a touchscreen and one hooked up to a relay board, with a server running on the relay unit communicating to a JavaScript frontend.

This has a few advantages over our current way of doing things:

  • It’s more unified (so no more hunting for the right switch)
  • It’s more intuitive (we can put instructions for things right there on the screen)
  • It’s more extensible (without needing to add yet more boxes to the front of the stage)
  • Because it’s software-defined we can update it easily (without having to go under the stage to visit the equipment rack)
  • Since it’s connected to our network we can control things remotely (for example by remotely assisting a Centre user if they’re having trouble)

When we’ve finished our proof-of-concept we’ll open-source the whole thing, as well as do a more complete write-up.

Making our orders of service more accessible

Our tagline begins with the words “where all find a welcome”, and we believe that this doesn’t just apply to the people greeting you at the door or to the people leading the service, but to the technology which supports us as well. That’s why when we heard from a member of our congregation with poor eyesight that our digital orders of service had given them a whole new way of experiencing services – by being able to scale up the text on their phone to a comfortable reading size – we decided to explore other ways we can make our orders of service more accessible.

We already try to make our website friendly for assistive technologies like screen readers (although we know we can do better, so we’re always making improvements), but we realised we could use the power of technology to put more accessible orders of service front and centre for a commonly sidelined disability: dyslexia.

Thanks to a typeface called OpenDyslexic which is designed specifically to combat some of the more common symptoms of dyslexia. Amongst other things, the letter shapes are all unique and have distinctive ‘heavy bottoms’ which help combat rotation and transposition.

Continue reading “Making our orders of service more accessible”

The impact of streaming

Advent is a time for looking forward, but it’s also a time for taking stock and reflecting. Given it’s the end of the year, it’s also a great time for us to do some maths and figure out the environmental impact of our service streaming.

In the last year, people watched around 3,200 hours of video on our YouTube channel. Streaming an hour of video online causes around 36g of CO2 emissions. This means our streaming was responsible for around 0.12 tonnes of carbon emissions.

We’re committed to reducing our impact (in fact, the PCC recently approved a whole new policy and strategy on it), so we decided to offset these emissions. Unfortunately, the company we use for offsetting can only offset in minimums of half a tonne at a time. So we decided to round up, and offset a full tonne to make sure we were also mopping up streaming from 2020 which wasn’t included in the total, as well as cover any unaccounted-for costs in equipment purchases (where we’ve previously offset larger items, but not some consumables).

Next, we counted up the number of likes which people had given our content in the last year – 153 overall – and we’ve funded the planting of a tree for each one, which you can see in our forest. These trees will continue to absorb carbon into the future, as well as boost the local ecosystem.

We hope to repeat this exercise every year, not only making streaming our services carbon negative, but helping to build greener future for everyone.

You can read more about our commitment to the environment, and discover what else we’re doing across the Church.

Oops: A trifecta of problems

Yesterday, we had three separate things go unexpectedly wrong during our services. Here are our explanations as to what happened, why it happened, and how we’re making sure it doesn’t happen again.

10.00 am Eucharist: No audio reinforcement in the church

What happened?

We didn’t switch on one of our equipment racks in the church before the start of the service, meaning that we had no radio microphones, in-church speakers or inductive hearing loop for the first few minutes.

Why did it happen?

We were distracted during our preflight checks by an unrelated problem with our video mixer, meaning we missed a check step for this equipment. Powering on this equipment rack is usually done by someone outside of the tech team, and we didn’t notice that it hadn’t happened.

How are we making sure it doesn’t happen again?

We’re moving the responsibility for this job to the technical team, and changing the wording in our pre-service checklist accordingly.

10.00 am Eucharist: No audio at the beginning of the video stream

What happened?

There was no audio on the live video stream at the start of the service.

Why did it happen?

We were distracted during our preflightchecks by an unrelated problem with our video mixer, meaning we didn’t properly switch on our audio mix at the start of the service.

How are we making sure it doesn’t happen again?

We’re moving towards always using a set of prerecorded, automated steps to begin a service, so we can’t forget to take this action again.

Choral Evensong: Low video quality at the beginning of the stream

What happened?

The beginning of our video stream for Choral Evensong was in lower quality than we usually stream.

Why did it happen?

As part of resolving the problem with our video mixer which led to issues in the morning service, we inadvertently reset some parts of our streaming configuration to their (low quality) default.

How are we making sure it doesn’t happen again?

We’re documenting this problem, so that anyone experiencing it in future has a single playbook they can use to fix it. This playbook includes making sure streaming settings are properly restored.

Bonus round: What went wrong with the video mixer?

We don’t know. But we did write about it publicly, and figured out how to fix it.

Post-mortem: Remembrance Sunday

We streamed Remembrance Sunday! This is a first for us, having pre-recorded the worship and act of remembrance last year, this year we wanted to make sure people could still take part even if they weren’t able or comfortable to join us in person.

We talked a bit about how in last week’s notes, but this is all about looking back at what happened; what went well, what didn’t go so well, and what we can do in the future.

Things that we liked

  • Feedback was almost immediate that it was widely appreciated, with people in the Community Centre who couldn’t make the procession to the War Memorial feeling like they were still part of the service
  • The audio system at the War Memorial – entirely battery powered – behaved exactly as we expected
  • The communication between different members of the Tech Team was useful in letting people know the current state of the broadcast

Things that we didn’t like

  • The radios we used to stay in touch caused some interference to our audio
  • One of our remote cameras failed to connect properly
    • Although this didn’t turn out to be a problem, because people then stood in front of it anyway
  • Mixing audio from remote camera sources didn’t work as smoothly as we would have liked
    • Although this didn’t turn out to be a problem, because we were down to one remote camera
  • We forgot to make sure the usual automated bells to signal the 10.00am service were switched off, so they started ringing 15 minutes into the Remembrance Sunday service.
  • Uniformed organisations weren’t in place in time, so we were left using less than ideal camera angles to cover the beginning of the service

Things we learned

  • We need to check that all our tripod mounts are compatible as part of our planning exercise
  • We need to have more planning for where people stand and when things happen, because mistakes are recorded for all time and are visible to the world
  • Where we have a roving camera who is focussing on action in front of them, they need a spotter if they’re moving around (especially if they aren’t moving forwards)
  • We need to make sure we have the necessary permissions in place for video work involving uniformed organisations further in advance

Things we’re going to do

  • Write a new checklist for hooking OBS into our usual stack ready for including external video sources
  • Write a new checklist for outside broadcast camera setup, to reduce unexpected configurations
  • Work out exactly how audio mixing should work, and make sure that’s part of the checklist
  • Start keeping common playbooks with churchwardens etc around these special services and events, so we have a shared understanding of what happens and when
  • Investigate ways to move the antenna for tech radios outside, to both improve reception and reduce interference

Weeknotes: Saturday 13 November

It’s been a busy week, with upgrades to our network and planning for our Remembrance Sunday service stream. here’s what we’ve been up to.

We upgraded our network edge equipment

Over the last couple of years our network has become slightly more important than it was before – instead of supporting an office PC, a couple of card payment devices and the occasional laptop it’s now also providing connectivity for weekly service streaming, and wireless networks in the church and Community Centre for both our own team and guests.

One of our guiding principles in the Tech Team is “do it right”, and we’ve been able to take advantage of a donation for technology in the church to replace some of the last pieces of borrowed equipment in our network. More specifically, we replaced our edge router (the bit which connects our internal network to the outside world) with a TP-Link ER605 Gateway, and we’ve replaced our temporary Omada controller on a Raspberry Pi with an OC200 Cloud Controller.

Together these pieces of equipment make our network faster and more secure, giving us improved reliability and more insight into what’s going on inside. They also give us a foundation for more improvements in the future.

We installed a network cabinet

Anyone who has ever worked with network equipment will know that the cables just love to tangle themselves at the first opportunity. No matter how neatly laid out they are, they’ll tie themselves in a knot and make future work far harder than it needs to be.

We’ve got a surprising amount of network equipment for a parish church, and with the new edge equipment above we wanted to make sure that it’s kept organised and protected.

So we cleared away our existing equipment in all its tangled glory, and in its place installed a 6U server rack. To this we added a proper surge-protected power supply, brush plate to help us keep our cables in order, and rack-mount shelving to keep our devices separated. When in future we install a new switch (as is part of our roadmap) we can pull out the shelf currently holding one, and rack the switch directly into the cabinet.

We planted some more trees

We know that technology – especially buying new technology – has a cost to the environment. To help negate this we carbon offset all our new technology purchases, as well as plant trees for the future.

To offset our new network equipment we’ve offset one tonne of CO2 emissions, and funded the planting of fifty trees in Mozambique. You can see all of St Mary’s offsetting and trees to date in our Ecologi forest.

We got ready for Remembrance Sunday

Remembrance Sunday is one of the key services we provide for the wider community in East Leeds every year. Once we have finished the eucharist service in the church, we proceed down Selby Road (ably assisted by West Yorkshire Police, who close the road for us) to the War Memorial where we lead the annual Act of Remembrance.

Last year we were unable to hold this service as we usually would, instead using a pre-recorded Act of Remembrance to give people a focus. This year, although we’re back to worshipping in person, we know that not everybody feels comfortable with crowds and might not be able to physically join us.

This left the Tech Team with a whole new challenge which we only briefly experimented with at Easter: can we get outside video reliably into our streaming services? The answer, after some planning and experimentation, is “yes”. Our aim is to move seamlessly from our usual eucharist service (starting earlier than usual at 9.30 am) into the procession, follow the procession down Selby Road to the memorial, and then pick up the Act of Remembrance.

To do this we’re using a combination of equipment, software and tools. The service is being mixed and streamed as usual via our ATEM mixer in the church, but we’ve also added a new laptop to the mix running OBS Studio. We’re taking the video mix from OBS and running it into the mixer in place of our font camera (which is very rarely used), so that we can pull in additional video from this external source.

The next piece of the equation is getting video into OBS from outside the church, and to do that we’re using a web-based tool called VDO.Ninja. By running this on a mobile phone hooked up to a regular mobile network, we can stream live video from the phone to our laptop running OBS over a technology called WebRTC.

We’re planning to use two phones for this – one will be mounted to a tripod at the War Memorial, and plugged into our sound reinforcement system so that we can get nice clear audio when people are speaking and when we play the Last Post. The second one will be strapped into a handheld mount along with an external microphone (with windshield), and will follow the procession down Selby Road. We’re using the mount so that our camera operator can use two hands to support the phone, which makes holding it for a period of time a lot easier and reduces shake. Once arriving at the Memorial, we’ll attach this phone to a second tripod to give us two camera angles, and be able to use the microphone from it for providing background noise.

We can use then OBS Studio to swap between these two external cameras and mix their audio outputs, and then feed this mix into ATEM where it can be encoded and sent on to the rest of the world.

Fingers crossed!

Weeknotes: Saturday 23 October

It’s been another week of technology here at Whitkirk. Here’s what we’ve been up to

Better order of service redirections

One thing we’ve been stuck doing manually each week is updating our “redirect to latest order of service” URL so that it actually went to the right place. We use this link for the QR Code in the church which points people to the order of service, so it’s important that it’s right.

We’ve now automated this process, so it’s one fewer thing to have to remember.

Buzz off

Our improved set of connections for in-house sound system.

Over the past few weeks an annoying buzzing sound has crept into the speakers in the Community Centre. We spent a bit of time fixing this (and tidying up some cables), but also took the opportunity to improve power management and connectivity.

To help save energy, our sound system is now on a timed power supply. A push-button switches the whole thing on for a few hours – plenty of time for bookings to make use of the speakers and hearing loop – and then it’ll turn itself off to help conserve energy.

As part of this we also added a new phono input and an additional 3.5mm jack input, giving more flexibility for our users.

Prepare for ludicrous speed!

via GIPHY

The lovely people at Openreach have finally switched on the ability for us to get fibre-optic cables run right into our Community Centre. This isn’t just about improving our internet speed (although we’ll certainly enjoy a speed boost), but also about making sure that we’re on more resilient digital infrastructure in the future.

We’re confident that we’ll be moving to this fibre-optic connection some time in the near future, but we’re still not sure exactly which provider and package we’ll be using. This last week we spent some time looking at options and putting together recommendations.

Weeknotes: Saturday 9 October

It’s been quiet on the tech front, but things are going to start getting busy again as we approach Advent and have to start thinking about things like carol concerts!

Fun on (and off) the stage

We do a lot of stuff in the church building, but we also take care of technology over in our Community Centre. Over the past week we’ve been thinking about work we need to do on stage to tidy things up and make it more usable for more people.

This is a big job which we’ll be breaking down and tackling in multiple stages, but we’re starting by rearranging some wiring to tidy things up. This is particularly a problem in the stage right corner where a tangle of cables is not only making it look a mess but is also making it unintuitive to use, hard to troubleshoot and is causing interference.

At the moment we’re still narrowing down exactly what we want to do on paper, thinking about our various Community Centre users to make sure we’re covering as many use cases as possible, and thinking about the future to make sure we don’t need to re-do this kind of work for a good few years.