Weeknotes

The period of Ordinary Time between Trinity Sunday and Advent is pretty empty in the Church calendar… but for the tech team it’s a chance to get on with all kinds of bits and pieces.

We automated the notices

Over the past few months, we’ve been progressively automating more of the things which happen every week in an easily repeatable way. The latest thing to be taken over by the computer is the process of generating our weekly notices both for our website and email.

We’ve had fibre installed

As part of our plans for continuous improvement, we’ve had fibre-optic internet fitted to the Community Centre. This gives us slightly higher speeds than before, but more importantly it offers improved reliability, future-proofing and scope for further improvement if needed.

Better continuity and disaster planning

Nobody likes thinking about things going wrong, but we’ve spent some time figuring out how we can improve the tech team’s resiliency in a number of situations.

It’s been a while since we had a weeknotes, hasn’t it? To cut a long story short, it’s been a busy winter for most of the tech team and we haven’t had as much time to do interesting stuff, let alone write about it. But we’re here now, and here’s a quick run-down of the past couple of months:

We did Christmas

There are three times in the year when we’re put under pressure to make sure things go seamlessly. Easter, Remembrance Sunday, and Christmas. This year for Christmas we went all-out and streamed every Christmas service, even streaming our crib service internally so we could use our Community Centre for socially distanced overflow if needed.

Most of these were streamed using our usual complement of equipment, but for the Festival of Lessons and Carols we built on things we had learned at Remembrance to add two whole new temporary camera angles. One of these gave us a closer view on the choir to help our viewers feel more connected to the music, and the other was an alternative angle on the lectern to give extra variety during readings. This went well for a couple of reasons; the extra angles make the process of streaming feel more creative and engaging both for the video operator and viewers, and it gives us a way to test out new angles when we’re thinking about the expansion of our permanent system before we commit to anything.

Our Omada write-up is being used

Way back last August we wrote a more in-depth dive into how we set up our Omada controller. Since then we’ve heard from a couple of people who followed the guide to get their own installation up and running. It’s always nice to know that we’re not just talking to the void!

We fixed some email configuration

As a sort of follow-up to the above Omada configuration, we identified a problem with our email settings where it wasn’t playing nice with Google’s two-factor authentication requirements. To resolve this we swapped our controller’s email configuration to use our Mailgun account instead.

We removed a load of equipment from the stage. Then we put it back.

Since our Community Centre was refurbished we’ve had a somewhat temporary kludge of stuff running the sound and stage lighting. During the pandemic there wasn’t much of drive to do anything about this, but now that we’re back up and running it was getting annoying.

To fix this we removed everything (including all our power and signal cables), came up with a tidier plan, and put it back again. We’ve got more work on this planned for the next couple of months to get things looking even smarter, keep them safer, and make them easier to maintain.

We started looking at better heating controls

Energy costs are going up, and our buildings are awkward and expensive places to heat. At the moment all our heating is controlled by fairly dumb thermostats with schedulers, and these need people to be physically stood next to them to update the schedule. We’re starting to look at connected solutions which not only let us update the schedule remotely, but which also support things like weather-aware pre-heating, and better zoning of our heat to get some potentially significant efficiency gains.

We built some tools to automate some workflows

One thing we dislike is having to repeat ourselves, which is why a lot of what we do relies on only having one ‘true’ source of information for each set of facts within the Church. For a lot of these things the source of truth is a thing called ChurchSuite – everything in our calendar comes straight from there, as does our list of what’s on (and we share the code we use for this). It knows about – at least in theory – all our upcoming services and events.

But quite a lot of our streaming services don’t yet rely on this single source of truth, instead relying on a series of checklists and our technical team manually stitching things together to make sure our orders of service and YouTube channel are in sync. This unfortunately involves quite a lot of repetition – we set things like dates and times in at least half a dozen places for every service, and write things like video titles in a very formulaic way.

Fortunately, computers are really good at copying things into lots of places in a formulaic way. So we built some bits of code which automate chunks of workflow for our Communications Team and Streaming Team. These pull events from ChurchSuite into another tool called Airtable, where these teams can then layer on top information specific to what they’re doing without needing to duplicate all the basics. And when something is updated in ChurchSuite, those replicated pieces of information are updated as well. Each team then uses Airtable to generate reports needed to do their job.

…and then we automated telling people the result

Once a week, another bit of tooling goes and grabs the information from Airtable, does some work to group it and highlight things of interest, and then emails the teams a summary. This improved visibility makes sure that our teams can more easily keep on top of what’s going on and what they need to do, stopping things from falling down the cracks and getting problems solved earlier.

…and then we automated some downstream processes

For the Streaming Team, though, we went a step further by automating to process of adding streaming services to YouTube. As soon as a service is marked as being streamed we now automatically create a new stream with the relevant details, ready to go.

Our plan for the next step once we’re confident in what we’ve built so far is to include the creation and interlinking of order of service entries as well, at which point we will have automated away the vast majority of weekly drudge work. This will free our time to worry about things that actually need a human’s attention, instead of copying and pasting.

We started working on an improved switcher for our stage equipment

As part of improving the house rig in the Community Centre, we’re building an improved mains switcher and interface so that Centre users can more intuitively manage the sound system and stage lighting. Our current plan is a pair of Raspberry Pis, one with a touchscreen and one hooked up to a relay board, with a server running on the relay unit communicating to a JavaScript frontend.

This has a few advantages over our current way of doing things:

• It’s more unified (so no more hunting for the right switch)
• It’s more intuitive (we can put instructions for things right there on the screen)
• It’s more extensible (without needing to add yet more boxes to the front of the stage)
• Because it’s software-defined we can update it easily (without having to go under the stage to visit the equipment rack)
• Since it’s connected to our network we can control things remotely (for example by remotely assisting a Centre user if they’re having trouble)

When we’ve finished our proof-of-concept we’ll open-source the whole thing, as well as do a more complete write-up.

It’s been a busy week, with upgrades to our network and planning for our Remembrance Sunday service stream. here’s what we’ve been up to.

We upgraded our network edge equipment

Over the last couple of years our network has become slightly more important than it was before – instead of supporting an office PC, a couple of card payment devices and the occasional laptop it’s now also providing connectivity for weekly service streaming, and wireless networks in the church and Community Centre for both our own team and guests.

One of our guiding principles in the Tech Team is “do it right”, and we’ve been able to take advantage of a donation for technology in the church to replace some of the last pieces of borrowed equipment in our network. More specifically, we replaced our edge router (the bit which connects our internal network to the outside world) with a TP-Link ER605 Gateway, and we’ve replaced our temporary Omada controller on a Raspberry Pi with an OC200 Cloud Controller.

Together these pieces of equipment make our network faster and more secure, giving us improved reliability and more insight into what’s going on inside. They also give us a foundation for more improvements in the future.

We installed a network cabinet

Anyone who has ever worked with network equipment will know that the cables just love to tangle themselves at the first opportunity. No matter how neatly laid out they are, they’ll tie themselves in a knot and make future work far harder than it needs to be.

We’ve got a surprising amount of network equipment for a parish church, and with the new edge equipment above we wanted to make sure that it’s kept organised and protected.

So we cleared away our existing equipment in all its tangled glory, and in its place installed a 6U server rack. To this we added a proper surge-protected power supply, brush plate to help us keep our cables in order, and rack-mount shelving to keep our devices separated. When in future we install a new switch (as is part of our roadmap) we can pull out the shelf currently holding one, and rack the switch directly into the cabinet.

We planted some more trees

We know that technology – especially buying new technology – has a cost to the environment. To help negate this we carbon offset all our new technology purchases, as well as plant trees for the future.

To offset our new network equipment we’ve offset one tonne of CO2 emissions, and funded the planting of fifty trees in Mozambique. You can see all of St Mary’s offsetting and trees to date in our Ecologi forest.

We got ready for Remembrance Sunday

Remembrance Sunday is one of the key services we provide for the wider community in East Leeds every year. Once we have finished the eucharist service in the church, we proceed down Selby Road (ably assisted by West Yorkshire Police, who close the road for us) to the War Memorial where we lead the annual Act of Remembrance.

Last year we were unable to hold this service as we usually would, instead using a pre-recorded Act of Remembrance to give people a focus. This year, although we’re back to worshipping in person, we know that not everybody feels comfortable with crowds and might not be able to physically join us.

This left the Tech Team with a whole new challenge which we only briefly experimented with at Easter: can we get outside video reliably into our streaming services? The answer, after some planning and experimentation, is “yes”. Our aim is to move seamlessly from our usual eucharist service (starting earlier than usual at 9.30 am) into the procession, follow the procession down Selby Road to the memorial, and then pick up the Act of Remembrance.

To do this we’re using a combination of equipment, software and tools. The service is being mixed and streamed as usual via our ATEM mixer in the church, but we’ve also added a new laptop to the mix running OBS Studio. We’re taking the video mix from OBS and running it into the mixer in place of our font camera (which is very rarely used), so that we can pull in additional video from this external source.

The next piece of the equation is getting video into OBS from outside the church, and to do that we’re using a web-based tool called VDO.Ninja. By running this on a mobile phone hooked up to a regular mobile network, we can stream live video from the phone to our laptop running OBS over a technology called WebRTC.

We’re planning to use two phones for this – one will be mounted to a tripod at the War Memorial, and plugged into our sound reinforcement system so that we can get nice clear audio when people are speaking and when we play the Last Post. The second one will be strapped into a handheld mount along with an external microphone (with windshield), and will follow the procession down Selby Road. We’re using the mount so that our camera operator can use two hands to support the phone, which makes holding it for a period of time a lot easier and reduces shake. Once arriving at the Memorial, we’ll attach this phone to a second tripod to give us two camera angles, and be able to use the microphone from it for providing background noise.

We can use then OBS Studio to swap between these two external cameras and mix their audio outputs, and then feed this mix into ATEM where it can be encoded and sent on to the rest of the world.

Fingers crossed!

It’s been another week of technology here at Whitkirk. Here’s what we’ve been up to

Better order of service redirections

One thing we’ve been stuck doing manually each week is updating our “redirect to latest order of service” URL so that it actually went to the right place. We use this link for the QR Code in the church which points people to the order of service, so it’s important that it’s right.

We’ve now automated this process, so it’s one fewer thing to have to remember.

Buzz off

Over the past few weeks an annoying buzzing sound has crept into the speakers in the Community Centre. We spent a bit of time fixing this (and tidying up some cables), but also took the opportunity to improve power management and connectivity.

To help save energy, our sound system is now on a timed power supply. A push-button switches the whole thing on for a few hours – plenty of time for bookings to make use of the speakers and hearing loop – and then it’ll turn itself off to help conserve energy.

As part of this we also added a new phono input and an additional 3.5mm jack input, giving more flexibility for our users.

Prepare for ludicrous speed!

via GIPHY

The lovely people at Openreach have finally switched on the ability for us to get fibre-optic cables run right into our Community Centre. This isn’t just about improving our internet speed (although we’ll certainly enjoy a speed boost), but also about making sure that we’re on more resilient digital infrastructure in the future.

We’re confident that we’ll be moving to this fibre-optic connection some time in the near future, but we’re still not sure exactly which provider and package we’ll be using. This last week we spent some time looking at options and putting together recommendations.

It’s been quiet on the tech front, but things are going to start getting busy again as we approach Advent and have to start thinking about things like carol concerts!

Fun on (and off) the stage

We do a lot of stuff in the church building, but we also take care of technology over in our Community Centre. Over the past week we’ve been thinking about work we need to do on stage to tidy things up and make it more usable for more people.

This is a big job which we’ll be breaking down and tackling in multiple stages, but we’re starting by rearranging some wiring to tidy things up. This is particularly a problem in the stage right corner where a tangle of cables is not only making it look a mess but is also making it unintuitive to use, hard to troubleshoot and is causing interference.

At the moment we’re still narrowing down exactly what we want to do on paper, thinking about our various Community Centre users to make sure we’re covering as many use cases as possible, and thinking about the future to make sure we don’t need to re-do this kind of work for a good few years.

More of a fortnightnotes than a weeknotes, but we’re here again!

The Temple Newsam Service

Tomorrow’s service at Temple Newsam House is an annual event in the calendar. Last year we stitched together a pre-recorded service to mark the occasion, but this year we’re venturing into uncharted territory with our first live outside broadcast.

All things going to plan, we’ll be leaving the relative safety of our tech desk and taking a pile of equipment with us; our trusty video mixer, a monitor, two cameras, a laptop and all of the associated cables. We’ve already done a couple of test runs, so we’re reasonably confident of things going to plan. In true Whitkirk Tech fashion, though, we’ve got a couple of contingency plans up our sleeves.

It’s been a few weeks since we last had an update from the tech team… because we’ve had a quiet few weeks! The period of Ordinary Time after Trinity has given us an opportunity to take a bit of a breather after the chaos of the last 18 months, but we’re starting to get back into things now.

The Heritage Open Days talk

As part of Heritage Open Days this year, we’ve got a talk! Although we’ve had plenty of talks given between the church and Community Centre over the years, this has been the first one since we’ve had our streaming equipment installed, so we’re sharing it with the world.

Since we’re not fans of doing things by halves, and since we’re always looking for opportunities to step up our game, we’ll be providing slides via projector and relay screen (to address some poor sightlines in the building), as well as interleaving them into the video stream using picture-in-picture features of our video mixer. We’ll also be providing a presenter display and clicker, for a little extra bit of quality of life.

The Temple Newsam Eucharist

Each year we hold a service at Temple Newsam House to celebrate the estate being acquired by the City of Leeds. This year, we’ve been asked to stream it live so that those who can’t attend in person can still join in.

Although we’re pretty confident that we can stream from outside the church building, we’ve been doing some planning and testing to make sure everything goes as planned.

More trees, less carbon

Both of the special events above have meant we’ve needed to get hold of some more equipment to help move video around, and as part of our ongoing commitment to reduce the harm we’re causing to the planet we’ve planted more trees and offset more carbon. Including our last two offsets, that’s a hundred trees and two full tonnes of carbon offset by the tech team.

Guest wifi in the Community Centre

We’ve finished up our resiliency and soft-launch testing, and we’re finally happy to roll out guest wifi in the Community Centre! If you’re visiting us, point your phone’s camera at one of our wifi QR Code posters to get connected, or manually enter the network details.

Getting technical with Raspberry Pi, Omada, SMTP and HTTPS

We did a load of stuff with our network controller, and we wrote a whole blog post about it.