Technology at Whitkirk

What happened?

Whilst streaming the Act of Remembrance from the War Memorial, our live stream didn’t include audio.

What happened?

Our Festal Evensong service on Sunday 3 November didn’t stream as expected.

It’s been a while since we did weeknotes! Here’s what’s happening:

We mixed up our prayers

If you subscribe to our weekly notices you’ll know they (usually) include a list of names to pray for. In the past these have always been sorted alphabetically, but we’ve made a change so they’re randomised on a week-by-week basis. This way helps prevent intercessions from being something you do ‘by rote’, encouraging more engagement with the act of praying for the individual.

We’re planning some audio improvements in the church

The audio equipment in the church is getting a bit long in the tooth, and we’re growing increasingly worried about its reliability. To address this we’re putting together a five-year plan for rolling repairs, replacements and upgrades.

We fixed timezones on whitkirk.com

The website for our Community Centre wasn’t properly understanding timezones and daylight savings, so we reminded it that we live in the Europe/London timezone and not UTC.

We started writing a handbook

We love openness, and we love documentation, so we started combining the two into our (work in progress) St Mary’s Church Handbook. The grand plan is that this will become the go-to source of “how do I do this thing” information for anyone on the entire St Mary’s team, not just the Technical Team.

We sped up some website behaviour

We were able to make some optimisations to how we serve images on our website which makes it snappier and more responsive for users.

After all of our big tech events, we like to review how things went and identify areas for improvement, even if we felt that nothing went wrong. We always do these in a no-blame way, following the Retrospective Prime Directive:

“Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.”

Norm Kerth, Project Retrospectives: A Handbook for Team Review

For our reviews we follow a format called The Good, The Bad and The Ugly, where we identify things which went well, things which shouldn’t have happened, and things which we could have done better.

The Good

• The stream from the War Memorial was good, with solid connectivity and clear audio.
• Using an additional speaker at the War Memorial gave clearer, crisper audio for those there.
• Relaying the Act of Remembrance into the Community Centre is appreciated by those unable to join the procession for whatever reason.

The Bad

• We lost our in-church connectivity just as the procession was forming, which meant we had to hurriedly switch to a backup. We were only able to do this because of performance issues earlier in the morning meaning we were in a state where this option was available. We should investigate ways of improving the resiliency of this connection.

The Ugly

• The internet connection in the church was performing poorly at the start of the service, prompting us to use a more adaptive but lower quality streaming process.
• There was a slight mis-timing at the start of the service due to some confusion over flag parties.
• We forgot to disable the wifi on our mobile camera, which led to a drop in connectivity as we left the church.
• We used a radio microphone at the War Memorial, but this wasn’t as reliable as we expected. We should spend the time to install a wired microphone in future.

A few weeks ago during a routine check on some of our equipment in our Community Centre we noticed that one of our wifi access points was no longer connected to the network and totally refused to reconnect no matter what we tried. This was annoying, but not entirely unexpected – the access points were installed in 2019, as a relatively cheap stop-gap solution until something more permanent was put in place. Then 2020 happened, and priorities changed slightly.

But still, a failing access point forced our hand. More specifically, it forced our hands to move the remaining access points around to try and maintain coverage where it was most needed – the bar, where it supports our till and card terminal, and the office, where people do a lot of parish administration.

Unfortunately, this deployment meant that the Main Hall suffered from poor coverage. It would sometimes work if you stood in the right place, but couldn’t really be relied on. Since we promote our guest wifi network, and advertise wifi coverage as one of the features of the building, this wasn’t great. Not only was it far below the standard we aim to provide, but it also began to affect the quality of events.

So we decided to fix it. Fortunately we already had the beginnings of a solution powering the wifi in our church building, a pair of TP-Link Omada hotspots (more specifically, EAP115s), along with a controller. Instead of spending time scoping out a new system, we knew we could just extend this. All we needed to do was decide on the new hardware for our Community Centre and get it installed.

During the refurbishment of our Community Centre in 2019 we installed some structural network cabling (the plan being to one day hook proper wifi access points up to it), which meant we knew the points in the building where hotspots could be easily installed. Being able to power them over ethernet was crucial because not all these locations had mains. Ideally, we wanted ones which could be powered by our network switch directly; the EAP115 access points need an external power injector, which in turn takes up space and needs sockets. We also considered the density of users, and if we should future-proof by supporting the newer WiFi 6 standards.

In the end we settled for four new WiFi 6 access points; three EAP620HD units which cover the main hall and bar, and an EAP615-wall unit which gives the Smeaton Room dedicated coverage, as well as maintaining access to physical network sockets in there should we ever need them. In an ideal world we would have ceiling-mounted all of the EAP620s, but limitations of our cabling means that two of them are wall-mounted instead. Fixing this would require a significant amount of work to move cabling, channeling out walls and ceilings and then making good again, and it’s simply not worth it for the marginal gains.

The next step once the hardware was installed was provisioning it and configuring our network. For the most part, we just added the Church’s existing network configurations to the new spots. Print off some updated signage with our new network details for guests, and we’re done!

Finally, because we know everything we do has an environmental footprint, we’ve offset a full tonne of CO2 emissions to cover the impact of manufacturing and shipping this new equipment. We’ve also funded the planting of another 25 trees, more specifically mangrove trees in Mozambique.

This morning we suffered a networking failure which meant we couldn’t stream the entire service, and only captured the first few minutes.

What happened?

Shortly after the start of the service, we began to see signs of unexpected buffering on our video stream. A few minutes after this we lost all internet connectivity in the church. We quickly tracked this down to a complete failure of the wireless bridge between the church and the Community Centre, but were unable to restore the connection.

Why did this happen?

We don’t know. It’s possible that the cold temperatures were having an impact on the equipment we use for the wireless bridge and the timing of the failure was entirely coincidence. The equipment, however, is intended for use outdoors and should be comfortable operating in a much wider range of temperatures. For this reason, we’re wary of naming the weather as the culprit.

What are we doing to fix it?

In the short term, we’ve re-angled one of the receivers which has become knocked during routine maintenance to make sure we always have the strongest possible connection between the two buildings.

In the long term, we’ll investigate the feasibility of installing a permanent fibre-optic link between the two buildings so that we’re unaffected by issues such as signal alignment and weather in future.

One thing we like to do in the technology team is look back at things we’ve done, and contemplate what we’ve liked, not liked, learned, and can do. We call this a post-mortem, and after a particularly busy weekend celebrating 100 years of Temple Newsam being owned by the City of Leeds, we’ve got two events to look at!

As always our reviews – especially around stuff which didn’t go quite right – are blameless. No one individual is held responsible, as we believe this is the best way to identify our weaknesses and get them fixed.

The Whitkirk Lecture

This was a lecture held in our Community Centre, for which we provided projection and sound reinforcement, as well as video recording for later release.

Things that we liked

• Having a clicker and presenter display made our presenter more confident, as well as keeping the podium clear

Things that we didn’t like

• Our projector’s VGA input wasn’t compatible with our chosen computer’s VGA adapter, so we had to cobble together an alternative on the fly
• There weren’t many good or unobtrusive places to put our tripods to record the talk

Things we learned

• We don’t have enough power distribution options to get power where we needed it without resorting to a mix of cabling

Things we’re going to do

• Encourage more investment in the Centre as a presentation venue, such as purchasing a proper podium
• Look at the costs involved in purchasing a new projector and screen suitable for the space
• Invest in some more power distribution options for the tech team

The Annual Eucharist

We streamed this service live from Temple Newsam House, as well as relaying it across the building to an overflow space in case the Long Gallery exceeded capacity.

Things that we liked

• Using a dedicated 4G router simplified our setup, meaning we could more confidently use multiple devices

Things that we didn’t like

• Our outbound network speed wasn’t as fast as we predicted, meaning we suffered some drop-outs and loss of quality
• The poor lighting in the room meant some of our shots weren’t as sharp as we would like
• The lack of internal networking in the building meant we were forced into maintaining two entirely separate mobile network connections

Things we learned

• Our internal network’s performance was impacted by the sheer volume of people when stood
• The mobile signal in the Long Gallery, despite our router having much larger antennas than a phone, still isn’t great

Things we’re going to do

• In future, make sure our hotspot is lifted above the crowd on a tripod
• Investigate an external antenna (or two) to let us maximise our external connectivity