Technology at Whitkirk

After all of our big services and events, we like to review how things went and identify areas for improvement, even if we felt that nothing went wrong. For our reviews we follow a format called The Good, The Bad and The Ugly, where we identify things which went well, things which shouldn’t have happened, and things which we could have done better.

The Good

• Having two operators meant that streaming was less stressful, and gave a more dynamic mix of both sound and video.

The Bad

• The soloist for the beginning of Once in Royal David’s City wasn’t mic’d correctly, meaning the sound wasn’t as clear as it should be.
  • In future, a member of the Tech Team should perform a mic check beforehand.
  • Microphone users who are unfamiliar with them should be told to always leave microphones switched on; the audio operator will manage levels.
• Changes had been made to the configuration of the sound desk at an unknown point in the past which affected our baseline mix, leading to a small amount of feedback.
  • Although this was quickly addressed by the sound operator, it shouldn’t have been an issue in the first place.
  • Replacement of the current mixing desk with a digital desk which can be locked or which has automatic feedback suppression should be made a higher priority.

The Ugly

• The microphone arrangement for the choir and organ is starting to show its weaknesses compared to the quality of the performance, and should be reviewed to provide a better mix.
  • This will likely require consultation with an acoustic engineer to ensure the best placement.
  • It is likely that this will also require engagement with the faculty process if anything more than the bare minimum is done, as microphones and cabling will need placing at high levels.
  • This will likely need either the inclusion of a submixer, or the replacement of the mixing desk, to allow for sufficient number of microphones.
• An incorrect expander configuration in our streaming pipeline meant that particularly quiet parts of singing or organ were being lost on the stream.
  • This configuration has now been changed to be less aggressive.
• A shortage of members of the Tech Team meant that we weren’t able to install the usual number of supplementary cameras, leading to a less engaging visual mix.
  • We should consider permanent installation of additional cameras in key positions, and potentially replacement of our video mixer with one supporting more inputs. This will provide a wider and more engaging range of shots for all services, not just ones where the Tech Team are able to install temporary supplementary ones.

What happened?

Our Festal Evensong service on Sunday 3 November didn’t stream as expected.

It’s been a while since we did weeknotes! Here’s what’s happening:

We mixed up our prayers

If you subscribe to our weekly notices you’ll know they (usually) include a list of names to pray for. In the past these have always been sorted alphabetically, but we’ve made a change so they’re randomised on a week-by-week basis. This way helps prevent intercessions from being something you do ‘by rote’, encouraging more engagement with the act of praying for the individual.

We’re planning some audio improvements in the church

The audio equipment in the church is getting a bit long in the tooth, and we’re growing increasingly worried about its reliability. To address this we’re putting together a five-year plan for rolling repairs, replacements and upgrades.

We fixed timezones on whitkirk.com

The website for our Community Centre wasn’t properly understanding timezones and daylight savings, so we reminded it that we live in the Europe/London timezone and not UTC.

We started writing a handbook

We love openness, and we love documentation, so we started combining the two into our (work in progress) St Mary’s Church Handbook. The grand plan is that this will become the go-to source of “how do I do this thing” information for anyone on the entire St Mary’s team, not just the Technical Team.

We sped up some website behaviour

We were able to make some optimisations to how we serve images on our website which makes it snappier and more responsive for users.

After all of our big tech events, we like to review how things went and identify areas for improvement, even if we felt that nothing went wrong. We always do these in a no-blame way, following the Retrospective Prime Directive:

“Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.”

Norm Kerth, Project Retrospectives: A Handbook for Team Review

For our reviews we follow a format called The Good, The Bad and The Ugly, where we identify things which went well, things which shouldn’t have happened, and things which we could have done better.

The Good

• The stream from the War Memorial was good, with solid connectivity and clear audio.
• Using an additional speaker at the War Memorial gave clearer, crisper audio for those there.
• Relaying the Act of Remembrance into the Community Centre is appreciated by those unable to join the procession for whatever reason.

The Bad

• We lost our in-church connectivity just as the procession was forming, which meant we had to hurriedly switch to a backup. We were only able to do this because of performance issues earlier in the morning meaning we were in a state where this option was available. We should investigate ways of improving the resiliency of this connection.

The Ugly

• The internet connection in the church was performing poorly at the start of the service, prompting us to use a more adaptive but lower quality streaming process.
• There was a slight mis-timing at the start of the service due to some confusion over flag parties.
• We forgot to disable the wifi on our mobile camera, which led to a drop in connectivity as we left the church.
• We used a radio microphone at the War Memorial, but this wasn’t as reliable as we expected. We should spend the time to install a wired microphone in future.

A few weeks ago during a routine check on some of our equipment in our Community Centre we noticed that one of our wifi access points was no longer connected to the network and totally refused to reconnect no matter what we tried. This was annoying, but not entirely unexpected – the access points were installed in 2019, as a relatively cheap stop-gap solution until something more permanent was put in place. Then 2020 happened, and priorities changed slightly.

But still, a failing access point forced our hand. More specifically, it forced our hands to move the remaining access points around to try and maintain coverage where it was most needed – the bar, where it supports our till and card terminal, and the office, where people do a lot of parish administration.

Unfortunately, this deployment meant that the Main Hall suffered from poor coverage. It would sometimes work if you stood in the right place, but couldn’t really be relied on. Since we promote our guest wifi network, and advertise wifi coverage as one of the features of the building, this wasn’t great. Not only was it far below the standard we aim to provide, but it also began to affect the quality of events.

So we decided to fix it. Fortunately we already had the beginnings of a solution powering the wifi in our church building, a pair of TP-Link Omada hotspots (more specifically, EAP115s), along with a controller. Instead of spending time scoping out a new system, we knew we could just extend this. All we needed to do was decide on the new hardware for our Community Centre and get it installed.

During the refurbishment of our Community Centre in 2019 we installed some structural network cabling (the plan being to one day hook proper wifi access points up to it), which meant we knew the points in the building where hotspots could be easily installed. Being able to power them over ethernet was crucial because not all these locations had mains. Ideally, we wanted ones which could be powered by our network switch directly; the EAP115 access points need an external power injector, which in turn takes up space and needs sockets. We also considered the density of users, and if we should future-proof by supporting the newer WiFi 6 standards.

In the end we settled for four new WiFi 6 access points; three EAP620HD units which cover the main hall and bar, and an EAP615-wall unit which gives the Smeaton Room dedicated coverage, as well as maintaining access to physical network sockets in there should we ever need them. In an ideal world we would have ceiling-mounted all of the EAP620s, but limitations of our cabling means that two of them are wall-mounted instead. Fixing this would require a significant amount of work to move cabling, channeling out walls and ceilings and then making good again, and it’s simply not worth it for the marginal gains.

The next step once the hardware was installed was provisioning it and configuring our network. For the most part, we just added the Church’s existing network configurations to the new spots. Print off some updated signage with our new network details for guests, and we’re done!

Finally, because we know everything we do has an environmental footprint, we’ve offset a full tonne of CO2 emissions to cover the impact of manufacturing and shipping this new equipment. We’ve also funded the planting of another 25 trees, more specifically mangrove trees in Mozambique.

This morning we suffered a networking failure which meant we couldn’t stream the entire service, and only captured the first few minutes.

What happened?

Shortly after the start of the service, we began to see signs of unexpected buffering on our video stream. A few minutes after this we lost all internet connectivity in the church. We quickly tracked this down to a complete failure of the wireless bridge between the church and the Community Centre, but were unable to restore the connection.

Why did this happen?

We don’t know. It’s possible that the cold temperatures were having an impact on the equipment we use for the wireless bridge and the timing of the failure was entirely coincidence. The equipment, however, is intended for use outdoors and should be comfortable operating in a much wider range of temperatures. For this reason, we’re wary of naming the weather as the culprit.

What are we doing to fix it?

In the short term, we’ve re-angled one of the receivers which has become knocked during routine maintenance to make sure we always have the strongest possible connection between the two buildings.

In the long term, we’ll investigate the feasibility of installing a permanent fibre-optic link between the two buildings so that we’re unaffected by issues such as signal alignment and weather in future.