TransformingTheChurch.org Public Blog Article

Our Largest Upgrade -- Ever!

8/19/2015 3:29 PM

We’re in the final stages of the largest upgrade of our website that we have ever performed. Overall, we have worked on this upgrade for the last eight months.

There is nothing much left to do except tweaks on a few pages. We’ve gone all of our final quality-assurance work and now are only clicking through all of the pages on the website to make sure everything is correct.

The work began just before Thanksgiving 2014 when we migrated off our old streaming platform to the new system. The November upgrade maintained all of our legacy functions and added some capacity. However, the functionality implemented in this upgrade was pretty much just maintained at our legacy levels. To fully exploit the power of the new platform, we were required to do considerable software development.

Click here for details of the November Migration

We did deep preparation work for the full upgrade during all of the winter season to get ready. We began by building and testing our new and improved media platform. The media platform is a separate mostly-independent system so we could build it parallel to the production-grade website. The final test of this work was our very successful Online Film Festival that we conducted March 8-9^th. We used the Film Festival to generate a large load on the new system to stress test it. We served almost 7-thousand requests for the festival’s high-quality video files over a 57-hour period.

Click here to see the film festival.

We spent the spring season remastering and re-encoding all of our videos. With our upgraded media platform completed and tested, remastering and re-encoding of all of our videos were necessary because of our planned conversion to the HTTP Live Stream (HLS) protocol. As intended in our upgrade, we moved away from the RTMP protocol and the use of the Adobe Media Server (AMS) that we have utilized in the past. AMS and its RTMP protocol served us faithfully and well for the last decade (and before that in other projects), but HLS offered many advantages in our architecture.

Click here for platform details.

The switch required re-encoding each of our videos using our new cloud-based encoder service that is an integral part of our new media platform. To get the most quality out of the switch over to HLS, we remaster new intermediate masters from the original edit masters for every video on the website. Remastering was a very time-consuming process, and we didn’t finish with this phase of the work until late in June.

With new intermediate masters finished, we uploaded all masters to our new cloud-based storage system. Once uploaded, we used our new cloud-based encoding service to make eleven HLS distribution file sets for each video on the website, ranging in speeds from 328-kbps to 8,884-kbps.

• 8,884-kbps - spatial size 1,920 x 1,080 (29.97 fps)
• 6.884-kbps - spatial size 1,280 x 720 (29.97 fps)
• 5,384-kbps - spatial size 1,280 x 720 (29.97 fps)
• 3,756-kbps - spatial size 960 x 540 (29.97 fps)
• 2,256-kbps - spatial size 896 x 504 (29.97 fps)
• 1,756-kbps - spatial size 768 x 432 (29.97 fps)
• 1,392-kbps - spatial size 640 x 360 (29.97 fps)
• 1,192-kbps - spatial size 640 x 360 (29.97 fps)
•    792-kbps - spatial size 640 x 360 (29.97 fps)
•    528-kbps - spatial size 480 x 270 (15 fps)
•    328-kbps - spatial size 416 x 234 (12 fps)

With both the storage system for the masters and the encoding service in the cloud, reading the masters in the encoding process and writing back the finished files could take place a core network speeds (10 gigabits-per-second.) In turn, the cloud-based encoding service utilizes massively-parallel processing, which is ideal for tasks like encoding. The speed and quality of the encoding were impressive.

Our new cloud-based encoding service performed beautifully, and we accomplished all of the re-encoding in only four days of 24-hour-per-day work. Using our old local encoding farm, this would have taken more than six weeks to achieve.

The output of the encoding service automatically placed all encoded distribution files into our new Content Deliver Network to be ready for immediate use.

We then replaced our entire set of embedded media players with the newest version of the player and reconfigured to use the new HLS streams. We finished loading the new media platform and cutover to the new HLS streams for all videos in the early morning of July 10^th although we began phasing in with new videos beginning on July 1^st.

Starting late in the evening of July 10^th, we then began converting all of our pages to a new approach to supporting both mobile and desktop devices called “responsive design.” For mobile devices, we have supported another technical approach called “adaptive design” for some time. However, after extensive testing, the responsive design proved to be superior and have many operational advantages such as “authoring once, serving all.”

Click here for details about conversion.

We did the heavy-lifting of this conversion over the weekend of July 10^th. Come Sunday night, the public facet of the site was 98% converted to the new responsive design and working well enough to serve most of our visitors without major disruption. Any visible artifacts on web pages of the conversion were only minor formatting anomalies.

By July 18th, we completed the conversion of the public facet, so it was 100% correct.

For the last three weeks, we have converted all of the non-public spaces of the website such as our courses, lectures, communities and events.

For the last week, we have refreshed our search engine optimization and also re-proofed the entire site.

We’re almost finished, or at least as one ever is for this type of work. Tweaking pages tends to become a continuous process.

During the last half of August, we will be the adding new content that has been piling up while we have been working on the upgrade. To make the upgrade work go as quickly as possible, we pretty much made an all-hands-on-deck action. All of our web editors have single-threaded through the upgrade work to the exclusion of almost everything else. As a result, our normal posting of new content fell behind. During the next few weeks, we will be playing catch-up.

For example, just this week, we add in all of the "Chuck Knows Church" episodes, both for the original series and The Committee, that have come out while we worked on the upgrade. With the new media platform in full production, we were able to add the episodes with all of the new features our new platform allows, including Adaptive Streaming. BTW, if you don't know, we simply don't post the episodes, we get the original 1080p high-definition files and encode them, just as we do with our own videos, using Adaptive Streaming. Doing so allows us to stream Chuck with all of the quality that is possible.