LocalTV+

LocalTV+ supports Airplay and it is an option to send the to your other devices. This works very well when working with Apple devices. Over the past year I have seen reports that it would randomly work on certain Roku devices. Alternatively when it does work people would report that there was no audio. Eventually I discovered that not all Roku devices support the codec we use (more on this later). To add more confusion some Roku devices while not supporting the codec directly do pass it through to your audio hardware, so if you have a receiver that does support it, it would work.

This bring us to Android, where I have discovered that the first party video player ExoPlayer does not directly support AC3 Audio (there is a way to build ffmpeg and pass it through). In fact, it seems stock Android itself does not support AC3 audio. To add more confusion here, some Android devices do have AC3 support that’s been added by the manufacturer. So while I do have a basic Android client that can play the video, there is no audio at the moment.

So what is AC3 and why do we use it? AC3 is a Dolby Digital 5.1 audio format. The ATSC signal we receive uses AC3 for audio and we pass it along unmodified. We could convert it to another format such as AAC however, converting from one lossy format to another will result in worse audio quality. The actual conversion process will also increase the load on out current encoders.

Why does Apple support it but not Google? I am not sure. My hunch is that it requires a license form Dolby who owns the format and Apple has baked it into the cost of every device.

Now that we have thousands of testers, this is what a typical NFL game looks like. This is the data usage from Sunday’s games. You may notice that we have also increased the video quality. So far it has been quite successful. We are not seeing many if any buffering/stalling events on the clients. We will continue to improve this. The green line is the hit on our servers. It doesn’t go up very much. The blue is what the CDN absorbs. That can go up until CDN reaches its limits, whatever they are. We pay 0.005  per GB so the 1.34 TB cost us about $6.80 to serve. So far in Sept we are at 9.7 TB total which is about $47 in bandwidth costs.

I am enabled Vision Pro support in the latest Testflight build for the iOS app. I do not own a Vision Pro and have only tested it in Apples simulator. It appears to work perfectly fine. If there is anyone out there who has one and tries it out, please let me know how it is for you. Also, yes I am watching CometTV here. This will be arriving in the latest build along with MeTV Toons and WGBX this week.

Looking at our statistics I notice that very few people watch WGBH in this app. It makes sense because there isn’t a lot of reason to watch this stream while PBS provides a high quality free stream themselves. In the interest of focusing on gaps in availability and also supporting broadcasters who make streams available, we will be moving to the official PBS streams linking directly to their app/web page. Everyone still gets to watch the channel for free and it frees up one tuner and one encoder and money that we spend on bandwidth for PBS. All of these are limited resources.

We entered a wider beta at Thanksgiving, midway through the NFL season last year and I immediately collected a lot of ‘data’ about my designs as the service scaled up from 3-4 concurrent users to dozens and everything melted down. After many adjustments I was able to scale up and could handle 65 concurrent viewers for the Super Bowl. We’ve continued to make improvements and I think the quality of the service has visibly improved. Thousands of people watched the NBA post season and the Olympics over the summer.

How many people have signed up recently? The first spike you see is for the NBA finals, the second spike is for the Olympics. There are now around 3000 users.

I measure viewership in streaming minutes. This is how much people are watching a channel. We count one viewer watching for 5 minutes as 1 unit for these graphs. This is what the NBA finals looked like.

Sporting events like the NBA and NFL provide bursts of activity. The olympics were different. Since NBC broadcast everything all day, it was a long term sustained level of traffic unlike anything I’ve seen.

Overall we have been able to handle both one time spikes and also sustained traffic. Looking forward, we will work to reduce stalling as much as possible. We will also switch to a kubernetes cluster that will hopefully allow us to both reduce costs and scale up even more. There is also a plan to add more tuner hardware for metv toons and WGBX.

We’ve have many stress tests over the last year or so. The NBA finals saw around 400 people watch the Celtics win. We’ve adjusted the server code to handle events like that without issues. The Olympics present a totally different stress test . Major sporting events like the NBA post season occur in bursts. There is a game or two for a few hours every other night that everyone watches. The olympics are the opposite of that. NBC broadcasts it all day so there are now people watching LocalTV all day in a way that we have never seen before — regular day time TV is never that compelling. This is a good long term stress test. At any given moment there appear to be a few hundred people watching the Olympics. But the world keeps moving, so while there is a new constant load from the Olympics the usual other bursty things are also happening like the Red Sox v Yankees game on Saturday night. So far nothing has broken.

The huge spike below consuming a 1 TB data in a day is the Olympics. The smaller bump on the 4th of July was the pops concert.

The Celtics being in the NBA finals has presented a new opportunity to watch what happens when the traffic spikes on the current architecture. Unlike other events the NBA finals are a series of matches which present an opportunity to test over time. I made a change recently to switch from 2.0 audio to 5.1 based on user feedback. The server went down for unrelated reasons during game 1. Game 2-4 saw record number of viewers watching. Game 4 last night has a high of around 400 viewers this is nearly an order of magnitude greater than the 65 viewers during the Super Bowl . However something seems to have gone wrong as highlighted by reddit posts. We aren’t sure what happened but the graphs below offer some clues. Game 2, 3 are the first two peaks but the last peak despite being a much higher number of viewers has half the bandwidth served. It’s also possible that people experienced errors but also possible that the Celtics were losing and many people turned off.

However the final clue that something went wrong is the large number of HTTP 400 error encountered unlike the earlier two games. We will try switching back to 2.0 audio to see if that improves things

There was a lot happening behind the scenes. Today there will be new iOS and tvOS betas released with notable changes. Everyone should notice that videos start up much faster. We have removed many delays that prevented video from starting as soon as you click on a show. There is a major UI update for tvOS that should be a much better user experience.

On iOS you will notice two new buttons below a show description. The share button lets you share information about the show with others either via chat or social media. The bell button allows you to set a reminder for the show. A clear bell means you can set a reminder and filled in bell indicates a reminder has been set. Since are very serious about privacy so this information never leaves your device. As always, we do not track what shows people watch or want to watch. A few min before the show starts you will receive an alert on your phone.

Finally we would like to thank all of the donors who have contributed to this project. If you are interested in donating, please use this link: https://givebutter.com/LocalTVPlus

As a matter of principle, we want to retain as little information as possible about our users. In order to minimize friction and retain almost no personally identifying information, we have rolled out Sign in with Apple. When you use this we recommend you pick the ‘hide email’ option. This way, you have an account on our system but we do know anything about you other thans the fake email address Apple provides. This address forwards to your real email and still allows us to contact you if we ever need to.

Thank you everyone who used the app to watch the Oscars last week. As expected the service ran without a hitch. Also, as expected, concurrent viewer numbers were not quite the Super Bowl but still more than we usually see. At its peak there were 37 concurrent viewers watching. I see this as more of an indicator of a wider user base from where we started.

The CDN continues to perform as expected. You can see we served the same amount of data (50gb) but the cache served another 130gb of data