SS 134 – How Accurate is your Fitness Watch?

Grant and Heavey discuss technology, specifically, how accurate fitness watches really are, the role of technology in addressing healthcare issues, and which brands win the best and worst fitness watches in the market today based on a Stanford study that seeks to validate the data generated by these devices.

 

 

[00.55] Heavey’s Birthday, Massive Flooding, and Other Natural Disasters

 

Heavey talks about how he celebrated his birthday out in the wilderness doing a canoe trip. A month ago, this river actually experienced massive flood and Heavey didn’t see the gravity of this until that float trip when he saw a full-sized picnic table hanging on a 40-foot tree. The water obviously got so high that it picked up a picnic table and took it to the top of the tree. Then as they kept going down along the river, the trees were all just knocked over. He also heard stories of a deer that got swept up and stuck in the tree 30 feet high. At one point, they passed by what used to be a bridge but it got annihilated and now all you see is mangled concrete and steel pushed off to the side.

 

Heavey grew up in a place where tornadoes seemed natural to him so they don’t scare him as much as he’s scared of earthquakes. California freaks him out due to the magnitude and intensity of what could happen and how crazy a bigger earthquake could be.

 

 

[05:45] Ticks Should Scare You

 

Grant, on the other hand, has this growing fear of tick-borne diseases. First, he read an article about a two-year-old girl who died of a tick-borne illness called Rocky Mountain spotted fever. The two biggest concerns when it comes to tick-borne illnesses are Lyme disease and Rocky Mountain spotted fever. Secondly, Grant has a close friend whose wife has Lyme disease and it was until interacting with them and seeing them that he saw how severe this stuff can be. His friend’s wife was also diagnosed with Lyme. She was so weak she couldn’t get up and down the stairs and upon doing this IV medicinal treatment which was a combination of antibiotics along with UV treatment, she got a little bit better. But by a little bit better, it means they can only go to Target for 30 minutes. It’s a terrible situation to be in. Worse, the diagnostic tests of these diseases are not that great. Heavey describes it as a very real disease that messes up a lot of people with no rock solid treatments that people have to deal with it for a lifetime.

 

Back to the article, this girl had a fever so she was taken to a hospital. She was advised to rest and her fever went up to 104. She was taken to a second hospital and her test came back positive for scarlet fever. She was given antibiotics but her condition grew worse. Seven days later, she died in her mother’s arms. It wasn’t until then that they found out she had Rocky Mountain spotted fever.

 

Another article talked about the counties in the country that have these ticks. Fifteen years ago, only 5% of counties were tick-infested areas. Today, this percentage has risen to 25% of counties having ticks including 100% of California counties. Basically, ticks are spread across the country and they attribute it to climate change and summer heat.

 

Now you’re probably wondering how this is related to our episode today but Grant and Heavey want all of us to pay attention to the things around us that can influence our lives including this email they got from one of their avid listeners, Dario.

 

 

[11:37] Fitness Watches: How Accurate Are These?

 

In an earlier episode, they mentioned being the early adopters of the Basis watch but Heavey didn’t like the heart rate monitor on it which seemed inaccurate compared to the chest strap that he normally wears for heart rate monitoring. Anyway, Dario sent Heavey this article which was actually a link to a Stanford study. It looked at seven different wrist-worn sensors for measuring heart rate and they looked at their accuracy.

 

They specifically studied the Apple Watch, Basis Peak, Fitbit Surge, Microsoft Band, Mio ALPHA 2, PulseOn, and Samsung Gear S2. They wanted to validate the data generated by these devices. More and more people are bringing the information from these devices to their healthcare providers who don’t know what to do with the information. In fact, some progressive health care systems are trying to bring that data into patient medical records. So there exists this really weird grey area where the doctors aren’t sure if they can trust the data generated from them and they want to make sense of the accuracy of these things.

 

 

[14:30] Measurement of Heart Rate and Energy Expenditure

 

The specific things they were looking at include their accuracy in estimating heart rate and estimating the amount of energy or calories burned each day. They compared energy expenditure throughout the day versus how much energy was actually spent using very robust clinical testing.

 

In terms of how these devices differ from each other, companies keep this stuff buttoned up. There’s a lot of proprietary pieces to the puzzle. These devices use infrared sensing technology but their processing or the accuracy of the sensors vary from device to device. So it’s uncertain whether it’s exclusively a software piece or if there is some hardware element to it but Heavey suspects a combination of both. That said, there is a big variation across devices in terms of accuracy.

 

 

[16:25] The Stanford Study on Fitness Watches – An Overview

 

This study conducted at Stanford University involves 60 different student volunteers, half were men and the other half were women coming from diverse age groups, wide range of height, weight, and fitness. They even had different skin tones to see if this had any impact on the accuracy of the sensors considering these devices use light in measuring the pulse.

 

They also looked at each individual in a variety of different activities such as sitting, running, cycling, etc. Each person was then put through this whole sequence of device and activity combination, conducting 80 different tests per individual. People had to wear four of the watches at a single time to expedite some of the testing but they validated it all the heart rate stuff with 12-lead electrocardiograph monitoring. Then they conducted continuous clinical grade indirect calorimetry for energy expenditure verification.

 

 

[18:10] The Protocol

 

The researchers got these medical lab grade tests compared to your watch to see how close they actually are. The protocol involves the person being seated for five minutes. They got up and walked for ten minutes on a treadmill followed by five minutes of faster walking, five minutes of slow running, five minutes of fast running, and then sit back for one minute to recover. Then they took two minutes to get onto a bike, five minutes of low-intensity cycling, five minutes of high-intensity cycling, and then one minute to recover. The intensity is then tailored based on each individual’s fitness level. So they’re basically going through this whole sequence with these watches on their wrist plus the medical grade equipment to validate it. This was the basis of the experiment.

 

 

[19:00] Bias and Findings

 

As probably expected, each device was not exactly on with the medical measurements. Some errors varied based on activity. They found that the watches are best at predicting heart rate during cycling and worst during walking. The also found that the error was greater in males, in higher BMI cases, and people with darker skin tones.

 

Out of the devices they studied, six had an error of less than 5% in the heart rate measurement during cycling. According to Heavey, the most important takeaway of this entire conversation is that no device achieved an error in energy expenditure of less than 20%. So when your watch tells you that you burn 2,000 calories today, it’s going to be off by at least 20%.

 

 

[21:16] The Best and Worst Devices Go To… (drum roll please)

 

The study found Fitbit Surge to be the best device at predicting energy expenditure. Its median error across all activities was 27%. On the other hand, PulseOn was the worst in terms of energy expenditure accuracy having a median error of 93%. Crazy, isn’t it? So if you burned 1,000 calories, this thing could have told you that you burned 2,000 calories. A lot of people say they’re going to the gym for an hour so they can eat xyz and so they use the information from these devices. Unfortunately, the information they’re getting is far from valid.

 

The study also showed that energy expenditure was most accurately measured on walking and running and was the worst on sitting. It’s interesting how the heart rate was worse on walking and energy expenditure was best on walking.

 

Out of these seven devices, the Apple Watch had the lowest error in heart rate overall at just 2%. So if your Apple Watch is saying that you’re heart rate is x, there’s a good chance it’s indeed accurate.

 

 

[23:50] Apple Watch Now Syncing with Your Treadmill

 

Grant being an Apple fan is obviously happy about this. He also shares that during the recent WWDC Conference, they presented how you can sync your watch with your gym equipment like treadmills, elliptical trainer, bike machines. It’s a two-way connection in that the device can give your watch the work input and the watch can give accurate heart rate information to the device. It does take new hardware but Apple has already partnered with the largest brands. Grant says 85% of all gym equipment in the country is covered by this new partnership.

 

 

[25:15] Going Beyond Short-Term Profits

 

Grant explains that when a company takes a longer outlook, none of that development of systems whether it’s the hardware required to communicate or the software and API to make it work right, is going to create much in the way of profit in the short term. Basically, you’re not going to move the needle with this. However, what they’re actually doing is they’re actually setting the stage for something that can really create some momentum in the right direction and downhill movement that can help people’s health in the long run. It will create profit eventually but it’s that mentality where “our work is more than short-term profits” and this is pretty impressive.

 

Heavey actually purchased a fitness watch, not an Apple Watch, but it’s a new device from Garmin that was recently released. It’s not on this list but he’s going to check back in on how this works for him.

 

 

[27:40] Worst Watch for Heart Rate Accuracy Goes To…

 

Samsung Gear S2 was found to have an error in heart rate at 6.8%.

 

Meanwhile, the authors of this study have a long-term perspective. They want to use this as an opportunity to start a website called precision.stanford.edu where they will begin publishing data for each of the devices. There is also an opportunity for device users to participate through their user upload options in order to grow the site’s knowledge base and figure out the accuracy.

 

 

[29:40] Apple versus Basis

 

As a previous Basis user, Grant is currently happy with his Apple Watch saying how much Apple Watch is better all the way around. What Heavey didn’t like about the Basis watch was the sparse granularity of the data. The device averages on a minute-by-minute basis so it actually doesn’t represent true data based on the intensity of your workout. So no matter how many burpees he was able to do for 30 seconds, it would still have his heart rate from a minute ago.

 

Grant wants to emphasize that not only is Apple Watch a good fitness tracker, but it also does have a pretty good number of other functions. He can either wear it as a watch or as a fitness band and go for a run. Heavey says he didn’t get an Apple because it didn’t have a built-in GPS but Grant quickly corrects him saying that Series 2 actually has built-in GPS plus it’s water-resistant to 50 meters. (So what now Heavey?)

 

 

[32:00] Technology Addressing Health Issues

 

Nevertheless, Grant hopes to see these devices becoming more and more robust and their energy expenditure measurement becoming more accurate over time. Grant is impressed with how technology is addressing health issues instead of just being consumer products.

 

Heavey points out a conflicted element to this in the sense that it gets people to focus on the wrong things but at least it’s getting people to pay attention. It’s making them decide to eat more than they should thinking they’re burning more calories than they actually do. However, he still sees this as a positive step long-term.

 

 

Links:

 

 

Article on 2-Year-Old Girl Who Died of a Tick-Borne Disease

 

Apple Watch Syncing With Gym Equipment

 

precision.stanford.edu

 

Study on Fitness Watches

 

Apple Watch, Basis Peak, Fitbit Surge, Microsoft Band, Mio ALPHA 2, PulseOn, and Samsung Gear S2