Brennan Sanford of Michigan State reaches for the puck against Tommy Novak of Minnesota in the first period during their game at Madison Square Garden on January 20, 2018 in New York City. (Photo by Abbie Parr/Getty Images)
The use of high-speed cameras has proliferated throughout sports, particularly baseball, as coaches, players, and researchers seek a better understanding of safe and effective technique. Clubs like the Houston Astros and private training facilities like Driveline Baseball are known to rely on them. High-speed cameras like Edgertronic, however, retail for thousands of dollars, pushing the use of this technology out of reach of many individuals or smaller teams.
Supercomputing company Nvidia’s research team has now created a system that can retroactively turn video footage from modest hardware—including mobile phones—into smooth, super slow-motion. The deep learning-based system can interpolate and create as many intermediate images as are necessary between the existing, real frames.
The sporting implications are myriad, both for broadcast and training. An example video produced by Nvidia features a clip of a college hockey game between Michigan State and Minnesota, and the research team noted that players could use its technology in self-scouting.
“You may want to see every single nuance,” said Jan Kautz, the senior director of visual computing and machine learning. “This is a way of looking at it in slow motion and making sure your form is correct. It’s better than manually slowing down your video.”
NVIDIA developed this technology in conjunction with researchers from UMass-Amherst and the University of California at Merced and presented at this week’s conference on Computer Vision and Pattern Recognition.
The technology draws on a pair of neural networks that discern the optical flow of pixels from one frame to the next while also accounting for the occlusion of objects. The demo also takes videos from the Slow Mo Guys—and slows them down even more.
“As a research organization, we try to solve problems that we believe are interesting from a scientific point of view as well as having applications in the real world,” Kautz said, adding later: “It’s a nice aesthetically pleasing effect you can achieve with it in addition to hopefully being useful to some specific domains such as sports, where you might want to slow things down to see how you’re performing.”
Nvidia AI system fills in missing frames between the left and right images (in green borders), enabling high quality video play back in slow motion. (Courtesy of Nvidia)
The research paper was submitted last November and was being presented on Thursday at the conference in Salt Lake City. Kautz said there are not yet any concrete plans to share the technology, nor any conversations with anyone in the sports world. But that surely will come in time.
“At Nvidia research, we try to push the state of the art forward, always with the goal of helping Nvidia or one of our partners to improve a product or technology that they use,” Kautz said.
