Fox had almost pulled the plug on Life of Pi midway through the $120 million production, fearing they were putting up blockbuster money for an arthouse movie with limited appeal, but it turned out to be an international hit, bringing in five dollars for every dollar spent. In his acceptance speech, Lee generously thanked the studio for "taking the leap" with him.

Having proved everyone wrong, he could now make any film he wanted to: another comparatively low budget indie movie like Brokeback Mountain or the Ice Storm; an epic martial arts sequel to Crouching Tiger, Hidden Dragon; a superhero franchise movie to prove that Hulk (an underrated film but a commercial flop) was an aberration. Instead, he chose to make a war movie in a daring, brand new format that required him to relearn basic movie-making skills.

Billy Lynn's Long Halftime Walk was shot at 120 frames a second, in 3D, at 4K resolution. Simply put, this means there is 40 times as much visual information on screen as there is in a normal film. "That's just how our brains work. Our brains like information, especially when it comes to studying faces. I think we crave detail," says Lee. He is betting that audiences can handle it. This is far from a sure thing.

"The knowledge and equipment hasn't caught up to the medium. We're not good enough to play with it yet, so we're chasing it, trying to make sure that it's watchable," he continues. "I would like to think I got to first base. I'd like to hit a home run but I'm lucky if I get to first base. It's a beginning of something. It's humbling."

Hollywood directors don't often say that they "barely made it watchable". Thankfully, there's no one from the marketing team in the room with us, but when he says it, I can feel every PR in Essex House New York flinch. Lee's candour is bracing.

The movie is the most astonishingly immersive experience I have ever had in a cinema, and I tell him so. By combining an ultra-fast frame rate and crisp resolution, in 3D, Lee has made a film that pulls viewers inside the action.

An early scene of US soldiers patrolling an Iraqi market set my heart pounding. The mole on Vin Diesel's forehead has never been so apparent and there is no disguising how much Chris Tucker has thickened in middle age. A third of the way through, I was disturbed to realise that sooner or later, we were going to watch someone die in distressing detail.

Billy Lynn's Long Halftime Walk is a returning soldier movie in the vein of Stop-Loss and In The Valley of Elah, about a squad that has been involved in heavy fighting, feted by clueless civilians on a trip home to Texas. Billy has killed a man. He has dragged his friend's body through enemy fire. At the football stadium, security guards tell him "I wear a uniform too", as if they understand.

Lee chose to shoot at the higher frame rate because he was frustrated by motion blur in Life of Pi, and more broadly by the tendency of 3D movies to make people feel seasick. Five times the frames means no strobing, no blur… and no make-up. The director encountered a problem familiar to pornographers working in hi-def for the first time: the viewer sees a little too much.

Half the filmmaking tricks he had relied on his whole career no longer worked. Together with his crew, he had to rethink his whole approach. "It's like, all of a sudden, I don't know how to make a movie," he says. "It's very frustrating because you see things. You see someone pulling focus. You see acting. You see lighting. Everything that you use to make a picture look good and dreamy sort of disappears. You're on your own. You feel naked. And that's where we started."

New technology demanded new rhythms: longer shots to give viewers time to take in the scene and feel part of it. "You're so invested in it that you don't want someone to twist your head, tell you where to look. You want to do your thing, make your own decisions. All the cuts still work but I think that you want to be left alone. And to do a shot that long is very hard."

A screening at the Future of Cinema Conference in Las Vegas, in April, gave him the confidence that the format could work. "That was nerve-wracking. The best news is, other than they they're startled by what they see, they're emotionally engaged, more so than a regular movie. Even people who don't like the movie they say 'I don't wanna be in there: I wanna stay outside' which means they're engaged," he says.

I put it to him that the hyper-clarity exposes the film's flaws. Somehow, even a clunky line of script – "Talk is cheap, I know, but money screams" – comes off worse when everything looks so lifelike. Is that fair comment? "It's fair. That's the beginning," he chuckles. "You know, there's considerable imperfection in our skin: that's a real person. I think your heart races to it. It's human nature. If you see make-up or you see acting you're less engaged."

The truly crazy thing about all the compromises Lee had to make to create this (so far) one of a kind experience is that there are hardly any cinemas that can show it in its full glory. Worldwide, there are about 10,000 screens capable of showing 120fps in 2D, or 60fps 3D, which is close enough. In Australia, viewers will see it at the standard frame rate and resolution. In short, it will look much like any other movie, but made by a director with one hand tied behind his back, standing on his head.

Peter Jackson shot The Hobbit at 48fps, only for people to complain it looked too much like TV. Director and special effects artist Douglas Trumbull pioneered a 60fps format called Showscan in the early 1980s, but it didn't take off, and he retreated to the Berkshire hills, in rural Massachusetts. Semi-retired, he remains an evangelist for higher frame rates.

The only two big name directors to visit Trumbull, to see the technology he is developing for themselves, are Lee and James Cameron. Cameron calls the combination of 120fps, 4K and 3D "the new platinum standard" and has been promising a sequel to Avatar shot this way for years.

"I think it's new territory worth trying," says Lee. "Some of my colleagues will join in. I don't know how long I can last – I want to keep discovering this medium but I cannot do it alone." Studios will be watching Billy Lynn keenly – the images and the box office returns – to determine whether the format has a future.

Billy Lynn's Long Halftime Walk is in cinemas from November 24.