Interview With "Welcome To Marwen" VFX Supervisor, Kevin Baillie

By Will Mavity 

One film that is currently shortlisted for the Oscar for Best Visual Effects and had a strong showing at the 2019 Visual Effects Bake-Off is Robert Zemeckis' new film "Welcome To Marwen."

​Despite a less than enthusiastic critical reception, the heavily computer generated film is receiving high remarks for its visual effects. Here to tell us a little bit more detail on how those effects were achieved, we were lucky enough to sit down and talk with Visual Effects Supervisor Kevin Baille.

​So this is a very VFX heavy film, yet you guys had what…50 mil to work with?

40 was the net budget for the film all in. It was about a tenth of what The Avengers had. This was certainly the most challenging film, I’ve done with Bob Zemeckis. I’ve been working with him since A Christmas Carol. That was full mocap and I would say this was more challenging than that.

Tell me a little more about that difficulty? How did you manage to make the dolls…not creepy?

I think one of the major accomplishments of this movie is that we did 46 minutes of the film inside Mark Hogencamp’s imagination with these dolls that are living and breathing beings without them ever falling into the uncanny valley. It was a pretty long journey getting there. We had to invent a bunch of new tech and workflows to make that happen.

When Bob first floated this script to me, this was back in 2013, I was barbequing in my backyard, and he called me all excited and said ‘hey I’ve got this wild script, can you tell me if it’s even doable?’ And it really took all the way from that point in 2013 all the way until we started shooting in 2017 to figure out how to bring these dolls to life. Initially, we started with the idea of either going motion capture like “The Polar Express,” but Bob was really adamant that he didn’t want to suffer any criticism for dead mocap eyes. So we initially set off thinking we were going to film live actors on these giant over-scale sets that were built to look like miniatures, and then we were gonna augment these actors with digital doll parts and kind of reshape their bodies to make them look more like dolls. So we ran a test with Steve Carell and tried that methodology. And it looked…horrifying. It was like someone in a really expensive Halloween costume.

So we went back and tried motion capture and that had all the problems that Bob was worried about. Especially because Bob said, ‘Kevin, if you can find a single film with a fully CG character in it for more than 10 minutes that really avoids these issues, I’ll trust you and we can go mo-cap.’ And lo and behold we weren’t able to find a movie that did a digital human at that scale and worked. And so eventually we had this aha moment and said, ‘instead of putting digital doll parts on Steve Carell, how about we put Steve Carell on a digital doll. So we took Steve’s eyes and mouth and fused those with a totally digital dolls body. And then gave the footage a treatment that made Steve’s body looked plastic and that worked. It really worked. You could feel Steve’s soul coming through in his character, but he looked 100% doll-like. So we ended up using that footage to get the movie greenlit, and that was the final methodology we used in the film.

So tell me a little bit more about keeping the soul, but essentially making them into Ken dolls.

It’s funny, one of the curses of computer-generated imagery is that the worst thing you can ever hear is ‘oh that looks like plastic.’ So our challenge here was to make the dolls actually look like plastic but also have them look high quality. So every doll started as a three-dimensional scan of the actor themselves. Then we worked with a fantastic makeup designer named Bill Corso. And he did a piece of concept art of each character, and said ‘here’s what each character would look like as a doll.’ So Steve Carell’s doll character Hogie, he has kind of a sharper jaw and a slightly smaller nose and is a little more symmetrical. Some of the women have button noses and smaller chins. So, every actor got a unique doll version of them that was designed.

We then digitally sculpted each doll. Bob approved each doll in the computer. Then we had a guy by the name of Dave Asling who has a company named creation consultants. We sent those scans to him and he 3D printed them. He built all these physical dolls and handpainted them. He even plugged in physical hairs three or four at a time. We then photographed and 3D scanned those dolls along with costumes that Joanna Johnston custom-built for them and round tripped those back into the visual effects pipeline. So the dolls you see on screen aren’t just digitally designed. They became dolls physically and then we took all the things we could learn about the physical dolls and then made sure our digital dolls matched exactly to that. I think that whole flow typifies all the departments working together in a very unusual way on this movie that helps the dolls and the surrounding town come to life.

Can you tell me a little bit more about the town itself?

So the town itself started in the very early concept phases with the visual effects team before a production designer was even involved. Bob just wanted a sense of how big this town was going to be. As soon as Stefan Dechant came on board, he took what we started and just made it a ton better with sketches and digital designs. And once we had the shape of the town down, we actually built it in full scale with foam core. Then Dave Asling and his team at Creation Consultants went about building two versions of the Marwen village. One that was gonna reside outside on a location and another that would be inside on a sound stage. They really used a lot of the same techniques that the actual Mark Hogencamp would use. They kind of took all their fancy tools and laser cutters and put them in the closet and really handcrafted the way Mark would. Once they finished that town, we photographed and 3D scanned every single building. And made an excessively detailed digital version of the Marwen town. There’s not a single frame of footage of the village that is the real village. It’s all our digital creation.

We also had to build a lightweight video quality version of the Marwen town for a few different purposes. I mentioned earlier that we were using actual footage of the actors’ faces fused with the dolls to bring these dolls to life. So in order to do that, the footage has to be beauty lit like it would be in a normal movie. Not only that, but we also had to film it from the EXACT camera angle that it was gonna appear in the final movie. So that meant on the motion capture stage, all of the actors had to be lit and filmed exactly as if this were a live-action stage. And of course, mo-cap stages are kind of gray voids. There’s nothing in them. So our DP, C Kim Miles, had to figure out what the lighting was going to be by using this video game quality of the town. He spent about two weeks working with the VFX team and Bob Zemeckis, figuring out what the blocking of each scene was gonna be. And then using a custom Ipad application, he was able to tweak and say “ok…the sun is at this height, and the sky is contributing this much fill light or I’m gonna have these custom lights that I’m gonna move around.” So before we shot a single frame on the motion capture stage, he had basically gone in and pre-lit every scene using this unreal game engine driven version of Marwen village. Not only did that give his team an idea of how to light the stage, but also, on stage, we had one monitors showing what the camera was seeing, actors with dots on their suits, but another would have this beautifully lit real-time version of the virtual world that the camera was seeing complete with moving dolls and everything.

So actors could do a take and then come back and look at the monitor and see what their dolls were doing in a rough version of what the final movie would look like right there on set. We call that process virtual production, and I think it helped the actors to get into their character, and it helped all the other departments to figure out why we had them doing things like putting on makeup when this is a mo-cap movie, and why we’re lighting them when this is a mo-cap movie. I didn’t have to explain any of that. They could just look at the monitor and see the impact these changes were having on the process we were creating. I know James Cameron has used similar technology on Avatar, but never before, has someone used tech likes this to inform live-action filming in the way we did, and certainly never before, has this degree of planning gone to filming a movie like this. Because we had to know everything about what we had to do before we got to the motion capture stage in terms of how our world was set up and lit in order to ensure that the actors and Bob had the freedom to do what they needed to. The amount of preparation we did for all this is the reason these dolls don’t look creepy.

You guys have CGI throughout pretty much the entire film, not just in Mark’s dream sequences. How many VFX shots are in the film, and where else did we see CGI at play that audiences might not immediately notice?

There are 655 visual effects shots in this film. Bob tends to have really long shots in his movies though. He gets a kick out of doing things in one single that most directors would need 3 or 4 to do. So I like to say that 600 VFX shots in a Zemeckis film are more like 1,400 shots in a normal movie.

Bob is a fan of shooting with as much control over the process as possible. That means often times, taking what another director might shoot outside and shooting it on a soundstage instead. Some of the things I’m really proud of are the scenes in Mark’s backyard where he’s out photographing his dolls out in the village were actually shot inside on a soundstage. There was a version of Mark’s trailer and Marwen town and a patch of grass and then everything else was just bluescreen. Watching that, even our cinematographer C Kim Miles had a hard time telling which scenes were shot on location and which were on a soundstage. And a lot of that was thanks to him. So many scenes in movies, even ones with amazing visual effects, when they have an exterior shots shot on a soundstage, it just doesn’t look right. Having the right amount of light pumped in to sell that there’s a sun there makes the difference. And C Kim Miles nailed that. So I think those scenes are really impressive.

There’s a scene towards the end where Mark pulls up at his gallery. And Method Studios did an amazing job turning Vancouver into New York for that scene. And every scene inside of Mark’s trailer is just blue screen out the windows. This all kind of speaks to something that’s underrated in films these days, which is consistency. In some incredible films this year, with some of the most amazing effects shots I’ve seen in a movie, my wife who doesn’t even work in the visual effects industry will roll her eyes saying “oof…well that’s a visual effect.” For me, the visual effects in a movie are only as good as the worst visual effects shot in it. Because that will take the audience out. So we really strived in this movie to make sure that we paid as much attention to bringing the simple window comps up to a level where they’re believable as we did to the full digital doll shots.

One other thing I’m pretty proud of is, one of the things that was really crucial was, making sure when we’re in Mark’s imagination that it feels like photography. That it doesn’t just feel like a normal movie. A lot of that has to do with the focus. When you’re shooting miniatures, things get soft really quickly. You know, you have a shallow depth of field. And traditionally, in a visual effects movie, that’s a post-process that’s done and there’s not a whole lot of thought that goes into it. In this movie, because we wanted it to look a lot like Mark’s photography, we actually had to pay special attention to focus and design for what the focus was going to be for every shot really early in the process.

To make the focus look as photographic as possible, we ended up using old school photographic techniques in a lot of the shots. We used tilt-shift lenses to make sure that two actors who are offset from each other at a diagonal line to the lens could both be in focus. One thing photographers loved to use as well when they have something in the foreground and something in the background that they want sharp, is called a split diopter. It’s like a bifocal for the camera lens. We used digital versions of those in this movie. And we even took that and invented a new thing called a variable diopter, which, technologically is one of the things I’m most proud of in this film. We were actually able to give the artists the ability to sculpt a curtain through whatever it is they want in focus. There’s a scene where the women of Marwen are sitting in a U-shape around a bar, and Bob wanted all of their faces to be in focus. So we were actually able to sculpt a curtain that went through all of the women’s faces that went away from the lens and back. Where that curtain is is what’s in focus. It’s almost like custom grinding a diopter for that specific shot. And the result looks really photographic and doesn’t look digital. More importantly, it achieved the story point that Bob wanted out of the tech. That’s the tech I love the most. Something that is unobtrusive and can feel natural to the viewer and the achieve the result the director wanted.

What are you working on next?

I’ll be working on Bob’s next project, which is shaping up to be “The Witches” with Warner Bros.

​"Welcome To Marwen" is currently shortlisted for the Oscar for Best Visual Effects and is playing in theaters now. To read more on the film's showing at the Visual Effects Bake-Off, click here.

You can follow Will and hear more of his thoughts on the Oscars and Film on Twitter at @mavericksmovies