The goal of my Innovation Fellowship is to create a new form of immersive 360-degree VR video. We are massive consumers of visual information, and as new forms of visual media and immersive technologies are emerging, I want to work towards my vision of making people feel truly immersed in this new form of video content. Imagine, for instance, what it would be like to experience the International Space Station as if you were there - without leaving the comfort of your own home. The Problem: To feel truly immersed in virtual reality, one needs to be able to freely look around within a virtual environment and see it from the viewpoints of one's own eyes. Immersion requires 'freedom of motion' in six degrees-of-freedom ('6-DoF'), so that viewers see the correct views of an environment. As viewers move their heads, the objects they see should move relative to each other, with different speeds depending on their distance to the viewer. This is called motion parallax. Viewers need to perceive correct motion parallax regardless of where they are (3 DoF) and where they are looking (+3 DoF). Currently, only computer-generated imagery (CGI) fully supports 6-DoF content with motion parallax, but it remains extremely challenging to match the visual realism of the real world with computer graphics models. Viewers therefore either lose photorealism (with CGI) or immersion (with existing VR video). To date, it is not possible to capture or view high-quality 6-DoF VR video of the real world. My Goal: Virtual reality is a new kind of medium that requires new ways to author content. My goal is therefore to create a new form of immersive 360-degree VR video that overcomes the limitations of existing 360-degree VR video. This new form of VR content - 6-DoF VR video - will achieve unparalleled realism and immersion by providing freedom of head motion and motion parallax, which is a vital depth cue for the human visual system and entirely missing from existing 360-degree VR video. Specifically, the aim of this Fellowship is to accurately and comprehensively capture real-world environments, including visual dynamics such as people and moving animals or plants, and to reproduce the captured environments and their dynamics in VR with photographic realism, correct motion parallax and overall depth perception. 6-DoF VR video is a significant virtual reality capability that will be a significant step forward for overall immersion, realism and quality of experience. My Approach: To achieve 6-DoF VR video that enables photorealistic exploration of dynamic real environments in 360-degree virtual reality, my group and I will develop novel video-based capture, 3D reconstruction and rendering techniques. We first explore different approaches for capturing static and dynamic 360-degree environments, which are more challenging, including using 360 cameras and multi-camera rigs. We next reconstruct the 3D geometry of the environments from the captured imagery by extending multi-view geometry/photogrammetry techniques to handle dynamic 360-degree environments. Extending image-based rendering to 360-degree environments will enable 6-DoF motion within a photorealistic 360-degree environment with high visual fidelity, and will result in detailed 360-degree environments covering all possible viewing directions. We first target 6-DoF 360-degree VR photographs (i.e. static scenes) and then extend our approach to 6-DoF VR videos. Project partners: This Fellowship is supported by the following project partners in the UK and abroad: Foundry (London) is a leading developer of visual effects software for film, video and VR post-production, and ideally suited to advise on industrial impact. REWIND (St Albans) is a leading cutting-edge creative VR production company that is keen to experiment with 6-DoF VR video. Reality7 (Hamburg, Germany) is a start-up working on cinematic VR video.

StoryFutures Academy is a genuine HEI-Industry collaboration between trainers and producers to develop the storytelling techniques and languages that will shape the future of immersive narrative. Led by the National Film & Television School and Royal Holloway, our bid is founded on research and training knowledge that places storytelling at its heart. We will provide core screen sector talent with the tools, space, creative freedom and cross-sector work structures to unlock the creative and commercial potential of immersive production. Partnered by Sir Lenny Henry, Destiny Ekaragha, Alex Garland, Georgina Campbell and more we will lead a charge of UK creative talent into immersive that embeds diversity into the development of the medium across writing, directing, producing, performance, cinematography, editing, animation and VFX. We will deliver training in action, providing opportunities for creatives to learn through taking part in immersive productions that tackle key creative and technical challenges. We link this to R&D in business model innovation and audience insight that combines electronic engineering, neuro- and cognitive psychology with long-sighted ethnography to provide a catalyst for growth of creative industries. We de-risk immersive production through 4 workstreams that provide £1.25m for collaborative projects with immediate impact: 1. Embedded Placements: Promoting talent development and commercial vitality by enabling placements of screen sector talent on immersive productions for cross sector innovation and work-based learning; 2. Collaborative Co-productions: Co-producing immersive experiences that tackle sector wide creative and technological barriers to growth, upskilling core screen sector workers via access to hands-on learning on live productions that build a cross-sector talent pool; 3. Experimental Labs: R&D-based productions that expose core screen sector talent partners to immersive and push technological and creative boundaries; 4. Developmental Training: Training a next generation of immersive storytellers and trainers that cascades knowledge to HEIs, FECs and industry across the country. We are unique in our industry credibility and relationships. The NFTS was awarded the BAFTA for Outstanding British Contribution to Cinema in 2018, and it is the only institution in the UK where industry already invest over £1.5m annually in CPD level training courses, enabling us access and partnership with internationally renowned on- and off-screen talent. Our partners are world-leaders in how story and new technologies combine to produce compelling and novel immersive experiences, including immersive theatre (Punchdrunk), VR (Rewind), gaming (Sony IE), film (BFI), television (Sky VR), advertising (McCann) visual effects (The Third Floor, Double Negative), performance capture (Imaginarium). We bring them together with advanced Original Equipment Manufacturers (Microsoft, Plexus) and sector experts (Digital Catapult) to place story and technology in tandem to explore, research, train and develop cross sector storytelling talent and business models. SFA will create over 60 ICE productions and generate nearly 1,000 direct beneficiaries. It will cascade benefits, insights and opportunities via collaborations with regional partners, including NFTS' base in Scotland alongside TRC Media and UK Games Fund as well as access to nationwide labs via Digital Catapult, and co-production bases in Manchester (McCann) and Yorkshire (BFI). It also gains significant advantage from the economies of scale and access to talent achievable from our Gateway Cluster base with its easy flows of talent and work in and out of London. StoryFutures Academy can make the UK a world-leader in immersive because it has unmatched access to mainstream creative screen sector talent, companies and technologies, allowing it to translate experimentation, training and R&D into tangible economic and creative ROI for the whole of UK Plc.