Dr. Lukasz Mirocha
Hong Kong Baptist University
Beyond the Lens: Leveraging Neural Radiance Fields and Generative AI for Crossmedia 3D Design
Recent AI-based advancements have significantly impacted the domain of 3D digital imagery. The intersection of photorealistic spatial reconstruction
and synthetic image generation has blurred the boundaries between indexical lens-based digital images and synthetically created digital photorealistic 3D content. Recent advancements in neural radiance fields, structure from motion, and simultaneous localization and mapping (SLAM) techniques have enabled the reconstruction of real- life objects and locations with unprecedented accuracy and visual fidelity. Concurrently, diffusion-based Generative AI models have exponentially increased the potential for generating still images, videos, and simple 3D assets. This convergence has given rise to a rhizomatic entanglement of styles, behaviors, and capture settings originating from lens-based cameras and their virtual counterparts. The resulting porosity of styles and aesthetics has created opportunities for innovative crossmedia image sensibilities, bridging the conceptual space between visual/digital twins and synthetic generations of real-life, fictional, and hybrid styles, characters, objects, and locations. This presentation showcases a series of work-in-progress projects that aim to investigate the potential of novel methods for capturing, processing, and displaying 3D reconstructions/generations of real-life locations and objects. The focus will be on leveraging 3D Gaussian Splatting, SLAM-enhanced digital twins, and large LED display volumes to preserve cultural heritage sites in Hong Kong and Poland. Additionally, another project will explore the creation of immersive experiences based on spatial scans of complex natural environments as well as photorealistic synthetic 3D assets combined in a unified game engine-based real-time environment. This contribution serves as both a technical ideation and a cultural intervention in emerging technologies. It reflects on the conceptual and technical challenges faced when working with rapidly evolving spatial 3D media. Based on the presented works and numerous educational activities undertaken in the discussed domain, this contribution investigates how the space between visual/digital twins and synthetic generations can be leveraged to create new crossmedia image sensibilities, thereby contributing to the broader discussion of synthography, and the role of camera and cinematography in today's media landscape. Additionally, by discussing these technological advancements through an artistic and educational lens, this contribution explores the potential of collaboration between human artists and AI tools, while also addressing ethical considerations in ownership and accessibility.
BIO
Dr. Lukasz Mirocha is a new media and creative software professional working with immersive and real-time 3D media. He is currently collaborating with Hong Kong Baptist University Visualization Research Centre, where he researches emerging real-time media design techniques for virtual production, interactive content and world-building as a Research and Practice Fellow in Augmented Visualization. He has been engaged in new media / media arts research, practice and tutoring for more than a decade, publishing on new media, digital design, exhibiting artworks and tutoring internationally. He holds a PhD from the School of Creative Media, City University of Hong Kong (2021).
<< previous | next >>
Hong Kong Baptist University
Beyond the Lens: Leveraging Neural Radiance Fields and Generative AI for Crossmedia 3D Design
Recent AI-based advancements have significantly impacted the domain of 3D digital imagery. The intersection of photorealistic spatial reconstruction
and synthetic image generation has blurred the boundaries between indexical lens-based digital images and synthetically created digital photorealistic 3D content. Recent advancements in neural radiance fields, structure from motion, and simultaneous localization and mapping (SLAM) techniques have enabled the reconstruction of real- life objects and locations with unprecedented accuracy and visual fidelity. Concurrently, diffusion-based Generative AI models have exponentially increased the potential for generating still images, videos, and simple 3D assets. This convergence has given rise to a rhizomatic entanglement of styles, behaviors, and capture settings originating from lens-based cameras and their virtual counterparts. The resulting porosity of styles and aesthetics has created opportunities for innovative crossmedia image sensibilities, bridging the conceptual space between visual/digital twins and synthetic generations of real-life, fictional, and hybrid styles, characters, objects, and locations. This presentation showcases a series of work-in-progress projects that aim to investigate the potential of novel methods for capturing, processing, and displaying 3D reconstructions/generations of real-life locations and objects. The focus will be on leveraging 3D Gaussian Splatting, SLAM-enhanced digital twins, and large LED display volumes to preserve cultural heritage sites in Hong Kong and Poland. Additionally, another project will explore the creation of immersive experiences based on spatial scans of complex natural environments as well as photorealistic synthetic 3D assets combined in a unified game engine-based real-time environment. This contribution serves as both a technical ideation and a cultural intervention in emerging technologies. It reflects on the conceptual and technical challenges faced when working with rapidly evolving spatial 3D media. Based on the presented works and numerous educational activities undertaken in the discussed domain, this contribution investigates how the space between visual/digital twins and synthetic generations can be leveraged to create new crossmedia image sensibilities, thereby contributing to the broader discussion of synthography, and the role of camera and cinematography in today's media landscape. Additionally, by discussing these technological advancements through an artistic and educational lens, this contribution explores the potential of collaboration between human artists and AI tools, while also addressing ethical considerations in ownership and accessibility.
BIO
Dr. Lukasz Mirocha is a new media and creative software professional working with immersive and real-time 3D media. He is currently collaborating with Hong Kong Baptist University Visualization Research Centre, where he researches emerging real-time media design techniques for virtual production, interactive content and world-building as a Research and Practice Fellow in Augmented Visualization. He has been engaged in new media / media arts research, practice and tutoring for more than a decade, publishing on new media, digital design, exhibiting artworks and tutoring internationally. He holds a PhD from the School of Creative Media, City University of Hong Kong (2021).
<< previous | next >>