3D skills are no longer confined to the highly competitive field of media and entertainment. Thanks to Universal Scene Description (OpenUSD), professionals equipped with these skills are finding groundbreaking opportunities in diverse industries.

OpenUSD, with its robust capabilities in 3D modeling and simulation, serves as a pivotal framework and ecosystem for professionals looking to expand and redefine their career trajectories. Since OpenUSD has become an interchange for 3D data from many sources, from Maya and Blender to Revit and Rhino, professionals in all corners of the globe are beginning to see utility for USD in their workflows.

Momentum around OpenUSD can be seen first-hand in the Alliance for OpenUSD, an open, non-profit organization that is fostering the standardization, development, evolution, and growth of OpenUSD. With founding members such as Pixar, Adobe, Apple, and Autodesk, the Alliance is continuously seeing membership grow from leading companies around the world.

By mastering OpenUSD, individuals can harness their expertise in computer graphics, physics, math, and game development to explore burgeoning fields where 3D applications are becoming increasingly important. This not only enhances their skill set but also opens doors to innovative industries where they can apply these competencies in diverse and impactful ways.

Digital Twins Are Captivating Industries

One of the most exciting applications of USD is in the creation of digital twins. These complex simulation environments are not just theoretical constructs but are being used in practical, real-world applications that extend far beyond their origins in animation studios.

For instance, manufacturing companies are using digital twins to simulate operational factory environments. This application aids in everything from layout planning to the optimization of robotic paths, all within a virtual space that mimics real-world physics and mechanics. The ability to create such detailed and interactive models showcases the critical role of USD in global industries.

Career Transformation and Opportunities

At SIGGRAPH 2024, a panel of veteran 3D professionals provided insights into how USD is transforming career paths beyond Hollywood.

Lana’s Transition from Animation to Synthetic Data and Avatars

Lana began her career as an effects artist at PDI/DreamWorks and transitioned to Pixar after the studio closed. At Pixar, she worked on developing crowd character pipelines using USD. Her career took a new direction when she joined TikTok, leading a team that worked on avatars and AR effects. Lana later moved to Meta Reality Lab Research as a tech art manager, where her team used USD to generate synthetic data for machine learning research.

Now, as a tech art manager at Meta’s Avatars team, Lana is supporting the development of complex dynamic character assembly workflows using OpenUSD. Her role involves promoting USD as a universal format for 3D asset exchange, facilitating better collaboration and innovation across various digital content creation tools and platforms.

Ken’s Journey Between Physics and Hollywood

Initially a physicist, Ken’s career took a dramatic turn when he began visualizing mission designs for NASA, sparking his interest in computer graphics. He combined his passion for physics and computer science at Caltech and eventually made the transition into creating digital twin productions for Hollywood movies. Then, he invented OpenVDB, an award-winning open-source C++ library that offers efficient storage and manipulation of sparse 3D grid-based volumetric data.

Today, he works at NVIDIA as Sr. Director of Physics Research, where he uses his previous experiences to bridge the gap between theoretical physics and practical, visual applications in computer graphics and machine learning.

Bobby’s Integration of Community and Technology

Bobby’s background in photojournalism and music played a crucial role in his career shift to immersive technologies. By integrating these community-based activities into computer games and later into more immersive technologies like VR and digital storytelling, Bobby demonstrates how diverse experiences can enrich technological innovation.

Today, he is a business development leader at FS Studio, a company visualizing and training robots in virtual environments.

Hunter’s Shift from Mechanical Design to Architectural Visualization

Hunter started in mechanical design but was drawn to the visual aspect of architecture. Inspired by a robotics talk, his career path veered into architectural visualization and he began exploring the use of 3D technology in various industries, far beyond traditional applications in movies and games.

Hunter now works at Amazon Robotics, using OpenUSD to help improve team efficiency, enhance graphic capabilities, generate synthetic data, and more.

You can view the on-demand panel from SIGGRAPH24 here.

Advice for Professionals Looking to Pivot Their Career with OpenUSD

As you consider pivoting your career with OpenUSD, it’s crucial to establish yourself as an expert in this dynamic field. Here are some strategies to help you make a successful transition:

1. Showcase Your Unique Skills: Identify and highlight the unique skills or perspectives you bring to OpenUSD. Whether it’s a deep understanding of 3D modeling, coding, or creative storytelling, make these skills visible.
2. Update Your Professional Profiles: Keep your LinkedIn and other professional profiles up-to-date. Recruiters and hiring managers often review these profiles when searching for potential candidates, so make sure yours stands out with relevant OpenUSD projects and skills.
3. Engage with the Technology: Dive deep into OpenUSD by engaging with the framework and ecosystem. Experiment with its features, try out different projects and understand its application in various industries. This hands-on experience will not only enhance your skills but also show your dedication and ability to adapt.
4. Demonstrate Willingness to Learn: The tech field is ever-evolving, and showing a willingness to continuously learn and adapt is crucial. Attend workshops, take online courses, and stay updated with the latest advancements in OpenUSD and related technologies.
5. Express Your Passion: Let your passion for technology and innovation shine through your work and interactions. Take on new challenges and projects that push the boundaries of what you can do with OpenUSD, making it clear to others why you are passionate about this field.
6. Understand Your Motivations: Knowing what drives you and what you aim to achieve in your career can help you navigate your path more effectively. This self-awareness is key to pursuing roles that align with your long-term goals and conveying your enthusiasm to potential employers or collaborators.

Looking Ahead: The Future of 3D Skills

The future looks bright for professionals skilled in 3D modeling and USD as sectors such as robotics, automation, medical systems, and even education find new innovative uses for digital twins.

The underlying skills that make a professional successful in USD — problem-solving, creativity, and an understanding of complex systems — are universally applicable, making them highly sought after. As industries continue to evolve, the demand for these skills will only grow, paving the way for innovative applications and new job opportunities.

To learn more about OpenUSD:

• Take courses in our Learn OpenUSD series
• Watch our video series about getting started with OpenUSD
• Access a collection of OpenUSD resources, including examples and samples
• Connect with the NVIDIA Omniverse community

Stay up to date on the latest in OpenUSD by subscribing to NVIDIA news, and following NVIDIA Omniverse on Discord and YouTube.

Meet the Author

Ashley Goldstein is lead tech marketing engineer at NVIDIA where she develops content that guides developers through integrating Omniverse and OpenUSD into their workflows. With a colorful background in areas such as academia, fine art, and 3D workflows, Ashley reaches far across the community in M&E, AECO, and education. She enjoys popping into the NVIDIA Omniverse livestreams and community forums, where her dogs make an occasional cameo.

In the fast-paced landscape of industrial digitalization, startups are increasingly leveraging Universal Scene Description (OpenUSD) and generative AI to push the boundaries of what’s possible in 3D content creation, digital twins, and AI-driven innovation.

At GTC, six pioneering startups shared how they are solving complex problems and creating new opportunities across various industries by leveraging OpenUSD and Generative AI. You can watch the session here.

These companies are supported by NVIDIA Inception, the free program for innovative technology startups. This global program of over 18,000 members offers startups engineering guidance, technical training, and introductions to funding opportunities to accelerate their business.

Here’s how each is driving the next era of advanced 3D solutions with OpenUSD redefining the boundaries of digital creation and consumption.

Rendered.ai: Open Standards for Synthetic Data Generation

Rendered.ai, an NVIDIA Inception member, is using physically accurate 3D simulations to generate synthetic computer vision data to help overcome bias and power innovation in AI. Their expertise spans domains like remote sensing, healthcare, and the public sector, enabling their customers to generate unlimited, fully labeled data for training AI.

Despite increasing volumes of data that are being collected to train AI, a persistent issue that machine learning (ML) engineers face is getting data that meets their specific criteria and can be reused across multiple projects. To address these challenges, Rendered.ai has adopted OpenUSD.

OpenUSD allows Rendered.ai to facilitate more simulations for customers with fewer data translations to speed up their iterative process, enabling faster physically accurate synthetic data generation for a wide range of applications, from agriculture to defense to manufacturing.

“One of the key stepping stones to solving problems optimally is open standards,” said Chris Andrews, COO at Rendered.ai. “Using standardized technology like OpenUSD allows us to participate in larger ecosystems that will enable us to do more than we could ever do alone.”

Evolver Interactive: Navigating the Future with OpenUSD Digital Twins

Evolver Interactive, has been utilizing NVIDIA Omniverse, the platform for connecting and developing OpenUSD applications, for over five years. Their focus lies in digital twins and unified management systems as both software and services. The startup’s use of USD for collaboration and enterprise solutions has been pivotal in scaling their projects across various industries.

By leveraging USD and Omniverse, they have created an asset store in the cloud, streamlining their processes and enhancing collaboration. Chris Scott, CEO of Evolver, highlighted the importance of starting small and gradually expanding the use of USD and Omniverse to solve complex problems. Today, Evolver is able to build large-scale digital twins of factories, working with various data sources like Lidar point cloud data for customers like the European Consortium.

“I urge you to consider where USD fits into your workflow and how it can propel your projects forward,” said Chris Scott. “OpenUSD and Omniverse offer versatile solutions, though no single solution covers all bases. It’s about leveraging the available tools effectively.

Ascon Systems: Bridging Digitalization, AI, and Industry Expertise

Today’s manufacturing industry faces numerous challenges, including traditional Programmable Logic Controller (PLC) and Supervisory Control and Data Acquisition (SCADA) systems, a shortage of skilled labor, and limited information technology and operational technology (IT-OT) convergence. To address these, Ascon Systems has developed Ascon Qube, an automation platform that integrates real-time data and USD-based 3D pipelines. By linking the physical and virtual worlds through digital twins, Ascon Qube enables live production control, predictive maintenance, and accelerated ramp-up processes.

This has been instrumental in projects like BMW Group’s Innovation Hub. BMW has used Ascon Qube to link several automated assembly and testing stations, allowing users to see a live view of operations and related information, run simulations and analysis, obtain forecasts, and much more.

“Looking ahead, we aim to shift away from linear programming and execution models, instead focusing on ecosystems and hyper-converged infrastructures, in collaboration with NVIDIA and our partners,” said Jens Mueller, CEO of Ascon Systems.

SyncTwin: Empowering Small Businesses to Build Digital Twins

SyncTwin aims to make the industrial metaverse accessible to small and medium-sized enterprises by leveraging OpenUSD and the Omniverse ecosystem. The catalyst for their journey was their partnership with BMW, which demonstrated the transformative potential of industrial digital twins. However, many small businesses lack dedicated IT departments for digital factory solutions, relying instead on tools like Excel and Powerpoint.

By converting existing layout information and process data from Excel sheets into OpenUSD components, SyncTwin creates interactive models that reveal the full potential of factory operations. This approach enables small businesses to easily engage with digital twin technology, empowering them to streamline operations, make data-driven decisions, and ultimately thrive in the digital age.

“The integration of OpenUSD into SyncTwin exemplifies our commitment to leverage cutting-edge technology to democratize access to the industrial metaverse for businesses of all sizes,” said Michael Wagner, CEO of SyncTwin.

Aireal: Immersive Pre-Construction Experiences with User-Centric Design

Aireal specializes in interactive and immersive technology platforms for pre-construction environments, providing visualization, customization, and exploration experiences before homes and communities are built. This extends to offering customers insights into amenities and lifestyles of future buyers.

By utilizing OpenUSD and Omniverse, they offer highly realistic and interactive experiences, allowing customers to visualize and customize their future homes. To ensure that product offerings align with customer expectations, Aireal connects data from ERP systems to OpenUSD so that showcased options are always available for production.

“OpenUSD provides incredible value to us through performant, realistic experiences, real-time collaboration capabilities, and improved customer engagement,” said Kevin Hart, Founder and CEO at Aireal.

Configuru: Generating Personalized Marketing Experiences

In the digital age of AI, traditional sales methods are becoming less effective as users now demand more personalized experiences and a comprehensive understanding of their challenges. To address this, Configuru, an NVIDIA Inception member, has developed various AI applications for image generation and integrated them with NVIDIA Omniverse. For example, using their most recent AI extension for Omniverse, users can effortlessly create complex scenes with drag-and-drop functionality, saving time and resources compared to conventional rendering methods.

“Integration with OpenUSD and Omniverse Nucleus enhances collaboration and data management across our different development environments,” said Aleksandr Evgrashin, Head of Software Development at Configuru.

These pioneering startups are at the forefront of leveraging OpenUSD and Generative AI to revolutionize industries. Their innovative solutions are transforming the way businesses leverage operational data, enable collaboration, and engage with customers. By harnessing the power of open standards and cutting-edge technologies like Omniverse, these companies are paving the way for a future where data-driven insights and immersive experiences drive success. The possibilities are endless, and these startups are leading the charge towards a more efficient, connected, and intelligent world.

Watch the full session with their presentations here: Building Tools for Digital Worlds: Startups Pioneering OpenUSD and Generative AI

Additional Resources:

• Get started with OpenUSD resources and documentation
• Watch more OpenUSD Day sessions from GTC
• Explore the on-demand OpenUSD training
• Learn more about the NVIDIA Inception Program

About the Author

Stephanie Rubenstein leads developer product marketing for Omniverse at NVIDIA and is responsible for driving adoption of the platform for building OpenUSD-based applications and 3D workflows. She has over a decade of experience helping both early-stage startups and global tech leaders successfully build and launch innovative products across spatial computing, cloud, and AI.

Universal Scene Description, known as OpenUSD, has transformed the way creators and developers compose, edit, and collaborate on complex 3D scenes. It’s become an indispensable tool for pushing the boundaries of creativity and innovation in various industries such as urban planning and manufacturing.

On March 19 at NVIDIA GTC, 3D experts and developers building with USD will participate in OpenUSD Day, a day devoted to networking and sharing knowledge with technical leaders advancing OpenUSD.

What makes USD so powerful are its four key features, which equip developers with superpowers to meet the demands of building virtual world applications:

• The composition engine, which enables non-destructive assembly of data from numerous sources as individual layers, allowing users to collaborate on multiple elements of a scene together in real-time.
• Custom schemas, which make USD’s data model fully extensible and allow developers to add new functionalities.
• Asset resolvers and file format plugins, which make it possible to connect any data source to OpenUSD scenes.
• Hydra, which allows USD to feed different renderers like Pixar Storm and Renderman out of the box.

You can learn about each of these superpowers in our short video series highlighting these 4 superpowers and how they can enhance your workflow.

A Day Devoted to OpenUSD

OpenUSD Day at GTC will feature 8 talks and panels showcasing how technical leaders are adopting and evolving OpenUSD, including how developers can build generative AI-enabled 3D pipelines and tools. Join in-person for networking opportunities with OpenUSD experts to learn how they are transforming their 3D workflows and the opportunity to win a GeForce RTX 4080 signed by NVIDIA Founder and CEO, Jensen Huang.

See a sneak peek at a few of the sessions below:

• Building Tools for Digital Worlds: Startups Pioneering OpenUSD and Generative AI: Explore the transformative impact of Universal Scene Description (OpenUSD) on the emerging landscape of generative AI-enabled 3D pipelines and tools with a panel of five cutting-edge startups.
• Omniverse-based Fab Digital Twin Platform for Semiconductor Industry: Samsung Electronics will present their Fab Digital Twin, the world’s first fab digital twin platform capable of reaching the smart factory Level 5 based on OpenUSD, NVIDIA Omniverse and its ecosystem.
• USD and Digital Twins for Immersive Location-Based Entertainment: See how Moment Factory leverages OpenUSD and NVIDIA technologies to elevate their collaborative pipeline. Learn how teams can foster creative collaboration, seamlessly integrate physical locations through an interactive workflow, and develop innovative solutions for real-time visualization of content or lidar tracking.
• Virtual Commissioning of AI Vision Systems with OpenUSD: Siemens is advancing the commissioning of AI vision systems by pre-training AI models entirely or primarily on synthetic data. With OpenUSD, these vision systems can be built, trained, and tested in a virtual world.
• Digitalizing the World’s Largest Industries with OpenUSD and Generative AI: Every industry has an opportunity to leverage the 3D internet to accelerate their path to automation and digitization. Hear this panel of major enterprise leaders and luminaries provide an update on their industrial metaverse projects with OpenUSD.

The Big Bang of Universal Scene Description

Originally developed by Pixar Animation Studios for feature films, OpenUSD is now being driven forward by an ecosystem of partners including Autodesk, Adobe, Pixar, NVIDIA, and others. Hear from luminaries at each of these companies about the potential for USD, and what their teams are doing to contribute to its evolution in a panel at GTC on Monday, March 18th at 10:00 AM.

USD for Synthetic Data Generation

Synthetic data generation is becoming a vital step for a wide range of AI use cases. With OpenUSD, developers can build photorealistic synthetic data pipelines for applications such as bootstrapping computer vision models, validating autonomous vehicles, and accelerating development of intelligent robots. At GTC, numerous USD-based synthetic data pipelines will be on display. You can see the full list of synthetic data sessions here.

Hands-on Training Labs at GTC

At GTC, developers and technical artists will have the opportunity to participate in training labs to get hands-on experience using OpenUSD for real-world use cases such as:

• Building a 3D Product Configurator with USD and Omniverse
• Training an Autonomous Mobile Race Car with OpenUSD and Isaac Sim
• Building a Native OpenUSD XR Application
• Building OpenUSD Applications for Industrial Digital Twins

The #1 Conference for Developers

These are just a few of the sessions at GTC that are setting the stage for the future of OpenUSD. Taking place from March 17–21, this year’s GTC will be in-person with free online access to sessions.

You can register now with this link for 25% off your conference pass: www.nvidia.com/de-de/gtc/?ncid=GTC-NVIUN2A7

About the Author

Aaron Luk is a Director of Product Management leading Universal Scene Description (OpenUSD) ecosystem engineering, including oversight, integration, and optimization of USD and Hydra paradigms across all components of NVIDIA Omniverse. Previously, Aaron co-developed and deployed USD into production pipelines at Pixar.

As innovators or leaders within our companies, we’re always on the lookout for the next big change — and there’s a fear of potentially missing it. In Architecture, Engineering, Construction, and Operations (AECO), significant technological advancements and evolving workflows have continuously changed the state of play and kept the industry on its toes.

As I take on projects for Fortune 500 companies and innovators in the space like UrsaLeo, a question I regularly ask myself is: am I making the best decisions for my team to ensure we build a scalable and sustainable future? After spending four years utilizing NVIDIA Omniverse and Universal Scene Description (OpenUSD), I can now confidently say yes — here are a few reasons why.

Diving Into RTX Rendering, the Future of Real-Time Design

I started using Omniverse with a simple need: I needed rendering to be done fast for project reviews with clients. I also required a platform that would allow my team to collaborate effectively. We all know adding more people to a project doesn’t necessarily mean it gets done faster, but that changed with Omniverse.

For architects, the ability to quickly and effectively communicate their designs is critical. It’s essential to visualize and share these designs to ensure client understanding, aid collaboration with stakeholders, secure regulatory approvals, facilitate design iterations, and maintain time and cost efficiency.

The Omniverse platform offers a powerful solution to aggregate data, and with its real-time rendering capabilities, it enables architects to produce immersive, photorealistic visualizations instantly. The Omniverse RTX Renderer provides an RTX real-time mode, which allows users to render more geometry and physically-based materials with high fidelity, in real time. Omniverse includes foundation applications that offer features such as annotation, design markup, and remote streaming. These foundation applications serve as templates of sorts from which users can develop their own tools for specific architectural use cases.

Most importantly, the platform empowers architects to convey their ideas to clients and stakeholders more effectively, whether from the cloud or locally, minimizing miscommunication.

The significance of RTX real-time rendering lies in its ability to enhance the speed and realism of these visualizations, streamlining the design process and transitioning to a platform powered by AI technology.

After finding out that real-time rendering was a great first win, my next question was: what else can Omniverse do for me?

Customizing Workflow Needs with APIs and Apps

The next challenge I wanted to tackle was asset management and the creation of custom tools to make our process even faster. This was something I had never done before, and I had no idea what to do. But with just a few of our developers, we managed to develop our own internal asset management tool with login and version control in just a week.

Architects often deal with complex datasets and multiple software tools, requiring efficient data management and workflow integration. OpenUSD and Omniverse address these challenges with powerful APIs that enhance collaboration and efficiency.

Users can integrate their preferred applications into the OpenUSD ecosystem and Omniverse, allowing for customized, optimized workflows and straightforward software integration.

OpenUSD and Omniverse are open and extensible, enabling designers to seamlessly incorporate these tools into their processes. This DIY approach allows architects to adapt technology to their unique needs, fostering innovation and precision in design.

At Evolver Interactive, with Chief Engineer Osamah Askari, we redesign and create extensions and connectors to streamline client workflows. We can combine these pieces together to build enterprise-ready applications for digital twin use cases in manufacturing, construction, and retail.

Omniverse’s codebase and extensions offer a broad spectrum of functionalities, enabling architects to implement any backend process they desire and focus on creative design.

Finally, the last question I had in mind is one that almost all business leaders are asking themselves: How can AI work for me?

Enhancing Design Intelligence and Automation with AI

The truth is, there’s so much AI out there that choosing just one solution would be difficult. Instead of choosing a single solution to enhance workflows, I decided to opt in for a platform from the world leader in AI (NVIDIA) , ensuring that I will always have the best new AI methods at my fingertips.

AI is revolutionizing architecture by enabling architects to automate tasks, enhance design intelligence, and gain deeper insights. The role of AI in generating innovative designs and optimizing energy efficiency represents just the beginning of its impact in the field.

Omniverse is a great starting point for using AI in architecture. NVIDIA Deep Learning Super Sampling (DLSS) uses AI to generate frames, generate additional pixels for intensive ray-traced scenes, and output higher-resolution frames from a lower resolution input. DeepSearch is another AI technology native to the platform that allows you to quickly retrieve 3D assets from extensive libraries using natural language or other images.

Not only is the platform powered by AI, but developers are building AI tools as Omniverse APIs for broader audiences to take advantage of.

The continuous advancement of AI promises to enhance Omniverse consistently, expanding the solutions and methods available to architects. This evolution is set to redefine architectural design and creativity, with Omniverse leading these developments.

Explore the Future of Architectural Design at GTC

Join NVIDIA for OpenUSD Day at GTC for a full day of expert-led sessions and panels to see how OpenUSD is transforming AECO and other global industries. View AECO industry conference sessions to explore solutions helping architects, engineers, and designers keep up with their advanced workflows. Register for GTC in-person or virtually to learn the latest in AI and attend in-depth sessions, workshops, and training on building OpenUSD-based extensions, apps, and services on Omniverse.

Get started with NVIDIA Omniverse by downloading the standard license free, access OpenUSD resources, and learn how Omniverse Enterprise can connect your team. Stay up to date on Instagram, Medium and Twitter. For more, join the Omniverse community on the forums, Discord server, Twitch and YouTube channels.

By: Gerard Andrews, Sr. Product Marketing Manager, NVIDIA

Every year, roughly 400,000 new robots enter global markets. They contribute to a growing industrial robotics market of nearly $45 billion which is expected to surpass $70 billion by 2028. From warehouses and factories to hospitals and delivery services, these robots are transforming the way we work and live.

Robotic systems have become integral in various sectors due to their ability to perform repetitive tasks with precision and efficiency. In industries such as logistics and manufacturing, they can optimize processes and help detect anomalies like product defects. The adaptability and versatility of AI-enabled robots have created a demand that is only set to increase in the coming years.

But training robots for real-world scenarios is a complex workflow comprising extensive data collection, training and simulation. Collecting such data can be time consuming, expensive, and even hazardous. That’s where synthetic data comes into play.

Synthetic data is artificially generated data that is designed to mimic real-world conditions. By employing synthetic data, teams can speed up the training process and ensure that robots are well prepared to face a variety of challenges.

Synthetic data can be generated using NVIDIA Omniverse Replicator, a developer framework that lets you build synthetic data generation (SDG) tools and pipelines. Built on Universal Scene Description (OpenUSD), Omniverse is a development platform for building 3D workflows, tools, and applications. Omniverse also powers NVIDIA Isaac Sim, an extensible robotics simulation application that enables roboticists to design, test, and train AI-based robots.

Omniverse Replicator is highly customizable and extensible driven by APIs that allow you develop your specialized SDG pipelines. As a developer, you can programmatically randomize various attributes such as lights, colors and textures. Additionally, you can build custom annotators and writers such as bounding boxes, segmentation masks for your training data.

Edge Impluse Customizes Omniverse Replicator for Visual Inspection

AI is already showing immense value in industrial visual inspection — helping to identify manufacturing defects. But developing accurate object detection models can be challenging due to limited real-world datasets.

To bridge the data gap, digital twins of factory lines can be created as a testing ground from which AI models can learn.

Jenny Plunkett, a lead developer from Edge Impulse, built her own custom pipeline to generate data for her defect detection use-case. She started with a scene of a conveyor belt transporting soda cans that contain defects. She then built routines in Replicator to programmatically randomize properties of the scene such as lighting, colors, texture, background, foreground, and location of the cans can be with Omniverse Replicator, generating a large and diverse set of of synthetic images for the model to learn from.

Once the synthetic data is ready, it can be fed into AI development platforms like Edge Impulse to annotate the data, build the model, and test it with real objects. Learn more about the workflow here.

Custom SDG pipeline for Training Warehouse Robots to Detect Objects

Autonomous robots are being extensively deployed in warehouses, where they can help with tasks such as efficiently and safely transporting goods or optimizing inventory management. Similar to the defect detection use case, developers can customize Omniverse Replicator for an object detection use case.

This blog shows how you can build a SDG pipeline to generate synthetic data to iteratively improve the performance of a model for autonomous mobile robots (AMRs) that detects pallet jacks in a warehouse.

The workflow starts by generating 5,000 synthetic images with color and camera position randomized. In the second iteration, various textures are added to the pallet jacks and lighting is randomized. To ensure that the model won’t get distracted by other objects in a warehouse, various random objects like traffic cones, wet floor signs, and barrels next to the pallet jacks are added to the synthetic data in the third iteration.

After generating synthetic images with various parameters randomized, the model could accurately detect ‌pallet jacks. Data that previously would have taken months to collect is generated in days. View the full workflow here.

Discrete Event Simulation with FlexSim

One way that robots such as automated guided vehicles (AGVs), wire-guided industrial robotics system planners, and AMRs are tested is through discrete event simulations.

Discrete event simulations model the operation of a system as a sequence of events in time. This allows engineers, designers, and simulation experts to optimize their processes by experimenting with complex models and testing them step-by-step.

To run discrete event simulations, many use FlexSim, a powerful modeling and simulation software tool that allows users to construct a 3D digital twin of a real-life system. FlexSim recently developed an Omniverse Connector, allowing teams to export 3D models and assets to OpenUSD.

FlexSim’s Omniverse Connector helps teams bridge the gap between their simulation data and real-time 3D visualization, and use that to analyze, visualize, and improve real-world processes.

Learn more about how FlexSim developed an Omniverse Connector and the benefits that USD is unlocking for their customers in this blog.

Get Started Training AI-Enabled Robots with Synthetic Data

With the Omniverse platform, teams can quickly get started designing, testing, and training AI-based robots.

NVIDIA Isaac Sim, an extensible robotics simulator powered by Omniverse, allows developers to create realistic robotics simulations to train and optimize AI robots for a wide variety of tasks. Isaac Sim 2023.1 was recently released with major updates to performant perception and high-fidelity simulation.

The core functionalities of Isaac Sim come from Omniverse Replicator, an extension of Omniverse that allows developers to build custom SDG tools and pipelines. Omniverse Replicator 1.10 was also recently released with new support for low-code, YAML-based workflows, scaled rendering, and greater flexibility.

Download NVIDIA Omniverse for free and get started. Developers can access Omniverse resources and learn more about OpenUSD.

Stay up to date with NVIDIA Omniverse by subscribing to the newsletter and following Omniverse on Instagram, LinkedIn, Medium, Threads, and Twitter. For more, check out our forums, Discord server, Twitch and YouTube channels.

About the Author

Gerard Andrews is a Sr. Product Marketing Manager focused on revolutionizing the way intelligent robots are developed, trained, tested, and deployed through driving the adoption of the NVIDIA Isaac Robotics platform. Prior to joining NVIDIA, Gerard was at Cadence where he was Product Marketing Director, responsible for product planning, marketing, and business development for licensable processor IP. He holds an MS in electrical engineering from Georgia Institute of Technology and a BS in electrical engineering from Southern Methodist University.

By: Guy Martin, Director of Open Source & Standards, NVIDIA

In the dynamic world of 3D content creation, the ability to seamlessly collaborate, share, and modify assets is unlocking new possibilities. Traditional pipelines often presented hurdles, forcing artists to wait for iterations and stifling creativity. Then came Universal Scene Description, also known as OpenUSD, a game-changing technology that’s revolutionizing the industry. At SIGGRAPH 2023, our team caught up with some industry leaders and 3D developers to hear their thoughts on OpenUSD.

OpenUSD, or USD, is an open-source, highly efficient, and scalable interchange format for 3D graphics data originally developed by Pixar Animation Studios. Its primary goal is to make it easier for artists and developers to move 3D content between various tools seamlessly. The impact of this innovation has not gone unnoticed by industry leaders, who are excited about the possibilities it offers.

A Paradigm Shift

OpenUSD represents a complete paradigm shift in the way we approach 3D workflows. This shift can be traced back to the original intent for the technology, as explained by Steve May, CTO at Pixar.

“We created USD at Pixar years ago because we were trying to solve a problem. The problem was, ‘How do we make it easier for artists to move 3D content, 3D data, between tools they use,’” said May.

Sir Wade Neistadt, a renowned 3D artist and YouTuber with over 200,000 subscribers, highlighted the bottlenecks of traditional 3D pipelines and how USD gives artists a newfound freedom to work without constraints.

“I used to think USD was just a file format, but I’ve come to understand it’s so much more than that. It’s a whole different way of working,” said Neistadt. “The moment I started moving things around in one application and each application is just updating in real time with me, I’m not sitting there waiting for some kind of propagating change. That was the moment that my brain broke and I realized ‘Oh my God, this is insane.’”

Ashley Goldstein, a Senior Engineer at NVIDIA for Omniverse Exchange, predicts widespread adoption of USD as it becomes an integral part of every creative process.

“OpenUSD is boundless,” said Goldstein. “I see it just taking over and everyone adopting it and everyone using it, and it no longer being about, ‘What can it do?’ Because it’s just going to be able to do everything.”

Uniting to Build a Universal Standard

All of the industry-leading companies mentioned in this blog have something in common: they are members of the Alliance for OpenUSD (AOUSD). A new organization that launched in August, AOUSD is dedicated to shaping the future of OpenUSD.

Members of this alliance collaborate with each other to promote the standardization, development, and evolution of 3D content through OpenUSD. By promoting greater interoperability of 3D tools and data, the alliance will enable developers and content creators to more easily describe, compose and simulate large-scale 3D projects and build a larger range of 3D-enabled products and services.

Check out the AOUSD website to learn more about the organization and explore the forum for more resources on OpenUSD.

USD in Geospatial and AEC

Patrick Cozzi, CEO of Cesium, is excited about the potential transformation that OpenUSD brings for 3D geospatial and architecture, engineering, and construction (AEC).

“We’re excited to see OpenUSD transform 3D geospatial and AEC,” said Cozzie. “We use USD to reference 3D tiles for the surrounding 3D geospatial context, such as the city and trees. Then it also references USD for an AEC model such as a proposed building.”

Autonomous Driving Simulation

Marc Petit, VP, Unreal Engine Ecosystem at Epic Games, shared how USD has gained traction in autonomous driving simulations.

“We saw a lot of adoption of USD in the autonomous driving simulation space where the challenge of creating large-scale cityscapes was really, really interesting to use USD and connect with a number of other tools,” said Petit.

Connecting Creativity

Jen Borucki, Omniverse Connector Evangelist at NVIDIA, emphasizes the importance of USD for connecting diverse creative teams.

“You’ve got the 3D modelers. Maybe you’ve got a surface artist,” said Borucki. “You’ve got your modeler coming in. You’ve got your substance designer going in painting and then you got your scene designer putting everything together.”

Learn more about OpenUSD

Learn from more industry experts on how OpenUSD is enabling custom 3D pipelines, easing 3D tool development and delivering interoperability between 3D applications in on-demand sessions from SIGGRAPH 2023.

For the latest USD resources and tutorials, visit our OpenUSD resources page. Try the free RunUSD Validation Service to validate the compatibility of your OpenUSD assets and applications against a range of OpenUSD versions and configurations.

If you’re a developer, start building OpenUSD-based apps and tools on the Omniverse platform.

To learn more about how OpenUSD can improve 3D workflows, check out a new video series about the framework.

Get started with NVIDIA Omniverse, the computing platform for developing OpenUSD-based 3D workflows and applications, by downloading the standard license for free or learn how Omniverse Enterprise can connect your team.

Developers can check out these Omniverse resources to begin building on the platform.

Stay up to date on the platform by subscribing to the newsletter and following NVIDIA Omniverse on Instagram, LinkedIn, Medium, Threads and Twitter.

For more, check out our forums, Discord server, Twitch and YouTube channels.

About the Author

Guy Martin is Director of Open Source & Standards at NVIDIA, where he works with the Omniverse team to help utilize and contribute to important open 3D projects & standards like Universal Scene Description (USD). He also serves as NVIDIA’s representative to the Alliance for OpenUSD (AOUSD) Steering Committee.

Guy brings a unique blend of 30 years’ experience as both software engineer and open source strategist to NVIDIA. He previously served as Executive Director for OASIS Open, an internationally-recognized open source & standards consortium and has participated in building numerous open source and standards bodies for technologies ranging from visual effects to Internet of Things.

By: Mario Viviani, Manager, Developer Relations, NVIDIA Omniverse

As technology continues to evolve, so does the way we create 3D models.

The integration of generative AI into 3D design is transforming the way 3D scenes are created, animated, and used across global industries. AI’s ability to analyze data quickly and generate sophisticated 3D models with realistic textures and materials is opening up a new world of possibilities. Various industries such as entertainment, architecture, automotive, and healthcare are applying generative AI to 3D workflows with unique goals in mind.

For example, photorealistic virtual characters with the ability to engage in real-time conversation can be created for video games or movies. 3D models of buildings can be generated from photos for architectural design. And images of shoes and other retail items can be turned into 3D models to create virtual shopping experiences.

AI has the potential to make 3D modeling more accurate and efficient than ever, unlocking incredible opportunities for custom 3D pipelines.

The framework and ecosystem that these innovative solutions are being built upon is Universal Scene Description (USD), also known as OpenUSD. An open and powerful interchange for 3D content creation tools, OpenUSD provides the flexibility to develop AI tools that can interoperate with various platforms for industrial metaverse applications. At SIGGRAPH 2023, NVIDIA announced ChatUSD, a generative AI copilot for OpenUSD development with NVIDIA Omniverse.

Transforming 2D images to 3D models with AI

Groundbreaking generative AI technologies like Alpha3D and BOOM Interactive’s Bubbles app are making it possible to quickly and accurately convert 2D images into high-quality 3D models. These technologies enable creators to save time and money while creating more accurate models.

Alpha3D, an NVIDIA Inception member, allows you to quickly and easily turn 2D images into high-quality 3D digital assets. It is designed to be used by anyone, even those without any 3D modeling expertise.

With the Alpha3D AI 3D model generator, users can either upload a single shot 2D image of a product or use the text-to-image feature to generate a new product image. The AI 3D model generator will then automatically transform the image into a 3D asset. The 3D model can then be downloaded, edited, and used immediately. Currently, Alpha3D has opened up one category for the public: shoes, and they are constantly adding more product categories.

Alpha3D developed an extension for NVIDIA Omniverse to allow creators to import their AI-generated models directly into Omniverse.

“Omniverse isn’t a tool; it’s a canvas where art meets technology,” said Abdallah Sham, Head of AI at Alpha3D. “Real-time collaboration and seamless integration with an array of platforms to build immersive experiences in one place with OpenUSD has opened up new horizons in creativity and efficiency for us.”

Bubbles™ is a 3D space design tool that allows you to use AI to bring your space into the digital world in seconds. Users can create scenes from 2D floor plan images, scan their space using LiDAR or AR, or build from scratch. The real-time 3D editor gives you the ability to move walls, add windows and doors, decorate and make decisions.

Bubbles™ is cross platform and includes communication features for co-designing. The interface is user friendly and designed for non-3D professionals. It allows anyone to capture, ideate, connect and export which ultimately speeds up time for 3D professionals, designers and clients.

Recently, NVIDIA Inception member BOOM Interactive developed a Bubbles Omniverse Connector that allows users to generate a USD project from Bubbles and open it in Omniverse. Using 3D scenes from Bubbles in Omniverse makes it easy to collaborate with others on your design and deploy it in virtual worlds.

“USD is the most flexible and best 3D format,” said Timber Barker, Founder and CEO at BOOM Interactive. “In our platform, any skill level can create, change, co-design, share and now export to USD, ultimately speeding up time for 3D professionals.”

Conversational AI meets generative AI

Convai is leveraging conversational and generative AI together to bring intelligent avatars to life in games and virtual worlds.

Their platform provides users with tools to create characters with a backstory, a voice, and certain areas of expertise. It also supports integrations with popular game engines, allowing users to easily connect their characters to existing NPC assets in a virtual world. With Convai, users can create characters that are able to have meaningful conversations with their players, providing a more engaging and immersive experience.

With Convai’s Omniverse extension, users can easily set up and converse with AI-based characters in virtual worlds via microphone input. The setup of characters requires no code experience.

“At Convai, we want to create NPCs that resemble humans. With Omniverse, these AI characters can be brought to life in virtual worlds and used for learning, training, socializing and other metaverse applications,” said Purnendu Mukherjee, CEO at Convai.

Face and motion capture animation with AI

AI is revolutionizing facial and motion capture animation for 3D characters. With the help of AI and machine learning, developers are building tools that generate realistic facial expressions and movements with unprecedented accuracy and detail.

For example, NVIDIA Inception member Move.ai’s AI-driven motion capture animation technology combines a library of real-world motions with deep learning algorithms to produce realistic motion and body language for 3D characters.

Their new Move One iOS app, currently in early access, transforms your iPhone into a powerful motion capture tool. The app uses advanced AI to convert videos into high-quality 3D animations that make 3D content creation much cheaper and more accessible for creators.

“With the newly release USD export — it is now possible to connect to a plethora of 3D apps via Omniverse, enabling our users to get into their 3D workflow immediately, which is really exciting,” said Tino Millar, CEO and Co-founder at Move.ai.

3D animations can also be generated from audio. NVIDIA Omniverse Audio2Face uses advanced deep learning techniques to generate realistic facial animations from audio recordings, allowing creators to match facial characteristics to any voice-over track.

AI automatically manipulates the face, eyes, mouth, tongue, and head motion to match your selected emotional range and customized level of intensity. After audio is uploaded to the app, it is fed into a pre-trained Deep Neural Network whose output drives the 3D vertices of your character mesh to create the facial animation in real-time.

Numerous new Audio2Face tutorials are now available on setting up custom characters, introducing new AI models, and more. You can also experiment with using Audio2Face with Unreal Engine 5.2 Metahuman, as shown by JSFILMZ.

AI texture generator wows audience at SIGGRAPH

At SIGGRAPH 2023, NVIDIA researchers won the Best in Show award with a demo of generative AI techniques to create and edit 3D materials. The techniques allow artists to use text or image prompts to generate custom textured materials such as fabric, wood, and stone.

Using Omniverse USD Composer, the researchers showed how brief text prompts can generate tileable materials that can be seamlessly replicated over surfaces at scale. The demo also showed how reference images could be imported and used to automatically generate custom textures for 3D scenes.

Generative AI for 3D scene backgrounds and world-building

Cutting-edge generative AI tools developed by Shutterstock using NVIDIA Picasso are reshaping 3D scene backgrounds and environment maps. Through simple prompts, the AI swiftly crafts custom 360-degree, 8K-resolution HDRi environment maps, aiding artists in setting scene backgrounds and lighting.

Autodesk, a provider of 3D digital content tools, is integrating generative AI content-creation services — developed using foundation models in Picasso — with its popular 3D software Autodesk Maya. Supercharging Autodesk customer workflows with AI allows artists to focus on creating — and to ultimately produce content faster.

Get started with generative AI for 3D modeling

The race is on to build exciting and powerful AI tools for custom 3D pipelines. Developers of various experience levels are using NVIDIA Omniverse, an open and flexible development platform, to build 3D workflows, tools, applications, and services.

To get started, visit the Omniverse Developer Resources page.

Join Alpha3D and BOOM Interactive on an NVIDIA Omniverse Community Livestream Wednesday, September 6, 2023.

Get started with NVIDIA Omniverse by downloading the standard license free, or learn how Omniverse Enterprise can connect your team. If you’re a developer, get started with Omniverse resources. Stay up-to-date on the platform by subscribing to the newsletter, and following NVIDIA Omniverse on Instagram, Medium, and Twitter. For resources, check out our forums, Discord server, Twitch, and YouTube channels.

About the Author

Mario Viviani is Manager of Developer Relations for Omniverse at NVIDIA, based in London, UK. His team focuses on helping developers and partners get familiar and onboard on NVIDIA Omniverse. Passionate technologist and hands on-developer, he’s ex-Amazon, where he led the global Apps and Games Tech Evangelism team; previously was co-founder of startups and led his own consulting company in mobile apps development. He is always projected into the future and into what is the next “big thing”!

By: Alex Qi, Mario Viviani, and Paul Cutsinger

Demand for 3D worlds and virtual environments is skyrocketing across industries, driving the need for streamlined workflows and creative solutions. However, the tedious and time-intensive nature of 3D design can pose challenges for artists and designers.

Generative AI tools for virtual worlds have emerged as game-changers, liberating creators from repetitive tasks and enabling them to focus on unleashing their creative potential.

Like many curious developers, members of the NVIDIA Omniverse team experimented with ChatGPT and the new GPT-4 large multimodal model to demonstrate the ease with which custom tools can be developed to rapidly generate 3D objects for virtual worlds. Combining GPT-4 with Omniverse DeepSearch, an intelligent AI librarian capable of searching vast databases of untagged 3D assets, they created an extension that allows developers and artists to retrieve 3D objects effortlessly using simple, text-based prompts and seamlessly incorporate them into their 3D scenes.

The extension, which is called AI Room Generator, showcases the immense potential of generative AI in populating realistic environments. With a few text prompts, high-fidelity objects are automatically generated and placed, saving hours of painstaking work. These objects, based on Universal Scene Description (OpenUSD) SimReady assets, are physically accurate and behave realistically in any simulation.

Recently, NVIDIA hosted an AMA event for developers to ask questions about different ways that ChatGPT, generative AI, and Omniverse can be leveraged to build tools for the metaverse.

Questions were asked about different types of generative AI models that can be used, specific use cases for ChatGPT, and tips for adding functionalities to generative AI tools in Omniverse.

Connecting Generative AI Models

Q: Can we mix LLMs with other models like Stable Diffusion (SD) for Gen AI?

LLMs can be used in various ways as a preprocess pipeline to connect to SD. For example, you might be able to ask LLM to stylize a prompt based on plain English, then it can be passed to SD. Technically, it’s all about connecting APIs together.

View in Forum

Q: If I don’t want to use ChatGPT, do I have any other options?

Omniverse is an open platform where other LLMs can be integrated. One such alternative is NVIDIA NeMo, which allows you to train on your own data and get responses that speak in your brand’s language and reflect your specific domain.

View in Forum

ChatGPT Metaverse Use Cases and Possibilities

Q: Would it be possible to create entire environments based on GPT prompts within Omniverse Replicator? The idea is to fine-tune computer vision models on synthetic datasets (models + environment).

You could certainly do that. You can even have GPT create USD directly rather than json files. However, creating meshes and textures from scratch requires long GPT responses that can use up your GPT tokens quickly.

A different approach would be to compose your scene, keeping in mind those things you would like to randomize. Then, ask GPT to give you random values for only those key items. That way, you’d be able to work with a much larger scene. That said, this is what Omniverse Replicator already does and I might use Replicator for this specific task over GPT.

View in Forum

Q: I would like to create an AI agent operating from visual feedback using python libraries such as YOLO or SegmentAnything. Is it possible to feed camera coordinates to a GPT prompt and then update the position based on the reply from GPT, similar to the 3D object placing pipeline?

You can incorporate pretty much any python package you want into your extension, the Omniverse platform is extremely flexible. You could definitely tap into one of the scene’s update callbacks, get camera coordinates, and feed those into GPT. Just be ready for a really, really slow frame rate because it takes a few seconds to get that reply back from GPT.

View in Forum

Q: Can we use NVIDIA Omniverse to build human-like Virtual 3D Assistants that talk using ChatGPT API as custom agent?

You can definitely build your own solution in NVIDIA Omniverse to animate virtual assistants, using technologies like Audio2Face in combination with speech synthesis tech like NVIDIA Riva and combine that with GPT-4.

There are also 3rd party solutions in Omniverse that allow you to easily create virtual assistants using generative AI like Convai. They have recently released an Extension for Omniverse that allows you to easily create your characters on their website and then interact with them in Omniverse.

View in Forum

Tips for Developing Generative Tools in Omniverse

Q: When creating 3D environments using prompts, are there strict constraints for properties such as the scale of an obj file? Or can we ask the prompt to use a specific obj file? How much control do we have on the domain?

There are two places in the process where you can add constraints. The first is in your prompt engineering and the second is in your post-processing of the prompt results.

When you craft your prompt, you could give GPT a list of acceptable responses and then ask it to only use those responses. You can see an example of this in prompts.py starting on line 32 where it says:

“For each object you need to store:

• object_name: name of the object
• X: coordinate of the object on X axis
• Y: coordinate of the object on Y axis
• Z: coordinate of the object on Z axis
• Length: dimension in cm of the object on X axis
• Width: dimension in cm of the object on Y axis
• Height: dimension in cm of the object on Z axis
• Material: a reasonable material of the object using an exact name from the following list: Plywood, Leather_Brown, Leather_Pumpkin, Leather_Black, Aluminum_Cast, Birch, Beadboard, Cardboard, Cloth_Black, Cloth_Gray, Concrete_Polished, Glazed_Glass, CorrugatedMetal, Cork, Linen_Beige, Linen_Blue, Linen_White, Mahogany, MDF, Oak, Plastic_ABS, Steel_Carbon, Steel_Stainless, Veneer_OU_Walnut, Veneer_UX_Walnut_Cherry, Veneer_Z5_Maple.”

You can use this type of pattern to apply other constraints to any of those properties.

When you get the results back from GPT you can enforce any other constraints you might want. The limit here is only what you can code in python.

View in Forum

Q: I was trying out ChatGPT to generate scene assembly based on the NVIDIA sample, but it would often get the height wrong. Are there best practices or a better way to generate a scene description so the geometry of objects put into a scene are taken into account?

There are a few ways to achieve this that include prompt engineering, result control and pre-object disposition checks. In the AI Room Generator example, we just take the XYZ position of objects and don’t provide specific instructions on how the Y axis should behave. However, you can be more strict in your prompt engineering about it. You could try adding specific behavior requirements like “If you are placing a tabletop object or an object that usually is on a wall or on top of furniture, make sure the Y axis value is not 0”.

You can also provide a few examples of desired results. For example, provide an example where a TV is placed on top of a TV stand. Once you receive the final results, you can also run your check to make sure that the desired objects are correctly placed with the right offsets ahead of placing them in the scene.

View in Forum

Q: The objects in the AI Room Generator Extension look perfectly arranged at right angles. How was this done?

GPT-4 has a good spatial awareness and can place objects at right angle positions and even rotations. A great way to have more control on this is to include the rotation angle on X, Y, Z, axis in your request to the LLM. For example “include the X,Y,Z rotation for each object” and store that information in the desired result such as a JSON variable.

View in Forum

So, where do I start?

A good place to begin developing AI-powered extensions using generative AI models is with the AI Room Generator Extension sample on GitHub. You can read a detailed walkthrough of the process for developing the extension in the blog, How ChatGPT and GPT-4 Can Be Used for 3D Content Generation.

Join NVIDIA at SIGGRAPH to learn about the latest innovations in graphics, AI and OpenUSD. Save the date for the NVIDIA Keynote with CEO Jensen Huang to see the latest award-winning research and announcements in OpenUSD and AI-powered solutions for 3D content creation.

See the latest Text2Material generative AI from NVIDIA at SIGGRAPH’s Real-Time Live and explore sessions in the NVIDIA Generative AI Theater.

Get started with NVIDIA Omniverse by downloading the standard license free, or learn how Omniverse Enterprise can connect your team. If you are a developer, get started with Omniverse resources. Stay up to date on the platform by subscribing to the newsletter, and following NVIDIA Omniverse on Instagram, Medium, and Twitter. For resources, check out our forums, Discord server, Twitch, and YouTube channels.

About the Authors

Alex Qi is a product manager for the NVIDIA AI Software Group. Her focus is on AI software and applications for a conversation AI framework (Riva) and AI/ML for multimedia streaming (Maxine). Prior to joining NVIDIA, she had extensive experience in leading challenging technical projects in technology and engineering organizations across various roles, such as data Scientist, Computational Modeling and Design Engineering. Alex holds a dual-degree masters’ from the Massachusetts Institute of Technology: an MBA from the MIT Sloan School of Management, and a master of science from the School of Engineering — Mechanical Engineering, where she focused her studies on robotics and artificial intelligence.

Mario Viviani is Manager of Developer Relations for Omniverse at NVIDIA, based in London, UK. His team focuses on helping developers and partners get familiar and onboard on NVIDIA Omniverse. Passionate technologist and hands on-developer, he’s ex-Amazon, where he led the global Apps and Games Tech Evangelism team; previously was co-founder of startups and led his own consulting company in mobile apps development. He is always projected into the future and into what is the next “big thing”!

Paul Cutsinger is director of Omniverse Exchange at NVIDIA, where he’s focused on tooling for real-time, true-to-reality simulation. With a career spanning Amazon, Disney, and Microsoft, Paul’s work centers on enabling creators to take their ideas into production.

By: Mario Viviani, Manager, Developer Relations, NVIDIA Omniverse

Demand for 3D worlds and virtual environments is growing exponentially across the world’s industries. 3D workflows are core to industrial digitalization, developing real-time simulations to test and validate autonomous vehicles and robots, operating digital twins to optimize industrial manufacturing, and paving new paths for scientific discovery.

Today, 3D design and world building is still highly manual. While 2D artists and designers have been graced with assistant tools, 3D workflows remain filled with repetitive, tedious tasks.

Creating or finding objects for a scene is a time-intensive process that requires specialized 3D skills honed over time like modeling and texturing. Placing objects correctly and art directing a 3D environment to perfection requires hours of fine tuning.

To reduce manual, repetitive tasks and help creators and designers focus on the creative, enjoyable aspects of their work, NVIDIA has launched numerous AI projects like generative AI tools for virtual worlds.

The iPhone moment of AI

With ChatGPT, we are now experiencing the iPhone moment of AI, where individuals of all technical levels can interact with an advanced computing platform using everyday language. Large language models (LLMs) had been growing increasingly sophisticated, and when a user-friendly interface like ChatGPT made them accessible to everyone, it became the fastest-growing consumer application in history, surpassing 100 million users just two months after launching. Now, every industry is planning to harness the power of AI for a wide range of applications like drug discovery, autonomous machines, and avatar virtual assistants.

Recently, we experimented with OpenAI’s viral ChatGPT and new GPT-4 large multimodal model to show how easy it is to develop custom tools that can rapidly generate 3D objects for virtual worlds in NVIDIA Omniverse. Compared to ChatGPT, GPT-4 marks a “pretty substantial improvement across many dimensions,” said OpenAI co-founder Ilya Sutskever in a fireside chat with NVIDIA founder and CEO Jensen Huang at GTC 2023.

By combining GPT-4 with Omniverse DeepSearch, a smart AI librarian that’s able to search through massive databases of untagged 3D assets, we were able to quickly develop a custom extension that retrieves 3D objects with simple, text-based prompts and automatically add them to a 3D scene.

AI Room Generator Extension

This fun experiment in NVIDIA Omniverse, a development platform for 3D applications, shows developers and technical artists how easy it is to quickly develop custom tools that leverage generative AI to populate realistic environments. End users can simply enter text-based prompts to automatically generate and place high-fidelity objects, saving hours of time that would typically be required to create a complex scene.

Objects generated from the extension are based on Universal Scene Description (USD) SimReady assets. SimReady assets are physically-accurate 3D objects that can be used in any simulation and behave as they would in the real world.

Getting information about the 3D Scene

Everything starts with the USD scene in Omniverse. Users can easily circle an area using the Pencil tool in Omniverse, type in the kind of room/environment they want to generate — for example, a warehouse, or a reception room — and with one click that area is created.

Creating the Prompt for ChatGPT

The ChatGPT prompt is composed of four pieces: system input, user input example, assistant output example, and user prompt.

Let’s start with the aspects of the prompt that tailor to the user’s scenario. This includes text that the user inputs plus data from the scene.

For example, if the user wants to create a reception room, they specify something like “This is the room where we meet our customers. Make sure there is a set of comfortable armchairs, a sofa and a coffee table.” Or, if they want to add a certain number of items they could add “make sure to include a minimum of 10 items.”

This text is combined with scene information like the size and name of the area where we will place items as the User Prompt.

“Reception room, 7x10m, origin at (0.0,0.0,0.0). This is the room where we meet 
our customers. Make sure there is a set of comfortable armchairs, a sofa and a 
coffee table”

This notion of combining the user’s text with details from the scene is powerful. It’s much simpler to select an object in the scene and programatically access its details than requiring the user to write a prompt to describe all these details. I suspect we’ll see a lot of Omniverse extensions that make use of this Text + Scene to Scene pattern.

Beyond the user prompt, we also need to prime ChatGPT with a system prompt and a shot or two for training.

In order to create predictable, deterministic results, the AI is instructed by the system prompt and examples to specifically return a JSON with all the information formatted in a well-defined way, so it can then be used in Omniverse.

Here are the four pieces of the prompt that we will send.

System Prompt

This sets the constraints and instructions for the AI

You are an area generator expert. Given an area of a certain size, you can generate a list of items that are appropriate to that area, in the right place.


You operate in a 3D Space. You work in a X,Y,Z coordinate system. X denotes width, Y denotes height, Z denotes depth. 0.0,0.0,0.0 is the default space origin.


You receive from the user the name of the area, the size of the area on X and Z axis in centimeters, the origin point of the area (which is at the center of the area).


You answer by only generating JSON files that contain the following information:


- area_name: name of the area
- X: coordinate of the area on X axis
- Y: coordinate of the area on Y axis
- Z: coordinate of the area on Z axis
- area_size_X: dimension in cm of the area on X axis
- area_size_Z: dimension in cm of the area on Z axis
- area_objects_list: list of all the objects in the area

For each object you need to store:
- object_name: name of the object
- X: coordinate of the object on X axis
- Y: coordinate of the object on Y axis
- Z: coordinate of the object on Z axis


Each object name should include an appropriate adjective.


Keep in mind, objects should be placed in the area to create the most meaningful layout possible, and they shouldn't overlap.
All objects must be within the bounds of the area size; Never place objects further than 1/2 the length or 1/2 the depth of the area from the origin.
Also keep in mind that the objects should be disposed all over the area in respect to the origin point of the area, and you can use negative values as well to display items correctly, since the origin of the area is always at the center of the area.


Remember, you only generate JSON code, nothing else. It's very important.

User Input Example

This is an example of what a user might submit. Notice that it’s a combination of data from the scene and text prompt.

"Reception room, 7x10m, origin at (0.0,0.0,0.0). This is the room where we meet 
our customers. Make sure there is a set of comfortable armchairs, a sofa and a 
coffee table"

Assistant Output Example

This provides a template that the AI must use. Notice how we’re describing the exact JSON we expect.

{
    "area_name": "Reception",
    "X": 0.0,
    "Y": 0.0,
    "Z": 0.0,
    "area_size_X": 700,
    "area_size_Z": 1000,
    "area_objects_list": [
        {
            "object_name": "White_Round_Coffee_Table",
            "X": -120,
            "Y": 0.0,
            "Z": 130
        },
        {
            "object_name": "Leather_Sofa",
            "X": 250,
            "Y": 0.0,
            "Z": -90
        },
        {
            "object_name": "Comfortable_Armchair_1",
            "X": -150,
            "Y": 0.0,
            "Z": 50
        },
        {
            "object_name": "Comfortable_Armchair_2",
            "X": -150,
            "Y": 0.0,
            "Z": -50
        }  ]
}

Connecting to OpenAI

This prompt is sent to the AI from the Extension via Python code. This is quite easy in Omniverse Kit and can be done with just a couple commands using the latest OpenAI Python Library. Notice that we are passing to the OpenAI API the system input, the sample user input and the sample expected assistant output we have just outlined. The variable “response” will contain the expected response from ChatGPT.

# Create a completion using the chatGPT model   
 response = openai.ChatCompletion.create(
         model="gpt-3.5-turbo",
         # if you have access, you can swap to model="gpt-4",
                    messages=[
                            {"role": "system", "content": system_input},
                            {"role": "user", "content": user_input},
                            {"role": "assistant", "content": assistant_input},
                            {"role": "user", "content": my_prompt},
                             ]
                    )
    # parse response and extract text
    text = response["choices"][0]["message"]['content']

Passing the result from ChatGPT to Omniverse DeepSearch API and generating the scene

The items from the ChatGPT JSON response are then parsed by the extension and passed to the Omnivere DeepSearch API. DeepSearch allows users to search 3D models stored within an Omniverse Nucleus server using natural language queries.

This means that even if we don’t know the exact file name of a model of a sofa, for example, we can retrieve it just by searching for “Comfortable Sofa” which is exactly what we got from ChatGPT.

DeepSearch understands natural language and by asking it for a “Comfortable Sofa” we get a list of items that our helpful AI librarian has decided are best suited from the selection of assets we have in our current asset library. It is surprisingly good at this and so we often can use the first item it returns, but of course we build in choice in case the user wants to select something from the list.

From there, we simply add the object to the stage.

Adding items from DeepSearch into Omniverse stage

Now that DeepSearch has returned results, we just need to place the objects in Omniverse. In our extension, we created a function called place_deepsearch_results() that processes all the items and places them in the scene.

def place_deepsearch_results(gpt_results, query_result, root_prim_path):
        index = 0
        for item in query_result:
            # Define Prim          
            stage = omni.usd.get_context().get_stage()
            
            prim_parent_path = root_prim_path + item[‘object_name’].replace(" ", "_")
            parent_xForm = UsdGeom.Xform.Define(stage, prim_parent_path)
            
            prim_path = prim_parent_path + "/" + item[‘object_name’].replace(" ", "_")
            next_prim = stage.DefinePrim(prim_path, 'Xform')


            # Add reference to USD Asset
            references: Usd.references = next_prim.GetReferences()
            
            references.AddReference(
                assetPath="your_server://your_asset_folder" + item[‘asset_path’])


            # Add reference for future search refinement 
            config = next_prim.CreateAttribute("DeepSearch:Query", Sdf.ValueTypeNames.String)
            config.Set(item[‘object_name’])
            
           # translate prim
            next_object = gpt_results[index]
            index = index + 1
            x = next_object['X']
            y = next_object['Y']
            z = next_object['Z']

This method to place items, iterates over the query_result items that we got from GPT, creating and defining new primitives using the USD API, setting their transformations and attributes based on data in gpt_results. We also save the DeepSearch query in an attribute in the USD, so it can be used afterwards in case we want to run DeepSearch again. Note that the assetPath “your_server//your_asset_folder” is a placeholder and should be substituted with the real path of the folder where DeepSearch is performed.

And voila! We have our AI-generated scene in Omniverse!

Swapping items using DeepSearch

However, we might not like all the items that are retrieved the first time. So, we built a small companion extension to allow users to browse for similar objects and swap them in with just a click. With Omniverse, it is very easy to build in a modular way so you can easily extend your workflows with additional extensions.

This companion extension is quite simple. It takes as argument an object generated via DeepSearch, and offers two buttons to get the next or previous object from the related DeepSearch query. For example, if the USD file contained the Attribute “DeepSearch:Query = Modern Sofa”, it would run this search again via DeepSearch and get the next best result. You could of course extend this to be a visual UI with pictures of all the search results similar to the window we use for general DeepSearch queries. To keep this example simple, we just opted for two simple buttons.

See the code below that shows the functions to increment the index, and the function replace_reference(self) that is actually operating the swap of the object based on the index.

def increment_prim_index():
            if self._query_results is None:
                return 


            self._index = self._index + 1


            if self._index >= len(self._query_results.paths):
                self._index = 0


            self.replace_reference()


def replace_reference(self):
        references: Usd.references = self._selected_prim.GetReferences()
        references.ClearReferences()
        references.AddReference(
                assetPath="your_server://your_asset_folder" + self._query_results.paths[self._index].uri)

Note that, as above, the path “your_server://your_asset_folder” is just a placeholder, and you should replace it with the Nucleus folder where your DeepSearch query gets performed.

This shows how by combining the power of LLMs and Omniverse APIs it is possible to create tools that power creativity and speed up processes.

From ChatGPT to GPT-4

One of the main advancements in OpenAI’s new GPT-4 is its increased spatial awareness in large language models.

We initially used ChatGPT API, which is based on GPT-3.5-turbo. While it offered good spatial awareness, GPT-4 offers much better results. The version you see in the video above is using GPT-4.

GPT-4 is vastly improved in respect to GPT-3.5 at solving complex tasks and comprehending complex instructions. Therefore we could be much more descriptive and use natural language when engineering the text prompt to “instruct the AI”

We could give the AI very explicit instructions like:

• “Each object name should include an appropriate adjective.”
• “Keep in mind, objects should be placed in the area to create the most meaningful layout possible, and they shouldn’t overlap.”
• “All objects must be within the bounds of the area size; Never place objects further than 1/2 the length or 1/2 the depth of the area from the origin.”
• “Also keep in mind that the objects should be placed all over the area in respect to the origin point of the area, and you can use negative values as well to display items correctly, since the origin of the area is always at the center of the area.”

The fact that these system prompts are being appropriately followed by the AI when generating the response is particularly impressive, as the AI demonstrates to have a good understanding of spatial awareness and how to properly place items. One of the challenges of using GPT-3.5 for this task is that sometimes objects were spawned outside the room, or at odd placements.

GPT-4 not only places objects within the right boundaries of the room, but also places objects logically: a bedside table will actually show up on the side of a bed, a coffee table will be placed in between two sofas, and so on.

With this, we’re likely just scratching the surface of what LLMs can do in 3D spaces!

Building your own ChatGPT-powered extension

While this is just a small demo of what AI can do once it’s connected to a 3D space, we believe it will open doors to a wide range of tools beyond scene building. Developers can build AI-powered extensions in Omniverse for lighting, cameras, animations, character dialog, and other elements that optimize creator workflows. They can even develop tools to attach physics to scenes and run entire simulations.

You can download and experiment with the AI Room Generator Extension Sample on GitHub. We encourage other developers to try building on the extension or creating their own generative AI extensions for Omniverse.

Using Omniverse Kit, you can start integrating AI into your extensions today. Download Omniverse to get started.

Get started with NVIDIA Omniverse by downloading the standard license free, or learn how Omniverse Enterprise can connect your team. If you are a developer, get started with Omniverse resources. Stay up to date on the platform by subscribing to the newsletter, and following NVIDIA Omniverse on Instagram, Medium, and Twitter. For resources, check out our forums, Discord server, Twitch, and YouTube channels.

About the Author

Mario Viviani is Manager of Developer Relations for Omniverse at NVIDIA, based in London, UK. His team focuses on helping developers and partners get familiar and onboard on NVIDIA Omniverse. Passionate technologist and hands on-developer, he’s ex-Amazon, where he led the global Apps and Games Tech Evangelism team; previously was co-founder of startups and led his own consulting company in mobile apps development. He is always projected into the future and into what is the next “big thing”!

USD is the HTML of 3D Worlds and will be key to unlocking the metaverse

By: Aaron Luk, Senior Engineering Manager, USD Ecosystem

Can the metaverse be a better ’verse? As the hype around the metaverse as a metaphor for all things futuristic coalesces into concrete efforts to enable the 3D evolution of the internet, it’s a good moment to reassess how both human beings and machine learning can reason about the scale and complexity of the data models and pipelines needed to represent virtual worlds in full fidelity.

A concept from the 1990s, the metaverse was first coined by science fiction author Neal Stephenson as a set of connected virtual worlds that extend the physical world. Like the 2D web, there are consumer and industrial uses of the metaverse. The 3D worlds of the metaverse will usher in a new era of design and simulation that unlocks new possibilities for AI and global industries.

So how is the metaverse evolving from speculative fiction to reality? The metaverse will require a common standard upon which to describe, compose, populate, simulate, and collaborate within 3D worlds. Again like the 2D web, the success of the metaverse’s 3D spatial overlay of the 2D internet will depend on its universal interoperability as governed by open standards and protocols.

Universal Scene Description (USD) will unlock the metaverse

Originally invented and open-sourced by Pixar Animation Studios to support large-scale visual effects of feature films, USD is an open scene description with APIs for creating, editing, querying, rendering, collaborating, and simulating in virtual worlds. Today, its highly scalable and interoperable qualities are enabling large, high fidelity scenes that accurately simulate the real world. We call these simulations digital twins.

A typical use of USD assembling scene elements to create a ground-truth view of a composed world is in automated factory assembly lines. With USD, numerous layers that describe properties of a robot in the factory can be combined together with sparse overrides to non-destructively modify source data. The geometry, shading, lighting, and non-graphical properties such as electrical info and sensor data are composed in a USD layer stack and presented as an autonomous robot.

The assembly line on which the robot sits can also have its own layer stack describing its properties. With USD, these can be easily stitched together into a composed factory world that brings together all the layer stacks in a USD stage. This stage can then be fed into any number of subsystems including rendering and physics to simulate real-world behaviors.

Digital twins for manufacturing and other use cases are already a reality. Companies like Siemens, Amazon Robotics, and BMW Group are leveraging USD to bring their virtual factories to life, finding new levels of operational efficiency for large-scale workloads.

Unique properties of USD

USD is much more than a file format. It’s a framework, ecosystem, and interchange paradigm that supports a wide variety of properties to define and render objects, while making scene structuring and editing more efficient and collaborative with sparse, non-destructive authoring.

There are four unique properties of USD that enable it to be highly extensible and meet the demands of virtual worlds:

1. Composition Engine: The composition engine enables the assembly of data from numerous sources as individual layers and the authoring of sparse, non-destructive overrides on top of those layers. This preserves the integrity of source data ingested in the USD ecosystem.

2. Custom Schemas: The data model of USD is fully extensible with custom schemas. USD itself comes bundled with core schemas such as geometry and shading. NVIDIA has worked with Pixar and Apple to create physics schemas for rigid bodies. Other custom schemas are being explored to further expand the ecosystem for digital twins and virtual worlds.

3. Asset Resolver & Data Storage: USD is entirely filesystem agnostic. The data is not tied to filesystems or any other persistent storage. It can even be procedurally generated. This is made possible by USD’s plugin system for asset resolvers and file formats.

4. Hydra: USD can feed different renderers out of the box like Pixar Storm and Renderman with Hydra, a generalized pipeline for custom renderers. NVIDIA has also integrated real-time ray tracing and physically accurate rendering support with Hydra.

Pushing the limits of USD with real-world testing

NVIDIA and our partners continue to push the boundaries of Universal Scene Description. The latest developments in USD are driven by real-world testing in applications for autonomous driving and robots and industrial and scientific digital twins.

The increased scale and complexity of the data models needed to mirror the real world leverages the relative scale and complexity of world-building for feature film animation. But the requirements for digital twins sprawl into entire families of use cases beyond media and entertainment. Here are some areas where digital twins require core enhancements to the USD ecosystem.

Unicode Identifier Syntax (UTF-8)

NVIDIA is developing Unicode Identifier Syntax for USD to bring international character support to USD workflows. USD supports the UTF-8 standard for content, but identifiers such as prim names are currently limited to ASCII, presenting challenges when the source data is encoded in non-Roman languages. UTF-8 identifiers will be encoded and validated in USD core and may allow leading digits, unlocking many use cases such as interchange of source data from Autodesk Revit with UTF-8 identifiers for manufacturing in Germany or architecture in Japan.

Geospatial Coordinates (WGS84)

NVIDIA is developing USD data models for geospatial posing with WGS84, a geodetic datum describing the curvature of the earth. This allows USD content, for use cases like railroad design or city mapping, to georeference real world geometry using geospatial coordinates. NVIDIA has prototyped an API schema for WGS84 reference frames and local offsets.

The USD development roadmap includes plugin points for geospatial kernels such that arbitrary coordinate reference systems can be plugged into the system.

The future of USD

NVIDIA is continuing to invest in USD, with numerous projects on the roadmap. Recent contributions from NVIDIA to the USD ecosystem include:

• Python 3 support: Updating from Python 2.7 (sunsetted in January 2020)
• Adding usd-core to PyPI: Easy installation of USD core for Python programmers
• Material Definition Language (MDL) materials support: Encoding physically accurate MDL materials in USD data models
• Schema for rigid body dynamics simulation (with Pixar and Apple): Standardizing the representation of mass distribution, collision behavior, and other data necessary for rigid body dynamics.

In the medium term, in addition to the UTF-8 and WGS84 support described above, NVIDIA is developing:

• glTF interoperability: A glTF file format plugin will allow glTF assets to be referenced natively as USD content. This means that users who are already using glTF can take advantage of the composition and collaboration features of USD without having to alter their existing assets or pipeline, just like users of Alembic and Draco. We will collaborate with other industry partners to productize this glTF file format plugin in the open-source USD GitHub repository.

• USD compatibility testing and certification suite: To further accelerate USD development and adoption, NVIDIA is building an open-source suite for USD compatibility testing and certification. Developers will be able to test their builds of USD and certify that their custom USD components produce an expected result. This is critical to maintaining an open ecosystem on top of the rapidly evolving nature of the USD software stack.

In the longer term, NVIDIA is working with partners to fill some of the larger remaining gaps in USD:

• High-speed incremental updates: USD was designed more for offline import/export interchange rather than large-scale real-time updates needed by digital twin simulations. NVIDIA is developing additional libraries on top of USD that optimize data layout and change tracking to support real-time simulation at scale.

• Real-time proceduralism: NVIDIA has begun to augment USD’s declarative content with a procedural graph-based execution engine called OmniGraph, as well as Warp, a framework for writing simulation kernels in Python that are JIT converted to to C++/CUDA and compiled at runtime and run directly on the GPU.

• Compatibility with browsers: To be accessible by everyone, everywhere, virtual worlds will need to be capable of running inside web browsers on any device. NVIDIA, alongside other USD stakeholders, is exploring client-side approaches such as WebAssembly builds with JavaScript bindings, as well as server-side approaches such as pixel streaming from Omniverse Cloud.

• Real-time streaming of IoT data: Industrial virtual worlds and live digital twins require real-time streaming of IoT data. NVIDIA is working on building USD connections to IoT data streaming protocols.

More information can be found in the recording of the GTC session that covered earlier iterations of the developments described in this article.

Getting Started with USD

You can access a wealth of USD resources from NVIDIA, available online for free. A good place to start is with NVIDIA’s hub of USD resources, which includes tutorials and pre-built libraries and tools, such as turnkey distribution of usdview in Omniverse Launcher. To learn the basics of USD with examples in USDA and Python in a step-by-step web tutorial, sign up for the USD DLI course.

NVIDIA Omniverse is a scalable computing platform for full-design-fidelity 3D simulation workflows and a toolkit for building USD-based metaverse applications. Omniverse was built from the ground up as a USD engine and open toolkit for building custom, interoperable 3D pipelines.

Learn more about USD at NVIDIA GTC

NVIDIA GTC, the conference for the era of AI and the metaverse taking place March 20–23, will feature numerous sessions on USD and developing with NVIDIA Omniverse.

Be sure to check out my session, An Overview of Universal Scene Description for Building Virtual Worlds, and a panel with leaders from Cesium, Open Geospatial Consortium, Autodesk, Adobe, Pixar, and NVIDIA about how USD can become the “HTML” of the Metaverse.

You can also access a hands-on training lab to experiment with the fundamental concepts of USD utilizing the USD Python API.

Subscribe to the Omniverse newsletter to receive updates about Omniverse Enterprise. Follow us on Instagram, Twitter, YouTube, and Medium to stay up to date with the next releases and use cases.

Visit the Omniverse Developer Resource Center for additional resources, view the latest tutorials on Omniverse, and check out the forums for support. Join the Omniverse community, Discord server, and Twitch Channel to chat with the community, and subscribe to get the latest Omniverse news.

About the Author

Aaron Luk, Senior Engineering Manager, Omniverse, NVIDIA

Aaron Luk is senior engineering manager leading Universal Scene Description (USD) ecosystem engineering, including oversight, integration, and optimization of USD and Hydra paradigms across all components of NVIDIA Omniverse. Previously, Aaron co-developed and deployed USD into production pipelines at Pixar.