Human Activity Behavior and Gestures Generation in vr systems Maarten Sierhuis, Ph. D.

Human Activity Behavior and Gestures Generation in VR Systems

• Maarten Sierhuis, Ph.D. (NASA ARC/RIACS)

• William J. Clancey, Ph.D. (NASA ARC/IHMC)

• Bruce Damer (Digitalspace, inc.)

Research Objective

• Ultimate Objective:

• Develop just-in-time uploadable VR training applications for crews on Mars or Space Stations.

• Technical challenges:

• Develop a generic agent language for modeling and “driving” bot-behavior in a Virtual World Environment.
• Create 3-D models of human work practices.

Overview

• Virtual Worlds

• Brahms

• Work Practice Analysis

• Brahms VE

• The FMARS Model

Virtual Worlds

• Multi-user web-based VR environments

• Allow people to interact
• Develop virtual communities

• Functionality

• Users enter as Avatars
• Bots (i.e. agents)
• Virtual spaces
• Virtual artifact and behavioral objects
• Links to web-sites
• Chat

Benefits for VR Training Systems

• Multi-user

• Trainer and trainee
• Chat capability

• “Light weight” VR application

• Web-based
• Applications are downloadable
• Training at a distance

• Multi-media functionality

• Different media as training material

• Commercial Virtual World Builders

• User community
• Consultants for building Worlds
• Development community

Issue with Virtual Worlds

• Limited Avatar  World Interaction

• Strongly scripted behavior
• Limited variations

• Limited behavioral programming environment

• VW development support
• Limited Bot language support

• What is needed is a way to model Avatar & Bot behavior in the Virtual World.

Brahms

Symbolic Discrete Event System

• Production Rules represent Qualitative Relationships!!

Multi-Agent Behavior

Brahms Engine

Reasoning with Beliefs of Agents and World Facts

Activity Scheduling and Subsumption in Brahms

Analyze the Work Practices of Astronauts and Field Scientists

• Research Method: Developing Models and Simulations of existing experiences of human collaboration.

• Deploying Scientific Instruments
• “Very sensitive instruments cannot tolerate the rough handling of robotic deployment.” (Spudis, 1999)
• 1969 – 1972 Apollo Project - ALSEP Deployment on the Lunar Surface
• Scientific Field Studies
• “The process that scientists use to conduct exploration in the field is poorly understood, and one cannot simulate what is not understood.” (Spudis, 1999)
• 1969 – 1972 Apollo’s Geo-Science Field Studies
• 1998 – 2003 Haughton Mars Project (FMARS)
• 2001 – 2003 Mobile Agents Project

Understand Work Practice

• Individuals and teams (CoP)

• Individual activities

• Collaboration between individuals

• Communication

• Tools and artifacts

• Environment/geography

Brahms VE

• Research Goal:

• 3-D Virtual Reality Display for Brahms Simulations
• Seamless integration between Brahms VM and VW Graphics Engine
• Bot language for Virtual Worlds

• Inspired by The Sims and Virtual World Communities (Active Worlds)

• Collaboration with DigitalSpace, Inc.

• Supported by Adobe Atmosphere Development Team

Coupling Behavioral Model with VW Model

• Agents => Bots

• Agent activities => Bot gestures

• Locations => 3D model

• Agent belief-state => Bot display

The FMARS Model

Flashline Mars Arctic/MDRS Research Station

• Mars Society: Research & Promotion

• “Holistic Sims”: Geology, biology, computing, medicine,… + Reporting + Mission support ops

• Investigate how we will live & work on Mars

• Six-person crew

• Crew rotations Dec-Aug 2001-2003

• Extensive publicity

Time Lapse Video Analysis

Activity Model

Brahms Model

Brahms Sim Output

• Brahms VM Output Format for Brahms VE:

• <Event Name>|<Activity Type>|<Start Time>|<End Time>|<Actor>|<Activity Name>|[|<Origin Area>|<Destination Area>]

• activity|move|156|164|projects.fmarsvre.RZ|Walk||projects.fmarsvre.WardroomTableArea|projects.fmarsvre.GalleyLadderArea

• activity|move|164|169|projects.fmarsvre.RZ|Upladder||projects.fmarsvre.GalleyLadderArea|projects.fmarsvre.WaterTankArea

• activity|primitive|169|256|projects.fmarsvre.RZ|CheckWaterLevel|

• activity|move|256|261|projects.fmarsvre.RZ|Downladder||projects.fmarsvre.WaterTankArea|projects.fmarsvre.GalleyLadderArea

• activity|move|261|269|projects.fmarsvre.RZ|Walk||projects.fmarsvre.GalleyLadderArea|projects.fmarsvre.WardroomTableArea

Gestures

• All scenarios, all agents

• MovingToArea
• GettingUp
• SittingDown
• OpeningCabinetDoor
• FetchingFood
• ClosingCabinetDoor
• EatingSnack

• Planning Meeting

• All agents
• SittingThroughPlanning
• SB
• ReportingGeneratorStatusToCommander
• ReportingRadioRepairStatusToCommander
• RZ
• AnnouncingStartOfPlanning
• RequestingGeneratorStatus
• RequestingWeatherReport
• RequestingRadioRepairStatus

VR: Data, Sim, Depiction

• Depicted three scenarios in Atmosphere action scripts

• Not coupled real-time: Hard-coded timing and waypoints, so agents don't run through objects and each other

• Next: Step into, inspect, & change running simulation; send personal agents to remote locations (e.g., from RST to Mars EVA) get back data & QT

Future: From “video” to “interactive” sims – issues and solutions

• Making things dynamic

• Dynamic interface
• Synching Brahms & VE clocks

• Interaction with simulation

• Movement
• Collision detection
• Line of sight
• Displaying mental states
• Avatar Interaction

Summary

• Research: “Just in Time” VR Training on Mars

• Combining Human Behavior Modeling with VR

• Brahms Agent Activity Language as Bot Language

• Brahms VE: Brahms + Adobe Atmosphere

• Example: FMARS Model

• Future: Dynamic Interaction