FINAL TECHNICAL PROGRESS REPORT
For SBIR I: NNA05AC13C
Simulation-Based Lunar Telerobotics Design, Acquisition and Training Platform for Virtual Exploration
Report Date: July 18, 2005
Reporting Period: January 25, 2005 – July 18, 2005
PI/Contact: Bruce Damer, DigitalSpace Corporation
NASA Program Recipients
NASA Ames Research Center
Accounting Operations Branch, M/S 203-18
Moffett Field, CA 94035-1000
CS Amine Hambaba, MS 241-1
COTR Mark Shirley, MS 269-3
SBIR Specialist MS 202A-3
SBIR Team Recipients: B Damer (DigitalSpace), G Brandt (DigitalSpace)
Phase I Project Summary
Firm: DigitalSpace Corporation
Contract Number: NNA05AC13C
Project Title: Simulation-Based Lunar Telerobotics Design, Acquisition and Training Platform for Virtual Exploration
Identification and Significance of Innovation:
To meet the objectives of returning to the moon by 2020 for extended stays leading to human exploration of Mars will require NASA to fly a series of telerobotic lunar orbital and surface explorers, some to prove the viability of In Situ Resource Utilization (ISRU) and others to engage in lunar exploration and human base precursor construction. For such a large number of vehicles and innovative designs, a real-time 3D tool that creates plausible simulations of space environments and vehicles can bring an urgently needed rapid prototyping capability to the new vision for space exploration.
Digital Spaces (DSS) is a 3D rapid prototyping platform that will allow distributed teams to iterate designs on a week to week basis to craft operational scenarios with real-time 3D simulations tightly linked to team meetings, teleconferences, web documents, CAD databases and other decision support tools. DigitalSpace completed the first phase of this SBIR project to model the Colorado School of Mines’ (CSM) prototype lunar bucket wheel excavator (BWE) to demonstrate that a real-time simulation of a lunar surface vehicle is possible. The BWE was chosen as it is one of the only lunar ISRU-oriented prototypes in the world and is the lead vehicle in a series that is likely to include trenching, boring and hauling machines. The BWE simulation allows users to run real-time traverses of the vehicle on a virtual landscape populated by elements of a hypothetical lunar base. A force feedback joystick and stereoscopic immersive display is supported to create the operator sensation of a real excavator and yet the environment runs on ordinary consumer computers. It is hoped that this tool will evolve from a design simulator into a prototype operator interface, a mission trainer, and finally a tool for day to day surface operations.
Technical Objectives and Work Plan:
This project sought to create a simulator of an existing prototype lunar bucket wheel excavator (BWE), the testing of that virtual robotic vehicle inside a virtual lunar regolith stimulant sand box and the final application of the combined virtual environment with haptic force feedback devices and multi-modal immersive displays. The work plan objectives were:
Prototype Bucket Wheel Excavator
Create dynamic 3D virtual model of the existing (CSM) prototype lunar analog BWE guided by experts from the CSM.
a virtual simulant “sandbox” to test the virtual excavator based on approximate
physical properties of the lunar regolith including gravity.
Haptic force feedback interfaces
Develop a driver for a simple control interface using a force feedback joystick to operate the virtual excavator in the sandbox.
Output the full version of the simulation to a 3D (stereo) display to show that a sense of immersion can be given to the operator.
Iteration of virtual prototype with expert team input
Provide a test group including the BWE’s designers with the work product software for testing and commentary guide us on Phase II project prospects.Technical Accomplishments:
This Phase I proposal successfully developed and tested a software-based virtual test fixture performing a 3D dynamic simulation of a lunar bucket wheel excavator (BWE) prototype built at the Colorado School of Mines (CSM). This simulation can be run in real-time on most PC workstations and supports a physics simulation of the vehicle on a lunar surface environment. The following work plan objectives were met:
Built 3D virtual model of the existing prototype lunar analog BWE and validated by experts from the Colorado School of Mines.
Created a virtual stimulant sandbox to test the excavator model. The sand box was validated by CSM testing with commentary on future simulation improvements.
A simple control interface using a force feedback joystick was developed to operate the virtual excavator in the sandbox.
The full version of the simulation was output onto a quad buffer auto-stereo display to show that a sense of immersion can be given to the operator.
The virtual prototype was iterated to match the properties observed by CSM and others in their original testing of the physical excavator in its simulant sand box. Guidance was obtained about future rapid prototypes of teleoperated lunar vehicles and missions.
NASA applications enabled by a collaborative real-time 3D rapid prototype platform include:
1. Design simulation of teleoperated surface robotic vehicles and missions on the Moon and Mars for exploration, site preparation and In-Situ Resource Utilization (shielding, materials, fuel and water extraction).
2. Concept development for teleoperated robotics for servicing for ISS, orbiting scientific observatories (Hubble) and long duration Crew Exploration Vehicles (CEV).
3. Simulation of astronauts and robotic agents within spacecraft or in surface facilities.
4. Telerobotics training for all classes of remote manipulator systems.
5. Training and mission operations using the virtual environments as an operator tool.
6. Design and prototyping of other types of vehicles through integrated modeling and simulation.
Non-NASA Commercial Application(s):
Any remotely controlled vehicle in mining, construction, hazardous waste handling, military operations and other commercial applications requires high fidelity virtual environments for development of viable designs, interaction scenarios, and eventually for the training of operators and day to day operations. The needs for teleoperations in the terrestrial mining, construction and manufacturing industries alone could create a multi-billion dollar annual business. We have identified the following applications in design simulation that could be served by the platform:
1. Collaborative engineering of automated mining especially deep mining where heat and hazards make ore extraction costly or prohibitive.
2. The design of automated construction including excavators, haulers, cherry pickers, and cranes.
3. Defense and security applications in the design, training and operations for teleoperated vehicles in the battlefield or for facilities surveillance.
4. Planning and training for emergency first responder hazardous area robotics.
5. Industrial design, training and operations applications.
6. Software games in the “robot wars” genre and for education/outreach in robotics education.
Name and Address of Principal Investigator:
Bruce Damer, DigitalSpace Corporation, 343 Soquel Ave, #70, Santa Cruz CA 95062.
Name and Address of Offeror:
DigitalSpace Corporation, 343 Soquel Ave, #70, Santa Cruz CA 95062
Part 1- Table of Contents
Part 2 Identification and Significance of the Innovation
2.1 Identifying the Need
On January 14, 2004, President George W. Bush enunciated a new vision for space exploration [1,2] which set forth a road map to return humans to the moon by 2020 and set the stage for the human exploration of Mars. One year later, the first NASA Robotic & Human Lunar Exploration Roadmap Meeting was held in Houston and NASA’s Chief Scientist, Jim Garvin, and others presented strategies and options for robotic and human exploration of the moon (including 14 day stays at a prepared site) as a test bed for the missions to Mars. Dr. Garvin’s Lunar Strategy , was echoed by the other participants including the members of the newly formed Lunar Exploration Analysis Group (LEAG). In each scenario, after the Lunar Reconnaissance Orbiter (LRO) mission of 2008 has identified candidate sites, a series of teleoperated fixed landers and long range rovers will be deployed to the moon to engage in geologic exploration of the Lunar regolith and searching for water ice and other elements for possible In-Situ Resource Utilization (ISRU) in support of human lunar visits.
In the lunar exploration community there is a strong consensus that ISRU is necessary for sustainable (and affordable) human presence in space . The LEAG recently identified the following priorities for early experimental test beds, research and flight missions :
A real-time 3D environment that creates plausible simulations of
space environments and vehicles can bring an urgently needed rapid prototyping
capability to any NASA mission and vehicle design effort. Going well
beyond static artist’s conceptions, this prototyping tool will allow distributed
teams to iterate designs on a week to week basis crafting operational scenarios
with real-time 3D simulations tightly linked to team meetings and teleconferences,
web documents, CAD databases and other decision support tools. DigitalSpace
proposed and completed the first phase of this SBIR to model the Colorado
School of Mines’ lunar bucket wheel excavator (BWE) to demonstrate that
a real-time simulation of a lunar surface vehicle is possible (figures
1-4). The BWE was chosen as it is one of the only lunar ISRU-oriented prototypes
in existence. The BWE is the lead vehicle of several classes of mining
machines which include further design iterations into trenching and boring
machines, all likely to be employed in lunar exploration and ISRU.
2.2. The Innovation
During the six months of this phase I project, DigitalSpace was called upon separately to support rapid turn-around 3D visualization and telemetry generation for a number of NASA ECP (extramural) projects (USC/Superbots shown in figure 5), a NASA CE&R presentation for a major subcontractor (Raytheon shown in figure 6) and an Exploration Systems Mission Directorate (ESMD) prototyping project (Distributed Observer Network). During this period, Mark Shirley, the COTR for this phase I SBIR effort, identified that there is a need for a platform to present shared 3D real-time simulations which allowing recording, playback and annotation and can be run by in-person or teleconferenced participants on desktop or laptop computers. It was against this backdrop that we completed work on this Phase I SBIR and discerned a vital need that could be met during a possible second Phase of this work.
Meeting ESMD’s collaborative rapid design prototyping needs
It is clear that over the next several years there will be a large number of mission trade studies, reference mission architectures, in-person conferences and teleconferences needed to support the implementation of the President’s new vision for space [1, 2]. Static images of mission hardware and scenarios or simple 3D animated movies have traditionally provided a starting storyboard for mission visualization but those media are too inflexible to support actual ongoing engineering design iterations and qualification of operational approaches. In addition, NASA and its contractor community will continue to rely heavily on voice conference calls backed up by email and document presentations. In near term, complex mission scenarios such as CEV flights, lunar telerobotic base construction and human lunar habitation, there will be increased challenges for team members to build effective mental models of overviews of proposed mission scenarios as well as detailed design options. We believe therefore that a gap exists between the static artist’s conceptions, teleconference notes and documents and the engineering commit stage of full computer aided design (CAD) and prototype construction (“bending metal”). It is in this gap where really clear understanding of complex vehicle/mission scenarios is so important that a rapidly iterated 3D prototyping environment can make a big difference.
The new enabling technology of real-time 3D
It has been only in the past several years that ubiquitous and powerful 3D chipsets have become standard in virtually all PCs and laptops. Therefore, real time 3D rendering is possible, with the end user able to change elements of a simulation on the fly. These environments are moving into the engineering mainstream. As an early investor in much of the serious real-time 3D industrial applications , NASA has a great legacy to draw on for 3D in mission simulation, training and mission operations . In the new era of exploration, NASA has a chance to again pioneer innovative uses of real-time 3D. In addition, in the era of open systems standards and open source, platforms can take advantage of tremendous economies of scale in development.
With support by NASA, USRA and others since 2000, DigitalSpace has built an open platform called “Digital Spaces” (or DSS) which is customized for a range of NASA mission and knowledge sharing needs. The architecture for DSS is shown in figure 7 below.
Figure 7: Digital Spaces (DSS) architecture, summer 2005
Bringing rapid prototyping to the design of space vehicles and mission scenarios
Virtually every field in which complex machines or parts are built on a short timeline utilizes some form of rapid prototyping. In manufacturing, expensive injection or press molds are machined only after iterations through 3D solid object “printers” which produce physical test articles out of low cost materials in time frames as low as a few minutes . This step greatly reduces errors and improves design prior to the costly machining of final parts. This kind of rapid prototyping is a metaphor for the platform proposed here. Real-time 3D environments that create plausible simulations of space environments and vehicles will bring a rapid prototyping capability to any early stage mission and vehicle design effort.
Proposed future platform features
As a result of this Phase I project, described in Part 3, we have derived the following feature set we believe would merge the world of rapid virtual prototyping with work practice norms at NASA.We plan to propose these in Phase II:
1. Synchronized simulations for a given vehicle or mission scenario, design and engineering team members will run through the Internet as a supplement to normal teleconferences and physical meetings.
2. Recording and playback of member-driven simulation or operations sequences: all simulation sequences (whether pre-programmed or driven dynamically by a team member) can be made available to project managers, engineers or participants who may have been absent for a given meeting. These recorded simulation sequences will also provide an important historical log of decisions made.
3. Annotation of these simulation sequences would be possible by direct user interaction. Meeting notes, decisions and attached media such as images, audio and outside references could all be attached to the 3D models or within the interface and stored in the Universal Model Repository (UMR) now in development under another SBIR project at DigitalSpace (figure 7 above). In a 2004 SBIR phase I project for NASA ARC, DigitalSpace created an online version of the SimStation real-time visualization of the ISS (reference “Sim Station Online” project at [10, 11]) Selecting all or part of the real-time model executes a SQL query and brings up any media type, database record or document. Changes to annotations or associations would be shared with team members in real time. In between meetings, any project member
Mission profiles served by the tool
NASA’s new exploration vision will require multiple classes of mission to move human explorers beyond Earth’s orbit. Since 1999 DigitalSpace has carried out a range of projects with NASA, its contractors and others which we believe puts us in a strong position to create a key tool to support multiple mission collaborative design needs. We will next examine how this tool will serve critical collaborative design needs for the following mission profiles:
· Surface exploration and ISRU validation missions
· Crew Exploration Vehicle
· Telerobotic and Human base operations
and ISRU validation missions
Early lunar missions after Lunar Reconnaissance Orbiter (LRO) flies in 2008 may include MER-style long duration surface resource explorers seeking deposits of water ice and other materials to lower mission costs and risks. The rapid prototyping of one or more designs for these vehicles will be a primary focus of this tool. The phase I project was directed mainly at this goal by modeling the lunar bucket wheel excavator (figure 9) and related work on sintering (figure 8).
Crew Exploration Vehicle
The CEV and its associated heavy lift cargo capability possibly based on Shuttle derived technology will bring crews and mission hardware to LEO and beyond in the post Shuttle 2010 time frame. DigitalSpace produced 3D visualizations of CEV for Boeing and Raytheon in 2004 (figure 10) and this tool should be well suited to rapidly rendering designs for CEV in ascent, in-orbit, docking and undocking, EVA and entry, descent and landing. A project for Raytheon and the program office of the Neutral Buoyancy Laboratory at JSC (figure 11) allowed us to experience creating a training simulator for crews in preparation for Shuttle/Station missions (in this case STS-114 return to flight). The ability to model IVA or EVA aboard the CEV will be a key element to design of both vehicle and missions.
Telerobotic and human
DigitalSpace’s prior work supporting the Brahms team at ARC and USRA for simulations of activities during their FMARS and MDRS analogue Mars base field seasons [12, 13, 14] (figures 12, 13) allowed us to get an early start in modeling human/robotics and the current MODAT ECP project puts it in an excellent position to support complex human-in-the loop human/robotics mission profiles. The parallel BrahmsVE project is now bringing Brahms agents to a lunar base simulation (figures 14, 15).
More on the recommendations from the testing of this Phase I work product and for future development tasks in part 7.2.
Part 3 Technical Objectives
3.1 The Objective
The solicitation for this topic called for innovation in the way that people will interact with both physical and virtual data sets using multi-sensory displays and interfaces (including force-feedback) to support richly endowed situational awareness and telerobotics.
Final work product:
Simulation of the BWE in an analogue lunar base environment
The prototype Lunar Bucket Wheel Excavator (BWE)  concept vehicle (figure 16) was developed at the Colorado School of Mines under the direction of Dr. Michael Duke, Dr. Bob King and was built by graduate student Tim Muff (now at Lockheed-Martin Corporation). This vehicle was tested in a Lunar regolith stimulant sandbox (figure 17) and its ability to transport (mine) stimulant was characterized (see reference 1 in section 6 below). DigitalSpace selected the Lunar BWE as the focus of this Phase I SBIR for the following reasons:
The CSM’s BWE is one reference design among many possible and represents one very early data point. The CSM’s lunar stimulant sandbox, based on but not using the stimulant JSC-1, is only a very rough analogue of lunar soils. In addition, the lunar environment of a near vacuum, one sixth of Earth’s gravity and the thermal and radiation environment, was not simulated for the CSM’s sandbox testing. Lastly the communications latency present with lunar teleoperations was not simulated in the CSM testing nor were operator interfaces such as haptic joystick controls typically found in terrestrial mining.
Despite these limitations, the CSM’s BWE is the only existing example of a concept lunar construction robot and serves as an excellent starting point for early research into mission-critical lunar resources utilization and telerobotic operations. The goal of this SBIR Phase I was to build on the strengths of this work to produce a virtual analog of the BWE in the lunar environment as a test fixture for the evolution of lunar telerobotic systems that may lead to flight hardware. The goals of the SBIR project are:
New additions made to the Digital Spaces platform
Figure 18: Current Digital Spaces architecture showing physics engine and haptic device extensions
Digital Spaces (DSS) is organized like an operating system and consists of three elements, core modules, akin to kernel elements in an operating system such as Linux, plug-ins which are extensions connected to the Core and other plug-ins by an abstracting API, and libraries, which may be databases of objects or groupings of functions. Figure 18 above illustrates the significant additions of modules, plug-ins and libraries during this phase I project:
1. A physics module which through an additional library called Gangsta, permits a wide range of open source and proprietary physics engines to drive physics in a common scenegraph.
2. A module for a force-feedback joystick (haptic device).
3. Enhanced full screen 3D rendering of the simulation with good performance and overlaid 2D graphical interface objects (Ogre renderer and CEGUI library). The renderer also supports immersive autostereo displays through NVidia quad-buffer hardware.
4. A dedicated BWE dynamics simulator plug-in to match the behavior of the physical BWE as described by its designer, Dr. Robert King.
5. Additional modules such as a new Camera Control, Entity State Display and Animation sequencers and playback elements were added. In the same period, a plug-in for the Brahms agent system was completed for a separate project. Brahms will be an important element of a Phase II proposal of this work.
The unique achievement of a multi-engine physics implementation
DigitalSpace determined that for any surface vehicle simulation, physics
simulation would play a key part. As there are a dozen open source and
proprietary physics engines available on the market, each having differing
strengths and weaknesses, we determined that it would be unwise to create
a dependency on only one solution. We hired a physics specialist, Ed Jones,
who built what we believe is the industry’s first multi-engine physics
module. Ed created and tested an interface (called Gansta) to four of the
leading engines (ODE, Novodex, TrueAxis and Newton) which is now integrated
into DSS. Figures 19-22 below illustrate our testing of three of the four
physics engines via this plug-in methodology. The before and after simulations
(tower of objects falling, robot arms lifting boxes) show the slightly
different performance of the three engines running in real time in the
same scenegraph. The final work product uses the open source ODE physics
Figures 19-22: three physics engines in before-after scenario testing via the generic DSS plugin
Tying a generic game force feedback joystick into a 3D scenegraph with physics
Another significant achievement of this 6 month effort is that we were
able to tie the generic DirectX support for low cost game joysticks having
a force feedback (haptic) capability into both the DSS 3D scenegraph and
the physics engine. Figure 23 illustrates how these modules interact to
drive physics by joystick or glove and output on an autostereo display
Delivery and test/evaluation schedule
The work product was delivered in stages by DM3D Studios (DigitalSpace’s subcontractor on the project) for alpha testing internally at DigitalSpace in March 2005 and the first on-site evaluation at Colorado School of Mines occurred during a site visit in April 2005 (see figure 5 above). There was a test and public demonstration given in Melbourne, Australia in May 2005 run for a week as part of an exhibit on virtual environments in the future of manufacturing sponsored by Swinburne University. Following the addition of the physics and force feedback and beta testing at DigitalSpace in June, the whole package was made available for test in early July, 2005 to Colorado School of Mines, NASA ARC, andStanford University.
Reasoning for not integrating Stanford Biocomputation’s Spring system
We opted (after discussing this with Dr. Kevin Montgomery of the Stanford Biocomputation Center) to not integrate their Spring system and hardware for reasons of cost, time and availability. We agreed that supporting standard force feedback game joysticks such as the Logitiech® Force3D Wingman within the standard Windows D3D driver would give us a good approximation (initial experience) with haptic control. In the course of the project three of these devices were acquired and shipped to developers and testers.
Final work product package
In conclusion, the work product delivered included:
The user experience
As the figures 25-32 below illustrates, the interface to the Digital Spaces’ lunar bucket wheel simulation permits a number of operating modalities: 1) camera tracking of the virtual vehicle from an independent viewpoint or tied to the vehicle motion, 2) operation of the simulation through a joystick or keyboard-based interface, 3) force feedback sensing in the joystick interfaces depending on the machine-regolith contact of the wheels/chassis of the virtual vehicle and of the bucket wheel contact with the simulated regolith, 4) teleoperation simulating the 2.6 second latency representing the communications time from Earth to moon and return (direct). In addition, the position and orientation of the virtual vehicle is reported.
We utilized a simple 14 day stay lunar base model from another DigitalSpace SBIR project as the backdrop for the BWE testing. Regolith occurred in a series of shallow hills and artificially piled up for radiation and rocket motor ejecta mitigation. Thus the virtual BWE was able to drive through a number of traverses over shallow or steep terrain, sintered level landing and trackways and was able to be tested backing up to a static ISRU processing facility model, which has a hopper bay designed for the BWE to offload material.
We used a simple “puff”
of dust to indicate contact of the bucket wheel with the simulated surface.
Colorado School of Mines has been awarded a Phase I ECP project to examine
dust mitigation strategies for the moon and Mars and has agreed to participate
with us in a Phase II SBIR proposal. Part of this participation will allow
us to produce particle simulations of likely dust conditions in the lunar
environment. These simulations, associated with lunar vehicles, will aid
“Project Dust” at the CSM to study mitigation strategies.
Reports on the testing of the BWE simulation
As this was a proof of concept project there was no statistical testing done. The Colorado School of Mines (CSM) was the main center for test although the COTR from NASA ARC also tested the simulation. The CSM testers were impressed by the work product and tested for several hours over a weeklong period. At the time of writing of this report they were still testing with several faculty. They felt we had captured the dynamics of the vehicle accurately and that it behaved as the physical robot does in its sandbox in terms of: a) the rocker-bogie mechanism was a plausible approximation, b) the behavior of the bucket wheel itself and its “bite in” felt realistic in the haptic interface, c) the felt that having the base elements such as the hab, blast ridge, and ISRU processor made for a valuable, more realistic test environment.
Where improvements could be made were:
1. A nonlinear interface from the joystick to the virtual vehicle drive (ie: the more the stick is pushed the faster the vehicle should driver) would be a better approximation.
2. The bucket wheel itself should move about a half to quarter of its current speed.
3. The bucket wheel should be able to be operated in “sweep” mode (moving side to side as normal mining BWEs do).
4. The first simulation of lunar regolith should be that if the bucket wheel digs in deeper it should be driven harder (resistance) and cause the vehicle to push back.
5. Collisions with base objects, such as the ISRU processor, should give some measure of whether the event damaged any equipment (metal stress points etc).
6. Simulation of dust should be more realistic (not “puffs” but ballistic arches) and that dust cannot “blow away in a breeze” as there is no atmosphere on the move to drive it but simply falls. The BWE wheels should also generate dust.
7. Simulation of regolith properties such as density, rock objects and material flows would be nice (although they pointed out how difficult a challenge this is).
The CSM testers and evaluators were very encouraging and looked forward to an opportunity to iterate the simulation again. We felt that this was a major endorsement of the concept of rapid virtual prototyping and forms the basis of our Phase II proposal.
Some of the results of the NASA ARC testing are as follows:
1. It would be very useful to have a pattern on wheel sidewalls to be able to determine if we are in motion (and a distinct shape on top of wheels per Mike Sims to match how they track MER's progress visually).
2. Collision and other geometry should be visible in a wire frame view mode.
3. We must improve the joystick interface (providing a definable template ideally) to allow the user to navigate their own point of view with the joystick while the rover drives via keyboard or somehow merge these modes. The SimStation modality (derived from Descent) is a good starting point for a better joystick control.
4. The joystick interface and haptic (force feedback) feedback was crude at best, given the use of sub-$100 game device technology. Due to the cost and custom programming interfaces, we opted not to use higher-end haptic devices such as those utilized at Stanford’s Biocomputation center. Testing with higher caliber human-machine interface devices in a subsequent phase of development could involve technology already in use for mining teleoperations [16, 17].
We believe that the Phase I project has proven the feasibility of producing a plausible real-time 3D visual simulation of a lunar surface vehicle and mission scenario in much higher fidelity. We are confident that DigitalSpace’s prior experience in building collaborative and multi-user platforms will enable us to create DSS-Prototyper, a marriage of the 3D rapid prototype simulation and multi-user collaborative tools (reference Phase II proposal).
See part 7.2 below for suggestions of future development made by the tester/evaluator teams.
Part 4 Work Plan
4.1 Technical Approach
The project commenced with extensive consultation with expert advisors at CSM/CCACS (the Center for Applications of Combustion in Space, now called the Institute for Space Resources (ISR), Stanford, and several NASA centers as other contractors and university collaborators. Thereafter the following staged implementation was be undertaken:
· Produce a 3D virtual model of the existing ISR prototype lunar analog bucket wheel excavator. Validate the virtual excavator with the ISR.
· Create a virtual stimulant “sandbox” to be able to test the virtual excavator. Endow the virtual sandbox with basic physical properties. Validate sand box with the ISR.
· Build a simple control interface and test operating the virtual excavator in the sandbox. Match the control interface to likely haptic force feedback controls.
· Iterate the virtual prototype to match the properties observed by ISR in their original testing of the physical excavator in its simulant sand box.
· Export the refined virtual model to the Spring system and work with Stanford Biocomputation center to create a first generation haptic force feedback version of the virtual excavator and stimulant sandbox.
· Test the haptic-enabled version in an immersive display interface (parabolic) and generate data/metrics to compare simulator/haptic and observed physical performance.
· Produce reports and recommendation for Phase II.
Throughout the second and third portions of the project, all consulted teams were be able to view progress and the final simulation through releases of the DSS platform and project content.
4.2 Task Descriptions
Table 1 below maps our allocation of hours by labor category for the eleven major tasks in this Phase I work plan and corresponded to our budget.
Table 1: Work Plan
Total of all hours:
890 where the roles are defined as:
Meeting the Technical Objectives
All of the technical objectives in this Phase I project were met.
4.4 Task Labor Categories and Schedules
This section describes the work schedule for the Phase I effort (see Table 2 below). DigitalSpace work was coordinated from its corporate offices located near Santa Cruz California. DigitalSpace development and testing teams are located at several places around the United States and internationally.
Table 2: Work Schedule
Part 5 Potential Applications
5.1 Potential NASA Commercial Applications
Surface operations on the Moon or Mars, teleoperated servicing robotics on long duration space station and crew exploration vehicle flights and a whole class of crew training and data rich control environments demand a new generation of haptically enhanced virtual environments. Key NASA applications enabled by an advanced closed loop virtual environment telerobotics simulator include:
All workstation/internet based versions of this environment can be offered as a public outreach (EPO) module for educational uses and the home. Telerobotics for Space Camp, schools based or internet based teleoperations experience for young students or high school/college level students will encourage a space engineering career direction.
5.2 Potential Non-NASA Commercial Applications
Any remotely controlled vehicle in mining, constructions, hazardous waste handling, military operations and other commercial applications requires high fidelity virtual environments for development of viable interaction scenarios, training of operators and day to day production. This project is an important step to making closed loop highly responsive telerobotics operations a commercial reality.
Mining and construction
The needs for teleoperations in the terrestrial deep mining [16, 17] and construction industries alone could create a multi-billion dollar annual business in a new generation of safer, and more effectively managed machinery operations.
Defense design, training and operations applications
The military will be using semi and fully autonomous agents working closely to support troops and command in surveillance and combat missions throughout the 21st Century . Therefore we expect a great deal of interest surrounding a product in this space. We are already in contact with the Naval Postgraduate School MOVES Institute about cooperation on and adopting a new XML based standard in simulation communications for virtual presence in the control of remote robotic vehicles.
Industrial design, training and operations applications
From factory floor robotic automation to security systems, construction, complex environments where humans work in operation with with mobile agents or other autonomous machine systems
Emergency first responder teleoperations
Chemical, biological or nuclear hazards will all require first responders to consider using teleoperated machines to enter affected areas first. This work could directly inform this type of technology for occupational safety.
Online games – educational and entertainment applications
Robot “wars” are one of the most popular forms of entertainment in the popular media and robot game competition are some of the finest learning events for K-12 and college engineering students and faculty. Massive multi-player online games are experiencing a large amount of investment and commercial interest. This work will be competent platform for the sourcing of a successful reality-based multiplayer online game both as a learning tool and as a pay-per-play tournament environment. With a robotics focus.
See Part 8 below for a more detailed discussion of potential applications.
Part 6 Contacts
6.1 Key Contractor Participants
The following brief resumes introduce management/technical staff members who participated in this phase I project.
Name: Bruce Damer (PI)
Years of Experience: 24
Education: Bachelor of Science in Computer Science (University of Victoria, Canada, 1984); MSEE (University of Southern California, 1986)
Assignment: Mr. Damer was Principal Investigator for the SBIR Phase I effort. He coordinated all interaction between DigitalSpace and its collaborators and NASA and other participants, responsible for all staffing, technical design, and review of work product.
Experience: Mr. Damer is the world's recognized expert on avatars and shared online graphical virtual spaces having created much of the early literature, conferences and awareness of the medium. Mr. Damer is a visiting scholar at the University of Washington HIT Lab and a member of the staff at the San Francisco State Multimedia Studies Program.
Name: Galen Brandt (Program Management, Marketing)
Position: New business development, DigitalSpace
Experience: 27 years including project management and marketing for Dun and Bradstreet, SUNY Fashion Institute of Technology, DoToLearn.
Assignment: Program management including in reviewing all aspects of the work product and documentation.
Name: Dave Rassmussen (SE)
Position: Principal of DM3D Studios, subcontractor to this project
Experience: 10 years experience in virtual world design, skills: 3DS Max, Java, Active
Worlds, Adobe Atmosphere, PHP/MySQL database development
Assignment: Dave has served as senior engineer for this phase, directing the team performing 3D modeling and animation, testing and delivery of the project
Name: Merryn Nielsen (Lead CD)
Position: Partner, DM3D Studios
Experience: 10 years experience in virtual world design and web page creation.
Assignment: Web & graphic design on project, 3D worlds, administrative support.
Name: Peter Newman (TE)
Position: DM3DS 5 years experience in C++, JS, PHP, HTML
Assignment: Chief technical architect of Digital Spaces (DSS) core and extensions.
Name: Ryan Norkus (CD & TG)
Position: DM3DS 5 years experience Graphic artist, 3d modeler and animator
Assignment: Focusing on 3D model construction & animated sequences
Name: Peter Meigs, Alex Grigny de Castro, Dean Strik (TE)
Position: Server support and infrastructure, DigitalSpace
Assignment: Focusing on the maintenance of DigitalSpace’s server infrastructure
6.2 Key NASA Participants
Key NASA participants in this project have included: project COTR, a number of NASA personnel at Ames Research Center including Michael Sims and others (see below).
6.3 NASA and Non-NASA Advisors and Evaluators
To facilitate both the live testing of the simulation of the lunar bucket wheel excavator, DigitalSpace recruited a team of individuals representing some of the world’s top expertise in robotics, the lunar environment and planetary surface exploration.
Table 3: Expert advisor/evaluator
team participants in the project
Brad Blair, of the Institute for Space Resources (ISR, formerly CCACS) at the Colorado School of Mines, is an internationally acknowledged expert in all aspects of lunar resources including the physical challenges of regolith handling and processing, the economics of ISRU and how it impacts whole mission architectures. Brad brought the depth of his knowledge of the space resources topic and community to bear on the project. Brad will also be assisting in directions for Phase II development and Phase III commercialization in the mining industry.
Dr. Robert King is Full Professor, Division of Engineering, Colorado School of Mines, and the lead designer of the lunar bucket wheel excavator working with Mike Duke and graduate student Tim Muff. Bob’s focus is all matters involving the dynamics and operability of machine/earth contact. Coming in with a focus on mining engineering, Bob applied his background to the BWE’s anticipated mission in a lunar setting. Bob was also involved in testing the Phase I deliverable..
Gary “Rod” Rodriguez, President of SysRAND, member of the ISR and Program Manager of “Project Dust” at the Colorado School of Mines brings years of experience of system design and ISRU background to the project. Rod brought his experience to bear during testing of the virtual BWE.
Dr. Masami Nakagawa, a professor at the Colorado School of Mines, brings his world-renowned lab and experience in dust and granular materials to the project. Working with Rod as PI on Project Dust, a NASA funded ECP efforts, he was one of the first testers of the environment during our site visit in April 2005 (figure 4).
Brahms/MODAT/Viz teams reviewed the Phase I deliverable and commented on the application of Brahms agents and Mobile Agents methodologies to future versions. In addition, the lunar base environment in which the BWE is being tested was developed as a BrahmsVE application for this team.
NASA ARC COTR Mark Shirley, and ARC/MER Science Co-Investigator Michael Sims both reviewed the platform.
Part 7 Technical Activities
7.1 CumulativeTechnical Activities
This Phase I proposal successfully developed and tested a software-based virtual test fixture performing a 3D dynamic simulation of a lunar bucket wheel excavator (BWE) prototype built at the Colorado School of Mines. This simulation can be run in real-time on most PC workstations and supports a physics simulation of the vehicle on a lunar surface environment. The following work plan objectives were met:
· Built 3D virtual model of the existing prototype lunar analog BWE and validated by experts from the Colorado School of Mines.
· Created a virtual stimulant sandbox to test the excavator model. The sand box was validated by CSM/ISR testing with commentary on future simulation improvements.
· A simple control interface using a force feedback joystick was developed to operate the virtual excavator in the sandbox.
· The full version of the simulation was output onto a quad buffer auto-stereo display to show that a sense of immersion can be given to the operator.
· The virtual prototype was iterated to match the properties observed by ISR in their original testing of the physical excavator in its simulant sand box. Guidance was obtained about future rapid prototypes of teleoperated lunar vehicles and missions.
7.2 Future Technical Activities
Reviewers identified the following immediate improvements in the BWE simulation and gave us a road map to the future applications of this kind of platform.
Immediate task 1: Improve user interaction modality with joystick, and keyboard
The overall user interface modality of joystick and keyboard needs to be improved.
Immediate task 2: Improve rendering views of the 3D scene
A wireframe view which shows the meshes for both the visible objects and physics collisions was requested. We will also add some visual cues suggested by Mike Sims and the project COTR including striations on the BWE wheels to be able to determine if there is movement occurring.
Future task 1: Enhancement of overall platform to add multi user synchronization
The ability of one user of the platform to drive a simulation from a particular point of view and then to save that and offer it to other users to play back was requested. Full contact-forces physics could allow sharing of the real-time simulation but it is more than likely that one “player” will simply send the telemetry to other viewers rather than relying on their local physics engines to faithfully duplicate the simulation.
Future task 2: Addition of multi user streams/presence
We have determined that the integration of the SILC (secure internet live conferencing) open source layer to DSS will give us a conferencing/chat and instant message capability necessary for geographically dispersed team collaboration.
Future task 3: Tie-ing in with existing NASA decision support and other web databases: component wrappering
Windchill, ICE, Nexiom and other tools mandated by NASA need to be tightly integrated with any platform that is supporting teams involved in mission design. Therefore the DSS platform needs to be tightly integrated with the web and other tools possibly as an ActiveX or Java Native Interface component.
Future task 4: Addition of HLA and other high level interfaces
High Level Architecture (HLA) has been suggested for addition to the platform to permit integration with the MODAT/Brahms/Viz tools and more.
Future task 5: A more comprehensive machine and environment simulation
Testers suggested that a more comprehensive simulation of the lunar regolith, soil mechanics and density, handling characteristics and modeling, and better dynamics (power train, physical behavior) of the BWE be modeled.
Future task 6: Addition of a scripting language
It was suggested to support the integration of SCHEME/Gambit to enable more flexibility in the platform and lessen the need to make programmatic additions for specific applications.
Future task 7: Open content/architecture
Pas experience guided us to create a plug-in architecture and open source core. A key indication of success will be when another team other than DigitalSpace takes up the tools and builds their own content. Two universities have now taken DSS in-house for projects.
Many of these future tasks will be addressed in the Phase II proposal in which we will detail the opportunity and benefits of a “DSS-Prototyper” platform.
Part 8 Potential Customer and Commercialization Activities
In anticipation of Phase III commercialization and Phase II pre-launch marketing and customer activity, we have identified the following classes of valuable applications for DSS and the components it encompasses. Companies in these fields include Multigen, Sense8, Virtualis, Planet9, RealSense, and CommonPoint.
8.1 Recent NASA Potential and Active Customer Activities
Project Dust at CSM
This ECP project has as its goals the simulation of dust conditions for tools and vehicles on the moon and will be using DigitalSpace’s platforms for that purpose in year two (2006). DigitalSpace is a member of this project.
MODAT/MDRS and Mobile Agents
This current ECP project at NASA ARC is planning to utilize the new capabilities in a model of a lunar base and JPL’s mission operations for MER. DigitalSpace is a team member of this project.
This ECP project is using DigitalSpace to visualize simulations of innovative modular robots on the lunar surface, in CEV and other tasks. The work product of this Phase I is key to the success of that effort as it relies on physics simulation.
ESMD collaborative visualization
DigitalSpace was invited to participate in the Distributed Observer Network (DON) project by KSC and Valador and we provided 3D models and telemetry. We expect that, as explained in Part 2.1 above, there will be a large need for collaborative rapid prototyping for a whole series of missions in the short term. DSS’ open source zero cost license should help many parts of NASA and its contractors adopt the platform. DigitalSpace and others hope to provide services and applications for this common open platform.
LEAG lunar exploration prototyping tools
The Lunar Exploration Analysis Group (LEAG) has invited DigitalSpace to present DSS and the BWE application at the October 2005 conference in Houston, TX. We will present this work in a plenary and it will be seen by the largest collection of lunar exploration and ISRU experts ever assembled. We hope it will become a tool for the LEAG.
SimEVA and SimIVA
Recent applications for the Neutral Buoyancy Laboratory at JSC and the VAST project for medical emergency training aboard ISS suggests there are a number of immediate applications for a real time rapid prototyping virtual environment in training and mission planning. We anticipate building Haptic and other device interfaces through partnerships built for the VAST project.
NASA educational outreach and Space Camp
8.2 Recent Non-NASA Potential Commercial Activities
K-12 and College, Education and Educational Web Services
The current set of DSS applications is being repurposed into educational course modules for schools, museums and other organizations. In discussions with the organizers of National Space Week, the Planetary Society, the Mars Society, and SPACE.COM, various modules of DSS projects have been made available and are now features in DVD, web and installation projects.
8.3 Other Recent Commercialization Activities
Public health applications
The National Institutes of Health, through the work of Dr. Dorothy Strickland and Do2Learn have contracted DigitalSpace to employ DSS to deliver safety games for children with autism. The first game was delivered for clinical trials at Emory University in Atlanta in July of 2004 and more work in this area is anticipated.
8.4 Future Potential Customer and Commercialization Activities
Space architect Constance Adams has invited DigitalSpace to use DSS for a September 2004 presentation of new designs for hotel rooms which embody principles learned while designing optimal work and living spaces for ISS.
Robot games – educational and entertainment applications
Robot “wars” are one of the most popular forms of entertainment in the popular media and robot game competition are some of the finest learning events for K-12 and college engineering students and faculty. Ames sponsors such events with CMU students and high schools. We have communicated with the organizers of the Ames events and demonstrated them DSS simulations. It is planned to partner with them and the local chapter of the Robotics Society of America to develop a kids’ robot design lab and competition space within the virtual spaces made possible by DSS. Massive multi-player online games are experiencing a large amount of investment and commercial interest. DSS is a competent platform for the creation of a successful multiplayer online game both as a learning tool and as a pay-per-play tournament environment. We plan to seek support for a commercial, online robot games application.
Defense design, training and operations applications
The military will be using semi and fully autonomous agents working closely to support troops and command in surveillance and combat missions throughout the 21st Century . Therefore we expect a great deal of interest surrounding a product in this space. We are already in contact with the Naval Postgraduate School MOVES Institute about cooperation on and adopting a new XML based standard in simulation communications.
Industrial design, training and operations applications
From factory floor automation to security systems, complex environments where humans work in tandem with mobile agents or other autonomous machine systems all need a comprehensive model-based environment with high fidelity 3D re-creation during both design, training and operations phases. Industrial training is a multi-billion dollar per year industry and DSS is uniquely suited to enter this market, running on industry standard platforms.
Consumer market research for personal robotics
The emerging era of consumer robotics as epitomized by Sony’s Aibo® and iRobot’s Roomba®  call for simulations for training and design for the general public..
Emergency first responder planning
Layout of cityscapes for planning first responder strategies in civil emergencies is an application well suited to DSS as it could employ the agent architecture of Brahms with detailed physics models for gas/dust dispersion.
Part 9 Resources Status
Up to the report date 100% of the work has been completed.
Part 10 References
 Office of Exploration Systems on the web at: http://exploration.nasa.gov and the President’s announcement of the new vision for space exploration on Jan 14, 2004 is at: http://www.nasa.gov/missions/solarsystem/bush_vision.html
 President's Commission on Moon, Mars and beyond, report on the web at: http://govinfo.library.unt.edu/moontomars/
 Garvin, Jim, “Strawman Robotic Architectures, Lunar Architectures” presented Jan 25, 2005 at the NASA Robotic & Human Lunar Exploration Roadmap Meeting 1, January 24-25, 2005, Space Center Houston. On the web at: http://www.moontoday.net/news/viewsr.html?pid=15540
 Space Resources Roundtable, LEAG and Lunar and Planetary Institute resources on the web at:
 G. Jeffrey Taylor, University of Hawai`I, Lunar Exploration Analysis Group (LEAG), presentation at the NASA Robotic & Human Lunar Exploration Roadmap Meeting 1, January 24-25, 2005, Space Center Houston. On the web at: http://www.hq.nasa.gov/office/apio/pdf/moon/05_leag_debrief_taylor.pdf
 LEAG contributions to NASA’s 60 Day Study from communication with Jeff Taylor, Jul 12, 2005.
 Multigen Paradigm library of projects on the web at: http://www.multigen.com/solutions/success.shtml
 R.B. Loftin and P.J. Kenney, Training the Hubble Space Telescope Flight Team, IEEE Computer Graphics and Applications, vol. 15, no. 5, pp. 31-37, Sep, 1995.
 Rapid Prototyping resources from the University of Utah on the web at: http://www.cc.utah.edu/~asn8200/rapid.html
 DigitalSpace projects and publications on the web at:
 Shirley, M., Cochrane, T., SimStation: A Knowledge-Integrating Virtual Vehicle, Virtual Iron Bird Workshop, NASA Ames Research Center, March 31, 2004. Available on the web at:http://ic.arc.nasa.gov/vib/day1/papers/Shirley_Cochrane.pdf
 M. Sierhuis, Modeling and Simulating Work Practice; Brahms: A multiagent modeling and simulation language for work system analysis and design, Ph.D. thesis, Social Science and Informatics (SWI), University of Amsterdam, SIKS Dissertation Series No. 2001-10, Amsterdam, The Netherlands, ISBN 90-6464-849-2.
 B. F. Damer, et al, Final Report, SBIR I: BrahmsVE: Platform for Design and Test of Large Scale Multi-Agent Human-Centric Mission Concepts, DigitalSpace Documents, July 2004. On the web at: http://www.digitalspace.com/reports/
 Mobile Agents and Brahms publications on the web at: http://homepage.mac.com/wjclancey/~WJClancey/WJCCogSci.html
 Muff, T., Johnson, L., King, R., and Duke, M.B., (2004), A Prototype Bucket Wheel Excavator for the Moon, Mars and Phobos, Proceedings of the 2004 Space Technology and Applications International Forum (STAIF-2004), Albuquerque, New Mexico, February 8-11, 2004.
 Automated Mining resources and publications on the web at: http://www.automatedmining.ca/news.html
 Victor Mallet, Miners are drilling to unimagined depths in the search for gold, Financial Times Weekend June 5 / June 6 1999
 S. Trentanelli, Armed Robots Provide Glimpse of Future Warfare, RDECOM Magazine, Sept 2003, on the web at: www.rdecom.army.mil/rdemagazine/200309/itf_armed_robots.html