Real-time Stitched Video for Contextual Path Planning

Metadata Updated: February 28, 2019

The crew and flight controllers who control the Space Station Remote Manipulator System (SSRMS) currently rely on multiple camera views to confirm or infer clearance between the SSRMS and the surrounding structural environment.  The same is true for other remotely operated equipment - surgical instruments, submersible vehicles, and telerobotics.  This project explores operational efficiencies gained through the use of an array of cameras surrounding an object (such as the SSRMS boom or Mars Rover) and pointed along the object's long axis or direction of travel.  This project will design and build a state-of-the art situational awareness tool for the operators of exploration vehicles.  An array of cameras placed around the boundary of the vehicle will feed video to a vehicle-embedded processor which will synthesize the scene data into a single composite view presented to the operator.  The composite view will encompass a 360° view around the vehicle from a bird's eye perspective.  Robotics platforms used to benchmark this technology include the Dextrous Manipulator Trainer (DMT), model SSRMS, and Multi-Mission Space Exploration Vehicle (MMSEV).Initial (ICA) Investigation: Preventing collisions is the first priority for safe operations of the Space Station Remote Manipulator System (SSRMS).  This depends on the ability of the crew and flight controllers to verify enough clearance exists between the SSRMS, its payload, and surrounding structure.  In the plan, train, and fly stages of each mission significant time is spent developing, documenting, and executing a camera plan that allows each portion of the SSRMS trajectory to be monitored.  This time could be decreased and operational situational awareness increased by using an array of cameras mounted around a boom on the SSRMS that point along a boom.  The output of the these cameras could be stitched together to provide a one composite view that provides clearance monitoring 360° around the boom.  Further, this technology could be used in any application where it is desirable to see proximity on two or more sides of an object - surgery, tele-robotics, deep sea exploration. This investigation will ask operators (crew and flight controllers) to compare clearance monitoring of a sample trajectory using conventional external camera sources versus a stitched video presentation from a camera array.  A test plan, script, and scoring for comparison will be used to determine if stitched camera arrays lend themselves to clearance monitoring.  The project investigator researched the required technology, including hardware and software, to perform video stitching to identify an approach that can be used for operator evaluation in the ICA project.  Initial (Innovation Charge Account Project (ICA)) results: A cadre of robotics professionals from JSC Robotics Operations and Astronaut Office participated in a benchmarking effort to quantify efficiency and safety metrics both with and without the use of a stitched camera array.  A modified Cooper-Harper scale was used to determine operator workload.  Other metrics included time required to perform task, motion stopped due to lack of clearance views, whether contact was made with external structures.  Results showed a reduced operator workload, faster completion of the task, and reduced contact with external structure.  Additionally, the technology was presented to the JSC community at Innovation Day 2012 where it won the People's Choice Award. Second Phase: Rearranging image pixels from multiple cameras to accomplish a perspective shift is computationally expensive.  In the last decade, advances in CPU performance and direct to memory image capturing methods have improved the frame rate and latency associated with video stitching.  In the previous phase (FY '12 ICA, People's Choice Winner),  the collaborator was able to achieve 10 frames per second with less than a second latency using off the shelf CPU and camera hardware.  The purpose of Phase 2 is to demonstrate the technology on a larger vehicle (Multi-Mission Space Exploration vehicle) using high-bandwidth (GigE) network, increased CPU/GPU resources, and high-performance cameras.  Second Phase Results: Ten video cameras (the minimum required to obtain coverage around the vehicle while providing enough image overlap) were placed around the upper surface of the MMSEV.  The video streams were piped to an on-board high-end PC where software written in MATLAB performed the perspective shifts and homographic alignment.  The resulting single view was displayed in Graphical User Interface (GUI) that allowed the operator to see the composite 'birds-eye' view or zoom in on a view from a particular camera when clearance was a concern.  The MMSEV was maneuvered around the simulated Martian landscape at JSC known as the Rock Pile.  To date the maximum achieved frame rate is 2 frames per second.  To increase frame rate current efforts are focused on transferring the homographic algorithms to a Xylinx field-programmable gate array (FPGA) processor.    

Access & Use Information

Public: This dataset is intended for public access and use. License: U.S. Government Work

Downloads & Resources


Metadata Created Date August 1, 2018
Metadata Updated Date February 28, 2019

Metadata Source

Harvested from NASA Data.json

Additional Metadata

Resource Type Dataset
Metadata Created Date August 1, 2018
Metadata Updated Date February 28, 2019
Publisher Space Technology Mission Directorate
Unique Identifier TECHPORT_12118
Maintainer Email
Public Access Level public
Bureau Code 026:00
Metadata Context
Metadata Catalog ID
Schema Version
Catalog Describedby
Harvest Object Id 9ab5bb8d-5251-40df-af5d-9d5005c2d641
Harvest Source Id 39e4ad2a-47ca-4507-8258-852babd0fd99
Harvest Source Title NASA Data.json
Data First Published 2013-09-01
Homepage URL
Data Last Modified 2018-07-19
Program Code 026:027
Source Datajson Identifier True
Source Hash a8924307226c8ce469aa247038c066ffbddeb6b2
Source Schema Version 1.1

Didn't find what you're looking for? Suggest a dataset here.