Extending Agentforce's Atlas Reasoning Engine into embodied AI for enterprise facilities management
Designing a solution using NVIDIA's perception stack and Agentforce's Atlas Reasoning Engine to implement a Cobalt AI facilities management robotics solution. It integrates NVIDIA's visual language models and Cobalt's mobile robot platform with Atlas's multi-step reasoning layer. The result is a Robot Safety Inspector proof of concept, where Atlas orchestrates autonomous room inspections for safety equipment, running entirely on local hardware.
System Architecture
Atlas plans and dispatches the robot. The robot physically inspects the room. NVIDIA's visual language pipeline reads what the robot sees, and everything routes through a DGX Spark unit for fully local processing. Video analysis never makes a round trip to the cloud.
Atlas dispatches and directs the robot; the robot inspects the room; NVIDIA's VLM pipeline reads the video feed for objects relevant to safety; DGX Spark keeps every step of vision processing on local hardware.
NVIDIA Blueprint
The full blueprint behind the Cosmos VLM and LLAMA pipeline referenced above. An ingestion pipeline turns raw video and audio into dense captions, CV metadata, and transcripts. A retrieval pipeline then answers questions, generates alerts, and produces summaries by combining a Vector RAG store with a Graph RAG knowledge graph, all behind NeMo Guardrails.
Camera and file video sources are chunked, demuxed, and split into a video branch (decode, frame selection, CV tracking, VLM NIM) and an audio branch (decode, convert, Riva ASR NIM). Both branches feed a multimodal dense caption used by the retrieval pipeline below.
The user's question passes through NeMo Guardrails to a Context Manager, which holds short term memory and can call a third party LLM NIM directly. Response Handlers query Vector RAG (reranking and a vector database) and Graph RAG (a graph retriever and knowledge graph database) in parallel, then route the combined answer back as a Q&A response, an alert, or a summary. Both RAG stores are populated from the multimodal dense caption produced by the ingestion pipeline above.
Robot Safety Inspector
Atlas orchestrates autonomous room inspections for safety equipment, closing the loop from detection to action without a human in the middle of the routine check.
Plan
Atlas's Task Planning, Topic Selector, and Action Orchestrator components determine what needs inspecting and dispatch the robot.
Move
The robot, powered by a Jetson Orin Nano, handles navigation and motion control, odometry and localization, and pose estimation as it moves through the space.
See
Video from the inspection runs through NVIDIA's Video Search and Summarization Blueprint, using Cosmos VLM for visual understanding and LLAMA for reasoning over what it sees. All of it runs on DGX Spark, with no round trip to the cloud.
Evaluate
Object detection identifies safety equipment in frame; safety evaluation checks it against what should be present.
Act
An inspection report is generated. When equipment is missing, Agentforce creates a Case and dispatches a Field Service Technician. The loop closes from autonomous detection to human action automatically.
Solution Stack
The multi-step reasoning layer plans the inspection, orchestrates the robot, and turns a detected gap into a dispatched action.
The mobile robot platform that navigates and inspects facilities autonomously, delivered as the facilities management robotics solution.
Vision capability via the Video Search and Summarization Blueprint, running Cosmos VLM and LLAMA locally on DGX Spark and Jetson Orin Nano.
All video analysis happens on device, and no footage leaves the facility for the model to reason over it.