Sophy: LLM Meets Robot¶
A local language model connected to a six-legged robot.
Sophy uses local inference to interpret sensor readings and propose actions. Software checks and operator approval control which commands reach the motors.
Local inference · Real sensors · Supervised motion
Hiwonder JetHexa
Local AI with supervised movement
From observation to authorized action¶
The model proposes. The agent validates. The operator authorizes physical movement. The robot checks its own permissions and clearance before executing.
Observation and approved movement ¶
Operator & development ¶
macOS workstation
Instructions → agent
Answer ← agent
Approval → agent safety gate
The operator selects the mode. The model cannot enable motion.
Local AI workstation ¶
Inference, agent & MCP
Jetson robot ¶
NVIDIA Jetson Nano · JetHexa
ROS publishes sensor data to the dashboard bridge. The gateway checks its age. MCP and the agent reject images with missing or stale metadata. After a slow model response, the agent reads safety observations again before requesting approval. The robot repeats its safety checks before executing.
How a request works¶
- Ask: the operator types “What is in front of you?” in the terminal or chat interface.
- Observe: ROS publishes LiDAR and camera data. The bridge records the readings and timestamps.
- Check freshness: the robot gateway rejects stale or unavailable frames. MCP exposes accepted observations to the agent.
- Reason locally: the agent supplies sensor facts and, when available, a fresh RGB snapshot to Qwen through Ollama. An image is a captured moment, not a continuous video feed.
- Respond or propose: the model answers, requests another allowed observation, or proposes a structured action. The agent validates the tool and its arguments.
- Authorize and act: in supervised mode, the agent refreshes safety observations and asks for approval of the exact forward command. Robot-side gates and clearance still apply. The agent then checks the execution status and reads new sensor data.
What works today¶
Implemented software¶
Local Ollama inference, structured tool calling, fresh-image handling, MCP observation tools, deterministic authorization, execution tracking, and operator-selected observe, plan, and supervised modes. Observe and plan never move the motors.
Physically verified¶
LiDAR, RGB/depth/IR sensing, servo telemetry and guarded tests. Forward gait testing was performed with the chassis supported and feet clear. Loaded-floor locomotion has not been verified.
Supervised / experimental¶
The agent can request one to five forward gait cycles after explicit approval and robot-side permission checks. The planner can describe a route without moving the robot. Other motion capabilities remain outside the agent’s allowlist.
Planned autonomy¶
Autonomous navigation is not implemented. Calibrated localization, loaded-floor gait validation, and continuous collision monitoring are prerequisites for future unsupervised movement.
Limitations¶
- Uncalibrated odometry. Position is estimated from commanded motion. A completed command does not measure how far the robot moved.
- Clearance checks run between movement segments. Obstacles are not continuously monitored during motion.
- Motion permissions are controlled outside the model. It cannot enable motion, change its mode, approve its own action, or start the sensors.
- Software stop and physical emergency stop are separate. Use the power switch for immediate physical intervention.
- Uncertain motion outcomes are not automatically retried. The agent checks the request ID for execution status.
Technical architecture¶
| Layer | Responsibility |
|---|---|
| Local inference | Ollama serves qwen3.5:35b on the AI workstation. The Jetson runs no LLM. |
| Perception & reasoning | The agent combines structured sensor facts with an optional RGB snapshot and validates each proposed tool call. |
| MCP tools | A local adapter exposes observations and permitted requests. The agent uses a restricted subset of the MCP tools. |
| ROS integration | ROS Melodic carries sensor topics. A dashboard bridge records samples. Freshness decisions belong to the gateway. |
| Physical actuation | A Python controller runs the gait and drives 18 bus servos outside ROS. |
| Safety boundaries | Operator-selected mode, per-action approval, robot-side motion gates, freshness checks, and stationary clearance checks. |
For implementation details, read the agent, MCP tools, and hardware status. Future work focuses on physical validation, measured localization, and collision monitoring before autonomous navigation.
Sophy is an independent Agentic Systems Lab project. Explore Agentic Experimentation Factory for the lab’s simulation and data infrastructure work.