Vision Language Action: Inference at the Edge

The Experiment

Why the Tower of Hanoi?

I assume if you are reading this website you have a fairly strong engineering background, probably in computer science, electrical engineering, computer engineering, etc. If so, then you have likely come across the Tower of Hanoi puzzle in undergrad and I bet you can recall when it all "just clicked" (I know I can!). For this work I needed a problem that was simple enough to reason about, but complex enough to prove out a system of systems without forcing a square peg into a round hole (see what I did there, peg, Tower of Hanoi?). I wanted something easy to conceptually understand, but at the same time not trivial to implement.

For the uninitiated and or a refresher, the classic Tower of Hanoi is a puzzle that consists of three pegs and a stack of game pieces arranged by size. The objective is to move the entire stack from one peg to another, following three simple rules:

1. Only one game piece can be moved at a time.
2. A larger game piece can never be placed on top of a smaller game piece.
3. All game pieces must be moved from the source peg to the destination peg, using an auxiliary peg.

Despite these simple rules, the puzzle produces a sequence of dependent decisions and a myriad of possible solutions where the minimal amount of moves can be expressed as the function f = { 2ⁿ-1 } where n is the number of game pieces. Therefore a tower with 3 game pieces has a minimum set of moves of 7 where (2³ - 1) == 7. There is a killer explanation over at Khan Academy: Tower of Hanoi Explained, which is where I sourced the image below.

Tower of Hanoi is useful because the rules of the puzzle don't require a machine-learning model, they can be represented deterministically in software. Once the current state of the board is known, conventional game logic can determine the next legal move. Additionally, I can easily decouple the game state; next-move generation; and controlling the robotic arm; thus allowing me to make this an extremely interesting example of a system of systems. Who would have thought? Well, I did!

Physical Limitations

As described above and shown in the image from Khan Academy, the classic Tower of Hanoi utilizes discs that are placed on 3 pegs. It resembles a child's toy with its simplicity. But, if you read Part 1 then you know that this research uses a less-than-precise robotic arm, the SO-101. As we have seen, this robot is a fantastic learning platform but it's dexterity is, well, it's just awful. But for sub $300 what did you expect? This means grasping the traditional discs is really out of the question.

I needed a way to physically solve the puzzel without having to grab discs from a peg. After days of riffing on ideas it dawened on me that I could compbine the peg and game piece, making manipulation orders of magnitude easier. See the images below:

In this design, the robot can pick a game piece by grasping the "peg" rather than a disc threaded onto a peg. The SO-101 then moves the piece to the correct location and drops it onto the peg extending from the piece below it. This design has the added benefit of self-centering each piece as they are dropped. A short video will make this clear:

System Architecture

Now that we understand the Tower of Hanoi, we can decompose the puzzle into a set of individual problems and map those problems onto one of the available processors within the target system (e.g. Intel Core Ultra 3). The important point is that the system does not treat the CPU, integrated GPU, and neural processing unit as interchangeable processors. Each device is designed for a fundamentally different class of computation. The goal for the system architecture is therefore to assign each part of the problem to the processor that can perform it most efficiently.

Central Processing Unit

The CPU is the general-purpose processor and acts as the system coordinator. Unlike the GPU and NPU, the CPU is designed to execute arbitrary sequences of instructions, including branches, loops, system calls, I/O operations, and application logic. This makes it particularly well suited to tasks that require decisions, coordination, and interaction with the rest of the system.

Integrated Graphics Processing Unit

The integrated graphics processing unit (iGPU) is a highly parallel processor designed to perform computations across large amounts of data simultaneously. Unlike the CPU, which is optimized for general-purpose sequential and control-oriented workloads, the GPU contains many parallel execution resources that can perform the same or similar operations on many pieces of data at the same time.

A standard discrete GPU (dGPU) is typically installed on a PCIe expansion card, containing its own dedicated high-bandwidth memory. For example, an Intel Arc Pro B70 contains its own GPU and 32 GB of GDDR6 memory. The dGPU therefore has a dedicated memory subsystem that is physically separate from the system memory used by the CPU.

By contrast, an integrated GPU (iGPU) is incorporated into the processor or system-on-chip and does not normally have a separate pool of dedicated graphics memory. The GPU instead shares the system's main memory with the CPU and other components of the processor. This avoids the need for a separate GPU memory subsystem and allows the GPU and CPU to operate on data within the same system-memory environment.

A discrete GPU can provide substantially greater memory bandwidth and dedicated memory capacity, making it well suited to large, compute-intensive models.

Neural Processing Unit

The Neural Processing Unit (NPU) is a specialized accelerator designed specifically for neural-network workloads. An NPU sacrifices much of the general-purpose flexibility of a CPU, and some of the programmability of a GPU, in exchange for highly efficient execution of the mathematical operations commonly found in neural networks. This allows neural-network inference to be performed with significantly greater power efficiency than would typically be possible using a general-purpose processor.

Mapping the Problem to the Processors

With the characteristics of each processor established, we can now return to the Tower of Hanoi and examine the individual problems that make up the system. The complete operation can be thought of as a pipeline:

1. The camera observes the board
2. The system determines the current state
3. The game logic determines the next legal move
4. The system describes the state to the user
5. A neural-network policy translates the next move into robot actions
6. The robot executes the movement.

Although these stages are part of a single application, they do not perform the same type of computation. Some require general-purpose program execution and decision-making. Others involve large amounts of mathematical computation that can be performed in parallel. Still others are specifically neural-network workloads.

1. The camera observes the board

The camera itself is not a processor workload in the conventional sense. It produces a stream of image data that must be captured and made available to the rest of the system. The CPU manages the camera interface and the movement of image data through the application. Once an image has been captured, however, the computational work required to interpret it is better suited to an accelerator.

2. The system determines the current state

Determining the state of the physical board is a computer-vision problem, but it does not require a neural network. The system uses conventional computer-vision techniques to analyze the camera image and determine the position of the game pieces, processing this workload on the CPU. Unlike some similar systems, we are conciously not asking a vision language model to decide where the pieces are and then trust its interpretation as the authoritative game state. Our board has a known physical structure and the pieces occupy known positions, allowing the state to be determined deterministically from the camera image.

3. The game logic determines the next legal move

Once the state of the board is known, determining the next move is fundamentally different from recognizing the board. The rules of the Tower of Hanoi are deterministic. Given the current state, the application can calculate which move should occur next using conventional program logic. This involves comparisons, conditional branches, data structures, and state management. This is exactly the type of workload for which the CPU is designed.

4. The system describes the state to the user

A vision language model provides a natural-language interface to the system, allowing the system to describe what it sees and communicate the state of the game to the user. This is a neural-network workload and is therefore well suited to execution on the NPU.

The key consideration here is the frequency and timing requirements of this inference. The VLM is not part of the robot's real-time control loop. There is no need to run it every 200 milliseconds, or even continuously. The system can invoke the VLM approximately once every one to two seconds, when an updated natural-language description is useful to the user. This makes the NPU a good fit. Its dedicated neural-network hardware can perform the VLM inference efficiently without occupying the GPU, while the GPU remains 100% available for the VLA, which has much more demanding timing requirements.

5. The neural-network policy translates the move into robot actions

The game logic produced a discrete instruction such as: Move the game piece from A3 to C1. The robot, however, cannot execute that instruction directly. It requires a continuous sequence of joint positions and gripper movements that describe how to physically perform the action. This is the role of the Vision-Language-Action (VLA) policy.

Like the VLM, the VLA is a neural network and therefore contains substantial amounts of tensor and matrix computation. However, the characteristics of this particular workload make the integrated GPU a good target. The GPU provides a large number of programmable parallel execution resources and is capable of efficiently executing the tensor operations required by the VLA. Unlike the NPU, it also provides a highly programmable environment that is well suited to the continuous and iterative inference workload of the robot policy.

6. The robot executes the movement

Finally, the CPU coordinates communication with the robotic arm and sends the generated actions to the robot. The actual physical movement is performed by the robot's motors and controllers, not by any of the processors in the Core Ultra system. The CPU manages this interaction, including timing, communication, and monitoring of the robot's execution.

A Deeper Look

Let's dive one level deeper into each major component.

Vision Language Action

The VLA is the component that determines how the robot moves. Recall our system design. I am not asking the VLA to solve the Tower of Hanoi, understand the rules, or even determine the state of the board. Those problems are being handled elsewhere. The VLA has a much simpler job: given a natural-language instruction describing a single legal move, determine the robot moves required to execute it. For example, the system may determine that the next move is: Move the game piece from A3 to C1.

That instruction becomes the task presented to the VLA. It receives the camera observations and the instruction, and generates the sequence of robot actions required to locate the piece, grasp it, move it to the destination, and release it.

Similar to Part 1 of this research, I trained the SmolVLA policy specifically for this type of pick-and-place operation using demonstrations with the SO-101. The training data focused on the physical manipulation task rather than the game itself. In other words, I am teaching the model how to perform the movement, not which movement it should choose. This keeps the policy focused on a relatively constrained problem and allows the rest of the system to handle the higher-level reasoning.

At runtime, the trained policy runs on the integrated GPU as described above. The CPU provides the observations and movement instruction, the VLA generates its action sequence, and the CPU then feeds those actions to the robot. Once the movement is complete, the system can observe the board again and determine whether the expected state transition actually occurred.

Vision Language Model

The Vision Language Model has a very different job in this system. I deployed SmolVLM to the NPU, but I deliberately did not use it to implement the game logic. Its job is to look at the board and describe what it sees. That sounds simple, but it provides a useful interface between the physical world and the rest of the system. The VLM can describe the current arrangement of the game pieces, identify what is visible on the board, answer questions about the pieces, and provide a natural-language description of the current scene. This capability allowed me to make the system a bit more interactive where I can questions about what the camera is seeing such as:

• Which pieces are on peg A?
• What is currently at position A3?
• Where is the red game piece?
• Describe the current state of the board.

The VLM generates the answers from the visual input and essentially provides a natural-language interpretation of the physical environment.

Computer Vision and Game Logic

This is where conventional software comes back into the picture. While the VLM is responsible for describing the board and providing a natural-language interface to the physical environment, I wrote a conventional computer vision algorithm to determine the actual game state. The cv system looks at the camera image and determines where the game pieces are located on the board. It identifies the pieces and maps their locations into the coordinate system used by the rest of the application: A1, A2, A3, B1, B2, B3, C1, C2, and C3.

One challenge with this is I can't simply look at a single frame and assume that it represents the current state of the board. The robot may still be moving a piece and the camera may capture the board while something is in motion. I therefore added a debouncing routine that waits for the board to reach a steady state before accepting the detected positions as the current game state. It's by no means perfect, but does the job for this demonstration.

Once the board is stable, the detected piece locations are mapped into an array representing the current board state. The game logic then compares that state against the known set of valid moves and determines what should happen next. The result is a simple source and destination instruction, such as Move the game piece from A3 to C1. This instruction is passed to the VLA, which generates the robot actions required to execute the move.

Training Data

Now that the system architecture is defined, the next question is: what data does the VLA actually need? As mentioned multiple times above, since the VLA is only responsible for physically moving a game piece from one location to another, I don't need to train it on the rules of Tower of Hanoi or the complete game. I only need demonstrations of the physical movements the robot will be asked to perform. Therefore, I trained the model on each legal movement required to move the tower from peg A to peg C. There are 18 unique movements in the complete sequence: 7 moves to solve the puzzle from A to C, and 11 moves to return from C to A. Wait, why 11 if the minimum solution for a three-piece Tower of Hanoi is only 7 moves. Let's look at how the game logic determines those moves.

Recall that I am using computer vision to determine the state of the board, and the game logic uses that state to deterministically generate the next move. If I simply reverse the seven moves from A to C, I would eventually arrive at board states that the system has already seen on the way there. For example, the final move from A to C is Move A1 to C3, leaving all three pieces stacked on C. If I then reverse that move and move the top piece back to A1, I have recreated a board state from the previous sequence. Since the game logic has one defined move for each state, I need to avoid previously visited states. Instead, the return sequence takes a different legal path, resulting in a nine-move sequence from C back to A where no states are repeated.

For each of the 16 movements, I recorded 50 demonstration episodes using the SO-101 as previously seen in Part 1. Each of these movements were recorded as its own dataset which I merged together before training the model, keeping the non-merged training data for future use. This gives me some flexibility as I continue testing the system. If I find that one particular movement isn't performing as expected, I can add more demonstrations for that movement without having to re-record 800 episodes (50*16). If necessary, I can also replace an individual dataset entirely and retrain the model with an updated merged collection.

Training, Re-Training, and Training Some More

I'm sure you have heard the mantra "you need clean training data for imitation learning" over and over again. This is true, and it is extremely important, but sometimes there is more at play than just clean data.

Take the third move, C3 to B2. This picks the smallest game piece from the farthest position on the board relative to the camer, and moves it to the middle position where it is placed on top of the green game piece already there. I went to great lengths to record this movement without interfering with the world camera. I even used two hands on the robot arm to steady it during the demonstrations, and after reviewing the data and associated videos, all looked good.

After training, however, the policy missed this move more than 90% of the time. It almost never successfully grasped the game piece. My first thought was simple: I need more data. So I threw out the original dataset (in case it was bad) for that one move and recorded 100 new episodes, twice as many as I originally recorded. I then retrained the policy, tested it again, and... zero improvement. I think it was actually worse, but that's subjective as I didn't actually record the stats during evaluation.

I went back and watched the demonstration videos carefully. Nothing obvious jumped out at me. The demonstrations looked clean, the robot was stable, and the movement looked consistent. I started wondering if the position of the world camera was making depth estimation difficult at that particular angle, especially since this was the smallest piece and the one farthest from the camera. Then I had another thought: what if the orientation I trained for this move was simply too difficult for the policy?

It was a bit counterintuitive, but I threw out the 100 new episodes and recorded another 50. This time, I approached the game piece from an easier-to-manipulate angle, even though that meant slightly occluding the world camera during the grasp. And guess what? It worked. The policy successfully performed the movement more than 95% of the time.

So, in the end, some of imitation learning really isn't an exact science, or at least it doesn't feel like one from this side of the mathematics. Sometimes the answer isn't more data. Sometimes the answer is changing what you are asking the model to learn.

Below is a quick video walking through this, and a few other tips that may help.

OpenVINO

At this point, let's fast forward a bit... I have described the architecture above, and we already discussed training SmolVLA in Part 1. So, let's assume we have the SmolVLA Vision Language Action policy trained on the NVIDIA CUDA stack and that I selected the underlying SmolVLM as my Vision Language Model for scene descriptions. Carry-on...

SmolVLA

Now I have a trained VLA model that works but there is just one problem: I didn't train it on the Intel edge device where I want to run it. The model was trained on my NVIDIA DGX Spark inside of the CUDA ecosystem. That's a great environment for model development and training, and it's the environment most AI development is currently executed, but it isn't the target environment for my system. The goal of this experiment is to run the complete Physical AI system at the edge on an Intel Core Ultra 3 platform, using the CPU, integrated GPU, and NPU as described in the System Architecture section above. Therefore, I needed to figure out how to deploy my work on a different hardware stack... hm.

This is where OpenVINO becomes important. OpenVINO is a free, open-source software toolkit by Intel used to optimize and deploy deep learning and AI models on multiple stacks. That is, I need to take the model I developed and trained in the NVIDIA ecosystem and transform it into something that can be deployed efficiently on Intel hardware. The model itself doesn't change its job, it is still the same VLA policy, but the way that model is represented and executed has to change to match the target hardware.

If you are familiar with embedded systems, this is similar to cross-compilation, where we develop the model is different from where we where we deploy it. In reality, the hardware used to train policies and the hardware used to run AI inference have very different requirements. I don't want my deployed system to require a physically large, hot, and costly, discrete GPU simply because that is where the model was trained. I want to train the policy where it is efficient to do so, then deploy where the application actually needs to run... at the edge.

Surveying what I actually needed at runtime made it clear that the entire LeRobot framework was unnecessary, and I certainly did not need any of the training code. At runtime, the VLA has one job: take the camera observations, the language instruction, and the current robot state, then generate the next chunk of robot actions. The challenge was therefore to identify exactly where that operation occurs within the much larger LeRobot and SmolVLA implementation so I could isolate it for deployment on the target hardware.

I started at the LeRobot policy level and traced the inference path down through the SmolVLA implementation rather than trying to guess which part of the framework was required. This led to the VLAFlowMatching model in modeling_smolvla.py, where I found the sample_actions() method. Unlike the surrounding policy code, this method contains the actual action-generation process. It takes the images, language tokens, masks, robot state, and noise, performs the prefix embedding and VLM processing, runs the flow-matching denoising loop, and finally returns the action chunk.

That was the boundary I was looking for. The code above sample_actions() is concerned with preparing inputs and managing the policy, while the code inside sample_actions() performs the computation that actually turns those inputs into robot actions. I therefore did not need to reproduce the LeRobot policy framework; I only needed to make this existing inference operation available to the OpenVINO conversion process.

But there was a complication. sample_actions() is already part of the loaded SmolVLA model, but it is not the model's standard forward() entry point that OpenVINO requires. PyTorch modules are normally invoked through forward(), and the PyTorch-to-OpenVINO conversion process uses that module interface to determine the computation it needs to convert. Simply giving the converter the existing model would therefore expose its normal forward() path, not the specific sample_actions() path I had identified as the runtime operation I wanted to deploy.

The solution was a very small adapter class. OpenVINO requires an object derived from torch.nn.Module, so I extended that class and implemented forward() to simply call the existing model's sample_actions() method. The adapter does not create another SmolVLA model or contain another set of weights. It simply holds a reference to the model loaded from the checkpoint and redirects the module's standard forward() call to the existing action-generation method. A snippet from the class I wrote can be seen below:

The forward() method is important because OpenVINO is not converting the Python class itself; it uses the method as the entry point to trace the computation that needs to be converted. When the OpenVINO converter invokes forward(), the call is now passed directly into SmolVLA's existing sample_actions() implementation. OpenVINO can then follow the operations performed by that method, including the model's embedding, VLM processing, and flow-matching denoising operations, and represent them as an OpenVINO computation graph. Once converted, that graph can be serialized as an OpenVINO model and executed without the original LeRobot policy machinery. The forward() method therefore isn't part of the VLA inference algorithm; it provides the entry point OpenVINO needs to identify and capture the inference computation I want to deploy. Here is a snippet from the simple class extension:

    
class SmolVLAInferenceWrapper(torch.nn.Module):

    def __init__(self, policy):
        super().__init__()
        self.model = policy.model

    def forward(
        self,
        images,
        img_masks,
        lang_tokens,
        lang_masks,
        state,
        noise,
    ):
        return self.model.sample_actions(
            images,
            img_masks,
            lang_tokens,
            lang_masks,
            state,
            noise=noise,
        )
    

With the inference boundary isolated, the next step was to convert that PyTorch computation into a form that could run natively through OpenVINO. This is where the ov.convert_model() function comes in:

    
ov_model = ov.convert_model(
    model,
    example_input=example_inputs,
)
    

Here, model is the small adapter class I created above, not another copy of SmolVLA. OpenVINO invokes its forward() method using the supplied example inputs. That call immediately enters the existing sample_actions() implementation, allowing OpenVINO to capture the computation that produces the action chunk. The example inputs provide the converter with representative tensors from which it can determine the inputs, shapes, data types, and operations involved in the computation.

OpenVINO then constructs its own representation of that computation as an OpenVINO model graph The trained SmolVLA parameters are carried into this representation along with the operations that use them. What began as a PyTorch model executing Python and PyTorch operations is therefore transformed into an OpenVINO graph that describes the neural-network computation independently of the original LeRobot policy runtime that can be executed on Intel hardware.

Once the conversion is complete, the resulting OpenVINO model can be serialized to disk:

    
ov.save_model(
    ov_model,
    "smolvla.xml",
)
    

The above produces the OpenVINO model files, with the XML file describing the graph and the associated binary file containing the model parameters. At this point, the original LeRobot policy object is no longer required to execute the converted inference graph. The deployment runtime can load the OpenVINO model directly and provide it with the same inputs—camera observations, language tokens, masks, robot state, and noise—to produce the action chunk.

In effect, the conversion process takes the inference computation I identified in the SmolVLA source code, captures it through the adapter's forward() interface, and turns it into a self-contained OpenVINO model that can be deployed on the target hardware.

SmolVLM

With the VLA running on the iGPU, I now have one more piece of the system to deploy. Remember that SmolVLA isn't operating by itself, it actually contains a Vision Language Model under the hood that provides the vision and language understanding needed by the action policy. For this system, I am also using that same family of models separately to provide scene descriptions for the Tower of Hanoi game.

The model I selected is SmolVLM2-500M-Video-Instruct. It is a relatively small Vision Language Model, which makes it an interesting candidate for edge inference on the NPU. In my system, its job is very different from SmolVLA. The VLM looks at the camera images and describes what it sees. It doesn't move the robot, and it doesn't determine what move the game should make. And I don't really care if it is slow as its output is not part of the control system, it's strictly for information purposes and really just to prove that we can run something like this on the NPU. In theory, I could put both models on the same accelerator (iGPU), but why? Since I can distribute the workloads across multiple devices, let's do it!

So now I have a different deployment problem. With SmolVLA, I started with a model I had trained myself and had to expose the specific inference path that generated the action chunk to OpenVINO. With SmolVLM, I am starting with an existing vision-language model and need to determine how its inference process can be represented in OpenVINO and executed on the NPU... not too dissimilar really.

The first question is therefore the same one I asked with SmolVLA, but the answer is going to be different: what exactly do I need to export?

Live System Visualization

At this point, the models are running and the different pieces of the system distributed across the Intel hardware, but there was still a problem... most of what was happening was invisible. The CPU was orchestrating the system, the NPU was interpreting the camera images, the iGPU was generating robot actions, and the SO-101 was physically executing those actions, but unless you were looking at the source code or the logs, you couldn't really see how all of those pieces were working together. This system was doing a very poor job of communicating to the world how great it is!

For the demonstration, I wanted everyone to see the system, to visualize what is going on under the hood. To that end, I built a small web-based interface that provides a live view of the system while the robot is running. I wanted this interface to provoke questions during the demonstration and, because things always go wrong in demos, give me a way to explain what happened using the system's actual output while showing how the robot responded. The simple UI seen below did the trick.

What's Next?

So... where does this leave us?

I started this experiment with a fairly simple question: can I take a Vision Language Action model that was developed and trained on a NVIDIA CUDA system and put it to work as part of a Physical AI deployment at the edge targetting Intel Core Ultra 3?

The answer is a resounding yes.

The interesting part for me (and hopefully, for you!) was figuring out how all of the pieces fit together. Training the VLA was one problem. Getting it onto different hardware was another. Understanding what the model was doing, deciding what should be handled by a model and what should be handled by conventional software, and then getting all of those pieces to operate together in real time... that's where this experiment became interesting to me.

The Tower of Hanoi gave me a useful way to test all of this because the problem is deliberately constrained. I know what the legal moves are. I know what the robot is supposed to do. I can observe the board before and after every move. That makes it possible to see exactly where the system succeeds and where it doesn't. And things don't always work.

That's actually one of the reasons I built the live interface. When something goes wrong, I don't want to simply say that the robot made a mistake. I want to know what the system saw, what the VLM reported, what move the game logic selected, what instruction was given to the VLA, and what the robot actually did. That's a fantastic conversation starter... and having presented technical demonstrations for the better part of two decades, I've found that there is nothing better than being able to explain why something failed and then riff on ideas for how the system could be improved. A failure becomes part of the demonstration rather than something you have to hide.

And that's really the point of this exercise. I'm not particularly interested in building a robot that can play the Tower of Hanoi. I'm interested in understanding what it takes to build a Physical AI system that can see, reason, and act on an edge device... and then figuring out what I can make it do next.

Onward...