Zero-Shot Deformation Reconstruction for Soft Robots

This section describes the team’s paper method. Its cage-label supervision differs from the point-displacement loss in the training implementation described above.

Creating the Training Supervision

The model was trained on a 10 × 10 cm compliant fabric substrate with a 3 × 3 cm tactile sensor attached to its central region. A mechanical fixture produced repeatable bending and twisting configurations while keeping each selected pose fixed.

For each deformation type, 20 representative poses were captured with a structured light scanner. Each reconstructed shape was then rendered from 100 fixed virtual viewpoints in Blender. The camera parameters remained consistent across poses so that the rendered images shared the same spatial reference.

Because 4D Gaussian Splatting requires a continuous image sequence rather than a small set of isolated poses, RIFE was used to generate 8 to 16 intermediate frames between neighboring poses. This produced approximately 160 time steps for each virtual view and about 16,000 rendered images for each type of motion.

The rendered sequences and camera parameters were then processed with 4DGS. The result was a dense set of Gaussian primitives whose centers remained temporally corresponding as the object changed shape. These Gaussian trajectories provided the geometric supervision used to train the tactile deformation model.

Scanning, Blender rendering, RIFE, and 4DGS were only used to construct the training supervision. They were not required during deployment.

Processing the Tactile Measurements

The tactile patch contained a 10 × 10 grid of electrodes, a Velostat piezoresistive layer, and TPU encapsulation. The electronics scanned the 100 sensing channels and generated one resistance matrix for each sensor frame.

Raw voltage measurements were first converted into resistance values. Each channel was then normalized by dividing its measured resistance R by its resting resistance R₀:

\[ R_{\mathrm{normalized}}=\frac{R}{R_0}. \]

A low pass IIR filter reduced temporal noise, while a 3 × 3 median filter reduced isolated spatial noise in the resistance matrix.

A lightweight convolutional encoder converted the normalized 10 × 10 matrix into a compact tactile feature. The model also received a six band sinusoidal encoding of normalized time. This encoding indicated the current position within the deformation sequence. It was not a camera signal; it provided temporal context that helped the network distinguish sensor patterns occurring at different stages of bending or twisting.

The tactile feature and temporal encoding were concatenated before being passed to the deformation network.

Representing Deformation with a Control Cage

Predicting the displacement of every Gaussian primitive directly would require the network to output hundreds of thousands of motion vectors. The framework instead represented deformation with a regular control cage containing 15 × 15 × 15 nodes, or 3,375 control points.

Each Gaussian center p_j was connected to its nearby cage nodes c_i. The connection weights were calculated in advance using inverse distance weighting:

\[ w_{ij} = \frac{ \frac{1}{\lVert p_j-c_i\rVert+\varepsilon} }{ \sum_{i' \in \mathcal{N}_k(j)} \frac{1}{\lVert p_j-c_{i'}\rVert+\varepsilon} }. \]

A nearby cage node therefore had more influence on a Gaussian than a distant node. The weights were normalized so that the contribution from the neighboring cage nodes summed to one.

This representation reduced the prediction problem from a very large set of Gaussian displacements to a compact and spatially structured set of control point displacements.

Extracting the Cage Training Labels

The 4DGS sequence provided the observed displacement of every Gaussian center at each time step. However, the graph network needed target displacements for the cage nodes rather than for the Gaussians.

Because the relationships between the cage nodes and Gaussian centers were already defined by the inverse distance weights, the system recovered the corresponding cage motion by solving a least squares problem for every training frame:

\[ \min_{\{\Delta c_i\}} \sum_j \left\| \Delta p_j^{\mathrm{gt}} - \sum_i w_{ij}\Delta c_i \right\|_2^2. \]

Here, Δp_j^gt is the Gaussian displacement obtained from 4DGS, while Δc_i is the unknown displacement of a cage node. The solution produced the cage motion that best reproduced the dense Gaussian deformation. These cage displacements became the target labels used to train the tactile model.

This step connects the image based 4DGS supervision to the sensor based deformation network.

Predicting Cage Motion with Graph Attention

The deformation cage was represented as a graph. Each node corresponded to a cage control point and was connected to its six spatial neighbors.

The combined tactile and temporal feature was passed to a graph attention network. The network contained two GAT layers with four attention heads and 256 hidden units, followed by a GraphConv layer and an MLP. Its final output was a three dimensional displacement vector for every cage node.

Graph attention allowed each node to combine information from its neighbors while assigning different importance to different spatial relationships. This encouraged neighboring parts of the cage to move coherently while still allowing bending and twisting to produce different deformation patterns.

The cage therefore served two purposes. It reduced the size of the network output, and it introduced a spatial structure that discouraged disconnected or irregular motion outside the area directly covered by the sensor.

Training Objective

The network was trained with a cage regression term and a temporal smoothness term:

\[ \mathcal{L} = \frac{1}{N_c} \sum_i \left\| \Delta c_i^{\mathrm{pred}} - \Delta c_i^{\mathrm{gt}} \right\|_2^2 + \lambda \left\| \Delta c_i^{\mathrm{pred}}(t) - \Delta c_i^{\mathrm{pred}}(t-1) \right\|_2^2. \]

The first term trained the network to reproduce the cage motion obtained from the 4DGS supervision. The second term penalized sudden changes between neighboring time steps, reducing visible jitter while preserving the underlying motion.

Gaussian noise was added to the sensor input during training to improve robustness to measurement variation. Optional exponential moving average smoothing was also applied during inference to improve temporal stability.

Propagating Cage Motion to the Gaussian Model

During inference, the network predicted the displacement Δc_i of every cage node. The precomputed inverse distance weights were then used to propagate that motion to each Gaussian center:

\[ \Delta p_j = \sum_{i \in \mathcal{N}_k(j)} w_{ij}\Delta c_i. \]

This operation allowed a relatively small number of cage nodes to control a dense Gaussian representation. It also propagated deformation beyond the sensor area while maintaining smooth transitions between neighboring regions.

The system therefore did not attempt to predict every point on the robot independently. It predicted a structured cage motion and used geometry to expand that motion across the complete object.

Rendering the Reconstructed Deformation

After propagation, the system updated the centers of the Gaussian primitives and rendered the resulting deformation with a 3D Gaussian rasterizer.

Only the Gaussian positions were updated at runtime. Their color, opacity, and covariance remained fixed from the representation prepared during training. The renderer therefore did not need to reconstruct the complete visual model for every incoming sensor frame.

This design allowed the system to run at approximately 30 FPS with the coarse Gaussian setting and 5 FPS at high resolution on a single NVIDIA RTX 3070 Ti.

Deployment on Unseen Soft Robots

The tactile model was trained only with deformation data collected from the compliant fabric platform. It was later deployed on two soft robots that were not present in the training data. One robot performed pneumatic bending, while the other performed pneumatic twisting.

For each robot, the system began with a static model of its undeformed geometry, such as the STL file used during fabrication. This geometry was converted into a canonical Gaussian representation, and the sensor region was aligned with the corresponding cage region.

The same tactile patch and trained network were then used directly. Each resistance matrix was encoded, the graph network predicted the cage motion, inverse distance weighting updated the Gaussian centers, and the renderer displayed the reconstructed deformation.

No deformation dataset was collected for either test robot. The network was not retrained or fine tuned for their geometry, and no camera image or rendered supervision was supplied during inference. This is the specific meaning of zero shot deployment in this project.