17 · Next Frame Generation
Next frame generation produces a surgical image at a specified future time from the current view and supplied conditions, modeling instrument motion and scene changes.
Input and output: A current RGB frame, time interval and action/motion text → a future RGB frame.
Source data and labels: Video frames, action labels and instrument GT masks. This example includes “hook–dissect–omentum”; the blue overlays mark the hook in two frames, 1 second apart.
Construction method: Compare the same instrument's mask positions to form motion-direction text. Combine the current frame, interval and action/motion conditions as input; use the later frame as the target. Future frames and masks are used for offline construction only, not as additional visual inputs. This pair uses 1 second; the subset includes intervals of 1–10 seconds.
Figure 6. Blue overlays show the corresponding instrument masks. The generation task receives only the plain current frame and text conditions.
Example instruction: Generate the surgical image 1 second after the current frame. The action is dissection and the hook moves right.
Target output: The reference frame at t + 1 second, shown at the lower right.