Vision Pro Surgery

Overview

The Stanford BDHG Vision Pro Surgery repository contains two applications which together enable streaming of surgical video tower feed to the Vision Pro. The objective of this project is to allow for immersive and dynamic viewing of video feed during surgical procedures.

The repository consists of a Spezi Server app which enables users to interface their computers with a surgical video tower and start a local network video stream. The Vision Pro Surgery app enables fast and easy connection to the live video stream on the Vision Pro headset.

Required Hardware and Setup

To get started, you will need to have the following hardware components:

1. Computer (Mac or Windows)
   Used for running SpeziServer app
2. Vision Pro
   Used for running VisionProSurgery app
3. Bidrectional Cables
   • VGA to HDMI (product recommendation)
   • DVI to HDMI (product recommendation)
4. 4K Capture Card (product recommendation)

Note

• Please check the ports on your surgical video tower, some use VGI out and others DVI out.
• If you are using a Mac, use a 4K Capture Card that goes from HDMI --> micro-hdmi
• For Windows, use a 4K Capture Card that goes from HDMI --> USB
• We recommend purchasing this 4K Capture Card that works for both platforms: Capture 1080P60 Streaming Recorder (Amazon)

Tip

• For improved streaming quality and lower latency, we recommend using a router to repeat the main WiFi signal
• The GL.iNet GL-SFT1200 (Opal) router is what we use in our experimentation for its small size and portability

Software Installation and Setup

Step 1: Downloading Repository

git clone https://github.com/StanfordBDHG/VisionPro/

Step 2: Installing Spezi Server Dependencies

pip3 install -r /SpeziServerApp/requirements.txt

Step 3: Launching Spezi Server App

Open the Spezi Server app using one of the executable files or by running:

python3 /SpeziServerApp/spezi_server.py

Tip

• You can create a local executable of the Spezi Server app by running the create_exe.py script
• Ensure you have PyInstaller installed before running the script

Step 4: Launching Vision Pro Surgery app

Open the Spezi Vision Pro app (VisionProSurgery.xcodeproj) through the Xcode simulator or by running it on a physical Vision Pro (learn more here).

Starting Video Stream

On the launch screen of the Spezi Server app as shown below, you can configure the following settings:

1. Video Port ID: this depends on the video stream and varies based on the surgical video tower output
2. Width and Height: this will determine the size/resolution of the video stream that will be recieved by the Vision Pro
3. FPS (frames per second): adjust for smoother video viewing

Note

• Increasing the FPS/width/height can lead to more delays and higher latency in video streaming
• For higher FPS and resolution, you will need a very stable WiFi connection

Warning

• Before starting the stream, ensure that your computer is connected to a WiFi network that the Vision Pro can also be connected to
• Both devices will need to be on the same WiFi for video streaming to work

After launching, the SpeziServer app will show you the port and IP address that you will enter into the Vision Pro for connection.

Connecting Vision Pro to Video Stream

After running the Vision Pro app on either the simulator or physical device, you will be presented with the start screen as shown below.

After tapping on Get Started, you will get to the set-up screen where you have to enter your port number and IP address.

Note

• You will find your local port number and IP address from the Spezi Server computer app (see "Starting Video Stream")

Contributors

You can find a list of contributors in the CONTRIBUTORS.md file.