Diagnostic and Clinical Endoscopy Processing

Deliver Intelligence and best performance per watt on superior
ISP and enhanced vision systems for surgical and GI video endoscopes


Advances in preventive medicine and minimally invasive surgeries are driving the need for enhancements in endoscopy systems. Traditional multichip implementations or general-purpose GPUs introduce latency which is counter-productive to the low-latency, high-bandwidth computationally intensive functions required in modern endoscopy.

A typical endoscopy system consists of an endoscope with a light source and a camera head, a camera control unit or CCU and an image management unit.

The camera head is connected to a light source. The endoscope with the camera and light source is inserted to take images of the concerned area. In the Camera Control Unit or the CCU, after performing image signal processing, the raw uncompressed images are captured and displayed real-time or near-real-time on one or more display monitors. The video or images are compressed and stored either in a DVR or other storage medium in various formats for later use and record keeping. Typically, the image management unit will perform video processing like encode and decode, jpeg conversion, do picture overlay, manipulation, picture-in-picture and other functions and compress and store the images for possible future use.

In a diagnostic GI video endoscope, the latency requirement is 50-150 milliseconds for an image to be captured, transferred, pre-processed and displayed. Surgical procedures using endoscopes needs a real-time or close-to real-time response. Various picture correction, color and noise corrections take place here followed by edge enhancement and scaling.

As the camera resolution requirements move from 2K to 4K to 4K-3D or even to 8K and AI-ML functions are required for assistance, the newly announced Xilinx Versal AI Edge Series ACAP devices can handle such extremely high-performance pre-processing while delivering the required low latency while maintaining a suitable thermal envelop.

Design Example

This design example discusses architecting a 4K video endoscope with Xilinx solutions. As mentioned above, an endoscopy system typically has a processing system consisting of a light source and a camera head, a camera control unit or CCU and an image management unit. This system supports a dual 4K video stream.

The dual-4K image sensor on the camera head performs the image capture and the image signal processing is done in a Xilinx FPGA/SOC.

The video streams are then fed to a one or more FPGAs-SoCs devices like Versal AI Edge or UltraScale+ devices to perform image pre-processing and the streams are displayed near real-time using a DisplayPort or quad SDI interface to high definition 4K monitors for the surgeon to view. The Camera Control Unit performs picture, color and noise corrections including white balance, automatic brightness, gain control, IRIS control etc. Some of the typical pre-processing functions are shown here. There are different ways to partition the pre-processing functions depending on what the requirements are.

Using Xilinx AI Edge or Zynq Ultrascale+ provides several distinct advantages:

  • Low latency, high bandwidth provides increased system performance with hardware acceleration for video analytics such as lens distortion correction, low latency rotation, point of view translation, object identification, comparison, counting, recognition, positioning and measurement
  • Integration of single-chip system processor and image processing reduces BOM cost and power consumption
  • Optimal hardware/software partitioning reduces latency and bottlenecks by combining image/video processing, analytics and camera control into a single device
  • Support for multiple connectivity standards – Quad SDI, HDMI, DisplayPort, SATA, USB, Ethernet and more

As part of post processing, various image management functions are performed on the raw video using an SOC like ZU+ MPSOC with built-in video codec engines. Xilinx solutions today have significant platform opportunities to be major part of a video endoscopy processing system.

  • 4K60 video encoder for network-based video transfer
  • Accelerate design productivity with a tightly integrated development tool flow using OpenCV, Vivado™ High-Level Synthesis, and IP Integrator tools
  • Reduce time to integration with specialized ecosystem IP and software, and operating systems

Typically, SoC-FPGA devices can produce much higher performance, bandwidth and real-time capabilities for image pre-processing functions over an ASSP device.  GPUs are widely used for image processing at the back-end, but Xilinx FPGAs can dissipate significantly lower power than GPUs.

Additionally, Xilinx is focused on reliability, safety, security and long-life support necessary for clinical equipment unlike competing GPGPUs or processors.