UPenn PRONTO Team Uses Meta SAM and DINO for AI-Powered Disaster Medical Triage

The Penn Robotic Non-contact Triage and Observation (PRONTO) team brings together experts in trauma from Penn Medicine as well as experts in robotics and computer vision from Penn Engineering and the General Robotics, Automation, Sensing, and Perception (GRASP) Lab at the University of Pennsylvania. By combining cutting-edge robotics with Meta SAM and DINO models, the PRONTO team enables autonomous, rapid detection and assessment of injuries in disaster scenarios. Given advancements in computer vision, robotics, and machine learning, DARPA announced a three-year challenge to spur innovation in medical triage. Using stand-off sensors onboard autonomous systems, the goal is to detect and identify physiological signatures in mass casualty incidents. PRONTO deployed the initial version of their system using a drone to quickly survey the scene to locate victims and a ground robot for more stable imaging and vital sign capture. Data was processed using Meta Segment Anything Model 2, which enables segmentation of objects in images or video. Unlike traditional models, DINO does not require labeled data, making it more efficient and scalable for a variety of tasks. The model can generalize across diverse domains including medical and satellite imagery, and can excel in environments where annotated data is scarce or unavailable. PRONTO uses DINO to extract visual features from robot images, which are then used to identify injuries via a customized deep neural network. DINO works alongside SAM and Grounding DINO to provide a comprehensive triage solution. Together with additional modules that estimate body pose and use wound-to-skeletal comparison algorithms to assess injury, the PRONTO multi-robot system detects and characterizes injuries, identifying a patient heart rate, respiration rate, awareness, and presence of wounds or amputations.