Projects

Reading time: 6 minutes and 24 seconds

Current projects

MarginDx

Real-time intraoperative assessment of cancer margins using advanced optical imaging and robotics techniques.

Overview

Margin Diagnostics (MarginDx), funded through the Advanced Research Funding Agency for Health (ARPA-H) Precision Surgical Interventions (PSI) program, combines optical imaging technologies with AI screening tools to ensure, in real time, that tumor tissue and cells are completely removed during surgery.

Individual Research Contributions

As the robo-optics lead, I am responsible for developing fundamental techniques and hardware required to bridge the gap between control theory and robotics, and biophotonics in in vivo imaging to operate under the effects of tissue deformation and motion. The project aims to develop a state-of-the-art device for high-resolution intraoperative optical imaging to enable cell-level identification of cancer margins and other forms of pathology, by leveraging nonlinear optical techniques and optical coherence tomography, coupled by adaptive precision robotic position and control. As part of my role, I have made the following contributions to the project:

Overview of the handheld and robotic components of the MarginDx system. I am responsible for the co-registration of images and pose between the handheld and robotic data, as well as for the complete robotic position and microscopic scanning hardware and software.

Spatiotemporal Distortion Mitigation for Low-Rate Single-Pixel Imaging in Fast Dynamic Large-Area Microscopy ↕

Developed novel approaches to mitigate temporal aliasing and motion-induced distortions in single-pixel and line-scan-based confocal imaging systems, addressing extreme scale separations between pixel dwell time and scanning motion speed.
Enabled high-resolution, large-area imaging of unstructured surfaces, facilitating the translation of confocal microscopy techniques (e.g., OCT, fluorescence imaging, nonlinear optics) to applications such as intraoperative diagnosis, material testing, and portable biological sensors.
Leveraged control theory to model scan mirror and exogenous motion uncertainties, enabling maximum likelihood reconstruction of anti-aliased images under spatiotemporal aliasing scenarios.
Introduced high-resolution pose estimation for handheld imaging probes using electromagnetic tracking systems with multiple distributed sensors.
Pioneered a real-time GPU-accelerated framework for low-rate, high-resolution imaging in dynamic settings with motion uncertainty, marking a first in the field.

Soft Body Deformation Compensation and Reconstruction using Intrinsic Elastography through Contact-Based Imaging ↕

Developed a novel framework to compensate for soft tissue deformation during contact-based imaging, enabling accurate intraoperative microscopic diagnosis by preserving spatial fidelity under mechanical loading.
Introduced intrinsic elastography techniques to estimate tissue elasticity in real-time, leveraging the imaging probe’s contact force, optical coherence tomography imaging features, and deformation dynamics inferred through dynamic loading to reconstruct undistorted tissue morphology.
Designed algorithms to model and correct for nonlinear tissue responses, ensuring robust reconstruction of microscopic features in the presence of varying contact pressures and probe orientations.
Integrated electromagnetic tracking systems with multi-sensor 6-DOF pose estimation to monitor probe-tissue interactions, providing real-time feedback for deformation compensation and elastographic mapping.
Demonstrated the application of this approach in intraoperative settings, enabling high-resolution imaging of soft tissues (e.g., breast, liver, and skin) for precise pathological assessment and surgical guidance, in addition to enabling mechano-optical palpation for pathology detection.

One-Shot Object Manipulation and Intrinsic Elastography through Visuo-Tactile Markers ↕

Pioneered the concept of visuo-tactile markers, combining 3D-printed relief tags with Aruco marker patterns to enable simultaneous vision-based navigation, robotic grasping, and precision tactile manipulation using GelSight Mini sensors.
Developed a non-intrusive framework for one-shot object manipulation, allowing both robotic and human use without obstructing tool functionality, particularly useful in surgical and industrial settings.
Enabled indirect intrinsic elastography by integrating force sensors on robotic manipulators with visuo-tactile markers, providing real-time measurement of tool pressure and deflection for precise soft tissue characterization and pressure-sensitive tool manipulation.
Demonstrated the application of these markers in surgical instruments, enhancing robotic-assisted surgery through improved tool localization, grasp stability, and tissue interaction feedback.
Validated the approach in diverse scenarios, including delicate object manipulation and intraoperative settings, showcasing its versatility and robustness in dynamic environments.

Semantic Image--Pose Matching for Soft Body Deformation Mitigation in Low-Rate Single-Pixel Fast Dynamic Large-Area Microscopy ↕

Developed a novel framework leveraging multimodal image-text embeddings based on VisualBERT/CLIP and hierarchical navigable small worlds (HNSW) search to enable robust pose-informed image matching for unstructured scanning on soft body surfaces.
Introduced transformer-based semantic descriptors as an alternative to classical feature-based methods (e.g., SIFT, SURF), overcoming limitations in multimodal settings with significant image attenuation across sequential scans.
Combined spatial location tagging from electromagnetic (EM) trackers with semantic embeddings to enable rapid 3D image reconstruction and comparison, even among disparate datasets, using techniques such as 3D Gaussian splatting.
Enabled seamless interplay between coarse handheld scanning and high-precision robotic scanning, facilitating information transfer in multimodal soft body imaging scenarios without requiring extensive pre-processing or calibration.
Demonstrated the framework’s effectiveness in mitigating soft tissue deformation during low-rate single-pixel imaging, ensuring accurate large-area microscopy under varying pressure and pose uncertainty conditions.

Recessed Augmented Reality Markers (RAMs)

Enabling one-shot precision manipulation of everyday objects using visual and tactile sensing.

Recessed AR Markers (RAMs) provide a new way to enable precision robotic interactions with everyday objects, while still keeping them usable by humans—without the need for reinforcement learning. These markers have been validated in practice using GelSight Mini sensors, with a manuscript detailing these techniques currently under preparation.

3D printed visuotactile recessed AR markers (RAMs).

Rapid SIFT-based feature matching with database markers after preprocessing.

Spatiotemporal grasp tracking using RAMs and adaptive Savitzy-Golay filtering.

Lodestar

C++ digital guidance, navigation and control framework

Lodestar provides a user friendly platform-agnostic framework for real-time linear control of dynamic systems.

Overview

Language: C++11 / Python (bindings)
License: BSD-3
Status: In active development.
Website: https://lodestarengine.org ; https://theory.ldstr.dev .

The Lodestar framework [1] consists of several templates that allow for control of linear discrete time systems. The following features are implemented:

Discrete-time linear time invariant systems modeling and control:
- Template-based object oriented framework for arbitrary finite dimensional linear systems
- Linear Quadratic Regulation (LQR)
- Linear Quadratic Estimation/Classical Kalman Filter (LQE/KF)

Planned features

The following capabilities are planned for Lodestar in the near future:

Continuous to discrete time zero-order hold conversion
Recursive algebraic Riccati equation (ARE) solver
Recursive Least Squares (RLS) system identification
Single-input single-output (SISO) Generalized Predictive Control (GPC)
Spatial value functions (SVF) based guidance and path planning [2]
Real-time plotting and performance evaluation (impulse, unit step repsonse, Bode/Nyquist plots) frontend
Google Protocol Buffers-based message passing interface for real-time logging and command
I/O interface for common sensors (e.g., IMUs, barometers, time-of-flight sensors) and servos (e.g., solenoids, brushless DC motors).

References

[1] H. El-Kebir, J. Bentsman, and M. Ornik, “Lodestar: An Integrated Embedded Real-Time Control Engine,” in 2023 Conference on Control Technology and Applications. Bridgetown, Barbados: IEEE, 2023.

[2] Mettler, B., Dadkhah, N., and Kong, Z. “Agile Autonomous Guidance Using Spatial Value Functions.” Control Engineering Practice, Vol. 18, No. 7, 2010, pp. 773–788. doi:10.1016/j.conengprac.2010.02.013.

Knifefish

A light-weight framework for robotic electrosurgery

Knifefish is a framework for real-time control and simulation of electrosurgical processes, allowing for sensor data processing and feedback control.

Overview

Language: C++11
License: TBD
Status: In active development

Past/shelved projects

Stratagem3D

A light-weight web-based game engine for 3D turn-based strategy games

Overview

Language: Haxe/Kha
License: TBD
Status: Shelved

Extendible Orbit System 3D (EOS3D)

Lightweight orbital mechanics calculation and visualization suite based on matplotlib

Overview

Language: Python
License: MIT
Status: Suspended

The eos3d library is a lightweight orbital mechanics calculation and visualization suite based on matplotlib. It includes orbit visualization, SPACETRACK Two-Line Element retrieval and current satellite position visualization, Solar System orbits and planetary position visualizations and real-time n-body simulations.

Documentation

The documentation of eos3d-mpl may be downloaded via the following link: documentation.

Projects

Current projects

MarginDx

Overview

Individual Research Contributions

Recessed Augmented Reality Markers (RAMs)

Lodestar

Overview

Planned features

Links

References

Knifefish

Overview

Past/shelved projects

Stratagem3D

Overview

Extendible Orbit System 3D (EOS3D)

Overview

Documentation