Conversational LLM-Based Decision Support for Defect Classification in AFM Images

Revolutionizing nanoscale imaging through AI-powered defect detection and real-time guidance

ANGONA BISWAS1§, JAYDEEP RADE1§, NABILA MASUD1, MD HASIBUL HASAN HASIB1, ADITYA BALU3, JUNTAO ZHANG2, SOUMIK SARKAR2, ADARSH KRISHNAMURTHY1,2, JUAN REN2, and ANWESHA SARKAR1
1Department of Electrical and Computer Engineering, Iowa State University, 2520 Osborn Dr., Ames, 50011, IA, USA 2Department of Mechanical Engineering, Iowa State University, 2025 Black Engineering, Ames, IA 50011, USA 3Translational AI Center, Black Engineering, 2529 Union Dr Suite 2024, Ames, IA 50011, USA §Equal Contribution

CORRESPONDING AUTHOR: Anwesha Sarkar (e-mail: anweshas@iastate.edu)

GitHub Paper

Abstract

Atomic Force Microscopy (AFM) has emerged as a powerful tool for nanoscale imaging and quantitative characterization of organic (e.g., live cells, proteins, DNA, lipid bilayers) and inorganic (e.g., silicon wafers, polymers) specimens. However, image artifacts in AFM height and peak force error images directly affect the precision of nanomechanical measurements. Experimentalists face considerable challenges in obtaining high-quality AFM images due to the requirement of specialized expertise and constant manual monitoring.


Another challenge is the lack of high-quality AFM datasets to train machine learning models for automated defect detection. In this work, we propose a two-step AI framework that combines a vision-based deep learning (DL) model for classifying AFM image defects with a Large Language Models (LLMs)-based conversational assistant that provides real-time corrective guidance in natural language, making it particularly valuable for non-AFM experts aiming to obtain high-quality images.


We curated an annotated AFM defect dataset spanning organic and inorganic samples to train the defect detection model. Our defect classification model achieves 91.43% overall accuracy, with a recall of 93% for tip contamination and 60% for not-tracking defects.

Two-Step AI Framework

Vision Model

Deep learning model classifies defects in AFM images with 91.43% accuracy

LLM Assistant

Conversational AI provides real-time guidance in natural language

Unified Interface

Seamless interaction between DL model and LLM-based guidance

Model Performance

91.43%

Overall Accuracy

93%

Tip Contamination Recall

60%

Not-Tracking Recall

Interactive Demo

AFM Assistant
I see flat lines in my topography image that suggest tracking failure —what environmental or scanner factors could be contributing to this, and how do I address them?
1. Adjust feedback parameters: increase the proportional/integral gains cautiously

2. Increase setpoint force to maintain better tip-sample contact

3. For persistent problems, try changing scan direction or starting position

4. Decrease scan speed to give the feedback loop more time
View on GitHub