**Abstract**

In this chapter, we explore a mathematical modelling that describes the nonlinear dynamic behavior of atomic force microscopy (AFM). We propose two control techniques for suppressing the chaotic motion of the system. The proposed model considers the interatomic interactions between the analyzed sample and the cantilever. These acting forces are van der Waals type, and we add a mathematical term that is a simple approximation to the viscoelasticity that possibly occurs in biological samples. We analyzed the behavior of the initial conditions of the proposed mathematical model, which showed a degree of complexity of the basins of attraction that were detected by entropy and uncertainty parameter, both detect if the basins have a fractal behavior. Numerical results showed that the nonlinear dynamic behavior has chaotic regions with the Lyapunov exponent, bifurcation diagram, and the Poincaré map. And, we propose two control techniques to suppress the chaotic movement of the AFM cantilever. First technique is the optimal linear feedback control (OLFC), which does not consider the nonlinearities of mathematical model. On the other hand, the control state dependent Riccati equation (SDRE) considers the nonlinearities of mathematical model. Both control techniques for a desired periodic orbit proved to be efficient.

**Keywords:** nonlinear dynamics, atomic force microscopy, control design, basins of attraction, OLFC, SDRE
