Preface

Multi-agent technologies (MATs) and machine learning (ML) are among the most advanced directions of modern computer science and its applications due to their ability to reduce the complexity of and time spent on knowledge acquisition and maintenance. They are also able to achieve high-quality solutions to various problems.

MATs, which arose from agent-based programming, are an efficient tool for the creation and evolutionary development of multi-agent systems (MASs), representing the logic of solution of various non-trivial tasks from multiple problem areas as well as the logic of behavior of various objects, operating in complicated and volatile environments. The great flexibility of MATs, achieved due to the representation of a task or a system as an easily modified set of interacting objects named agents, enables their wide application as well as their software and even hardware implementations. Today, many successful applications of MATs in engineering, physics, medicine, energy, economy, banking, social management, ecology, and more demonstrate the great power of this tool.

Machine learning ML as a segment of artificial intelligence (AI) arose primarily from neural networks (NNs), which are efficient and widely used for solving tasks requiring massive parallelism for processing intensive data flows originating primarily from various sensors. The most well-known application of NNs is image processing, which enables online monitoring of wide areas covered by video cameras and/or measurement devices. A great achievement of ML associated with NNs is its implementation in teaching NNs by input–output sets ("training samples"). Thus, a work of a learning person in this case is reduced to the presentation of an a priori accumulated set of training samples to an NN, transferred to a teaching regime. Moreover, there are ML paradigms such as unsupervised learning and reinforcement learning (or learning with feedback) that do not require a person at all or require them in a very limited capacity.

Since some period of mutually isolated research of MATs and ML, it has become clear that there is much promise in combining the advantages of both approaches: the generality and flexibility inherent to MATs with the low-cost knowledge acquisition inherent to ML. There may be different approaches to constructive development and implementation of this basic idea, for example, the creation of MASs with self-learning agents or MASs with adaptive agents set, and thus investigation of ways of integrating MATs and ML is an interesting and promising area of research.

This book consists of six chapters in three sections. Section 1, which includes Chapters 1 and 2, discusses the integration of MATs and ML. Chapter 1 reviews current research in MASs combined with various ML techniques. It analyzes features of MASs from the ML point of view, classifies applications of MASs integrated with ML methods, and presents a density map of applications in manufacturing, commerce, and E-learning. Chapter 2 discusses deep multi-agent reinforcement learning methods for solving problems with MASs consisting of very large numbers of heterogeneous agents. Such methods have significant scalability potential and

inter-agent coordination capabilities in large-scale and complex multi-agent settings. The chapter uses air traffic management as a practical background for comparative consideration of the described methods.

Section 2, which includes Chapters 3 and 4, describes the most interesting and refined applications of MATs combined with ML. Chapter 3 is a comparative study of different design approaches to a corporative multi-agent system for optimal scheduling. The authors propose creating a dataset using multiple algorithms with different performance metrics to find the best one. This dataset may be imported into some ML tools for training and predicting, based on the selected performance metrics. The chapter examines three approaches: first come first serve, round robin, and ant colony. The chapter shows that the ant colony algorithm achieves the best results. Chapter 4 describes an architecture of a knowledge-based multi-agent system (KBMAS) demonstrating inter alia capabilities, which open due to the integration of advantages of MATs and ML. It proposes an approach to the development of applied software agents that combines knowledge-based reasoning with neural network models. It also considers the method of reinforcement learning, the system of rules, and the queries to the knowledge base.

Section 3, which includes Chapters 5 and 6, describes advanced results in MATs and ML that may be used as a background for their further integration and extension of an area of their combined theoretical consideration and application. Chapter 5 discusses the multi-agent modeling of a charging station of an electric vehicle. Power engineering applications, such as power system optimization and restoration, electric energy market modeling, and smart grid control are among the most topical and successful MATs. The described MAS illustrates how MATs enable adequate modeling of power-supplying devices with a non-trivial physical background. Such models of the lowest-level devices may be easily implanted into a model of a power system of any static or mobile powerconsuming object, thus demonstrating an "additivity" of MATs, enabling their efficient application in multiple problem areas. Chapter 6 presents the useful application of ML to the development of one of the most interesting and practically valuable problems of operations research: approximate dynamic programming. It proposes an efficient ML algorithm for two-stage stochastic programs. The optimization framework makes it easy to introduce changes to the already obtained decisions as well as to capture the collective intelligence of the experienced decisions. Such features are hardly (or even not) available inside classic operations research approaches. Moreover, as computational results indicate, the proposed ML-based algorithm explicates rapid convergence.

Overall, this volume provides a comprehensive overview of the current state and perspectives of MATs–ML integration and application in a wide set of practical and theoretical problems.

> **Igor A. Sheremet** Full Member of the Russian Academy of Sciences, Scientific Supervisor and Chair of the Science-Technological Board of the National Computer Corporation (NCC), Moscow, Russia

## Section 1
