**1. Introduction**

202 Recurrent Neural Networks and Soft Computing

Bernard, O.; Hadj-Sadok, Z.; Dochain, D.; Genovesi, A. & Steyer, J.P. (2001). Dynamical

Bialecki, B. & Fairweather G. (2001), Orthogonal spline collocation methods for partial

Boskovic, J.D. & Narendra, K. S. (1995). Comparison of linear, nonlinear and neural-

Deng H. & Li H.X. (2003). Hybrid intelligence based modeling for nonlinear distributed

Deng H., Li, H.X. & Chen G. (2005). Spectral-approximation-based intelligent modeling for

Padhi, R.; Balakrishnan, S. & Randolph, T. (2001). Adaptive critic based optimal neuro-

Padhi, R. & Balakrishnan, S. (2003). Proper orthogonal decomposition based optimal neuro-

Pietil, S. & Koivo, H.N. (1996). Centralized and decentralized neural network models for

Wan, E. & Beaufays, F. (1996). Diagrammatic method for deriving and relating temporal

13, No. 5, (September 2005) page numbers (686-700), ISSN 1063-6536 Gonzalez-Garcia, R; Rico-Martinez, R. & Kevrekidis, I. (1998). Identification of distributed

13-273350-1, Upper Saddle River, New Jersey 07458

(August 2001) page numbers (1223-1234), ISSN 0005-1098

1-2, (March 2001) page numbers (55-82), ISSN 0377-0427

numbers (424–438), ISNN 1097-0290

page numbers (160-165), ISSN 0941-0643

3511), ISSN 1062-922X

728), ISSN 0893-6080

ISBN 1-843-39550-9, Montreal, Canada

numbers (182–201), ISSN 0899-7667

model development and parameter identification for an anaerobic wastewater treatment process. *Biotechnology and Bioengineering*, Vol. 75, No 4, (June 2001) page

differential equations, *Journal of Computational and Applied Mathematics*, Vol. 128, No

network-based adaptive controllers for a class of fed-batch fermentation processes. *Automatica*, Vol. 31, No 6, (June1995) page numbers (817-840), ISSN 0005-1098 Bulsari, A. & Palosaari, S. (1993). Application of neural networks for system identification of

an adsorption column. *Neural Computing and Applications*, Vol. 1, No 2, (June 1993)

parameter process with applications to the curing process. *IEEE Transactions on Systems, Man and Cybernetics*, Vol. 4, No. 4, (October 2003) page numbers (3506 -

distributed thermal processes. *IEEE Transactions on Control Systems Technology*, Vol.

parameter systems: a neural net based approach. *Computers and Chemical Engineering*, Vol. 22, No 1, (March 2003) page numbers (965-968), ISSN 0098-1354 Haykin, S. (1999). *Neural Networks, a Comprehensive Foundation*, Second Edition, Section 2.13,

page numbers (84-89); Section 4.13, page numbers (208-213), Prentice-Hall, ISBN 0-

control synthesis for distributed parameter systems, *Automatica*, Vol. 37, No 8,

control synthesis of a chemical reactor process using approximate dynamic programming. *Neural Networks*, Vol. 16, No 5-6, (June 2003) page numbers (719 -

distributed parameter systems. *Proceedings of CESA'96 IMACS Multiconference on Computational Engineering in Systems Applications*, page numbers (1043-1048), ISBN 2-9502908-9-2, Lille, France, July 1996, Gerf EC Lille, Villeneuve d'Ascq, FRANCE Ray, W.H. (1989). *Advanced Process Control*, page numbers (133-242), Butterworths Publ., ISBN 0-409-90231-4, Boston, London, Singapore, Sydney, Toronto, Wellington Schoefs, O.; Dochain, D.; Fibrianto, H. & Steyer, J.P. (2004). Modelling and identification of a

partial differential equation model for an anaerobic wastewater treatment process. *Proceedings of 10th World Congress on Anaerobic Digestion,* page numbers (343 - 347),

neural networks algorithms. *Neural Computations*, Vol. 8, No.1, (January 1996) page

A distributed computer system (CS) is a set of elementary computers (ECs) connected by a network that is program-controlled from these computers. Each EC includes a computing module (CM) (processor with a memory) and a system unit (message router). The message router operates under CM control and has input and output ports connected to the output and input ports of the neighboring ECs, correspondingly. The CS structure is described by the graph *G (V ,E ) sss* , where *Vs* is the set of ECs and *E =V V s ss* is the set of connections between the ECs.

The topology of a distributed system may undergo changes while the system is operating, due to failures or repairs of communication links, as well as due to addition or removal of ECs (Bertsekas, Tsitsiklis, 1989). The CS robustness means that failures and recoveries of the ECs bring only to increasing and decreasing time of a task execution. Control on resources and tasks in the robust distributed CS suggested solution of the following problems (Tarkov, 2003, 2005): the CS optimal decomposition to connected subsystems; mapping parallel program structures onto the subsystem structures; static and dynamic balancing computation load among CMs of the computer system (subsystem); static and dynamic message routing (implementation of paths for data transfer), i.e. balancing communication load in the CS network; distribution of program and data copies for organization of fault tolerant computations; subsystem reconfiguration and redistribution of computation and communication load for computation recovery from failures, and so on.

As a rule, all these problems are considered as combinatorial optimization problems (Korte & Vygen, 2006), solved by centralized implementation of some permutations on data structures distributed on elementary computers of the CS. The centralized approach to the problem solution suggests gathering data in some (central) EC, solving optimization problem in this EC with the following scattering results to all ECs of the system (subsystem). As a result we have sequential (and correspondingly slow) method for the problem solution with great overhead for gathering and scattering data. Now a decentralized approach is significantly developed for solution of problems of control resources and tasks in computer

Optimization of Mapping Graphs of Parallel

Fig. 2. Example of a 2D-torus

of the system graph *Gs* ).

scalability.

neural networks.

**systems by recurrent neural networks** 

neural network by the Lyapunov function

Programs onto Graphs of Distributed Computer Systems by Recurrent Neural Network 205

In this paper, we consider a problem for mapping graph *G (V ,E ) ppp* of a parallel program onto graph *G (V ,E ) sss* of a distributed CS, where *nV V <sup>p</sup> <sup>s</sup>* is a number of program branches (of ECs). The mapping objective is to map nodes of the program graph *Gp* onto nodes of the system graph *Gs* one-to-one to carry out mapping *Gp* edges onto edges of *Gs* (to establish an isomorphism between the program graph *Gp* and a spanning subgraph

In section 2, we consider a recurrent neural network as a universal technique for solution of mapping problems. It is a local optimization technique, and we propose additional modifications (for example, penalty coefficients and splitting) to improve the technique

In section 3, we propose an algorithm based on the recurrent neural network and WTA ("Winner takes all") approach for the construction of Hamiltonian cycles in graphs. This algorithm maps only line- and ring-structured parallel programs. So, it is less universal than the technique proposed in section 2 but more powerful because it implements a global optimization approach, and hence it is very more scalable than the traditional recurrent

**2. Mapping graphs of parallel programs onto graphs of distributed computer** 

Let us consider a matrix *v* of neurons with size *n n* , each row of the matrix corresponds to some branch of a parallel program and every column of the matrix corresponds to some EC. Each row and every column of the matrix *v* must contain only one nonzero entry equal to one, other entries must be equal to zero. Let the distance between the neighboring nodes of the CS graph is taken as a unit distance and *ij d* is the length of the shortest path between nodes *i* and *j* in the CS graph. Then we define the energy of the corresponding Hopfield

systems with distributed memory (Tel G., 1994), and in many cases this approach allows to parallelize the problem solution.

Massive parallelism of data processing in neural networks allows us to consider neural networks as a perspective, high-performance, and reliable tool for solution of complicated optimization problems (Melamed, 1994; Trafalis & Kasap, 1999; Smith, 1999; Haykin, 1999; Tarkov, 2006). The recurrent neural network (Hopfield & Tank, 1985; Wang, 1993; Siqueira, Steiner & Scheer, 2007, 2010; Serpen & Patwardhan, 2007; Serpen, 2008; da Silva, Amaral, Arruda & Flauzino, 2008; Malek, 2008) is a most interesting tool for solution of discrete optimization problems. A model of a globally converged recurrent Hopfield neural network is in good accordance with Dijkstra's self-stabilization paradigm (Dijkstra, 1974). This signifies that the mappings of parallel program graphs onto graphs of distributed computer systems, carried out by Hopfield networks, are self-stabilizing (Jagota, 1999). An importance of usage of the self-stabilizing mappings is caused by a possibility of breaking the CS graph regularity by failures of ECs and intercomputer connections.

For distributed CSs, the graph of a parallel program *G (V ,E ) ppp* is usually determined as a set *Vp* of the program branches (virtual elementary computers) interacting with each other by the point−to−point principle through transferring messages via logical (virtual) channels (which may be unidirectional or bidirectional) of the set *E VV p pp* . Interactions between the processing modules are ordered in time and regular in space for most parallel applications (line, ring, mesh, etc.) (Fig. 1).

For this reason, the maximum efficiency of information interactions in advanced highperformance CSs is obtained by using regular graphs (,) *GVE sss* of connections between individual computers (hypercube, torus) (Parhami, 2002; Yu, Chung & Moreira, 2006; Balaji, Gupta, Vishnu & Beckman, 2011). The hypercube structure is described by a graph known as a *m*-dimensional Boolean cube with a number of nodes 2*<sup>m</sup> n* . Toroidal structures are *m*dimensional Euclidean meshes with closed boundaries. For *m* = 2, we obtain a twodimensional torus (2D-torus) (Fig. 2); for *m* = 3, we obtain a 3D-torus.

Fig. 1. Typical graphs of parallel programs (line, ring and mesh)

Fig. 2. Example of a 2D-torus

systems with distributed memory (Tel G., 1994), and in many cases this approach allows to

Massive parallelism of data processing in neural networks allows us to consider neural networks as a perspective, high-performance, and reliable tool for solution of complicated optimization problems (Melamed, 1994; Trafalis & Kasap, 1999; Smith, 1999; Haykin, 1999; Tarkov, 2006). The recurrent neural network (Hopfield & Tank, 1985; Wang, 1993; Siqueira, Steiner & Scheer, 2007, 2010; Serpen & Patwardhan, 2007; Serpen, 2008; da Silva, Amaral, Arruda & Flauzino, 2008; Malek, 2008) is a most interesting tool for solution of discrete optimization problems. A model of a globally converged recurrent Hopfield neural network is in good accordance with Dijkstra's self-stabilization paradigm (Dijkstra, 1974). This signifies that the mappings of parallel program graphs onto graphs of distributed computer systems, carried out by Hopfield networks, are self-stabilizing (Jagota, 1999). An importance of usage of the self-stabilizing mappings is caused by a possibility of breaking the CS graph

For distributed CSs, the graph of a parallel program *G (V ,E ) ppp* is usually determined as a set *Vp* of the program branches (virtual elementary computers) interacting with each other by the point−to−point principle through transferring messages via logical (virtual) channels (which may be unidirectional or bidirectional) of the set *E VV p pp* . Interactions between the processing modules are ordered in time and regular in space for most parallel

For this reason, the maximum efficiency of information interactions in advanced highperformance CSs is obtained by using regular graphs (,) *GVE sss* of connections between individual computers (hypercube, torus) (Parhami, 2002; Yu, Chung & Moreira, 2006; Balaji, Gupta, Vishnu & Beckman, 2011). The hypercube structure is described by a graph known as a *m*-dimensional Boolean cube with a number of nodes 2*<sup>m</sup> n* . Toroidal structures are *m*dimensional Euclidean meshes with closed boundaries. For *m* = 2, we obtain a two-

parallelize the problem solution.

regularity by failures of ECs and intercomputer connections.

dimensional torus (2D-torus) (Fig. 2); for *m* = 3, we obtain a 3D-torus.

Fig. 1. Typical graphs of parallel programs (line, ring and mesh)

applications (line, ring, mesh, etc.) (Fig. 1).

In this paper, we consider a problem for mapping graph *G (V ,E ) ppp* of a parallel program onto graph *G (V ,E ) sss* of a distributed CS, where *nV V <sup>p</sup> <sup>s</sup>* is a number of program branches (of ECs). The mapping objective is to map nodes of the program graph *Gp* onto nodes of the system graph *Gs* one-to-one to carry out mapping *Gp* edges onto edges of *Gs* (to establish an isomorphism between the program graph *Gp* and a spanning subgraph of the system graph *Gs* ).

In section 2, we consider a recurrent neural network as a universal technique for solution of mapping problems. It is a local optimization technique, and we propose additional modifications (for example, penalty coefficients and splitting) to improve the technique scalability.

In section 3, we propose an algorithm based on the recurrent neural network and WTA ("Winner takes all") approach for the construction of Hamiltonian cycles in graphs. This algorithm maps only line- and ring-structured parallel programs. So, it is less universal than the technique proposed in section 2 but more powerful because it implements a global optimization approach, and hence it is very more scalable than the traditional recurrent neural networks.
