**5. Conclusions**

14 Computational and Numerical Simulations

(a)

asynchronous updating

synchronous updating

(b)

processing units and **c***<sup>k</sup>* be the outer-input of processing units in *Sk*. **c***<sup>k</sup>* contains nonzero elements if there exists a processing unit in *Sk* that is connected with units not in *Sk* and those nonzero elements are determined by mean activations of processing units outside *Sk*.

dense connectivity among processing units inside each *Sk* and sparse connectivity among

In each cluster *Sk* when there is a processing unit connecting to processing units outside *Sk* according to the approach in section 2, all processing units inside *Sk* are evaluated directly by synchronous computations for fixed **c***k*. The approach which combines synchronous update

*<sup>k</sup>*=1. Ideally, there is

*<sup>k</sup>*=<sup>1</sup> is proposed for

**Figure 12.** The histograms of cutsize obtained by 50 executions of synchronous update and asynchronous update.

After *K*-set graph partition, all nodes are reindexed according to {*Sk*}*<sup>K</sup>*

*<sup>k</sup>*=<sup>1</sup> as illustrated in Figure 13.

of mean activations in side each *Sk* and sequential update among {*Sk*}*<sup>K</sup>*

{*Sk*}*<sup>K</sup>*

*<sup>k</sup>*=<sup>1</sup> through {**c***k*}*<sup>K</sup>*

This paper has proposed a novel approach for tracking mean field dynamics by synchronous computations of recurrent multilayer perceptrons. The strategy is to introduce time delays and auxiliary variables and constructs equivalent recursive relations. This strategy essentially constructs recurrent multilayer perceptrons for tracking densely coupled mean field dynamics. The proposed approach is also extended to deal with large-scale sparsely interconnected mean field dynamics. In the beginning, all processing units are partitioned into *K* clusters by solving graph partition. The task is decomposed to *K* subtasks of synchronous computations and different clusters are sparsely connected by outer-inputs. The work combines synchronous updating inside each cluster with sequential updating among *K* clusters.

Numerical simulations show that the proposed approach has successfully translated mean field equations of solving the graph bisection problem to a system of post-nonlinear recursive functions, and verified the consistency between the original mean field equations and corresponding recurrent computations.

10.5772/57217

299

http://dx.doi.org/10.5772/57217

 

<sup>2</sup> *N* is a constant, the energy function is rewritten as

Let *Wij* = *Tij* − *<sup>A</sup>* where *Wii* = 0. Since *<sup>A</sup>*

A linear system is given by

Let

*A* =

 

be a sparse matrix and

*A*<sup>11</sup> *A*<sup>12</sup> *A*<sup>13</sup> *A*<sup>21</sup> *A*<sup>22</sup> *A*<sup>23</sup> *A*<sup>31</sup> *A*<sup>32</sup> *A*<sup>33</sup> =

  *<sup>E</sup>*(*S*) = <sup>−</sup><sup>1</sup>

**6.2. Appendix B. An example decomposing sparse interconnection**

2

*x*<sup>1</sup> = 0 + *a*12*x*<sup>2</sup> + *a*13*x*<sup>3</sup> + 0 + 0 + 0 + 0 + 0 + 0 *x*<sup>2</sup> = *a*21*x*<sup>1</sup> + 0 + *a*23*x*<sup>3</sup> + 0 + 0 + 0 + 0 + 0 + 0 *x*<sup>3</sup> = *a*31*x*<sup>1</sup> + *a*32*x*<sup>2</sup> + 0 + 0 + 0 + 0 + 0 + *a*38*x*<sup>8</sup> + 0 *x*<sup>4</sup> = 0 + 0 + 0 + 0 + *a*45*x*<sup>5</sup> + *a*46*x*<sup>6</sup> + 0 + 0 + 0 *x*<sup>5</sup> = 0 + 0 + 0 + *a*54*x*<sup>4</sup> + 0 + *a*56*x*<sup>6</sup> + 0 + 0 + 0 *x*<sup>6</sup> = 0 + *a*62*x*<sup>2</sup> + 0 + *a*64*x*<sup>4</sup> + *a*65*x*<sup>5</sup> + 0 + 0 + 0 + 0 *x*<sup>7</sup> = *a*71*x*<sup>1</sup> + 0 + 0 + 0 + 0 + 0 + 0 + *a*78*x*<sup>8</sup> + *a*79*x*<sup>9</sup> *x*<sup>8</sup> = 0 + 0 + 0 + 0 + 0 + 0 + *a*87*x*<sup>7</sup> + 0 + *a*89*x*<sup>9</sup> *x*<sup>9</sup> = 0 + 0 + 0 + 0 + 0 + 0 + *a*97*x*<sup>7</sup> + *a*98*x*<sup>8</sup> + 0

> *<sup>S</sup>*<sup>1</sup> = {*x*1, *<sup>x</sup>*2, *<sup>x</sup>*3} *<sup>S</sup>*<sup>2</sup> = {*x*4, *<sup>x</sup>*5, *<sup>x</sup>*6} *<sup>S</sup>*<sup>3</sup> = {*x*7, *<sup>x</sup>*8, *<sup>x</sup>*9} **v**<sup>1</sup> = [*x*<sup>1</sup> *x*<sup>2</sup> *x*3]

**v**<sup>2</sup> = [*x*<sup>4</sup> *x*<sup>5</sup> *x*6]

**v**<sup>3</sup> = [*x*<sup>7</sup> *x*<sup>8</sup> *x*9]

*T*

*T*

*T*

*N* ∑ *i*=1

*N* ∑ *j*�=*i*

*Wijsisj*

Tracking Mean Field Dynamics by Synchronous Computations of Recurrent Multilayer Perceptrons

 *a*12*x*<sup>2</sup> *a*13*x*<sup>3</sup> 000000 *a*21*x*<sup>1</sup> 0 *a*23*x*<sup>3</sup> 000000 *a*31*x*<sup>1</sup> *a*32*x*<sup>2</sup> 00000 *a*38*x*<sup>8</sup> 0 *a*45*x*<sup>5</sup> *a*46*x*<sup>6</sup> 000 *a*54*x*<sup>4</sup> 0 *a*56*x*<sup>6</sup> 000 *a*62*x*<sup>2</sup> 0 *a*64*x*<sup>4</sup> *a*65*x*<sup>5</sup> 0000 *a*71*x*<sup>1</sup> 000000 *a*78*x*<sup>8</sup> *a*79*x*<sup>9</sup> *a*87*x*<sup>7</sup> 0 *a*89*x*<sup>9</sup> *a*97*x*<sup>7</sup> *a*98*x*<sup>8</sup> 0

**Figure 14.** The flow chart of parallel and distributed processes for tracking mean field dynamics of sparse connectivity.

### **6. Appendix**

### **6.1. Appendix A. Rewriting energy function of graph bisection problem**

$$\begin{split} E(S) &= -\frac{1}{2} \sum\_{i=1}^{N} \sum\_{j \neq i}^{N} T\_{ij} s\_i s\_j + \frac{A}{2} \left( \sum\_{i=1}^{N} s\_i \right)^2 \\ &= -\frac{1}{2} \sum\_{i=1}^{N} \sum\_{j \neq i}^{N} T\_{ij} s\_i s\_j + \frac{A}{2} \left( \sum\_{i=1}^{N} s\_i^2 + \sum\_{i=1}^{N} \sum\_{j \neq i}^{N} s\_i s\_j \right) \\ &= -\frac{1}{2} \sum\_{i=1}^{N} \sum\_{j \neq i}^{N} (T\_{ij} - A) s\_i s\_j + \frac{A}{2} \left( \sum\_{i=1}^{N} s\_i^2 \right) \\ &= -\frac{1}{2} \sum\_{i=1}^{N} \sum\_{j \neq i}^{N} (T\_{ij} - A) s\_i s\_j + \frac{A}{2} N \end{split}$$

Let *Wij* = *Tij* − *<sup>A</sup>* where *Wii* = 0. Since *<sup>A</sup>* <sup>2</sup> *N* is a constant, the energy function is rewritten as

$$E(\mathbf{S}) = -\frac{1}{2} \sum\_{i=1}^{N} \sum\_{j \neq i}^{N} \mathcal{W}\_{ij} \mathbf{s}\_i \mathbf{s}\_j$$

### **6.2. Appendix B. An example decomposing sparse interconnection**

A linear system is given by

$$\begin{aligned} \mathbf{x}\_1 &= \begin{array}{c} 0 & + & a\_{12}\mathbf{x}\_2 + a\_{13}\mathbf{x}\_3 + & 0 & + & 0 & + & 0 & + & 0 & + & 0 & + & 0 \\ \mathbf{x}\_2 &= a\_{21}\mathbf{x}\_1 + & 0 & + a\_{23}\mathbf{x}\_3 + & 0 & + & 0 & + & 0 & + & 0 & + & 0 \\ \mathbf{x}\_3 &= a\_{31}\mathbf{x}\_1 + a\_{32}\mathbf{x}\_2 + & 0 & + & 0 & + & 0 & + & 0 & + & a\_{33}\mathbf{x}\_8 + & 0 \\ \mathbf{x}\_4 &= 0 & + & 0 & + & 0 & + & a\_{45}\mathbf{x}\_5 + a\_{46}\mathbf{x}\_6 + & 0 & + & 0 & + & 0 \\ \mathbf{x}\_5 &= 0 & + & 0 & + & 0 & + a\_{54}\mathbf{x}\_4 + & 0 & + a\_{56}\mathbf{x}\_6 + & 0 & + & 0 & + & 0 \\ \mathbf{x}\_6 &= 0 & + a\_{62}\mathbf{x}\_2 + & 0 & + a\_{64}\mathbf{x}\_4 + a\_{65}\mathbf{x}\_5 + & 0 & + & 0 & + & 0 & + & 0 \\ \mathbf{x}\_7 = a\_{71}\mathbf{x}\_1 + & 0 & + & 0 & + & 0 & + & 0 & + & a\_{78}\mathbf{x}\_7 + & 0 & + a\_{89}\mathbf{x}\_9 \\ \mathbf{x}\_8 &= 0 & + & 0 & + & 0 & + & 0 & + & 0 & + & a\_{97}\mathbf{x}\_7 + & 0 \\ \mathbf{x}\_9 &= & 0 & +$$

Let

16 Computational and Numerical Simulations

**6. Appendix**

k :1 K

condition Halting

k k 

sub - clusters .

<sup>k</sup> <sup>k</sup> K S

Form , Form , **u**

1 K

k B k

divide all nodes into Apply graph partition to

<sup>k</sup> update **c**

 <sup>k</sup> B kk <sup>k</sup> **v** tanh 

**6.1. Appendix A. Rewriting energy function of graph bisection problem**

*Tijsisj* +

*Tijsisj* +

(*Tij* − *<sup>A</sup>*)*sisj* +

(*Tij* − *<sup>A</sup>*)*sisj* +

*A* 2

*A* 2

� *N* ∑ *i*=1 *si*

 *N* ∑ *i*=1 *s* 2 *<sup>i</sup>* + *N* ∑ *i*=1

> *A* 2

*A* 2 *N*

� *N* ∑ *i*=1 *s* 2 *i*

�2

*N* ∑ *j*�=*i sisj*

�

 

*N* ∑ *j*�=*i*

*N* ∑ *j*�=*i*

*N* ∑ *j*�=*i*

*N* ∑ *j*�=*i*

*<sup>E</sup>*(*S*) = <sup>−</sup><sup>1</sup>

2 *N* ∑ *i*=1

<sup>=</sup> <sup>−</sup><sup>1</sup> 2 *N* ∑ *i*=1

<sup>=</sup> <sup>−</sup><sup>1</sup> 2 *N* ∑ *i*=1

<sup>=</sup> <sup>−</sup><sup>1</sup> 2 *N* ∑ *i*=1 **u c**

**Figure 14.** The flow chart of parallel and distributed processes for tracking mean field dynamics of sparse connectivity.

EXIT

$$A = \begin{bmatrix} A\_{11} \ A\_{12} \ A\_{13} \\ A\_{21} \ A\_{22} \ A\_{23} \\ A\_{31} \ A\_{32} \ A\_{33} \end{bmatrix} = \begin{bmatrix} 0 & a\_{12} \mathbf{x}\_{2} & a\_{13} \mathbf{x}\_{3} & 0 & 0 & 0 & 0 & 0 & 0 \\ a\_{21} \mathbf{x}\_{1} & 0 & a\_{23} \mathbf{x}\_{3} & 0 & 0 & 0 & 0 & 0 & 0 \\ a\_{31} \mathbf{x}\_{1} & a\_{32} \mathbf{x}\_{2} & 0 & 0 & 0 & 0 & 0 & a\_{38} \mathbf{x}\_{8} & 0 \\ 0 & 0 & 0 & a\_{45} \mathbf{x}\_{5} & a\_{46} \mathbf{x}\_{6} & 0 & 0 & 0 \\ 0 & 0 & 0 & a\_{54} \mathbf{x}\_{4} & a\_{65} \mathbf{x}\_{5} & 0 & 0 & 0 \\ 0 & a\_{62} \mathbf{x}\_{2} & 0 & a\_{64} \mathbf{x}\_{4} & a\_{65} \mathbf{x}\_{5} & 0 & 0 & 0 \\ a\_{71} \mathbf{x}\_{1} & 0 & 0 & 0 & 0 & 0 & 0 & a\_{78} \mathbf{x}\_{8} \ x\_{79} \mathbf{x}\_{9} \\ 0 & 0 & 0 & 0 & 0 & 0 & a\_{87} \mathbf{x}\_{7} & 0 & a\_{88} \mathbf{x}\_{9} \\ 0 & 0 & 0 & 0 & 0 & 0 & a\_{97} \mathbf{x}\_{7} \ a\_{88} \mathbf{x}\_{8} & 0 \end{bmatrix}$$

be a sparse matrix and

$$\begin{aligned} S\_1 &= \{ \mathbf{x}\_{1\prime} \mathbf{x}\_{2\prime} \mathbf{x}\_3 \} \\ S\_2 &= \{ \mathbf{x}\_{4\prime} \mathbf{x}\_{5\prime} \mathbf{x}\_6 \} \\ S\_3 &= \{ \mathbf{x}\_{7\prime} \mathbf{x}\_{8\prime} \mathbf{x}\_9 \} \\ \mathbf{v}\_1 &= \begin{bmatrix} \mathbf{x}\_1 \ \mathbf{x}\_2 \ \mathbf{x}\_3 \end{bmatrix}^T \\ \mathbf{v}\_2 &= \begin{bmatrix} \mathbf{x}\_4 \ \mathbf{x}\_5 \ \mathbf{x}\_6 \end{bmatrix}^T \\ \mathbf{v}\_3 &= \begin{bmatrix} \mathbf{x}\_7 \ \mathbf{x}\_8 \ \mathbf{x}\_9 \end{bmatrix}^T \end{aligned}$$

10.5772/57217

301

http://dx.doi.org/10.5772/57217

**References**

1982.

1987

2008

3088-3092, 1984.

pp. 55–77, Sept. 2000.

problems," Biol. Cybern., vol. 52, p. 141, 1985.

Amer., vol. 257, no. 6, pp. 104–115, 1987

no. 4, pp. 671–684, 1996.

neural network," Int. J. Neural Syst., vol. 1, p. 3, 1989.

[1] J. J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities", Proceedings of the National Academy of Sciences 79:2554-2558,

Tracking Mean Field Dynamics by Synchronous Computations of Recurrent Multilayer Perceptrons

[2] J. J. Hopfield, "Neurons with graded response have collective computational properties like those of two-state neurons", Proceedings of the National Academy of Sciences 81:

[3] J. J. Hopfield and D. W. Tank, "Neural computation of decisions in optimization

[4] C. Peterson and B. Söerberg, "A new method for mapping optimization problems onto

[5] J. M. Wu, "Potts models with two sets of interactive dynamics," Neurocomput., vol. 34,

[6] J. M. Wu, "Annealing by Two Sets of Interactive Dynamics," IEEE Trans. on Systems

[7] R. J. McEliece, E. C. Posner, E. R. Rodemich, and S. S. Venkatesh, "The capacity of the Hopfield associative memory," IEEE Truns. Inform. Theory, vol. IT-33, pp. 1-33, July

[8] T. Isokawa, H. Nishimura, N. Kamiura, and N. Matsui, "Associative memory in quaternionic Hopfield neural network," Int. J. Neural Syst., vol. 18, no. 2, pp. 135–145,

[9] D. W. Tank and J. J. Hopfield, "Collective computation in neuronlike circuits," Sci.

[10] C. Y. Liou and J. M.Wu, "Self-organization using Potts models," Neural Netw., vol. 9,

[11] K. Rose, E. Gurewitz, and G. C. Fox, "Statistical mechanics and phase transitions in

[12] K. Rose, "Constrained clustering as an optimization method," IEEE Trans. Pattern Anal.

[13] J. M. Wu and S. J. Chiu, "Independent component analysis using Potts models," IEEE

[14] A. V. Rao, D. J. Miller, K. Rose, and A. Gersho, "A deterministic annealing approach for parsimonious design of piecewise regression models," IEEE Trans. Pattern Anal.

[15] J. M. Wu, Z. H. Lin and P. H. Hsu, "Function approximation using generalized adalines,"

clustering," Phys. Rev. Lett., vol. 65, no. 8, pp. 945–948, 1990

Trans. Neural Networks, vol. 12, pp. 202–211, Mar. 2001.

IEEE Trans. On Neural Networks,Vol.17 No.3, 541-558, May 2006

Machine Intell., vol. 15, pp. 785–794, Aug. 1993.

Machine Intell., vol. 21, pp.169–173, Feb. 1999.

Man and Cybernetics Part B-Cybernetics 34 (3): 1519-1525, Jun 2004.

**Figure 15.** Dense interconnection in each cluster and sparse interconnection among three clusters.

Based on graph partition of *K* = 3, the system **x** = *A***x** has dense interconnection of *Sk*, ∀*<sup>k</sup>* = 1, 2, 3 and sparse interconnection among {*Sk*}<sup>3</sup> *<sup>k</sup>*=<sup>1</sup> as shown in Figure 15. Let

$$\begin{aligned} d\_1 &= d\_2 = d\_4 = d\_5 = d\_8 = d\_9 = 0 \\ d\_3 &= a\_{38} x\_8 \\ d\_6 &= a\_{62} x\_2 \\ d\_7 &= a\_{71} x\_1 \end{aligned}$$

and **c**<sup>1</sup> = [*d*<sup>1</sup> *d*<sup>2</sup> *d*3] *<sup>T</sup>*, **c**<sup>2</sup> = [*d*<sup>4</sup> *d*<sup>5</sup> *d*6] *<sup>T</sup>* and **c**<sup>3</sup> = [*d*<sup>7</sup> *d*<sup>8</sup> *d*9] *<sup>T</sup>* be the outer-input of three clusters {*Sk*}<sup>3</sup> *<sup>k</sup>*=1. *di* is nonzero if there is a node *xj* connected to *xi* with weight *aij* �<sup>=</sup> 0 where *xi* and *xj* belong to different clusters. Therefore, the updating rule of {**c***k*}<sup>3</sup> *<sup>k</sup>*=<sup>1</sup> is

$$\mathbf{c}\_{k} = \sum\_{j \neq k}^{3} A\_{kj} \mathbf{v}\_{j}$$
