**3. Reproducing fear conditioning using mobile robots**

In this section, we introduce our experimental design for reproducing fear conditioning using mobile robots. In specific, we select visual signals, the brightness of light as the conditional stimulus, and vibration signals as the unconditional stimulus. Vibration signals emulate electric shock applied on the foot of rats in fear conditioning experiments, while light signals serve as a neutral stimulus. We select the Leaky

Integrate and Fire (LIF) neuron model to build neural assemblies for both UC and CS signal pathways due to its simplicity. In our experiments, the Nengo simulator has been used to implement the system [40]. Our mobile robot is controlled with the Robot Operating System (ROS) [41].

#### **3.1 Neuron model**

In our experiment, a mobile robot is placed on a vibration platform and unconditionally responds to the acceleration signals, which will be detected with an *inertial measurement unit* (*IMU*). The acceleration signals emulate the unpleasant stimulus in the fear conditioning experiment in rats. The conditional stimulus is mimicked with the brightness of a light. Two neural assemblies will be built for processing these UC and CS signals. At last, the movement of a mobile robot is controlled by the motion neurons. We design several specific neurons for precepting the brightness of lights, detecting vibrations, and controlling mobile robots. All these neurons are customized from classic Leaky Integrate and Fire (LIF) neurons, which are characterized using the following eqs. [42]:

$$C\_m \frac{dV\_m}{dt} = G\_L(E\_L - V\_m) + A \ast I\_{app},$$

$$\sharp^\circ V\_m > V\_{th} then \, V\_m = V\_{reset},\tag{1}$$

$$
\mathfrak{r}\_{\rm RC} = \mathbb{C}\_{m} / \mathbb{G}\_{L},\tag{2}
$$

where *Cm* is the membrane capacitance, *Vm* is the membrane potential, *GL*is the leak conductance, *A* is the input signal gain, *EL*is the leak potential, *Iapp* is the input current, and *τRC* is the RC time constant. The specific values of these parameters in our design are summarized in **Table 2**.

For all the neurons in **Table 2**, the membrane potential is fixed at 1 V, and input gain is modified instead. They all use the Nengo LIF model's default *τRC* of 0.02 seconds because it is sufficient for our desired functionality. The other two parameters are calculated and optimized based on our experimental setups so that they can produce the desired responses for their respective uses. In specific, for the vibration detection neuron, gain (A) and bias (*Vreset*) are empirically derived so that it fires with vibration stimulus input instead of a small sudden move. The movement response neuron is a typical LIF configured to spike whenever it receives any sustained input spikes, either from vibration neurons or from brightness neurons. The brightness neuron is the Layer 3 neuron in **Table 3** and is a LIF neuron with empirically derived gain and bias so that it fires only when the Light Feature neurons have a high enough collective output. More details regarding how these neurons work will be introduced in subsequent sections.


**Table 2.** *LIF neuron parameters.*

#### *Neuromorphic Computing*


#### **Table 3**

*LCA neuron parameters in layers 2 and 3.*

#### **Figure 2.**

*(a) Acceleration data of IMU for three dimensions denoted as X-axis (blue), Y-axis (orange), and Z-axis (green). (b) Resultant acceleration.*

#### **3.2 Vibration perception**

The accelerometer within the onboard IMU of the mobile robot is used to measure the acceleration. In our experiment design, acceleration is used for evaluating the degree of vibration. Our vibration platform generates vibration signals with 1.2 mm amplitude at 25 Hz. In general, the IMU measures three-dimensional accelerations from x, y, and z directions as shown in **Figure 2a**. In **Figure 2a**, the acceleration in the z-axis (vertical) has the largest magnitude as it includes the intrinsic gravity of the earth (9.8 m/s<sup>2</sup> ). Because the mobile robot should only count the acceleration from the vibration platform, the gravity effect is removed from z-direction by subtracting the standard gravity (9.8 m/s<sup>2</sup> ). Thus, the resultant acceleration, which is used for evaluating the vibration states, is calculated by the equation:

$$a\_{\rm res} = \sqrt{a\_{\rm x}^2 + a\_{\rm y}^2 + \left(a\_{\rm x} - \text{9.8}\right)^2},\tag{3}$$

where *ares* is the resultant acceleration and *ax*, *ay*, and *az* are the accelerations in Xaxis, Y-axis, and Z-axis, respectively. The resultant acceleration measured by our mobile robot is depicted in **Figure 2b**. The resultant acceleration is imported into the vibration detection neuron. A vibration detection neuron, implemented with LIF, is connected to the accelerometer, and it fires if the detected acceleration is larger than its threshold (shown in **Figure 3**). At last, a movement neuron is specifically designed to control our mobile robot moving away from the vibration platform with specific direction and speed. The motion neuron is also implemented by a LIF neuron. The parameters of motion neuron are listed in **Table 2**. The active motion neuron will trigger a specific movement (escape) response that commands the mobile robot to move away from the vibration platform at a speed of 0.3 m/s.

*Implementation of Associative Memory Learning in Mobile Robots Using Neuromorphic… DOI: http://dx.doi.org/10.5772/intechopen.110364*

#### **Figure 3.**

*Vibration detection neuron response to the acceleration input: (a) input vibration signals; (b) membrane potential of the vibration detection neuron; (c) output spiking signals of the vibration detection neuron.*

#### **3.3 Visual perception with sparse coding and locally competitive algorithm**

In order to process visual stimuli, we designed an artificial neural network (ANN) to activate the output neuron if the light is on. **Figure 4** illustrates the brightness of the light captured by the stereo camera equipped with our mobile robot. The stream of visual signals captured by the camera is sent to the computer via ROS. As the images arrive, their resolution is 24x48 pixels, and the pixel brightness is normalized to the range between �1 and 1.

An ANN model based on 2D sparse coding is used for detecting the brightness of lights to further determine whether the light is on or off. The goal of sparse coding is to represent an input vector with a linear combination of features from a dictionary. This can be modeled by the LASSO optimization function:

$$E = \frac{1}{2} \|\mathbf{x} - \Phi \cdot \mathbf{a}\|\_2^2 + \lambda \cdot \|\mathbf{a}\|\_1,\tag{4}$$

$$a^\* = \underset{a}{\text{argmin}} E(a),\tag{5}$$

where the features Φ*<sup>i</sup>* are columns of the dictionary matrix Φ, and the sparse code *a⃰*is the set of coefficients *ai* for which the reconstruction (Φ � *a*) of the input *x* minimizes the cost (*E*). The sparsity penalty λ reduces the amount of non-zero terms in *a* by penalizing its 1-norm.

Typically, the sparse code *a⃰*is used to reconstruct *x* or as the input to a classifier. In 2D image sparse coding, we divided the image into patches the size of the features

**Figure 4.** *Sights of the light off and on.*

Φ*j*. Thus, each patch is a sparse-coding optimization problem. As shown in **Figure 5**, the light image is divided into 9 sub-regions, which are further partitioned into patches. The Spiking Locally Competitive Algorithm (LCA) model [43] is used to solve Eq. (6) and Eq. (7):

$$\dot{u} = \frac{1}{\tau} \left( \Phi^T \mathbf{x} - u - \left( \Phi^T \phi - I \right) \cdot a \right), a = T\_\lambda(u) \tag{6}$$

$$T\_{\lambda}(u) = 0 \text{ if } u \le \lambda \text{, else } T\_{\lambda}(u) = u - \lambda \tag{7}$$

where *ai* is the firing rate of neuron *i*, *ui* is the average soma current, *τ* is the discrete time step, and *Tλ* is the thresholding function that determines if neuron *i* will fire.

**Figure 6** illustrates a spiking LCA network for solving one image patch (**Figure 5**). The model consists of the first layer for the input *x* and the second layer for the sparse code a<sup>∗</sup> . The firing rates of each neuron are the coefficients *ai.* The neurons in the second layer are referred to as feature neurons as each of them is associated with one feature Φi. Typically, an overcomplete dictionary is used in sparse coding, resulting in *a* having a larger size than *x*. Our model represents each patch with only two features, light and dark. Thus, the dictionary in **Figure 6** only contains two features. The dictionary could be made complete by having more feature neurons than the size of x,

### **Figure 5.**

*Image region layout and patch structure.*

*Implementation of Associative Memory Learning in Mobile Robots Using Neuromorphic… DOI: http://dx.doi.org/10.5772/intechopen.110364*

9 in this case. There could be copies or variations of light and dark features to make the dictionary overcomplete.

The convolutional stride between the image patches is equal to the width of one patch (3 pixels), resulting in no overlap between them. This simplifies the LCA model by removing connections between feature neurons in overlapping patches. The neural network contains a third layer with one neuron for each of the 9 regions in the image as illustrated in **Figure 7**. The third layer has one neuron as an output neuron integrating the light feature neurons of every patch. In **Figure 7**, each neuron at layer 1 simply has a firing rate proportional to the pixel intensity it represents, serving as a spike generator. The neurons at layer 2 and layer 3 are Integrate and Fire neurons with the parameters listed in **Table 3**. The layer 2 neurons are the LCA neurons in a singlelayer LCA configuration. The parameters of the Integrate and Fire LCA feature neuron in layer 2 are implemented via the *Vreset* parameter, which was empirically adjusted until the desired response was achieved from the feature neurons. The other parameters are assigned according to the LCA model [43]. The neurons in layer 3 have the parameters derived for brightness neurons from **Table 2**.

When the high brightness signals of the light (**Figure 4a**) enter the network, the light feature neurons in the center patches start to fire. The light-off image (**Figure 4b**) subsequently reduces the activity in the neurons. These images are alternately presented as inputs, shown in **Figure 8**. This causes the neurons in layer 3 to fire for the center region from the time periods 0 s to 2 s and 4 s to 6 s, as shown in **Figure 9**. Those are the times the light is on in our experiment.

The sparse coding network operates at Intel Loihi neuromorphic chip for power and energy estimation. Our associative memory neural networks are implemented with Nengo simulator and further deployed into Loihi chips by Intel's NxSDK as a backend [44]. The NxSDK contains a built-in LCA network implementation, which can be connected to the rest of the SNN. The parameters from **Table 3** were used to create the same LCA network from Nengo for deployment on Loihi. In the Loihi chip, the synaptic weights only have a 4-bit resolution, instead of a 24-bit resolution in typical

**Figure 7.**

*Neural network for light detection in the center region. Note: The dark feature neurons in Layer 2 are not shown.*

**Figure 8.** *Membrane potential of light feature neurons (Layer 2) in center region image patches.*

**Figure 9.** *Spiking signals of Layer 3 neuron.*

**Figure 10.**

*Power consumption for sparse coding network with the Loihi chip. The VDD represents the compute logic; VDDM is the SRAMs, and VDDIO is the IO interface.*

computational platforms. Thus, the time of one simulation is reduced from 1 ms to 20 ms at the Loihi platform. The measured powers are shown in **Figure 10**. The average VDD power is 30 mW, and the average VDDM power is 29 mW. In **Figure 10**, the VDDIO has a relatively negligible contribution to the total power, while VDD and VDDM have effectively equal contributions. The power is measured and reported with the average consumption across every 8 timesteps while the experiment is under operation.

*Implementation of Associative Memory Learning in Mobile Robots Using Neuromorphic… DOI: http://dx.doi.org/10.5772/intechopen.110364*

#### **3.4 Associative memory learning experiment and results**

**Figure 11** illustrates the comparison of signal pathway modification between our SLN system and fear conditioning of rats. In our design, the brightness signal from light serves as a conditional stimulus, and the acceleration from the vibration platform is the unconditional stimulus. These two signals emulate the electric shock and sound signals in fear conditioning experiments. The movement of our mobile robot away from the vibration platform emulates the fear response of rats. The red arrows (**Figure 11**) represent the weak synaptic connections in the conditional signal pathway, while the blue arrows represent the unconditional pathway with weaker synaptic connections. During associative memory learning, the synaptic connection (red arrow) will be strengthened by Hebbian learning.

In the brains of rats, shock signals and tone signals are processed at different neural regions: auditory thalamus and somatosensory thalamus. The output signals from these two regions converge at lateral nucleus [1]. Originally, the rat has no fear reaction to a neutral tone. However, when the tone is presented with a foot shock associated with tone signals, the rat starts memorizing the relationship between the tone and the shock. After multiple times, the tone alone will stimulate a fear response, indicating the accomplishment of the associative memory learning [1].

Our experimental setup to emulate fear conditioning using our mobile robot is shown in **Figure 12**. The mobile robot is placed on a testing platform, which is constructed with 9 wooden boxes. The dimension of each box is 23 in (L) 23 in (W) 8 in (H). A vibration plate is installed underneath the center platform, which is marked in red square in **Figure 12a**. The vibration plate can provide vibration signals (15–40 Hz) emulating an unpleasant stimulus (the electrical shock) in fear learning on the rats. The other eight platforms with no installed vibration plate are marked in green-dashed squares in **Figure 12**.

For the fear response, the robot is expected to move away from the vibration platform to the neutral location if unpleasant stimuli, such as vibration, are applied.

#### **Figure 11.**

*(a) Our SLN system for associative memory learning implementation. (b) Fear conditioning in rats.*

**Figure 12.**

*Fear conditioning imitation experimental setup with the mobile robot: (a) top view; (b) side view.*

The specific experimental procedures aiming to reproduce the experimental process of rats in fear conditioning are:


The synaptic weights are modified based on Hebbian learning [42, 45]. Hebbian learning states that when the pre- and postsynaptic neurons are both active at the same time, the synaptic weights between them will be modified using the eqs. [42, 45]:

$$w = \eta r\_i r\_j,\tag{8}$$

where the *ri* and *rj* are firing rates of pre- and postsynaptic neurons, respectively, and the *η* is the learning rate, determining the changing rate of synaptic weight. In our experiment, the learning rate *<sup>η</sup>* is 2 � <sup>10</sup>�4. **Figure 13** shows that the initial synaptic weights between the brightness detection neuron (CS) and the movement neuron are small. Consequently, the brightness stimulus of the light cannot be delivered to the movement neuron to stimulate it to fire. As a result, the synaptic weights between the brightness detection neuron and movement neuron stay constant. When the vibration (US) is applied to the vibration detection neuron, the movement neuron starts to fire. When the vibration and light are applied to the system, both the brightness detection neuron and the movement neuron are active, resulting in the synaptic weight's

*Implementation of Associative Memory Learning in Mobile Robots Using Neuromorphic… DOI: http://dx.doi.org/10.5772/intechopen.110364*

**Figure 13.** *Change of synaptic weight during associative memory learning.*

increase. **Figure 13** illustrates that the synaptic weights increase when the vibration and light stimuli are both applied. However, the first overlapping time frame is not long enough to establish a significant synaptic weight modification. Thus, a sequence of weak spiking signals is observed from the response neuron after the vibration is removed, which is marked in **Figure 13**. In contrast, the second overlapping period is longer than the first one, which leads to a larger increase in synaptic weights. Thereby, after learning, the response neuron (movement detection neuron) will fire with a visual stimulus (light) even with no vibration stimulus. This demonstrates an accomplishment of associative memory learning.

In the experiments, the synaptic weights between the brightness detection neuron and the movement neuron are modified during the training process. As a result, after associative memory learning, the mobile robot will move away from the vibration platform under the stimulus of light, even with no vibration signal presented, demonstrating successful online learning in real-time. Compared to other state-of-the-art associative memory works listed in **Table 4**, we reproduce the classic fear conditioning experiments of rats using a mobile robot and the Loihi chip rather than simply simulation. In addition, the scales of our neural networks outperform other works.
