*3.2.4. Volumetric local binary patterns*

Gabor filter with a specified alpha, depending on how much background suppression is

To obtain an image that contains only the strongest edges and corresponding orientations, we

The resulting output of Anisotropic Inhibited Gabor Filter is an image that is *M* × *N*. Results are

We build upon the work in Ref. [46], but the proposed approach is significantly different. The anisotropic Gabor energy filter (AIGF) further computes the orientations corresponding to

**Figure 2.** (a) Original frame, (b) result of Gabor energy filter (Eq. (15) with *α* = 0), and (c) result of Anisotropic Gabor

*θ*

A soft histogram is computed from Θ with votes weighted by the maximal edge response

Local binary patterns (LBP) encode local appearance as a microtexture code. The code is a func‐ tion of comparison to the intensity values of neighboring pixels. Some formulations are invari‐ ant to rotation and monotonic grayscale transformations [31]. At present LBP and its many variations are one of the most widely used feature descriptors for facial expression recognition.

*AIGF*. For the proposed approach, we use *AIGF* and do not compute a soft histogram.

the number of pixel neighbours. The LBP code of a pixel at (*x*, *y*) is given as follows:

{*u*,*v*}∈*Nx*,*<sup>y</sup> LBP* ^

(*x*, *y*; *θ*) (16)

*g*˜(*x*, *y*; *θ*) (17)

where *n* is a parameter that controls

; sign(.) is the sign of the expression;

is the neighborhood of

sign(*I*(*u*, *v*) − *I*(*x*, *y*)) × 2*<sup>q</sup>* (18)

*LBP*

*LBP*

needed. We follow [46] where a value of *α* = 1 was empirically selected.

*AIGF*(*x*, *y*) = max *<sup>θ</sup> g*

14 Emotion and Attention Recognition Based on Biological Signals and Images

given in **Figure 2**.

Energy Filtering.

the maximum edges as follows:

*3.2.3. Local binary patterns*

Θ(*x*, *y*) = argmax

LBP result in a texture descriptor with dimensionality of 2*<sup>n</sup>*

where (*u*, *<sup>v</sup>*) iterates over points in the neighborhood of *Nx*,*<sup>y</sup>*

*<sup>q</sup>* is a counter starting from 0 that increments on each iteration; and *Nx*,*<sup>y</sup>*

*LBP*(*x*, *y* ) = ∑

take the edges with the strongest magnitude across *N* different orientations:

Volume local binary patterns (VLBP) and local binary patterns in three orthogonal planes (LBP‐TOP) are variations of LBP that were developed to capture dynamic textures for video data. In VLBP, the circle of neighboring points in LBP is scaled up to a cylinder. VLBP com‐ putes code values as a function of three parallel planes centered at {*x*, *y*, *t*}. That is, the middle plane contains the center pixel. VLBP coding is obtained by the following equation:

$$VLBP(\mathbf{x}, y, t \mid \mathbf{y}) = \sum\_{\text{lue}[-L, 0, 1]} \sum\_{\|\mathbf{u}, \mathbf{v}\| \le N^{\text{max}}\_{\text{v}\boldsymbol{\theta}}} \text{sign}\{I(\mathbf{u}, \mathbf{v}, \mathbf{k}) - I(\mathbf{x}, y, t)\} \times \mathbf{2}^{\text{q}} \tag{19}$$

where *k* iterates over three time points: *t*, *<sup>t</sup>* <sup>−</sup> *<sup>L</sup>*, and *<sup>t</sup>* <sup>+</sup> *<sup>L</sup>*. *Nx*,*y*,*<sup>t</sup> VLBP* is the set of spatiotemporal neigh‐ bours of {*x*, *<sup>y</sup>*, *<sup>t</sup>*} (see **Figure 3B**). A large set of *Nx*,*y*,*<sup>t</sup> VLBP* results in a large feature vector while a small *Nx*,*y*,*<sup>t</sup> VLBP* results in a small feature vector. As with LBP, a histogram is taken for further compact‐ ness. The maximum grey‐level from Eq. (19) is 2(3*n*+2) , thus VLBP are more computationally expensive to calculate and require larger feature vector.

### *3.2.5. Local binary patterns in three orthogonal planes*

LBP‐TOP was developed as an alternative to VLBP. VLBP and LBP‐TOP differ in two ways. First, LBP‐TOP uses three orthogonal planes that intersect at the center pixel. Second, VLBP considers the cooccurrences of all neighboring points from three parallel frames, which make for a larger feature vector. LBP‐TOP only considers features from each separate plane and then concatenates them together, making the feature vector much shorter when compared to VLBP for large values of *n*. LBP‐TOP performs LBP on the three orthogonal planes cor‐ responding to the *XY*, *XT*, and *YT* axes (see **Figure 3C**). The *XY* plane contributes the spatial information and the *XT* and *YT* frames contribute the temporal information. These planes intersect at the center pixel. Whereas in Eq. (19), VLBP captures a truly three‐ dimensional microtexture, LBP‐TOP computes LBP codes separately on each plane. The resulting feature vector dimensionality of LBP‐TOP is 3 × 2 *<sup>n</sup>* .

#### *3.2.6. Local anisotropic inhibited Gabor patterns in three orthogonal planes*

In the proposed method, the computational efficiency of LBP‐TOP is applied to images filtered with the anisotropic‐inhibited Gabor filter. The suppression of background texture provides an image that only contains the edges separate from the background texture. These edges are the significant boundaries of facial features that are useful when determining expression and emotion. Local anisotropic binary patterns' (LAIBP) code values are computed as follows:

$$LAIBP(\mathbf{x}, y) = \sum\_{(\boldsymbol{\mu}, \boldsymbol{\nu}) \in \mathbb{N}\_{\boldsymbol{\nu}\_{\boldsymbol{\nu}}}} \text{sign} \{AIGF(\boldsymbol{\mu}, \boldsymbol{\nu}) - AIGF(\mathbf{x}, y)\} \times 2^{\boldsymbol{\epsilon}} \tag{20}$$

where *g*(*u*, *v*) is the maximal edge magnitude from Eq. (16). LAIBP‐TOP features are extracted in a similar fashion to LBP‐TOP: Compute *LAIBP* codes from Eq. (20) in *XY*, *XT*, and *YT* planes and concatenate the resultant histograms. A comparison of AIGF, LBP, and the proposed method, LAIBP, are given in **Figure 4**. The proposed method (LAIBP‐TOP) is significantly different from LBP‐TOP because we introduce background texture removal from Eq. (16).

**Figure 4.** From left to right: The original frame, anisotropic inhibited Gabor filter (AIGF), local binary patterns (LBP), and the proposed method local anisotropic inhibited binary patterns (LAIBP). Note that the proposed method has more continuous lines compared to AIGF. LBP is susceptible to JPEG compression artifacts.
