**1. Introduction**

350 Applications of Digital Signal Processing

Mohan P. V. A. & Premkumar, A. B. (2007). RNS-to-Binary Converters for Two Four-Moduli

Molahosseini, A.S., Dadkhah, C., Navi, K. & Eshghi, M. (2009). Efficient MRC-Based Residue

1}. *IEICE Transactions on Information and Systems*, vol. E92-D, pp. 1628-1638. Molahosseini, A.S. & Navi, K. (2010). A Reverse Converter for the Enhanced Moduli Set {2n–

Molahosseini, A.S., Navi, K., Hashemipour, O. & Jalali, A. (2008). An efficient architecture

Navi, K., Molahosseini, A.S. & Esmaeildoust, M. (2011). How to Teach Residue Number

Omondi, A. & Premkumar, B. (2007). *Residue Number Systems: Theory and Implementations*,

Parhami, B. (2000). *Computer Arithmetic: Algorithms and Hardware Design*, Oxford University Press. Piestrak, S. J. (1994). Design of residue generators and multioperand modular adders using carry-save adders. *IEEE Transactions on Computers*, Vol. 423, No. 1, pp. 68-77. Piestrak, S.J. (1995). A high speed realization of a residue to binary converter. *IEEE* 

Soderstrand, M.A. & et al. (1986). *Residue number system arithmetic: modern applications in* 

Stouratitis, T. & Paliouras, V. (2001). Considering the alternatives in lowpower design. *IEEE* 

Taylor, F.J. (1984). Residue arithmetic: a tutorial with examples. *IEEE Computer*, Vol. 17, pp.

Vinod, A.P. & Premkumar, A.B. (2000). A residue to binary converter for the 4-moduli

Wang, Y. (2000). Residue-to-Binary Converters Based on New Chinese remainder theorems. *IEEE Transactions on Circuits and Systems-II*, Vol. 47, No. 3, pp. 197-205. Wang, Z., Jullien, G. A. & Miller, W. C. (2000). An Improved Residue-to-Binary Converter. *IEEE Transactions on Circuits and Systems-I*, Vol. 47, No. 9, pp. 1437-1440. Wang, Y., Song, X., Aboulhamid, M. & Shen, H. (2002). Adder based residue to binary

Wang, W., Swamy, M. N. S., Ahmad, M. O. & Wang, Y. (2000b). A high-speed residue-to-

Zhang, W. & Siy, P. (2008). An efficient design of residue to binary converter for four moduli

*Circuits and Systems-II*, Vol. 47, No. 12, pp. 1576–1581.

superset {2n-1, 2n, 2n+1, 2n+1–1}. *Journal of Circuits, Systems and Computers*, Vol. 10,

numbers converters for (2*n*-1, 2*n*, 2*n*+1). *IEEE Transactions on Signal Processing*, Vol.

binary converter and a scheme of its VLSI implementation. *IEEE Transactions on* 

set (2*n*–1,2*n*+1, 22*n*–2, 22*n*+1–3) based on new CRT II. *Elsevier Journal of Information* 

*Annual Symposium on VLSI (ISVLSI'10)*, Kefalonia, Greece, Jul. 13-15. Molahosseini, A.S., Navi, K., Dadkhah, C., Kavehei, O. & Timarchi, S. (2010). Efficient

*and Systems-I*, Vol. 54, No. 6, pp. 1245-1254.

*Systems-I*, vol. 57, no. 4, pp. 823-835.

Imperial College Press, London.

*digital signal processing*, IEEE Press.

*Circuits and Devices*, Vol. 7, pp. 23-29.

54, pp. 156-163.

50–62.

pp. 85-99.

50, No. 7, pp. 1772-1779.

*Sciences*, Vol. 178, No. 1, pp. 264-279.

*Journal of Systems Architecture*, vol. 54, pp. 929-934.

*Transactions on Circuits and Systems-II*, Vol. 42, pp. 661-663.

Set {2n-1, 2n, 2n+1, 2n+1–1} and {2n-1, 2n, 2n+1, 2n+1+1}. *IEEE Transactions on Circuits* 

to Binary Converters for the New Moduli Sets {22n, 2n–1, 2n+1–1} and {22n, 2n–1, 2n–1–

1, 2n+1, 22n, 22n+1–1} Using CRT and MRC, *Proceedings of IEEE Computer Society* 

Reverse Converter Designs for the New 4-Moduli Sets {2n–1, 2n, 2n+1, 22n+1–1} and {2n–1, 2n+1, 22n, 22n+1} Based on New CRTs. *IEEE Transactions on Circuits and* 

for designing reverse converters based on a general three-moduli set. *Elsevier* 

System to Computer Scientists and Engineers. *IEEE Transactions on Education*, vol.

**Abstract**: A complexity metric for concurrent software controlled systems is defined and derived. It is equivalent or comparable to the Shannon information metric, which essentially measures entropy of a system, but uses a novel and efficient technique based on a FFT to calculate its value. This can be extended to other temporal realizations of behaviour.

For concurrent software, the amount of context switching that occurs indicates the level of complexity of the program. If the program consists of a fixed period cyclic scheduler, context switches will occur very predictably at those same fixed intervals. However, if the program consists of a mix of periods, some long, some short, and some perhaps sporadically aperiodic, the complexity of such a program will be much greater. Further, the greater the spread between short periods and long periods will indicate that the program will be much harder to verify, as the shorter cycles will accumulate more testing coverage than the longer or sporadic interval context switching.

In some sense, this is what makes clock-driven synchronous logic much easier to test. The state space of possible events is reduced in direct relation to the reduction of available slots for computation. As the context switching and potential interactions between threads occurs only at these slots, a simplification of the temporal behaviour here will result in a better chance to verify the correctness of its execution.

By the same token, any concurrent program will show greater non-determinism than an equivalent sequential program. The benefits of making a concurrent program more synchronous grants a greater predictability in its execution semantics. In terms of meeting hard real-time constraints, a synchronously designed program will have predictable points for schedulability, allowing for techniques such as rate-monotonic scheduling (Klein, 1993) to meet strict timing deadlines. These categories of techniques grant a concurrent program the same possibilities for verification as a sequential one.

Even though these strict scheduling programs are effective for their problem domains, they are difficult to maintain and do not scale that well for the large software systems required. In fact most interactive programs and other soft real-time systems use the concept of eventdriven semantics, which can allow interruption at any point in time. These have the benefit of dealing with interactions only upon demand, and so scale better, especially in terms of not placing a huge computational or communication load on the system when not needed.

Entropic Complexity Measured in Context Switching 353

**Periodic Impulse** 

**Disordered Autocorrelation with Mean** 

**Train**  ߜሺݐ െ ݊ܶሻ

**Noise** Random(u(t))

transformation process.

Thread #

ୀି low

Low Entropy Context Switching

(right). The low entropy signals are more regular.

where *N* is the order of the FFT.

Time

ߙή݁ିఈ௧ 1+ln(D)

Maximum entropy over time interval

Table 1. Mapping of signal entropies to their corresponding frequency domain entropies. The trend of increasing entropies with increasing disorder is maintained through the

Fig. 1. Example of context-switching waveforms with low entropy (left) and high entropy

In practice using a discrete Fourier transform such as the Fast Fourier Transform (FFT) (Brigham & Morrow, 1967) allows us to then use the summation variant of the entropy,

ܵൌെ σ ሺ݂ሻ ή ൫ሺ݂ሻ൯ ே

**Signal Entropy FT(Signal) Entropy** 

ߨʹ

ܶ

Random

Thread #

ߨʹ

ୀି low

Time

High Entropy Context Switching

ୀଵ (5)

ܶ ൰

<sup>ߙ</sup><sup>ଶ</sup> ߱<sup>ଶ</sup> ~ 2+ln(D)

Maximum entropy over frequency range

݇ െ ߱൬ ߜ

ߙ

in frequency domain

New events can also be handled well, either by adding a new thread or task or interrupt to watch for their occurrence or by adding an event handler to a queue.

Unfortunately these benefits of scalability and flexibility detract from being able to reason about the execution semantics of a program, thus leading to an element of non-determinism. One can thus consider the obvious potential for introducing stochastic ideas to qualitatively and quantitatively understand how complex a design has become. If we can attach a simple and fundamental measure to this complexity and thus generate a useful metric, it has the capacity for comparing software designs or of providing a continuous benchmark of growing potential complexity. That leads to the notion of supplying an entropy-based measure to characterize the behaviour of a concurrent software system.

#### **2. Approach – The context-switching entropy measure**

The premise is to evaluate the complexity of a concurrently executing program. A novel approach of creating such a complexity metric involves the analysis of the context-switching frequency spectrum. We take a Fourier transform (*FT*) of the temporally distributed context switching events, *c(t),* and treat that as a probability density function (Feller, 1957) in frequency space (i.e. a normalized power spectrum).

$$p(f) = \left| FT(c(t)) \right| = \left| \int\_{-\infty}^{\infty} c(t) \cdot e^{-i2\pi ft} \, dt \right| \tag{1}$$

Then the Shannon information entropy (*S*) of *p(f)* (Leon-Garcia, 2008) will generate a simple complexity measure.

$$S = -\int\_0^\infty p(f) \cdot \ln\left(p(f)\right) df \tag{2}$$

The entropy of a spectrum has meaning because the power spectrum itself transforms the autocorrelation of the signal in the time domain. The autocorrelation is nothing more than the probability of a time-shift occurring between points of interest. Thus by transforming the time domain information to the frequency domain we do not lose the essential information pertaining to order and disorder in the signal, in particular when it comes to comparing one waveform against another. The information containing the essential order is largely retained across the transformation.

$$prob(t) \approx autocorrelation(c(t)) = \int\_0^\infty c(t) \cdot c(t+\tau)d\tau \tag{3}$$

So the Fourier transform represents the same probabilities as that of the autocorrelation but computed in the frequency domain. For any captured sequence of events, the Fourier transform of the autocorrelation is practically obtained as the magnitude of the Fourier transform of the waveform.

$$p(f) = FT\{prob(t)\} = |FT(c(t))|\tag{4}$$

For a given mean frequency <*f*>, the value of *S* will be the greatest when the probability spread of the context switching frequency components is maximized. In this case, complexity tracks entropy and the maximum entropy principle (Jaynes, 2003) can be applied to understand the trend toward greater complexity. By Parseval's theorem, we do not lose the integrated signal when we take the Fourier transform.

This general approach is also known as Frequency Domain Entropy (Bao, 2004).

352 Applications of Digital Signal Processing

New events can also be handled well, either by adding a new thread or task or interrupt to

Unfortunately these benefits of scalability and flexibility detract from being able to reason about the execution semantics of a program, thus leading to an element of non-determinism. One can thus consider the obvious potential for introducing stochastic ideas to qualitatively and quantitatively understand how complex a design has become. If we can attach a simple and fundamental measure to this complexity and thus generate a useful metric, it has the capacity for comparing software designs or of providing a continuous benchmark of growing potential complexity. That leads to the notion of supplying an entropy-based

The premise is to evaluate the complexity of a concurrently executing program. A novel approach of creating such a complexity metric involves the analysis of the context-switching frequency spectrum. We take a Fourier transform (*FT*) of the temporally distributed context switching events, *c(t),* and treat that as a probability density function (Feller, 1957) in

ሺ݂ሻ ൌ หܨܶ൫ܿሺݐሻ൯ห ൌ ห ܿሺݐሻ ή ݁ିଶగ௧݀ݐ <sup>ஶ</sup>

Then the Shannon information entropy (*S*) of *p(f)* (Leon-Garcia, 2008) will generate a simple

ܵൌെ ሺ݂ሻ ή ൫ሺ݂ሻ൯ ݂݀ <sup>ஶ</sup>

The entropy of a spectrum has meaning because the power spectrum itself transforms the autocorrelation of the signal in the time domain. The autocorrelation is nothing more than the probability of a time-shift occurring between points of interest. Thus by transforming the time domain information to the frequency domain we do not lose the essential information pertaining to order and disorder in the signal, in particular when it comes to comparing one waveform against another. The information containing the essential order is largely retained

ݎܾሺݐሻ ൎ ܽݐݑܿݎݎ݈݁ܽݐ݅݊൫ܿሺݐሻ൯ ൌ ܿሺݐሻ ή ܿሺݐ߬ሻ݀߬ <sup>ஶ</sup>

So the Fourier transform represents the same probabilities as that of the autocorrelation but computed in the frequency domain. For any captured sequence of events, the Fourier transform of the autocorrelation is practically obtained as the magnitude of the Fourier

For a given mean frequency <*f*>, the value of *S* will be the greatest when the probability spread of the context switching frequency components is maximized. In this case, complexity tracks entropy and the maximum entropy principle (Jaynes, 2003) can be applied to understand the trend toward greater complexity. By Parseval's theorem, we do not lose

This general approach is also known as Frequency Domain Entropy (Bao, 2004).

ିஶ <sup>ห</sup> (1)

(3)

(2)

ሺ݂ሻ ൌ ܨܶ൫ݎܾሺݐሻ൯ ൌ ȁܨܶሺܿሺݐሻሻȁ (4)

watch for their occurrence or by adding an event handler to a queue.

measure to characterize the behaviour of a concurrent software system.

**2. Approach – The context-switching entropy measure** 

frequency space (i.e. a normalized power spectrum).

the integrated signal when we take the Fourier transform.

complexity measure.

across the transformation.

transform of the waveform.


Table 1. Mapping of signal entropies to their corresponding frequency domain entropies. The trend of increasing entropies with increasing disorder is maintained through the transformation process.

Fig. 1. Example of context-switching waveforms with low entropy (left) and high entropy (right). The low entropy signals are more regular.

In practice using a discrete Fourier transform such as the Fast Fourier Transform (FFT) (Brigham & Morrow, 1967) allows us to then use the summation variant of the entropy, where *N* is the order of the FFT.

$$S = -\sum\_{l=1}^{N} p(f\_l) \cdot \ln\left(p(f\_l)\right) \tag{5}$$

Entropic Complexity Measured in Context Switching 355

nature. When the concurrency becomes an integral aspect of the system and in particular when threads must interact with one another does the complexity start to increase quickly.

System designers will use tools such as state diagrams, activity diagrams, and Petri nets (Peterson 1981) to architect concurrency behaviors. A typical Petri net is shown below which illustrates the parallel activities and synchronous behaviors. These have various levels of contol, in that they can abstract some of the temporal scale differences by encapsulation of behaviours (Pukite & Ludwig, 2007). However, the actual synchronization will always

Fig. 3. A typical concurrent automata represented by a Petri net. Multiple threads are represented by filled tokens (places), and context switches occur when threads meet at synchronization points (bars). For large systems, the complexity of interactions canl grow

Degas (Ludwig & Pukite, 2006) provides such an instrumentation facility.

To effectively put the temporal complexity measure into practice, the program or simulation implementation will need a way to time stamp and data log context switches during execution. An implementation of a discrete event task scheduler such as the public source

The simplest input stream is a list of events signified by time-stamps, which will serve to capture the *c(t)* signal. The values for *c(t)* can be as simple as delta functions with a value of

This algorithm needs a FFT routine. The following pseudo-code guides the calculation calling a generic FFT routine named **Fourier8**. This assumes that each line contains the number of events that occur in the time step and is read through standard input. (This is set to read in exactly 2N lines, where N is an order 15 FFT) If sparse input is provided with time stamps then the data must be transferred into a discrete set of lines with either 0 or N events

That becomes the essence of the disorder or entropy that we try to capture.

reveal itself in the realized system.

well beyond this level.

unity indicating a context switch.

on each line, due to the nature of the discrete time FFT.

**2.1 Application** 

**2.2 Computation** 

The utility of the frequency domain approach is apparent when we consider that the frequency domain is partitioned into FFT bins and we have a very fast way of computing the metric, and in many cases more convenient than by calculating the autocorrelation, or by traditional periodogram (Schuster, 1898). If we did have the extra processing time, it may make sense to calculate the entropy off the autocorrelation as that would definitely retain more of the information measure, and thus not losing information due to the transformation process. In practice, however, many times we want to know only *relative* entropy measures, so the loss in precision is not as important as long as we maintain the relative ordering or rank.

As an intuitive interpretation of this measure consider that a value of *S* near 0 (if <*f*> normalized to 1) indicates that the program is very specialized and is only trying to a few things at once. As *S* approaches 1 it becomes a general purpose application that attempts to control many different behaviours or responds to events of a more random nature. If *S* goes much greater than 1, then the temporal complexity turns even more random, even approaching fractal. It becomes progressively harder to test such systems because of the diverging spread in time scales.

This turns into a useful metric since any sufficiently capable controller program or eventdriven system will try to accommodate as much real-world functionality as possible, and since the real world tends to disorder and to maximize entropy, this will track an increasingly complex scenario. So in practice, we will have fast cycles interspersed with slower cycles corresponding perhaps to human events or sporadic signals.

Fig. 2. Variation of power spectrum with increasing disorder. (a) Ordered signal with harmonics generates a low entropy signal. (b) Disordered signal generates a larger intropy.

In further practical terms, to apply the metric we need to run a simulation or collect data from a real or prototype system to gather the statistics. It may be possible to evaluate a program statically but that would require an introspective analysis of a program's structure, a task much more involved than collecting the statistics during typical use or as a test scenario.

The possibility of using this approach on non-concurrent programs such as a single-tasking event-driven program exists but these have less inherent complexity due to their sequential 354 Applications of Digital Signal Processing

The utility of the frequency domain approach is apparent when we consider that the frequency domain is partitioned into FFT bins and we have a very fast way of computing the metric, and in many cases more convenient than by calculating the autocorrelation, or by traditional periodogram (Schuster, 1898). If we did have the extra processing time, it may make sense to calculate the entropy off the autocorrelation as that would definitely retain more of the information measure, and thus not losing information due to the transformation process. In practice, however, many times we want to know only *relative* entropy measures, so the loss in

As an intuitive interpretation of this measure consider that a value of *S* near 0 (if <*f*> normalized to 1) indicates that the program is very specialized and is only trying to a few things at once. As *S* approaches 1 it becomes a general purpose application that attempts to control many different behaviours or responds to events of a more random nature. If *S* goes much greater than 1, then the temporal complexity turns even more random, even approaching fractal. It becomes progressively harder to test such systems because of the

This turns into a useful metric since any sufficiently capable controller program or eventdriven system will try to accommodate as much real-world functionality as possible, and since the real world tends to disorder and to maximize entropy, this will track an increasingly complex scenario. So in practice, we will have fast cycles interspersed with

precision is not as important as long as we maintain the relative ordering or rank.

slower cycles corresponding perhaps to human events or sporadic signals.

Fig. 2. Variation of power spectrum with increasing disorder. (a) Ordered signal with harmonics generates a low entropy signal. (b) Disordered signal generates a larger intropy. In further practical terms, to apply the metric we need to run a simulation or collect data from a real or prototype system to gather the statistics. It may be possible to evaluate a program statically but that would require an introspective analysis of a program's structure, a task much more involved than collecting the statistics during typical use or as a test scenario. The possibility of using this approach on non-concurrent programs such as a single-tasking event-driven program exists but these have less inherent complexity due to their sequential

diverging spread in time scales.

nature. When the concurrency becomes an integral aspect of the system and in particular when threads must interact with one another does the complexity start to increase quickly. That becomes the essence of the disorder or entropy that we try to capture.

System designers will use tools such as state diagrams, activity diagrams, and Petri nets (Peterson 1981) to architect concurrency behaviors. A typical Petri net is shown below which illustrates the parallel activities and synchronous behaviors. These have various levels of contol, in that they can abstract some of the temporal scale differences by encapsulation of behaviours (Pukite & Ludwig, 2007). However, the actual synchronization will always reveal itself in the realized system.

Fig. 3. A typical concurrent automata represented by a Petri net. Multiple threads are represented by filled tokens (places), and context switches occur when threads meet at synchronization points (bars). For large systems, the complexity of interactions canl grow well beyond this level.
