**8. Advances in software technology**

As seen in the clinical validation, the sensitivity observed with visual analysis is acceptable, but specificity is lower than desired with visual interpretation. Further, visual observations and heuristic categorization are subject to human error and variation through subjective inter‐ pretation. To solve these problems, there are automated and semiautomated approaches for diagnostics [35]. We review some of the software tools available from companies who are intending to provide a replicable method of interpreting thermal images.

We review the technology used in three such software tools from Niramai health, Total vision and Mammo vision. All these approaches use static images obtained after cooling the subject with discrete imaging protocols.

#### **8.1. Visualization tools for thermal interpretation**

Given that a thermologist has to look at five colored images, where the temperature differ‐ ences between neighboring regions need to be identified by minute color variations, inter‐ pretation of thermal breast images is a huge cognitive overload and very error prone. So, software tools that aid in visualization and capturing of the observations about thermal pat‐ terns are becoming available.

Total vision software from Med‐hot.com gives an excellent visualization of the thermal images and additional support for a thermographer to systematically look for specific abnormal ther‐ mal pattern alongside a rule‐based decision‐making support to simplify the interpretation process. However, it does not have any automation of the diagnosis.

Mammo vision [31] is a semi‐automated tool that tries to identify the non‐vascular abnormal thermal patterns during dynamic thermography with cold challenge. It considers 10 images in total, 5 images before cooling and 5 images after cooling, for the analysis. An elliptical grid is used to approximate breast region, and it automatically extracts the lateral symmetry, isothermia in each quadrant, areolar temperature, nipple temperature, temperature decrease with cooling and hotspot parameter. Additionally, the clinician can manually identify the vascularity in the breast by looking at grayscale thermal image, which is then used by the tool to categorize the subjects into five groups. The tool defines assessment criteria called Breast Infrared Assessment System (BIRAS) with which they categorize the images into five groups with BIRAS 1 being low risk and BIRAS 5 being high risk.

#### **8.2. Use of sophisticated computer-aided diagnostics**

Use of sophisticated artificial intelligence algorithms for enabling automatic diagnosis or clinical interpretation guidance is most needed to reduce subjectivity in interpretation [37]. Niramai Thermalytix software is one such advanced software tool with a technology that enables end‐to‐end fully automated approach for the diagnosis [38–40]. The Niramai tool uses complex computer algorithms for the following five key aspects of automated diagnostics.

#### 1. Autotagging

Since one single view may not be sufficient to capture tumor region in different parts of the breast region, multiple views are taken. Typically, there are five thermal images in mul‐ tiple views that are captured; one of the common mistakes done by clinicians is to name the image wrongly. It is observed that many a times humans are confused with classification of right and left sides of breast in the image correctly and resulting in improper tagging of lateral and oblique views. Hence, Niramai software provides an automated tagging support. This reduces the error in naming or false tagging, which in turn would have resulted in other errors such as segmentation error and misclassification of subjects. Their software automati‐ cally tags the views based on the body border curvature and body area.

#### 2. Detecting the region of interest

We review the technology used in three such software tools from Niramai health, Total vision and Mammo vision. All these approaches use static images obtained after cooling the subject

Given that a thermologist has to look at five colored images, where the temperature differ‐ ences between neighboring regions need to be identified by minute color variations, inter‐ pretation of thermal breast images is a huge cognitive overload and very error prone. So, software tools that aid in visualization and capturing of the observations about thermal pat‐

Total vision software from Med‐hot.com gives an excellent visualization of the thermal images and additional support for a thermographer to systematically look for specific abnormal ther‐ mal pattern alongside a rule‐based decision‐making support to simplify the interpretation

Mammo vision [31] is a semi‐automated tool that tries to identify the non‐vascular abnormal thermal patterns during dynamic thermography with cold challenge. It considers 10 images in total, 5 images before cooling and 5 images after cooling, for the analysis. An elliptical grid is used to approximate breast region, and it automatically extracts the lateral symmetry, isothermia in each quadrant, areolar temperature, nipple temperature, temperature decrease with cooling and hotspot parameter. Additionally, the clinician can manually identify the vascularity in the breast by looking at grayscale thermal image, which is then used by the tool to categorize the subjects into five groups. The tool defines assessment criteria called Breast Infrared Assessment System (BIRAS) with which they categorize the images into five groups

Use of sophisticated artificial intelligence algorithms for enabling automatic diagnosis or clinical interpretation guidance is most needed to reduce subjectivity in interpretation [37]. Niramai Thermalytix software is one such advanced software tool with a technology that enables end‐to‐end fully automated approach for the diagnosis [38–40]. The Niramai tool uses complex computer algorithms for the following five key aspects of automated diagnostics.

Since one single view may not be sufficient to capture tumor region in different parts of the breast region, multiple views are taken. Typically, there are five thermal images in mul‐ tiple views that are captured; one of the common mistakes done by clinicians is to name the image wrongly. It is observed that many a times humans are confused with classification of right and left sides of breast in the image correctly and resulting in improper tagging of lateral and oblique views. Hence, Niramai software provides an automated tagging support. This reduces the error in naming or false tagging, which in turn would have resulted in other errors such as segmentation error and misclassification of subjects. Their software automati‐

cally tags the views based on the body border curvature and body area.

process. However, it does not have any automation of the diagnosis.

with BIRAS 1 being low risk and BIRAS 5 being high risk.

**8.2. Use of sophisticated computer-aided diagnostics**

with discrete imaging protocols.

100 New Perspectives in Breast Imaging

terns are becoming available.

1. Autotagging

**8.1. Visualization tools for thermal interpretation**

The thermal image is captured with the patient sitting about three feet from the camera. This captures the thermal signature of the top part of body of the subject starting from neck region. A tool like Niramai that does automatic analysis of breast cancer has to accurately crop the region of interest (ROI), namely the breast tissue region. For this, Niramai tool removes inframammary fold, axilla, sternum and thyroid regions that are usually warm regions and might unnecessarily cause false positives. Additional heuristic based on the shape of body gives accurate segmentation of the ROI as shown in **Figure 3**. There is considerable research in the detection of ROI for single view [35], and tools that provide manual support through freehand segmentation and adjustable and draggable ellipse that the clinician can use mark the region of interest. Niramai software automatically detects the breast region using a poly‐ gon approximation of region that makes it easier for a clinician to edit, if needed.

#### 3. Tumor localization

Once the region of interest for analysis is determined, next technical challenge is to accurately identify the exact location of an abnormality or a lesion. This usually means detecting regions having warm and hot temperature pixels in the image and analyzing the heat pattern around the same. The heat patterns found in the thermal images are then analyzed for specific tumor properties. Tumor‐specific patterns include multiple important thermal patterns or features that typically help in discriminating malignancy versus benign conditions [38].

Symmetry plays a significant role in detecting whether a hot patch is abnormal. So, a subset of the ROI showing a significant increase in temperature as compared to the neighboring areas and contralateral sides is identified. In NIRAMAI, two varieties of abnormal regions

**Figure 3.** Results of automated segmentation in different views. (a) Frontal (b) Left Oblique (c) Right Oblique (d) Left Lateral (e) Right Lateral.

are extracted, hot‐spots and warm‐spots, based on the degree of their thermal response. This categorization helps to increase sensitivity with low thermal response tumors without increasing the false positives. Hot‐spots correspond to high‐temperature regions segmented using a combination of temperature‐based thresholds. Warm‐spots correspond to slightly lower temperature regions as compared to hot‐spots with a change in parameters. One way of categorizing the same is using the modes and maximum temperature values, as shown in Eqs. (1) and (2).

$$T\_a = \ T\_{\text{overallaux}} - \mathbf{T} \tag{1}$$

$$T\_b = \Gamma + \mathcal{P}(T\_{\text{overallmax}} - \Gamma) \tag{2}$$

In above equations, Γ refers to the mean of the modes of the ROI temperature histograms in all views, and *T*overallmax represents the overall maximum temperature in all views. (*Ρ*, *<sup>Τ</sup>*) are param‐ eters chosen depending on the dataset.

Niramai tool detects hot‐spots and warm‐spots in each view of the subject. The best views of hot‐spots and warm‐spots are defined as the view in which the normalized size of the detected abnormal regions with respect to the ROI is maximum. **Figure 4** shows some sample subject images with their corresponding hotspots identified by NIRAMAI tool. From the detected hot‐spots in multiple views, the hot‐spots and warm spots corresponding to the best view are usually used to extract core features. Since symmetry places an important role, features are also extracted using the best view and its contralateral side view.

**Figure 4.** Sample subject images for (a) hormone‐sensitive tissues showing warm‐spots, (b) lactating case showing warm‐spots, (c) malignant case showing hot‐spots, (d) benign case showing warm‐spots and (e) normal case.

#### 4. Feature extraction

are extracted, hot‐spots and warm‐spots, based on the degree of their thermal response. This categorization helps to increase sensitivity with low thermal response tumors without increasing the false positives. Hot‐spots correspond to high‐temperature regions segmented using a combination of temperature‐based thresholds. Warm‐spots correspond to slightly lower temperature regions as compared to hot‐spots with a change in parameters. One way of categorizing the same is using the modes and maximum temperature values, as shown in

*Ta* = *Toverallmax* − Τ (1)

*Tb* = Γ + Ρ(*Toverallmax* − Γ ) (2)

In above equations, Γ refers to the mean of the modes of the ROI temperature histograms in all views, and *T*overallmax represents the overall maximum temperature in all views. (*Ρ*, *<sup>Τ</sup>*) are param‐

Niramai tool detects hot‐spots and warm‐spots in each view of the subject. The best views of hot‐spots and warm‐spots are defined as the view in which the normalized size of the detected abnormal regions with respect to the ROI is maximum. **Figure 4** shows some sample subject images with their corresponding hotspots identified by NIRAMAI tool. From the detected hot‐spots in multiple views, the hot‐spots and warm spots corresponding to the best view are usually used to extract core features. Since symmetry places an important role, features are

**Figure 4.** Sample subject images for (a) hormone‐sensitive tissues showing warm‐spots, (b) lactating case showing

warm‐spots, (c) malignant case showing hot‐spots, (d) benign case showing warm‐spots and (e) normal case.

Eqs. (1) and (2).

102 New Perspectives in Breast Imaging

eters chosen depending on the dataset.

also extracted using the best view and its contralateral side view.

Once the hot‐ and warm‐spots showing potential lesion is detected, three high‐level proper‐ ties of the lesion are extracted. These are boundary features, thermal symmetry and tempera‐ ture distribution.

Malignant tumor cells are aggressive in nature, which makes them to invade surrounding tissues by rupturing through the boundary formed by basal laminas [19]. This makes the boundary irregular for malignant cases compared to non‐malignant and benign cases which behave similar to normal cells.

In the case of malignant tumors, benign tumors, inflammation or wound‐healing cases, an increase in temperature in the abnormal regions is observed. This leads to a difference in ther‐ mal heat patterns compared to the contralateral breasts. However, similarity in thermal heat patterns is seen for normal, hormonal, lactating conditions [12, 22, 36] due to the presence of similar hormone‐sensitive tissues in both the breasts. This property is captured by including symmetrical features.

Finally, the mean temperature difference between the detected abnormal region and the remaining region of interest is calculated to get the relative increase in temperature compared to the neighboring region. In addition, many other temperature parameters of the abnormal region can be used for analysis.

#### 5. Automated classification

Computer algorithms based on artificial intelligence and machine learning are making huge inroads in automated diagnostics [38]. Many methods of supervised classification are being developed where a small group of patient data is used to train a probabilistic model that represents the decision criteria based on the extracted features. A simple such classifier is a random forest that is able to identify the significant discriminatory features and learns a com‐ bination of the features and feature groups that helps decide on malignancy subjects. Other classifiers include support vector machines, Kmeans classifiers and deep learning.
