**4. Data representation ability of 4K stereo image**

In the data visualization using the 4K image, detailed information can be represented based on the high resolution image compared with the conventional visualization using the SXGA or WUXGA resolution image. Particularly, when the 4K stereo image is used, it is expected that the detailed three-dimensional information can be displayed with high accuracy in the three-dimensional virtual space. In this study, the representation ability of the 4K stereo image in the immersive visual data mining was examined.

#### **4.1 Perception of spatial resolution**

156 Applications of Virtual Reality

In order to generate and render the 4K stereo image, two high-end graphics workstations (Dell Precision T7400, 2xQuad Core Xeon 3.2GHz) with the graphics engine (NVIDIA Quadro Plex 1000 Model IV) that has a genlock function are used for right eye image and the left eye image, respectively. In this system, interface devices such as a USB game controller and a physical MIDI controller are connected to the graphics workstation. The USB game controller is used to walk through the visualized data space or to change the visualization method. On the other hand, the physical MIDI controller is used to change the visualization parameter minutely. By using these interface devices, the user can perform accurate and intuitive interaction with the visualized data, and the immersive visual data

Figure 4 shows the software structure of the immersive visual data mining environment. As for the software library to develop the visual data mining application in the immersive virtual environment, Open CABIN Library was used (Tateyama, 2008). Open CABIN Library is a software platform to develop the immersive virtual reality application. This

application 2

plug-in mechanism

master

……

renderer

device driver device driver OpenCABIN library

immersive projection display

The master-renderer programming is a style that constructs an application program consisting of a master part that calculates the state of the virtual world and renderer parts that render the image of the virtual world. This software configuration is effective to construct the network application that needs to access to the database or needs to transmit data to other system through the network, because master process has only to communicate

On the other hand, the plug-in mechanism is a method to construct an application program in the form of dynamically loaded library. When API of the usual library is used, it is necessary to rewrite the source code of the program to add a certain function to improve it. For example, in order to construct an application program that visualizes several kinds of data simultaneously, the integrated visualization program must be developed specially

Fig. 4. Software structure of immersive visual data mining application.

library has features of master-renderer programming and plug-in mechanism.

mining using 4K resolution stereo image can be realized.

application 1

master

renderer

**3.2 Software structure** 

with the remote processes.

First, the spatial resolution with which the visualized data can be perceived in the threedimensional space was experimentally measured. In this experiment, the subjects sat at a position 3m away from the screen, and two vertical parallel lines with the gaps of 0.5mm, 1mm, 2.5mm, 5mm, or 10mm were displayed randomly in the direction from the front to left as shown in figure 5. The subjects were asked to move the parallel lines in the depth direction and to stop them at the boundary position where they can recognize the displayed image as two lines by using the USB game controller. The boundary positions of the parallel lines with various gaps and in various directions were recorded.

Fig. 5. Condition of experiment on the perception of spatial resolution.

Figure 6 shows the result of the experiment for five subjects with visual acuity of more than 1.0. In this graph, the contour lines are drawn by connecting the average values of the

Immersive Visual Data Mining Based on Super High Definition Image 159

evaluate the three-dimensional sensation that was felt from the displayed image using the three-grade system (2: clear three-dimensional sensation, 1: unclear three-dimensional

Figure 8 shows the result of this experiment for nine subjects. Each plotted point means average value of the evaluation for each depth condition and for each near distance condition. From the results, there was significant difference among the condition of the distance to the near side at 1% level, though there was no significant difference among the condition of the depth of volume. From the graph, we can see that the evaluation of the depth sensation felt by the subjects decreased according to the increase of the number of the displayed points. This means that the subjects could not recognize the three-dimensional scene from the left eye image and the right eye image when too many point clouds were displayed. Namely, the representation ability of the super high definition display for the three-dimensional point cloud image would decrease when a large amount of point data are displayed simultaneously. In addition, the ability of representing three-dimensional point

depth

subject

number of point data (x 10,000)

1.0, 1.5, 2.0 or 2.5m

depth=1.0, near=1.0 depth=1.5, near=1.0 depth=2.0, near=1.0 depth=2.5, near=1.0 depth=1.0, near=2.0 depth=1.5, near=2.0 depth=2.0, near=2.0 depth=2.5, near=2.0 depth=1.0, near=3.0 depth=1.5, near=3.0 depth=2.0, near=3.0 depth=2.5, near=3.0

3m distance

Fig. 7. Condition of experiment on representation of point cloud data.

0 10 20 30 40 50

Fig. 8. Result of experiment on representation of point cloud data.

screen

rectangular solid area

1, 2 or 3m

point cloud data

field of view

0.0

0.5

1.0

three-dimensional sensation

1.5

2.0

sensation, 0: no three-dimensional sensation).

boundary positions for each direction and for each gap of the parallel lines. From this graph, we can see that the subjects could recognize 0.5mm gaps between lines when the parallel lines were displayed at the distance of about 50cm, and they could recognize 10mm gaps between lines when they were displayed at the distance of about 10m. This recognition ability is equivalent to visual acuity of about 0.3, and it is a little bad compared with the visual acuity (more than 1.0) of the subjects. From this result, we can consider that the effect of stereo image caused the decrease of the visual acuity in the visualization environment. However, the resolution of the parallel lines recognition at near distance (0.5mm) was higher than the pixel width (0.97mm) of the projected image. We can consider that this result is also caused by the effect of high resolution stereo image.

Therefore, it is understood that the 4K stereo display can represent the detailed information with high accuracy, though the recognized spatial resolution of the displayed image depends on the distance and direction from the user. Namely, this means that the visual data mining using the super high-definition stereo image has an ability to transmit a large amount of information from the computer to the user with high accuracy.

Fig. 6. Result of experiment on perception of spatial resolution.

#### **4.2 Representation of point cloud data**

Though this system can visualize detailed data with high accuracy using the super highdefinition stereo image, it has a limitation in transmitting large amount of information from the computer to the user. Next, in this study, the ability of representing three-dimensional information in the visualization of large amount of point cloud data was experimentally examined.

In the experiment, the subjects stood at the position 3m away from the screen and the image of the colored point cloud that was distributed uniformly in the rectangular solid area was displayed. In this case, the number of point cloud data was changed up to 500,000 points, and the distance to the near side of the rectangular solid area was changed among 1m, 2m and 3m. In addition, the depth of the volume of the rectangular solid area was changed among 1.0m, 1.5m, 2.0m, and 2.5m, as shown in figure 7. Then, the subjects were asked to

boundary positions for each direction and for each gap of the parallel lines. From this graph, we can see that the subjects could recognize 0.5mm gaps between lines when the parallel lines were displayed at the distance of about 50cm, and they could recognize 10mm gaps between lines when they were displayed at the distance of about 10m. This recognition ability is equivalent to visual acuity of about 0.3, and it is a little bad compared with the visual acuity (more than 1.0) of the subjects. From this result, we can consider that the effect of stereo image caused the decrease of the visual acuity in the visualization environment. However, the resolution of the parallel lines recognition at near distance (0.5mm) was higher than the pixel width (0.97mm) of the projected image. We can consider that this

Therefore, it is understood that the 4K stereo display can represent the detailed information with high accuracy, though the recognized spatial resolution of the displayed image depends on the distance and direction from the user. Namely, this means that the visual data mining using the super high-definition stereo image has an ability to transmit a large

0 2 4 6 8 10 12 14

Though this system can visualize detailed data with high accuracy using the super highdefinition stereo image, it has a limitation in transmitting large amount of information from the computer to the user. Next, in this study, the ability of representing three-dimensional information in the visualization of large amount of point cloud data was experimentally

In the experiment, the subjects stood at the position 3m away from the screen and the image of the colored point cloud that was distributed uniformly in the rectangular solid area was displayed. In this case, the number of point cloud data was changed up to 500,000 points, and the distance to the near side of the rectangular solid area was changed among 1m, 2m and 3m. In addition, the depth of the volume of the rectangular solid area was changed among 1.0m, 1.5m, 2.0m, and 2.5m, as shown in figure 7. Then, the subjects were asked to

distance from the subject (m)

5 deg.

10 deg.

15 deg.

20deg.

10 mm interval

30 deg. 25 deg.

result is also caused by the effect of high resolution stereo image.

2.5 mm interval

Fig. 6. Result of experiment on perception of spatial resolution.

out of view field

screen

(m) 6

5

4

3

front left side

0.5mm interval 1 mm interval

**4.2 Representation of point cloud data** 

2

1

0

examined.

amount of information from the computer to the user with high accuracy.

5 mm interval evaluate the three-dimensional sensation that was felt from the displayed image using the three-grade system (2: clear three-dimensional sensation, 1: unclear three-dimensional sensation, 0: no three-dimensional sensation).

Figure 8 shows the result of this experiment for nine subjects. Each plotted point means average value of the evaluation for each depth condition and for each near distance condition. From the results, there was significant difference among the condition of the distance to the near side at 1% level, though there was no significant difference among the condition of the depth of volume. From the graph, we can see that the evaluation of the depth sensation felt by the subjects decreased according to the increase of the number of the displayed points. This means that the subjects could not recognize the three-dimensional scene from the left eye image and the right eye image when too many point clouds were displayed. Namely, the representation ability of the super high definition display for the three-dimensional point cloud image would decrease when a large amount of point data are displayed simultaneously. In addition, the ability of representing three-dimensional point

Fig. 7. Condition of experiment on representation of point cloud data.

Fig. 8. Result of experiment on representation of point cloud data.

Immersive Visual Data Mining Based on Super High Definition Image 161

In the data visualization, it is important that the user can interact intuitively with the visualized data to recognize the phenomenon. In addition, in the visual data mining, it is also important that the user can operate the visualized data precisely. Therefore, in this system, GUI interface and physical USB controller are used to realize the intuitive and

Fig. 10 shows the screen image of the GUI interface. The visualization parameters X, Y, Z, C,

X=*a*11*d*1+*a*12*d*2++*a*13*d*3+*a*14*d*4+*a*15*d*5+*a*16*d*6+*a*17*d*7+*a*18*d*8

Y=*a*21*d*1+*a*22*d*2++*a*23*d*3+*a*24*d*4+*a*25*d*5+*a*26*d*6+*a*27*d*7+*a*28*d*8

Z=*a*31*d*1+*a*32*d*2++*a*33*d*3+*a*34*d*4+*a*35*d*5+*a*36*d*6+*a*37*d*7+*a*38*d*8

C=*a*41*d*1+*a*42*d*2++*a*43*d*3+*a*44*d*4+*a*45*d*5+*a*46*d*6+*a*47*d*7+*a*48*d*8

R=*a*51*d*1+*a*52*d*2++*a*53*d*3+*a*54*d*4+*a*55*d*5+*a*56*d*6+*a*57*d*7+*a*58*d*8 where *d*1,.., *d*8 are data values in the floating point number fields of the database table, and *a*11,…, *a*58 are coefficients that are specified by the user. Each tab in the right side on the GUI screen corresponds to the visualization parameter, and the user can define these

The data is displayed at the position specified by parameters of X, Y, Z. In this case, visualized data can be selected by specifying the rectangular region in the three-dimensional space. This area is defined by giving the center position of the rectangular region and the width along each axis using the sliders displayed on the left side of the GUI interface.

visualization parameters by moving the sliders to adjust the coefficients.

**5.2 Interface** 

precise operation.

Fig. 10. GUI interface.

and R are defined using the linear equation,

cloud image decreased steeply when the data were displayed near the subjects. We can consider that this is because the binocular parallax was large when the point image was displayed close to the subject in front of the screen. Therefore, we can understand that the control of the number of the visualized data is very important when a large amount of point cloud data are visualized.

#### **5. Visual data mining platform**

In general, effective visualization method used in visual data mining depends on the visualized data or the purpose of data analysis. In particular, in the early phase of the data mining, visualization method or visualization parameter are often changed in trials and errors. Therefore, the development of the platform of immersive visual data mining in which the user can easily analyze data by changing the visualization method or visualization parameter interactively is required. In this study, the platform of immersive visual data mining for the point cloud data was developed, because the super highdefinition display can be used effectively to represent the point data.

#### **5.1 Process of visual data mining**

In the platform of immersive visual data mining, multi-dimensional data that is stored in the database table managed by RDBMS (relational database management system) can be visualized. This database table consists of ID field and eight floating point number fields (*d*1, *d*2, …, *d*8). Then, data in various application fields can be visualized in the immersive visual data mining environment by storing them in the database table.

Figure 9 shows the data flow in the process of immersive visual data mining. In order to visualize the data in the three-dimensional space, several attributes such as the threedimensional position, color, and filtering condition should be specified. In this system, five visualization parameters such as the position (X, Y, Z), color (C) and reference parameter for filtering (R) are made from the values in the floating point number fields of the database table, and the data is visualized based on the five parameters. Thus, the user can perform the visual data mining independently from the domain or type of the target data.

Fig. 9. Data flow of visual data mining in platform.

#### **5.2 Interface**

160 Applications of Virtual Reality

cloud image decreased steeply when the data were displayed near the subjects. We can consider that this is because the binocular parallax was large when the point image was displayed close to the subject in front of the screen. Therefore, we can understand that the control of the number of the visualized data is very important when a large amount of point

In general, effective visualization method used in visual data mining depends on the visualized data or the purpose of data analysis. In particular, in the early phase of the data mining, visualization method or visualization parameter are often changed in trials and errors. Therefore, the development of the platform of immersive visual data mining in which the user can easily analyze data by changing the visualization method or visualization parameter interactively is required. In this study, the platform of immersive visual data mining for the point cloud data was developed, because the super high-

In the platform of immersive visual data mining, multi-dimensional data that is stored in the database table managed by RDBMS (relational database management system) can be visualized. This database table consists of ID field and eight floating point number fields (*d*1, *d*2, …, *d*8). Then, data in various application fields can be visualized in the immersive visual

Figure 9 shows the data flow in the process of immersive visual data mining. In order to visualize the data in the three-dimensional space, several attributes such as the threedimensional position, color, and filtering condition should be specified. In this system, five visualization parameters such as the position (X, Y, Z), color (C) and reference parameter for filtering (R) are made from the values in the floating point number fields of the database table, and the data is visualized based on the five parameters. Thus, the user can perform

> visualization parameter

visual data mining

MIDI controller/ game controller

the visual data mining independently from the domain or type of the target data.

parameter definition

GUI interface/ MIDI controller

definition display can be used effectively to represent the point data.

data mining environment by storing them in the database table.

cloud data are visualized.

**5. Visual data mining platform** 

**5.1 Process of visual data mining** 

database table

Fig. 9. Data flow of visual data mining in platform.

pre-processing

data in application fields In the data visualization, it is important that the user can interact intuitively with the visualized data to recognize the phenomenon. In addition, in the visual data mining, it is also important that the user can operate the visualized data precisely. Therefore, in this system, GUI interface and physical USB controller are used to realize the intuitive and precise operation.

Fig. 10 shows the screen image of the GUI interface. The visualization parameters X, Y, Z, C, and R are defined using the linear equation,

> X=*a*11*d*1+*a*12*d*2++*a*13*d*3+*a*14*d*4+*a*15*d*5+*a*16*d*6+*a*17*d*7+*a*18*d*8 Y=*a*21*d*1+*a*22*d*2++*a*23*d*3+*a*24*d*4+*a*25*d*5+*a*26*d*6+*a*27*d*7+*a*28*d*8 Z=*a*31*d*1+*a*32*d*2++*a*33*d*3+*a*34*d*4+*a*35*d*5+*a*36*d*6+*a*37*d*7+*a*38*d*8 C=*a*41*d*1+*a*42*d*2++*a*43*d*3+*a*44*d*4+*a*45*d*5+*a*46*d*6+*a*47*d*7+*a*48*d*8 R=*a*51*d*1+*a*52*d*2++*a*53*d*3+*a*54*d*4+*a*55*d*5+*a*56*d*6+*a*57*d*7+*a*58*d*8

where *d*1,.., *d*8 are data values in the floating point number fields of the database table, and *a*11,…, *a*58 are coefficients that are specified by the user. Each tab in the right side on the GUI screen corresponds to the visualization parameter, and the user can define these visualization parameters by moving the sliders to adjust the coefficients.

Fig. 10. GUI interface.

The data is displayed at the position specified by parameters of X, Y, Z. In this case, visualized data can be selected by specifying the rectangular region in the three-dimensional space. This area is defined by giving the center position of the rectangular region and the width along each axis using the sliders displayed on the left side of the GUI interface.

Immersive Visual Data Mining Based on Super High Definition Image 163

In this study, in order to evaluate the effectiveness of the developed system, the platform of immersive visual data mining was applied to the seismic data analysis. In Japan, a lot of earthquakes occur every year. Then, it is important to analyze large amount of past data to predict next earthquake and estimate damage (Furumura, 2005). Particularly, this problem has been very important after Great East Japan Earthquake on March 11, 2011. As original data for the immersive visual data mining, the seismic hypocenter data in last five years that is stored in Hi-net system (Okada, 2004) constructed by National Research Institute for Earth

In the visualization using the conventional two-dimensional display, it is difficult to observe both the individual location and the overall distribution of the hypocenter data and to understand the feature of three-dimensional distribution of the earthquake. Therefore, first, the seismic hypocenter data was visualized using the platform of representing point cloud data in the super high-definition immersive visual data mining environment. In this system, about 600,000 seismic hypocenter data with magnitude and occurrence time were stored in the database table and they were visualized as point clouds. From the hypocenter data, statistical data of b-value in Gutenberg-Richter (GR) relation were calculated. GR relation is a typical model to represent the distribution of magnitude of the earthquakes. In this case, more than 2,000,000 b-value data were calculated in the area specified by time and place, and they were stored in another database table to visualize them in the platform. In this application, these data were visualized overlapped with the coastal line of Japan using the

Fig. 12. Operation using GUI interface and physical MIDI controller.

**6. Application to seismic data analysis** 

Science and Disaster Prevention was used.

**6.1 Visualization of hypocenter data** 

plug-in function of Open CABIN Library.

Point cloud data can be displayed in various colors according to the value of the color parameter C. In this system, the color of the displayed data is defined by specifying minimum value and maximum value of the color parameter using the sliders, and the value of the color parameter is mapped to the displayed color. In this case, when the color value is less than minimum value, the data is visualized with gray color, and when the color value exceeds the maximum value, the data is visualized with white color.

The displayed data can be filtered using the value of the reference parameter R. When the center value and the range are specified using the sliders on the left side of the GUI screen, the visualized data is filtered by comparing the value of reference parameter with the filtering range.

In addition, the view point for the visualization image in the three-dimensional space can be controlled by dragging the cursor in the lower left area of the GUI interface. This function supports the rotation of the view point around the arbitrary axis as well as the movement along the screen and perpendicular to the screen. Then, the user's view point can be controlled minutely using the GUI interface as well as the user can walk through the visualization space using the game controller.

Though the parameter values are controlled using the sliders on the GUI interface, these sliders are assigned to the MIDI control channels. Then, the parameter values can also be controlled by using the physical MIDI controller connected to the interface PC shown in Figure 11. Since the MIDI controller communicates with the interface PC using USB protocol, the control data are sent to the visual data mining system through the interface PC. Therefore, the user can control the plural parameter values interactively in the immersive virtual environment by operating the physical MIDI controller without looking at the console. Figure 12 shows that the user is operating the platform of immersive visual data mining using the GUI interface on the interface PC and the MIDI controller.

Fig. 11. Physical MIDI controller.

Point cloud data can be displayed in various colors according to the value of the color parameter C. In this system, the color of the displayed data is defined by specifying minimum value and maximum value of the color parameter using the sliders, and the value of the color parameter is mapped to the displayed color. In this case, when the color value is less than minimum value, the data is visualized with gray color, and when the color value

The displayed data can be filtered using the value of the reference parameter R. When the center value and the range are specified using the sliders on the left side of the GUI screen, the visualized data is filtered by comparing the value of reference parameter with the

In addition, the view point for the visualization image in the three-dimensional space can be controlled by dragging the cursor in the lower left area of the GUI interface. This function supports the rotation of the view point around the arbitrary axis as well as the movement along the screen and perpendicular to the screen. Then, the user's view point can be controlled minutely using the GUI interface as well as the user can walk through the

Though the parameter values are controlled using the sliders on the GUI interface, these sliders are assigned to the MIDI control channels. Then, the parameter values can also be controlled by using the physical MIDI controller connected to the interface PC shown in Figure 11. Since the MIDI controller communicates with the interface PC using USB protocol, the control data are sent to the visual data mining system through the interface PC. Therefore, the user can control the plural parameter values interactively in the immersive virtual environment by operating the physical MIDI controller without looking at the console. Figure 12 shows that the user is operating the platform of immersive visual data

mining using the GUI interface on the interface PC and the MIDI controller.

exceeds the maximum value, the data is visualized with white color.

visualization space using the game controller.

Fig. 11. Physical MIDI controller.

filtering range.

Fig. 12. Operation using GUI interface and physical MIDI controller.
