The 2018 AAAI Spring Symposium Series IoT-Based Emotion Recognition Robot to Enhance Sense of Community in Nursing Home Shintaro Nagama, Masayuki Numao Department of Communication Engineering and Informatics The University of Electro-Communications n1411227@edu.cc.uec.ac.jp Abstract Senior isolation is becoming a major social problem in Japan, as a super-aged society where more than a quarter of population is over 65 years old. Many elderly people are living in single-resident homes without family or social support. Even in nursing home, residents stay in their private bedrooms lonely without participating social activities, such as chatting, playing game, watching TV together at a living room, etc. Since social isolation leads to serious consequences such as disuse syndrome, mental depression, suicide etc., maintaining person s sense of community is very important. But measuring sense of community is difficult because it is a mental process and many kinds of activities and interactions are involved in the process. In this paper, we define Social Activities of Daily Living (SADL) to focus on social activities to enhance the sense of community. We also propose a multimodal sensor based recognition method for SADL, which is implemented in the IoT-based emotion recognition robot for nursing environment. The robot monitors the daily activities and emotions of the residents, estimates the social relationships of the residents, takes care of the residents who are isolated from the community, and reduces their loneliness feelings by forming a good relationship in community. Introduction According to the United Nations, the population aging is progressing all over the world (United Nations 2015). As of September 2013, Japan became a super aging society where one quarter of the population is aged 65 years or older (Ministry of statistics 2013). Consequently, from April 2000 to April 2013, the number of nursing home residents has increased 1.71 times (Ministry of Health, Labor and Welfare 2014). In the survey of personnel shortfall for nursing home in 2017, 62.6% of nursing home answered that the caregiver was short (Person nursing labor stability Copyright 2018, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. center 2016). Despite the fact that the number of care recipients is increasing, care workers are in short supply. Therefore, the government encourage to introduce ICT in nursing environment to mitigate the workload of caretakers. In this context, many types of monitoring system are developed for nursing home. However, the monitoring system in the nursing home in the past is many in the bedroom and the area to be monitored is narrow. The ideal monitoring system monitors the entire nursing home such as the bathroom, the toilet, the dining room, the discourse room as well as the bedroom. The dining room and discourse room can be accessed during the day and anyone can use it. Sadly, there exist some residents who spend most of their time in the bedroom without participating in the social activities such as television appreciation, conversation and recreation in the lounge, etc. These people are suffered from social isolation, which leads to many harmful health conditions (Nicholas 2012), which is becoming a problem not only in Japan but also worldwide. The social isolation is caused by various incidents, such as physical weakness, physical disorder, mental illness, deterioration of psychological function, social loss, and when these occur simultaneously, the elderly people often face to a deep social isolation (Karatsu 2012). It is also involved in depression, which is caused by a big life event such as relatives disappearing and chronic stress (Ministry of Health 2009). We consider that creating a good relationship in the community is a useful mean to prevent social isolation. By creating a good community relationship, social activities are increased such as dining with others, chatting, playing game, watching TV together, helping others etc., which, in turn, decreases the loneliness feeling. In this paper, we define Social Activities of Daily Living (SADL). Compared with ADL and Instrumental ADL (IADL), the former focuses on person s physical self-care abilities to perform independent living. And the latter fo- 253
cuses on person s mental-involved complex activities, SADL focuses on the social activities to measure the person s sense of community. Then, we propose a multimodal sensor-based recognition method for SADL, which is implemented in the IoT-based emotion recognition robot for nursing environment. The robot first recognizes person s activities and change of emotions by integrating the sensors data: microwave sensor for vital data, video camera for the facial expressions, microphone for speech tone, environmental sensors for temperature, and brightness etc. Then, it evaluates the ADL, IADL index. SADL is also evaluated by monitoring the social relationships of the residents. The robot takes care of the residents who are isolated from the community by chatting on health-related talk. It also advises to create a good relationship in the community. Figure 2 use case diagram Related Work Relationship between Loneliness and ADL Relationship between loneliness or social isolation and various patterns of daily living are investigated(goonawardene et al. 2017). In their research, Activity of homebound elderly people are analyzed. It is shown that the time spent in the living room is significantly correlated with the emotional loneliness. This suggests that single living elderly person who spends most time in living room feels lonely. Correlation of daytime napping duration with social loneliness is also analyzed, suggesting that if elderly people lack of sense of community, they sleep more during day time. From these studies, it can be considered that social isolation and loneliness can be measured by actively in daily living (ADL). Furthermore, this study shows that depression in the elderly people correlates with the loneliness and social isolation. Emotion Recognition Emotions can be recognized from various things such as facial expressions, voice, sentences, body temperature and so on. In patented technology by Panasonic (Panasonic), we recognize feelings form talking content and sounds of voice. In addition, the Empath API (Smartmedical) recognized feelings using the physical sound of speech. Microsoft s Emotion API (Microsoft)can recognize emotions from facial expressions. Recognition of facial expressions is also implemented as a function in the Omron camera module(omron) that is used in our system. Techniques for recognizing emotions like these are actively researched and these are applied in various fields. Figure 1 Ideal Monitoring System Proposed Monitoring System The monitoring system in nursing home needs to be widespread including dining room and discourse room. The use case diagram of the monitoring system is shown in the Figure 1. "People", "Room" and "Items" on the right side indicate objects to be monitored. In the Target Model, it refers to the state to be monitored. The "Sensor" in the Health Monitoring System reads the information in the Target Model and recognizes what kind of state it is. "User" is the person who actually uses the monitor system. If you look at these in a nursing home, "People" are carerequiring persons who use a nursing home, "Room" is a room of a nursing home such as a bedroom, a living space, and "Items" is toothbrushes, dishes used for meals, furniture, and so on. "User" corresponds to a caregiver or a doctor. In Based on this use case diagram, we propose a monitoring system targeting the entire facility including the living space used by multiple people. Illustration of a desirable monitoring system in our proposal is shown in the Figure 2. It monitors the entire facility, including bedrooms 254
that are often found in the conventional products. In addition to abnormality detection, we propose a system that utilizes information such as behavior recognition, conversation and expression, which leads to enhance community and living in the nursing home of the elderly. In this paper, we describe the implementation, experiment and evaluation of the IoT based robot with multiple sensors, which is developed to realize the proposed functions. Proposed Model of Loneliness In this research, we aim to measure the loneliness of the elderly automatically by the system, which in turn acts to support a creation of a good sense of community. The loneliness depends on several things, such as mental process and physical process. So, we propose to measure the loneliness of elderly people in three axes: physical loneliness, mental loneliness, and social loneliness. Physical loneliness is simply based on whether or not there are people around the elderly. Mental loneliness is measured from emotions such as expression. Also, actions such as reading and talking are related to mental loneliness. Social loneliness is measured by the amount of social activity such as recreation. Model of Loneliness Calculation of the loneliness of each of the three axes is performed by recognizing the actions in daily living. Consideration is given to the relationships between typical actions in daily living and the physical loneliness, mental loneliness and social solitude. Table 1 gives examples of points of senior citizens' behavior and loneliness levels related to them. For example, consider the act of eating with others in the dining room at meal time, that action is considered to be beneficial to reduce the physical loneliness and social isolation, thus the good points are given to these two axes. The higher the score of each axis, the lower the loneliness of elderly people feel. Define a table that combines such actions and points for typcical activities in daily living. Table 1 relationship between behavior and loneliness Behaviors Physical Mental Social Eating with everyone +1 +1 +1 Speak using cell phone +0 +1 +0 Visit discourse room +0 +0 +1 Join Recreation +1 +1 +1 Figure 3 Mapping of Loneliness Mapping of Loneliness Using the table shown in Table 1, we can summarize the all activities in daily living and map the results into the three axes shown in the Figure 3 as the degree of loneliness in a day. By analyzing this three-dimensional figure, it is possible to grasp the tendency of which axis the solitude degree is high or low. After mapping, we can make a personalized care plan to reduce the solitude which is different peron by person. Monitoring System Design and Implementation System Design Functional requirements of the monitoring system are as follows: (1) Status Monitor The most basic function is to monitor person various status such as position, posture, activity, and vital sign. Different kinds of sensors should be used, thus integration of sensor data to reason higher level status is necessary. (2) Emergency Watch If the person is in a critical status, such as stroke and falling, the system should recognize the status and put alert within a specified time, such as 1 minitute, that means the system s response time requirement is essential. (3) Forecasting Forecasting of status change is also desirable. For daily living, prediction of wake up time or urination is important for a caregiver.to realize it, the system should store the activity log and analyze to find a daily life pattern. (4) Dialog Normal functions of monitoring are passive. The monitoring should be performed without bothering the residents. But for isolated elders, active monitoring function should be considered, such as dialog and greeting. The reactive planning capability is necessary to realize it. 255
Configuration Figure 4 System Configuration The system configuration diagram is shown in the Figure 4. All sensors are mounted on RaspberryPi3: camera module, non-contact temperature sensor, microwave sensor. We use Fluentd S/W toolkit which is originally developed for a Web server log collection, for our data-driven architecture. It can processes and integrates data flowing asynchronously from each module, and utilizes them for recording of vital data, communication by utterance function, and behavior recognition. The log data and the name of the elderly are recorded in the database. Experiment and Evaluation Experiment Method We actually installed a IoT based robot in a nursing home, and conducted an experiment. The purpose of the experiment is two-fold. First, since the camera module, the non-contact temperature sensor, and the microwave sensor operate asynchronously, it is necessary to integrate them and perform simple communication with the elderly using the integrated data. The verification of this basic function should be checked. The communication performed this time is to detect the face of a person and speak with data of heart rate, respiratory rate, expression, obtained from each module if it is a person registered in the database. Figure shows implemented scenarios. This scenario assumes that the robot will greet the elderly. If there is only one face detected, Figure 5 Implemented Scenario speak the name, the expression at that time, and vital data. If two or more people, call their name and greet. In this scenario, the robot uses camera module and microwave sensor for detecting face and sensing the heart rate and breathing rate. The second is to recognize one SADL, "visit a discourse room" and to measure the social loneliness degree of elderly people in a simple way. For recognition of this SADL, a camera module is used. When the robot monitors the entire discourse room and recognizes the face, it records the ID of that person and the time visited in the database. Count the number of visits by elderly people and measure the social loneliness with the number of visits as a score. For the experiment, we registered the face of four facility users in the robot. The experiment was conducted for about 2 days from 13:00 on January 18, 2018 to January 19, 2018. Results of Face Recognition After registering the face of the person and verifying it in the discourse room, The IoT-based robot could communicate with the elderly using sensor data. Table 2 shows a part of vital data such as the heart rate of the user actually obtained. Even the elderly talked to the robot, and many laughing facial expressions were observed. Figure 4 shows the state of the experiment at the actual nursing home. Table 2 Vital Log Data 256
Figure 6 The robot is talking to user Results of SADL Recognition It was possible to obtain the time and the staying time of visiting the lounge room of the subject whose name was registered in the database. The Figure 5 shows the result of recognition of SADL. The horizontal axis is time, which is a point indicating that a round plot has visited the discourse room. The top one is the log data of user 1, her visiting was at 14:22 on 18th and never on 19th. Likewise, user 3 never visited the lounge on the 19th. On the other hand, user 2 and user 4 were visiting the lounge room on the 19th. Based on Figure 5, Table 2 shows how many times each user visited the lounge room manually. If the intervals of the plots are short, it is assumed that they are recognized multiple times by one visit, and a set of plots in which one hour or more is free is taken as "one visit". Based on the proposed Model of Loneliness, we will measure the loneliness of each user. Assuming that the behavior of visiting the discourse room is related only to the degree of social isolation, it can be said that user 1 has the most social degree of solitude among the four. Table 3 User Visits User User Visits User1 2 User2 5 User3 4 User4 4 Figure 7 Visitor to discourse room Discussion In this experiment, we are only able to observe the visitor in the discourse room, we cannot map the social degree of loneliness because we cannot currently recognize other actions. By increasing the recognizable behavior, it is possible to measure significant degree of loneliness. In order to increase the number of recognizable behaviors, it is necessary to handle not only the sensor built into the robot but also the data of the remote sensor installed at a location away from the robot, such as a bedroom or dining room. By doing so, the scope of monitoring as a monitoring system will also become wider, and it will be possible to increase the number of types of behaviors that can be recognized and to be able to map loneliness to significant 3 axis diagrams at the same time. Moreover, by using the vital data obtained by the experiment and the recognition result of SADL for visualization and measuring the degree of loneliness, it is expected to realize a system which can support to reduce the caregivers work load. It is thought that an interface using speech recognition technology becomes essential. When we were experimenting at a facility, we often saw robots are talked back. we actually got an opinion from the residents that "I wat to continue conversation after the robot talked to me. Since speech recognition can be used by a caregiver who both hands are occupied in doing work, we will also consider application in the field of ICT to convey work efficiency and information on the elderly. 257
Conclusion In this paper, we propose a next-generation monitoring system in a nursing home, and developed a monitoring robot which implemented a part of the desirable functions. We also proposed a model for measuring elderly loneliness. The robot incorporates multiple sensors and a speaker so that communication can be taken. As a result of the experiment using the robot, it was possible to consolidate the sensor data sent to fluentd asynchronously and to store the log data such as the expression of the elderly at that time in association with the person. We were able to communicate easily with the elderly people. Also, we were able to recognize SADL, by monitoring who and when the visitor comes. By using the result of recognizing SADL, we could measure social loneliness degree of elderly person in a simple manner by counting the number of visiting times in the discourse room. Currently, recognizable behaviors are limted. Although few measurements of the degree of solitude have been made, future data of the remote sensor will be utilized to increase the recognizable behavior. Microsoft Corporation. Emotion API. Retrieved November 8, 2017. https://azure.microsoft.com/ja-jp/services/cognitiveservices/emotion/. OMRON Corporation. HVC-P2. Retrieved November 8, 2017. http://plus-sensing.omron.co.jp/product/hvc-p2.html. Acknowledgments This work was supported by JSPS KAKENHI Grant Number JP17H01823 Development of Watching System by Integration of Non-Restrictive Sensors. We would like to thank WCL and Long-Term Care Health Facilities Yui for their support. Furthermore, we would like to thank Masayuki Numao giving valuable comments on the manuscript. References United Nations Department of Economic and Social Affairs. 2015. World Population Ageing. Ministry of statistics. 2013. Population of elderly people. Ministry of Health, Labor and Welfare. 2014. The situation surrounding the long-term care insurance system. Person nursing labor stability center. 2016. Result of nursing care labor survey. Nicholas R. Nicholson. 2012. A Review of Social Isolation. The Journal of Primary Prevention, 33(2-3):137-152. Hiroshi Karatsu. 2012. A Study on Elderly People s Isolation in the Super Aging Society (in Japanese). Nara Bunka Women's Col-lege. pp189 (in Japanese). Ministry of Health, Labor and Welfare Report. 2009. Basic knowledge of depression of the elderly (in Japanese). http://www.mhlw.go.jp/topics/2009/05/dl/tp0501-siryou8-1.pdf Panasonic Corporation. PASTA. Retrieved November 8, 2017, https://feeling.pas-ta.io/. Smartmedical Co., Ltd.. Empath API. Retrieved November 8, 2017. https://webempath.net/lp-jpn/. 258