A concurrency control mechanism for collaborative work is a key element in a mixed reality environment. However, conventional locking mechanisms restrict potential tasks or the support of non-owners, thus increasing the working time because of waiting to avoid conflicts. Herein, we propose an adaptive concurrency control approach that can reduce conflicts and work time. We classify shared object manipulation in mixed reality into detailed goals and tasks. Then, we model the relationships among goal, task, and ownership. As the collaborative work progresses, the proposed system adapts the different concurrency control mechanisms of shared object manipulation according to the modeling of goal–task–ownership. With the proposed concurrency control scheme, users can hold shared objects and move and rotate together in a mixed reality environment similar to real industrial sites. Additionally, this system provides MS Hololens and Myo sensors to recognize inputs from a user and provides results in a mixed reality environment. The proposed method is applied to install an air conditioner as a case study. Experimental results and user studies show that, compared with the conventional approach, the proposed method reduced the number of conflicts, waiting time, and total working time.
Mixed reality (MR) is an environment that can be flexibly changed according to the purpose of a service that provides both virtual reality (VR) and augmented reality (AR) environments. Because of the rapid development of device technology, including head-mounted displays (HMDs) and electromyography (EMG) sensors, MR environments have been widely used in various applications such as games, entertainment, and industrial fields [
However, if users repeat the same work while manipulating shared objects, a conflict phenomenon in which the result of the shared object is suddenly changed may occur. If this phenomenon occurs continuously, the consistency of participants in the collaborative work will become problematic, and the work would have to be repeated. Consequently, the work time would be prolonged, and the satisfaction of the participants would diminish [
This study proposes an adaptive concurrency control approach for shared objects in MR environments in which the ownership of subdivided tasks is significantly changed while users perform work on shared objects in an MR environment so that the control over the shared objects can be changed to achieve the relevant goal. The proposed method consists of goal–task–ownership for the manipulation of shared objects. As the collaborative work progresses, the proposed system adapts the different concurrency control mechanisms of shared object manipulation according to the modeling of goal–task–ownership. For example, a concurrency control mechanism is not necessary when two users are moving a shared object in an MR environment. However, the owner may want to restrict specific tasks of the non-owner to focus on assembling small parts of the shared object after the moving state. Finally, the non-owner may support the owner’s task such as changing the visual appearance and testing the shared object. With the proposed system, detailed concurrency control mechanisms of the shared object can be adapted when the state of the goal is changed.
The proposed system was applied to the collaborative installation of an air conditioner in an MR environment. As the interactions of users in MR are similar to the real world, a sensor-based hand and upper body motion recognition method is applied to the part of actions to achieve specific motions. Hence, we used MS Hololens [
The main contributions of this study are as follows:
The proposed system provides an adaptive concurrency control mechanism that enables users to significantly change the ownership rules for tasks while they are performing collaborative work to achieve certain goals. The proposed system provides a sensor-based hand and upper body motion recognition method to recognize inputs and provide appropriate interactions because an MR environment has the potential to provide natural behaviors of users in the real world.
MR environments visualize virtual objects, show them to users through HMDs, and provide interactions with the user. In addition, MR environments provide a solution of VR or AR environments to users according to the purpose of the provided application [
To manipulate objects in an MR environment, the user can use various interfaces and interactions. The first method enables users to manipulate objects through finger touches using the sensor on the HMD. Although this method is relatively simple except for manipulating the 2D UI visible in the user’s field of vision, 3D objects, which are important in MR environments, cannot easily be manipulated using this method [
In an MR environment, users can interact with 3D objects using the interaction methods. Interactions with objects can be concretely classified into several tasks depending on the movement of the 3D position of the object, rotation, size transformation of the object, changes in the appearance of the object, simulations of objects, and combining and disassembling the objects [
Task | Definitions |
---|---|
Transition | Transition of 3D position of the object |
Rotation | Rotation of the object |
Scaling | Change the size of the object |
Merge | Two or more shared objects are combined into one shared object |
Fission | One shared object is divided into two or more shared objects |
Simulation | Calculate specific conditions of a shared object |
Appearance | Change visual properties of a shared object |
-Change the color of a shared object | |
-Change the texture of a shared object |
When two or more users perform collaborative work in an MR environment, 3D objects must be synchronized. That is, if a user executes one of the tasks mentioned in
In an MR environment, when interactions with a shared object are conducted simultaneously by multiple users, conflicts may occur. As such, the result of the task on the object can be changed before the interaction of a user is completed depending on the interactions of other users. Consequently, the consistency of the tasks that had been worked on is affected. Concurrency control approaches intended to address these challenges are classified into three types. The first type is an optimistic method that allows users interactions with a shared object. However, if any conflict occurs among the users, this method recognizes only the work of the first user [
As collaborative work environments have become more complex and diversified, methods to divide ownerships into small pieces and provide the divided ownerships according to individual interactions rather than providing the ownership of shared objects per se to users have been proposed. Lee et al. proposed a method to allow other users to access and perform other interactions such as painting the relevant object when the owner of the object is holding the object [
Moreover, in an MR environment, collaborative work in an extremely complex industrial field is complex. For example, in an MR environment, multiple users can perform collaborative work to build an office [
Although users have one goal when performing collaborative work in an MR environment, diverse concurrency control approaches to the interaction methods of users may occur. For example, when two users conduct collaborative work to move an air conditioner, the ownership of the movement of the shared object can be processed so that two or more users can operate it. Thereafter, while fixing other parts through additional screws to the air conditioner in a situation where one user is holding the air conditioner after it has moved, only the user who is holding the object has the authority to move the object. Moreover, the authority to assemble the object can be provided to the other users.
To provide users with collaborative work on shared objects in an MR environment, the characteristics of the user action model are applied in this study to identify users’ intentions to achieve a specific goal. Further, a gorilla troops optimizer (GTO) model is defined accordingly so that the ownership of users’ actions can be divided, or the ownership can be owned jointly. First, concerning the definition of modeling terms, a goal that a user wants to achieve and detailed goals that must be achieved to achieve the foregoing goal in an application program exist. Moreover, the aforementioned goals are classified into goals and sub-goals, respectively. Actions that appear in the process of users’ recognition of what must be performed to achieve detailed goals are defined as tasks. In addition, the part that sets the ownership of the tasks is defined as “Ownership.”
In this study, the collaborative work performed in an MR environment is classified into Goal, Sub-Goal, and Task, and a collaborative work platform that can adaptively transform ownership management appropriate for relevant tasks according to the progress of the tasks and sub-goals is proposed. The proposed platform consists of a client that visualizes MS Hololens and a server in charge of collaborative work, as shown in
The server analyzes the UDP information transmitted from the client through a task analyzer to analyze the information on the tasks performed by users. Thereafter, based on the analyzed information on users’ tasks, the currently shared objects are selected using an object manager that manages the shared objects, for which work is currently performed; the owner of the shared objects is identified, and the input of the task by the user is performed by a task manager. At this time, the goals of participants in the current collaborative work and the information on the progress of the sub-goals to achieve the goals are identified using the goal manager. Further, the concurrency of the collaborative work is controlled according to the ownership policy set for the tasks in the sub-goals currently in progress through the ownership manager, and the relevant results are informed to the user. Thereafter, even if the tasks of users are the same, the users are allowed to accomplish the following functions: perform the tasks even if they are not owners according to the ownership policy set to fit the “Sub-goals,” in progress, access the tasks through the owner’s permission, or not to access (lock).
For the control of the concurrency of the shared object proposed in this study, users use a direct object manipulation method involving hand usage through upper body motion recognition. Thus, the proposed system is structured to facilitate interaction with users after they have worn the MS Hololens and Myo devices, which are EMG sensors, on both arms. In the case of MS Hololens, when voice input is received from the user or when the user moves in an MR environment, the 3D position is tracked, and information on the rotation value of the user’s head wearing MS Hololens is extracted. Moreover, a user can perform various tasks on 3D shared objects while moving in an MR environment and use Myo devices made by combining EMG and IMU sensors to recognize the foregoing. When the user is attached to the Myo sensors, the user’s movements from the head to both arms can be manipulated in response to the movement of individual joints, as with real arms. Moreover, shared objects can be selected and manipulated through the recognized hand.
To manipulate other tasks, the speech recognition function of MS Hololens can be used, or the menu using ray casting can be clicked to visualize the current progress in the goal of the object and the information on other available tasks. Moreover, the relevant tasks can be selected through voice or ray casting to perform other tasks.
When two or more users perform collaborative work, the ownership of the object is judged. If they are not the owner, three types of concurrency controls are performed on the task according to the state of the goal of the current collaborative work, as shown in
First, the work to grab, move, and rotate a shared object can be performed. However, when two or more users move or rotate while holding a shared object, to prevent the conflict phenomenon in which the position or rotation information is changed according to the movement of one user, a concurrency control algorithm for the movement and rotation of the object is provided, as shown in the following equation. When two users grab an object, if the owner is assumed to be PA and the non-owner is assumed to be PB in the order in which the users hold the object, the middle point PC is obtained using
Second, with the consent of the owner, tasks, such as simulations of shared objects, can be performed. In this study, to obtain the owner’s consent, an approach was set so that consent would be considered to have been obtained in cases where the owner permitted through the speech recognition of MS Hololens or clicked ray casting in the relevant menu. Third, the owner’s work was blocked.
In this study, collaboration contents for air conditioning systems were developed.
Item | Specs |
---|---|
MR HMD | MS Hololens 1 |
PC (Server) | i7-8700 CPU (3.2 GHz, 8 cores)/GTX2070 |
Upper body tracking sensor | Myo, muscle electrical signal detection sensor, 6-axis gyro sensor |
Software | Unity 2018.4 (LTS) and UWP (universal window platform) |
Network | Local area network with 100 Mbps |
This study experimented on three concurrency control approaches including the proposed method. The first, Method A, blocks the access tasks of non-owner until the owners release their ownerships. The second, Method B, is a task-based concurrency control approach in which the ownership of shared objects exists and permission from the owner is obtained every time a task is used to perform work. The third, Method C, applied the GTO-based adaptive concurrency control approach proposed in this study. Users in the individual groups experimented in the order described in
User group | Order of experiment |
---|---|
User1 | A–>B–>C |
User2 | B–>C–>A |
User3 | C–>A–>B |
In Method C, the relationship to a GTO can be defined. Thus, based on the literature [
Sub-goals | Task | Ownership management |
---|---|---|
Moving | Translation | Allow |
Rotation | Allow | |
Scaling | Lock | |
Merging | Lock | |
Fission | Lock | |
Simulation | Allow | |
Appearance | Allow | |
Assembling | Translation | Allow with permission |
Rotation | Allow with permission | |
Scaling | Lock | |
Merging | Allow | |
Fission | Allow with permission | |
Simulation | Lock | |
Appearance | Allow | |
Testing | Translation | Lock |
Rotation | Lock | |
Scaling | Lock | |
Merging | Allow with permission | |
Fission | Allow with permission | |
Simulation | Allow | |
Appearance | Allow with permission |
The second stage constitutes assembling the accessories of the air conditioning system. In this case, the user with the ownership primarily fixes the air conditioner, and other users assemble accessories such as filters and hoses for the air conditioner. Users without ownership can join accessories without the permission of the owner. However, if they wish to disassemble the accessories because of a wrong connection or adjust the location or direction of the air conditioner, they should obtain permission from the owner to perform the collaborative work.
The third involves starting the air conditioner and checking it upon completion. At this stage, users without ownership can access the simulation to turn on and view the wind of the air conditioner without the owner’s permission. However, disassembling and assembling tasks can be performed only with the owner’s permission.
Users who participated in the experiment completed the processes for air conditioner movement, installation, and testing. However, in the process of jointly moving or rotating the air conditioner, the methods proposed in this study were commonly applied to all experimental methods. In the experimental process, the number of conflicts between users’ collaborative work was measured. The experimental results of the foregoing show that conflicts occur when Method A is used, as shown in
The average completion times of the participants are shown in
We conducted an analysis of the variance (ANOVA) test on the performance evaluations for statistical analysis, as described in
Condition | ANOVA result (pa = 0.01) | Mean and SD results, stored as in |
---|---|---|
Conflicts | F = 4362.24, | A (M = 190.4, SD = 46.82857) |
pa = 2.31E − 49 | B (M = 56.7, SD = 7.78) | |
MSw = 24.59365 | C (M = 34.8, SD = 19.17143) | |
Waiting time | F = 5172.116, | A (M = 334.3, SD = 119.6667) |
pa = 5.53E − 51 | B (M = 120.4, SD = 56.54286) | |
MSw = 61.33 | C (M = 56.7, SD = 7.78) | |
Total working time | F = 11477.72, | A (M = 724.5, SD = 199.1238) |
pa = 3.11E − 58 | B (M = 341.1333, SD = 89.8381) | |
MSw = 106.7111 | C (M = 165.8, SD = 31.17143) |
After the experiment, the participants were interviewed to qualitatively evaluate the proposed system. When a question concerning the most uncomfortable concurrency control approach was asked, participants indicated that Method A was the most inconvenient. There were inconveniences when the consistency of collaborative work was interrupted because of other users’ work. Although the precautions for this experimental method were earlier explained, some participants thought it was an error of the system. Additionally, some users indicated inconveniences in Method B. Users with ownership complained of the inconvenience of having to grant permission to each task requested by a non-owner during collaborative work, whereas those without ownership felt inconveniences as they had to request ownership every time they performed a task. Moreover, they had to watch work when their work was not permitted. Users generally evaluated Method C, the proposed method, as being satisfactory and efficient in that the concurrency control approach to tasks was changed according to sub-goals. However, some users were unfamiliar with both the air conditioner and air conditioning system. Consequently, they wrongly manipulated the system without recognizing changes in the concurrency control approach to tasks because of changes in the sub-goal and proposed that the guidance UI for relevant parts should be improved.
This study proposed an adaptive concurrency control mechanism with modeling of goal–task–ownership to enable users to perform collaborative tasks in an MR environment. Hence, the proposed system classified the collaborative tasks in an MR environment into detailed goals and tasks. The classified goals and tasks are mapped to appropriate ownership strategies. By modeling goal–task–ownership, the proposed system adapted the ownership strategy of a shared object according to the changes in the detailed goals.
To provide the collaborative tasks of shared objects in an MR environment, MS Hololens and Myo devices were proposed. Users could perform collaborative tasks on the shared object using both hands and upper body motions. The proposed concurrency control method was implemented and applied to install an air conditioner system, which could be one of the most promising applications of our approach. To verify the proposed method, two users were asked to perform virtual air conditioner movement, installation, and test processes in an MR environment as collaborative work. The experiment results showed that, compared with conventional concurrency control mechanisms, the proposed method could minimize numerous conflicts. Interestingly, the number of conflicts affected both the waiting and total working times. According to user studies, the participants adopted the proposed adaptation of concurrency controls of shared objects and voice communication for requesting permission for shared objects in an MR environment.
In future research, we will study appropriate UI for collaborative work in MR environments. We expect to apply a deep learning-based motion recognition method, that is, a system that will naturally enable collaborative work even when no devices are attached to the body.
The authors thank HyungJun Cho and DongHyun Kim (Hoseo University) for their hard work in supporting the experiments of the proposed system.
These results were supported by “
The authors declare that they have no conflicts of interest to report regarding the present study.