There have been numerous works proposed to merge augmented reality/mixed reality (AR/MR) and Internet of Things (IoT) in various ways. However, they have focused on their specific target applications and have limitations on interoperability or reusability when utilizing them to different domains or adding other devices to the system. This paper proposes a novel architecture of a convergence platform for AR/MR and IoT systems and services. The proposed architecture adopts the oneM2M IoT standard as the basic framework that converges AR/MR and IoT systems and enables the development of application services used in general-purpose environments without being subordinate to specific systems, domains, and device manufacturers. We implement the proposed architecture utilizing the open-source oneM2M-based IoT server and device platforms released by the open alliance for IoT standards (OCEAN) and Microsoft HoloLens as an MR device platform. We also suggest and demonstrate the practical use cases and discuss the advantages of the proposed architecture.
The Internet of Things (IoT) has made the world smarter, is continuously evolving, and has pervaded all aspects of life, being more innovative in homes [
The combination of MR with rich visualization related to real objects and IoT with feasible information about things is expected to produce new applications while overcoming limitations. However, IoT has been developed based on standards that interconnect several heterogeneous systems and services, while AR/MR ones are being developed for specific applications tightly coupled with those devices, primarily head mounted displays (HMDs) developed by various vendors in their own ways. These different characteristics between the development processes of IoT and AR/MR may cause interoperability problems when combined and served for common applications [
There have been numerous works to merging AR/MR and IoT in various ways, which will be explained in detail in Section 2. The studies in [
In this paper, we propose a novel architecture of oneM2M-based convergence platform for AR/MR and IoT systems and services to solve interoperability and reusability problems that existing studies have. Since the proposed architectural concept is to combine machines (or devices) and services from/to IoT and MR, we call the proposed architecture the machine to MR, M2MR in short. The main contributions of the paper are the following:
The proposed architecture adopts the oneM2M IoT standard [ The proposed architecture can provide intuitive interfaces and usages to users by exchanging data complying with oneM2M IoT standard between MR and IoT systems and applications. We implement the proposed architecture utilizing the Mobius and the nCube as open-source oneM2M-based IoT server and device platforms, respectively, released by the open alliance for IoT standards (OCEAN). Besides, we consider Microsoft HoloLens as an MR device platform and add oneM2M device interface by modifying the nCube platform in it. We suggest and demonstrate the practical use cases of the proposed architecture, then give the discussion about the advantages of the proposed architecture.
The remainder of this paper is organized as follows. In Section 2, backgrounds on the proposed method are described. The proposed architecture and implementation are presented in Section 3. In Section 4, the validation of the proposed architecture is given. In Section 5, discussions on the case studies and the advantages of the architecture are illustrated. Finally, Section 6 concludes the paper.
Many studies have been conducted to augment IoT information to AR/MR devices for specific applications and domains. Belen et al. [
These works mainly focused on the implementation in specific situations or elementary technologies. Although few of them dealt with architecture or platform, their output also had domain dependency. For example, [
Several studies have considered the IoT architecture for integrating MR and IoT. Martin et al. [
However, they only pertain to implementation-level standards and protocols, which are conceptual and ambiguous for commercial use. Hence, our study progresses toward applying well-known IoT standards instead of specifying entire practices in the IoT architecture. Among the standards, we choose the oneM2M standard, which is open-sourced and well designed for application developers.
The primary goal of oneM2M is to unify fragmented IoT ecosystems that operate industrially. This project has defined requirements, architecture, API specifications, security solutions, and interoperability for IoT. To provide for multivendor interoperability, the architecture provides a vendor-independent middleware consisting of common service functions (CSFs) commonly required in IoT services. In addition, CSFs allow the application developer to focus on IoT applications. The list of CSFs are Registration, Communication Management and Delivery Handling, Data Management and Repository, Device Management, Application and Service Layer Management, Discovery, Group Management, Subscription and Notification, Location, Network Service Exposure, Service Execution, and Triggering, Security, and Service Charging and Accounting.
In oneM2M standard, communication between entities is defined as a reference point. M2M communication with AE (Mca) reference points refer to communication flows between CSE and AE. Similarly, M2M communication with CSE (Mcc) reference point refers to communication flows between two CSEs. The oneM2M primitives are service layer messages transmitted over the reference points. They follow a representational state transfer (REST) architecture style that allows requesting systems to create, retrieve, update, and delete (CRUD) data to access resources. This style provides ease of use and interoperability to developers and can bind to multiple protocols such as HTTP, Constrained Application Protocol (COAP), and webSocket.
In a typical IoT architecture, the entire data is stored in a database on a central server and the user can access the information of the entire system by only interacting with the server. In oneM2M, IN-CSE stores data from all entities such as CSE and AE in a hierarchical structure. In oneM2M architecture, each entity is represented as a resource and consists of multiple attributes and child resources. Attributes refer to a set of information related to the resource, such as identifiers, tags, creation and update times, groups, and values. In addition, each resource has a unique and addressable URI to access the RESTful system.
Domain | Node | Entity | Mapping to IoT architecture |
---|---|---|---|
Infrastructure | IN | IN-CSE | Central server |
IN-AE | User application | ||
Field | MN | MN-CSE | Gateway functions |
MN-AE | Gateway application | ||
ASN | ASN-CSE | Aggregator functions | |
ASN-AE | Sensor/actuator application without aggregation function | ||
ADN | ADN-AE | Sensor/actuator application with aggregation function |
There are various resource types defined in oneM2M; the following three types are essential to map oneM2M to the context of an IoT system: CSEBase (
This section explains the architecture of the oneM2M-based convergence platform for AR/MR and IoT, called M2MR. First, the requirements for M2MR design are described. Then, the overall system design and procedure are presented. Finally, the implementation of M2MR with the Microsoft HoloLens and the open-source oneM2M-based IoT platforms by OCEAN is illustrated.
The most typical application domains and functions commonly considered in exiting studies on adapting AR/MR in association with IoT systems are shown in
Category | Study | Domain | Object tracking | Data monitoring | Actuator control | MR data transmission |
---|---|---|---|---|---|---|
Domain-dependent Application and Architecture | Belen et al. [ |
Smart home | O | O | O | - |
Mahroo et al. [ |
O | O | O | - | ||
Pargmann et al. [ |
Smart plant | - | O | O | - | |
Zhang et al. [ |
Smart city, smart building | - | O | - | - | |
Rashid et al. [ |
O | O | - | - | ||
Jang et al. [ |
- | O | O | - | ||
Mylonas et al. [ |
O | O | O | - | ||
Jo and Kim [ |
- | O | O | - | ||
Mizutani et al. [ |
Industry, smart factory, robot | O | O | O | - | |
Guhl et al. [ |
O | O | - | - | ||
Hoppenstedt et al. [ |
- | O | - | - | ||
Stark et al. [ |
O | O | O | - | ||
Park et al. [ |
Fitness | O | O | - | - | |
Phupat- tanasilp et al. [ |
Smart farm | O | O | - | - | |
Architecture and protocol | Martin et al. [ |
- | O | - | O | O |
Koutitas et al. [ |
- | O | O | - | - | |
Croatti and Ricci [ |
- | - | - | - | - | |
Blanco-Novoa et al. [ |
Smart home | O | O | O | O |
To build the OT platform, developers should choose one of two methods to track the real objects: marker-based (MB) and marker-less (ML). The marker refers to a specific image pattern; MB-OTS maps a predefined pattern to images captured by a camera on the HMD. ML tracking refers to all techniques that exclude markers such as simultaneous localization and mapping (SLAM), deep learning-based object detection, and geographic information. Either MB-OTS or ML-OTS can be selected according to several considerations. Computing power is one of the essential considerations, because usually MB-OTS consumes less computing power than ML-OTS. To overcome the lack of the power, OTS can be placed on the cloud instead of running on the HMD. In this case, network latency is also another essential metric because the systems must guarantee real-time.
To design the M2MR architecture that guarantees the requirements, we derived detailed procedures for each requirement with sequence diagrams illustrating how entities interact with each other.
Let us assume that there is the HMD
To build an IoT environment, we used the open-sourced platforms by OCEAN, which is responsible for releasing the platforms certificated by oneM2M. Each entity in our architecture runs the OCEAN's open-sourced platform corresponding to its own type as follows: IN-CSE platform Mobius, AE platform nCube Thyme, ASN-CSE platform nCube Lavender, and MN-CSE platform nCube Rosemary. We built IN-CSE with the oneM2M server platform, Mobius 2.4.X, and MySQL-based database on a personal computer. For Windows system, we used MySQL, and for the Linux (Ubuntu) system, we used MariaDB. Nodes in the field domain were deployed on Raspberry Pi 3 and 4. Every platform we used was implemented with Node.js.
Reference points were set as follows as the default values of Mobius and nCube. Sensors and actuators interact with nCube Thyme through the socket. Between CSE and AE in the field domain, MQTT is the primary communication protocol. nCube Thyme publishes sensor data with its own MQTT topic and Lavender or Rosemary subscribe it. For the actuator data, it is vice versa. Mosquitto was used for the MQTT service. The reference points of Mobius are HTTP Requests with REST API.
We chose Microsoft HoloLens for the MR device. The initial output was developed with HoloLens 1, and the output was developed with HoloLens 2 after the release. Because there is no service to interact with oneM2M in HoloLens, we implement the application consisting of two AEs using Unity, one of the most popular game development tools. Both AEs in HoloLens connect to IN-CSE with HTTP.
OTS has been implemented in various ways, which will be covered in the next section with practical applications. The reference point between ADN-AEs in OTS and HoloLens also depends on implementation.
A
The hardware layer consists of optics and sensors, and the input and output hardware. In the development platform layer, Windows OS kernel services and the HoloLens interaction services are abstracted to the Windows Universal Windows Platform (UWP). For instance, identifying gestures and voice is one of the HoloLens interaction services pre-developed in Windows UWP.
The open-source layer consists of a Mixed Reality Toolkit (MRTK) and other open sources running on Unity. MRTK is supported by Microsoft to develop VR/AR/MR windows applications. Various services of MRTK are used to develop HoloLens applications, such as camera handlers, input handlers, and scene managers. However, only the REST and UX building blocks are shown in the diagram because the components reference and use them primarily as external libraries. Other open sources include object pool, HoloLens CameraStream, and SimpleJSON. Object pool is a set of scripts that implement the object pool design pattern. This pattern is used to improve performance with pre-instantiated ready-to-use objects on a pool instead of creating new objects whenever required. HoloLens CameraStream is the Unity plugin that makes the HoloLens video camera frames available to a Unity app in real time. SimpleJSON provides serializing (building) and deserializing (parsing) JSON type messages in C#.
The component layer is described by dividing it into three entities according to the proposed architecture. IN-AE consists of a data visualizer, sensor/actuator data model, and oneM2M handler. The data visualizer displays holographic objects on a see-through screen using UX building blocks. Because holographic objects depend on the performance of the HoloLens CPU and GPU, dynamically created UI objects are managed as an object pool. The sensor/actuator model refers to data type and logic to manipulate IoT sensors and actuators. The model is referenced by both the data visualizer and the oneM2M handler. The oneM2M handler includes Web Request methods to Mobius based on MRTK REST. Similarly, ADN-AE contains the oneM2M handler to update MR data to IN-CSE. OTS I/O contains image handler that encodes an image byte array using HoloLens CameraStream, sending it to OTS and receiving the tracking result.
In this section, experiments and results are described to verify our architecture, M2MR and prototype.
ID | Requirements | Subcategories | Test cases |
---|---|---|---|
TC1 | Object tracking | Realtime image transmission | HMD sends image to OTS |
TC2-1 | Object tracking methods | MBS (Marker based OTS) | |
TC2-2 | MLS (Marker-less OTS) | ||
TC3-1 | Data monitoring | OneM2M interaction |
HMD retrieves |
TC3-2 | Actuator control |
HMD creates |
The experimental environment is equivalent to the system environment of the prototype. IN-CSE was run on Windows 10. MBS runs as a component on HoloLens 2 and MLS runs independently on Ubuntu OS. In addition, nodes communicate through 802.11ac in the local area network.
Because image streaming is a feature that delivers video content, it is difficult to describe whether streaming works. Instead, the captured image from the HMD and the picture in the third person are compared. Therefore, there is no allowance for the metric; however, pass/fail is decided based on human perception and experience.
Streaming service was required to prove that the image was successfully delivered to the OTS. We implemented a Unity application including the function to decode byte arrays into an JPH format. In addition, communication between HMD and OTS is developed with a TCP socket sending byte array.
The test case aims to verify the OTS, including MBS and MLS. We implemented the MBS with quick response (QR) code tracking running in HMD and MLS with convolutional neural network (CNN) based object tracking running on the remote server. MBS was implemented through Vuforia, one of the most popular marker-based AR SDKs, and MLS was implemented with TensorFlow. Similar to TC1, it verifies the system through the screen capture of the actions from the outputs.
The purpose of TC3 is to prove that the interaction between oneM2M and HMD is working properly. Two subcases are listed to determine whether it has been successfully verified: TC3-1 is for sensor data visualization and TC3-2 is for actuator control and MR data transmission. For both cases, the rate of HTTP response messages to HTTP request messages was selected as a metric. The successful responses to all requests, “200 OK” for HTTP Get and “201 Created” for HTTP Post, can be considered appropriate for implementing the two interactions. Therefore, it was hypothesized that the number of request packets and the number of successful response packets would be the same. Because network performance is not considered, the metric allowance is set to 100%, which does not fail. To show this more effectively, the interval between HTTP messages is reduced from 1,000 milliseconds to 1 millisecond. That is, the number of packets per second increases exponentially.
To ensure the feasibility of our architecture, M2MR and its implementation, two practical IoT applications are demonstrated in this section. The first application [
This is based on our first IoT application using the Mobius platform and HoloLens presented in January 2018. OneM2M sensor data retrieval and actuator control functions were simply implemented as a HoloLens application. This application used MBS with Vuforia for OTS.
This system is a marker-less based IoT system that satisfies all requirements. In addition, the HoloLens application in this system follows the component layer architecture. We used the latest versions of IoT platforms, Mobius 2.4.36 and nCube Thyme 2.3.2. As a marker-less based system, the object tracking system runs in the infrastructure domain. We used HoloLens CameraStream to capture images and sockets to send them as a byte stream. MLS runs the faster R-CNN model [
We presented the feasibility of M2MR with a verification experiment and a case study. Lastly, we highlight the core advantages by integrating the oneM2M standard with MR.
In this paper, we proposed the oneM2M-based convergence platform, called M2MR, to operate AR/MR and IoT systems and services in general-purpose application environments without being subordinate to specific systems, domains, and device manufacturers. The M2MR platform has been implemented utilizing the open-source oneM2M-based IoT server and device platforms and Microsoft HoloLens as an MR device platform. It was validated for three practical cases where IoT and AR/MR are involved together and showed the potential efficacy for future use of AR/MR and IoT integration.
AR/MR provides the capabilities of rich visualizations and interactions on human, tangible objects, locations, and all things. IoT makes the world more innovative by making it possible to monitor and control everything, including various sensors, edge devices, systems, and so on, remotely, autonomously, and in time. The combination of MR and IoT is expected to produce new applications while overcoming each other's limitations. The proposed method enables interoperable data exchanges for applications running on heterogeneous MR devices and IoT elements developed by different vendors in different environments. The M2MR platform we have developed so far is at the prototype implementation of the architecture and primary operations verification. It needs more elaborate structural design and functional supplementation, and we will study the issue further.