Based on various digital technologies, the world we live in today is now highly interconnected, with humans, intelligent systems, machines, and devices acting together, trying to evolve into new forms of collective intelligence. Therefore, the future industry is expected to revolutionize the way humans interact with progressively more intelligent machines. In the forthcoming years, the industry faces the challenge of adjusting to and taking advantage of essential change drivers that are already very knowable. With the development of cyber-physical systems, the Internet of Things (IoT), and sensor networks, we now see a greater interpenetration between the physical (real world) and the virtual world.
The “Reality-Virtuality continuum”
Published in 1994, but still quite actual, Paul Milgram’s seminal article defines a continuum between Reality and Virtuality, where solutions as VR and AR are represented. This continuum maps different combinations of real and virtual elements within what Milgram refers to as Mixed Reality (MR). They are concepts currently embodied in the broader picture of Extended Reality (XR).
On the right side of the reality-virtuality continuum come the virtual environments, where the user is fully immersed and feeling to be present in a world distinct from the current one. Within the scope of an industrial project, INESC TEC developed a virtual environment where we can observe a training situation in which a maintenance technician practices a procedure. The user is completely isolated from the real world by the equipment s/he uses, feeling “transported” to a virtual environment, where s/he can operate a set of tools to repair electrical equipment. Immersion and the sense of presence that can result from it are two very relevant characteristics of a VR system.
On the left side of the continuum comes augmented reality, in which virtual three-dimensional objects are positioned in an integrated way with real objects. Here arises a strong connection of the user with the context in which s/he is inserted or operates. For example, the next image shows a paper map augmented with virtual information such as routes and green areas. The tablet is, in this case, used as a “magnifying glass”…
And what about solutions as something intermediate in the reality-virtuality continuum? Mixed Reality intermediate solutions combine virtual environments augmented by real objects present in the actual environment where the user is in reality. The real room walls and the furniture can be integrated into the virtual environment, allowing the user to touch the wall or lean against a table to perform a specific task. Next image demonstrates walking on a step of a virtual walkway. Although the user is only a few millimeters off the ground, on a wooden board, his brain feels present in a virtual environment hundreds of meters high, providing a physiological sensation of vertigo.
The technology maturity of nowadays and the challenges for its current adoption
Despite immersive systems research area is several decades old ( from Morton Heilig’s “Sensorama” in the ’50s to Ivan Sutherland’s “Sword of Damocles” in the ‘60s), only in the last decade, the technology and its production reached higher levels of maturity. This maturity is concerned with allowingto allow, on the one hand, the generation of real-time content, with quality and response times feasible to create a good sense of presence and immersion, and comfortconfort. On the other hand, it also focuses on the production ofproduce the devices at prices that facilitate broader adoption.
It is possible to find several systems on the market today with different device configurations. One of the most appealing is the “Head-Mounted Display” (HMD). It includes a viewing and listening device placed on the user’s head and a pair of controls to be used in the hands — called “wands” (among other auxiliary devices).
These configurations accommodate:
(a) Stereoscopic vision — the user has a sense of depth, since each eye receives the image from a slightly different viewpoint, as happens in an actual situation;
(b) Synchronization between head movements and the virtual viewpoint/audience — the view received follows the head movement, and sounds are reproduced according to the orientation relative to the sound sources;
(c) Control of hand position and orientation, access to buttons for symbolic interaction, and vibration feedback;
(d) Position the user in space (in some configurations) — the user has the sensation of moving in virtual space in the actual area where s/he is located.
In some cases, users may feel nauseous when using these devices, which mainly occurs when the synchronization between actual movements and what is visualized is not well achieved. Motion synchronization is one of the challenges shared with augmented reality systems. Additionally, the integration — also referred to as registration — of virtual elements into the real elements of the augmented scene is another challenge of AR. For example, a virtual box seems to be correctly placed on a real table, maintaining visual coherence even when the user or the device moves.
The challenges humans will face in the industry of the future
In today’s factories, humans find increasingly autonomous and intelligent environments and productive systems that involve a considerable diversity of technologies.
This trend brings new challenges, mainly concerning the relationship between human beings and these industrial environments. An increasing amount of data and information available has to be understood, increasing the complexity of management and decision-making. The revolution in the way humans interact with machines translates into solving several challenges, of which we can highlight the following:
● Transformation of the workforce, promoting the development of new skills in humans that enable them to manage work digitally with the support of cyber-physical systems;
● Development of human-centered industrial systems, enabling more significant action in all industrial processes and providing them with more adequate resources to extend their decision-making and action capabilities;
● Promotion of more significant, more efficient, and more effective collaboration of the human being with intelligent machines, systems, and robots, also enabling the increase of the added value of final products and services.
Given these challenges, immersive systems technology has been developing to increase human beings’ ability to intervene in this new work environment through technologies such as Virtual Reality (VR) or Augmented Reality (AR).
INESC TEC has been developing training scenarios in the scope of Industry 4.0. One of the applications that have been gaining interest is the use of “digital twins.” Real machines and operating systems are virtually replicated and augmented with real-time information about their status and task planning. Also, it has been possible to test future industrial layouts in virtual environments at the planning and management level and evaluate their effectiveness and cognitive load, focusing the new industrial systems on the human being. And in maintenance tasks or even in jobs on an assembly line, augmented reality allows more efficient performance, through augmented information on the machine itself, or remote collaboration with experts, in a mixed reality environment to solve more complex problems.
Through immersive technologies, humans can increase their ability to intervene in the industrial environment more effectively and with less cognitive overload.
The future industry will most likely be the stage for the emergence of an augmented human being, integrating immersively into industrial systems that are more intelligent and autonomous.
António Coelho, A. Augusto Sousa, Rui Rodrigues, Filipa Ramalho
Short BIOs
Filipa Ramalho is a Ph.D. candidate at the University of Porto, Portugal, and she is also a researcher at INESC TEC.
She studied Information Science (Bachelor’s Degree and Master’s Degree, 2005–2010). Then she worked on projects as business processes and information systems consultant for nine years with companies from different industrial and service sectors (2008–2017). Filipa Ramalho now works as a researcher and Ph.D. Candidate at Centre for Enterprise Systems Engineering (CESE) from INESC TEC in the scientific domains of Information Management, Knowledge Management, Collaboration, Immersive Systems (AR, VR, MR), Human-Centered Manufacturing Systems, and Industry Technology Adoption.
António Fernando Coelho is an Associate Professor with Habilitation at the Department of Informatics Engineering of the Faculty of Engineering (FEUP), University of Porto (UP), where he teaches Computer Graphics, Programming, and Digital Games. He is also the director of the Doctoral Program in Digital Media at the University of Porto and academic Leader of Work Package 4 of the European University Alliance for Global Health (EUGLOH).
He is also a Senior Researcher at the Center for Information Systems and Computer Graphics (CSIG) of INESC TEC (formerly INESC Porto) since 1995. He is currently the coordinator of the Computer Graphics and Virtual Environments area. His research interests are in the areas of Computer Graphics, Serious Games, and Geospatial Systems.
A. Augusto de Sousa got his PhD in 1996 in FEUP (Faculty of Engineering of University of Porto, Portugal), in the area of Computer Graphics/Image Synthesis and Parallel Computing. He has been teaching in the same Faculty since 1983.
Since 1985, he has been a researcher in INESC/INESC TEC, where he was the Coordinator of the Information Systems and Computer Graphics Unit. He was a member of the Executive Board of the Portuguese EUROGRAPHICS Chapter, being President, Vice-President and Treasurer, as well as a member of the Executive Committee of the EUROGRAPHICS Association itself.
Currently, he is a member of the Executive Board of FEUP, being the Vice-President of the Academic Affairs Council. Between 2008 and 2014, he was the director of MIEIC — Master in Informatics Engineering and Computing (a 5 years long master program). Between 2000 and 2003 he was a member of the executive board responsible for the installation of the undergraduate degree on Journalism and Communication Sciences.
Rui Rodrigues graduated in Systems and Informatics Engineering at University of Minho in 1998. During his PhD he researched in 3D reconstruction from Images both at Philips Research (Eindhoven) and University of Minho, until he concluded in 2006. He has worked in the industry in the field of interactive systems and games until 2009, when he joined the Department of Informatics Engineering (DEI) at the Faculty of Engineering of University of Porto (FEUP) as Assistant Professor. Between 2015 and 2019 he was the director of the Multimedia Masters of University of Porto. He teaches and researches in the areas of Computer Graphics, Interaction and Game design, and is one of the responsibles for the GIG group at DEI/FEUP and its labs. In that context, he has supervised/co-supervised several MSc and PhD thesis. He is also a senior researcher at INESC TEC’s CSIG, through which he participates in research projects of various dimensions.
