HI-Drones! – Human interaction for drones universal solution of gesture control
Data
2021-09-14
Autores
Título da revista
ISSN da revista
Título do Volume
Editora
Resumo
A crescente popularidade dos Veículos Aéreos Não Tripulados (UAV) é o resultado dos últimos desenvolvimentos científicos, bem como empresariais, para tirar partido das suas capacidades de voo únicas. A flexibilidade de voo dos drones promove o fornecimento de vários serviços que, de outra forma, seriam perigosos, morosos, e de difícil execução, mesmo com robôs terrestres. No entanto, existem desafios na integração dos drones na nossa sociedade, alguns deles relacionados com uma interação complexa. O controlo remoto (RC) tradicional requer destreza e treino para pilotar UAVs com sucesso, especialmente em ambientes densos e perigosos, o que leva as pessoas a terem receio de o usar, consequentemente reduzindo a interação. Como tal, o desenvolvimento de uma alternativa de controlo intuitiva, simples, e fácil de aprender, tem o potencial de inverter este contexto e aumentar a presença dos UAVs na sociedade. Uma revisão da literatura da área da interação humano-drone (HDI), um ramo da interação humano-computador (HCI) dedicada à interação com drones, mostra que nos últimos anos, as interfaces gestuais e cérebro-computador (BCI) são as que apresentam maior interesse. Ambas são consideradas interfaces naturais de utilizador (NUI) por serem intuitivas e garantirem interações simples, com um processo de aprendizagem mais rápido, entre outras vantagens. No entanto, os gestos demonstram o maior potencial para o controlo de drones em tempo real, enquanto as BCI ainda estão numa fase prematura relativamente à sua fiabilidade. Como tal, o objetivo principal desta tese é a proposta de uma solução para o controlo de drones em tempo real, constituída por uma nova interface e uma arquitetura que a suporte. A arquitetura foi definida de forma modular, considerando uma comunicação universal e a centralização do controlo. Esta caracterização resulta numa solução compatível com tantos drones diferentes quanto possível, que permite ao piloto selecionar um ou vários drones para controlar, através da mesma interface, ao mesmo tempo. Em relação à interface, esta consiste no uso de duas smart bands customizadas para proporcionar um controlo simples e intuitivo através de gestos manuais dinâmicos e naturais. Para este fim, foi criada uma biblioteca de gestos, definida para a pilotagem de drones em tempo real, onde os gestos correspondem a rotações em torno dos eixos transversal e longitudinal das mãos, usadas individualmente, ou combinadas. A validação desta nova interface e arquitetura foi executada através do desenvolvimento de um protótipo funcional, aplicado em dois casos de uso, que correspondem à pilotagem de dois quadcópteros diferentes, um do mercado consumidor, e um montado por módulos customizados. Os resultados obtidos através de testes técnicos e com utilizadores validaram a solução proposta aplicada ao controlo de drones em tempo real, em relação ao controlo e à latência da comunicação, e demonstraram os benefícios dos gestos manuais, em relação ao RC tradicional, especialmente para participantes com limitações na interação (e.g., reumatismo, défice de atenção). Esta avaliação de usabilidade provou o sucesso da interface wearable proposta, como uma alternativa fiável ao RC tradicional. Durante a implementação do protótipo e dos testes funcionais, foi identificada uma limitação. As pessoas têm receio de fazer colidir um drone real, pela possibilidade de causar ferimentos pessoais ou estragos à propriedade. Como tal, pedir a um utilizador principiante para pilotar um drone ao longo de uma trajetória com obstáculos é uma decisão arriscada. Assim, para mitigar os riscos de acidente com o drone durante sessões de treino em ambientes reais, este trabalho também apresenta uma solução de treino e avaliação para desenvolver trajetórias e obstáculos virtuais, que podem ser usados em tempo real, com drones reais, em ambientes reais. Através de tecnologias de Realidade Aumentada (AR) para a web, esta solução adapta áreas abertas em trajetórias de voo complexas através de marcadores impressos, os quais possibilitam a criação de desafios de treino com um risco reduzido de colisão, uma vez que os obstáculos são virtuais e apresentados ao piloto através de uma sobreposição do ecrã que reproduz a stream do drone. Os resultados dos testes com utilizadores demonstraram que esta proposta pode melhorar as sessões de treino realizadas no mesmo espaço, através da definição de trajetórias virtuais variáveis. Esta tese resultou numa caracterização detalhada de uma solução de controlo para drones em tempo real, que usa gestos manuais naturais para potenciar a interação dos utilizadores e a sua educação para o controlo de drones, o que leva à promoção da popularidade dos UAVs na sociedade como fornecedores de serviços. Resultou também num guia científico que fornece o conhecimento e as metodologias necessários para definir, demonstrar, e avaliar soluções de controlo de drones em tempo real, suportado pela caracterização de diversos tópicos teóricos de HCI, UAVs, e HDI, que descrevem o contexto. Por exemplo, novas soluções podem beneficiar da modularidade da arquitetura definida para adaptar a solução proposta utilizando uma interface de controlo diferente, ou até controlar dispositivos diferentes, para além dos drones. Por outro lado, a interface wearable proposta pode ser integrada em soluções diferentes, para o controlo de drones ou outros dispositivos, ou considerar um conjunto diferente de gestos manuais, enquanto a própria biblioteca de gestos pode ser reutilizada com diferentes interfaces de gestos manuais.
The increasing popularity of Unmanned Aerial Vehicles (UAV) is a result of the latest developments in the scientific and business worlds to benefit from their unique flight capabilities. The increased flight flexibility of drones promotes the provision of several services that otherwise would be dangerous, time-consuming, and difficult to perform, even with ground robots. However, the integration of drones into our society faces challenges, some of them related with a complex interaction. The traditional remote controller (RC) requires high dexterity and training to fly UAVs with success, especially in dense and dangerous environments, often making people being afraid to use it, and consequently reducing engagement. As such, the development of an intuitive, simple, and easy to learn, control alternative has the potential to revert this context and increase the presence of UAVs in our society. A literature review of human-drone interaction (HDI), the field of human-computer interaction (HCI) dedicated to the interaction with drones, shows that in recent years, gestures, and brain-computer interfaces (BCI) became the interface types of greatest interest. Both are considered natural user interfaces (NUI) for being intuitive and ensuring simpler interactions with a faster learning process, among other advantages. However, gestures demonstrate the greatest potential for the control of drones in real-time, whilst BCIs are still in an early stage and unreliable. Therefore, the main goal of this research work is the proposal of a control solution for realtime drone piloting, which comprises a novel wearable interface and an architecture to support it. This architecture was designed modularly considering a universal communication and the centralization of the control. This characterization results in a solution compatible with as much different drones as possible, that allows the pilot to select and control a single or multiple drones, using the same interface, at the same time. On its turn, the interface, which consists in two custom smart bands, provides means to achieve a simple and intuitive control through natural dynamic hand gestures. With this purpose, a gestures library was created, designed for real-time drone piloting, where the gestures correspond to pitch and roll hand rotations, used individually, or combined. The validation of this novel interface and architecture was performed after the development of a functional prototype, applied in two use cases, corresponding to the piloting of two different quadcopters, a consumer-grade, and a custom-built one. The results obtained through technical and user tests validated the proposed architecture for the piloting of drones in real-time, in terms of control and communication latency, and demonstrated the benefits of hand gestures relative to the traditional RC, especially for the participants with interaction limitations (e.g., rheumatism, attention deficit). This usability evaluation proved the custom wearable interface to be successful as a reliable alternative for the traditional RC. During the prototype implementation and functional tests, a limitation was identified. People are afraid of crashing a real drone, for the possibility of causing personnel injuries and damage to property. As such, asking a neophyte to fly a drone in a trajectory with obstacles is a risky decision. Therefore, to mitigate the risks of crashing a real drone during training sessions in real environments, this research work also presents a training and evaluation solution to develop virtual trajectories and obstacles that can be used in real-time, with real drones, in real environments. Through Augmented Reality (AR) web technologies, this solution adapts open areas into complex flight courses through printed markers, which enable the creation of training challenges with a decreased risk of crashing the drone, as the obstacles are virtual and presented to the pilot as an overlay in the display reproducing the drone stream. The results of the user tests demonstrate that this proposal can improve training sessions performed in the same space through the definition of variable virtual trajectories. This research work results in the detailed characterization of a real-time drone control solution that uses natural hand gestures to potentiate the engagement of users and their education for the control of drones, promoting the popularity of UAVs in our society as service providers. It also works as a scientific guide that provides the knowledge and methodology required to design, demonstrate, and evaluate real-time drone control solutions, supported by the characterization of several theoretical topics from HCI, UAVs, and HDI, which describe the scope. For instance, new solutions can benefit from the modularity of the defined architecture to adapt the proposed solution using a different control interface, or even to control different devices besides drones. On their turn, the proposed wearable interface can be integrated in different solutions for the control of drones or other devices, or consider a different set of hand gestures, whilst the gesture library itself can be reused with different hand gesture interfaces.
The increasing popularity of Unmanned Aerial Vehicles (UAV) is a result of the latest developments in the scientific and business worlds to benefit from their unique flight capabilities. The increased flight flexibility of drones promotes the provision of several services that otherwise would be dangerous, time-consuming, and difficult to perform, even with ground robots. However, the integration of drones into our society faces challenges, some of them related with a complex interaction. The traditional remote controller (RC) requires high dexterity and training to fly UAVs with success, especially in dense and dangerous environments, often making people being afraid to use it, and consequently reducing engagement. As such, the development of an intuitive, simple, and easy to learn, control alternative has the potential to revert this context and increase the presence of UAVs in our society. A literature review of human-drone interaction (HDI), the field of human-computer interaction (HCI) dedicated to the interaction with drones, shows that in recent years, gestures, and brain-computer interfaces (BCI) became the interface types of greatest interest. Both are considered natural user interfaces (NUI) for being intuitive and ensuring simpler interactions with a faster learning process, among other advantages. However, gestures demonstrate the greatest potential for the control of drones in real-time, whilst BCIs are still in an early stage and unreliable. Therefore, the main goal of this research work is the proposal of a control solution for realtime drone piloting, which comprises a novel wearable interface and an architecture to support it. This architecture was designed modularly considering a universal communication and the centralization of the control. This characterization results in a solution compatible with as much different drones as possible, that allows the pilot to select and control a single or multiple drones, using the same interface, at the same time. On its turn, the interface, which consists in two custom smart bands, provides means to achieve a simple and intuitive control through natural dynamic hand gestures. With this purpose, a gestures library was created, designed for real-time drone piloting, where the gestures correspond to pitch and roll hand rotations, used individually, or combined. The validation of this novel interface and architecture was performed after the development of a functional prototype, applied in two use cases, corresponding to the piloting of two different quadcopters, a consumer-grade, and a custom-built one. The results obtained through technical and user tests validated the proposed architecture for the piloting of drones in real-time, in terms of control and communication latency, and demonstrated the benefits of hand gestures relative to the traditional RC, especially for the participants with interaction limitations (e.g., rheumatism, attention deficit). This usability evaluation proved the custom wearable interface to be successful as a reliable alternative for the traditional RC. During the prototype implementation and functional tests, a limitation was identified. People are afraid of crashing a real drone, for the possibility of causing personnel injuries and damage to property. As such, asking a neophyte to fly a drone in a trajectory with obstacles is a risky decision. Therefore, to mitigate the risks of crashing a real drone during training sessions in real environments, this research work also presents a training and evaluation solution to develop virtual trajectories and obstacles that can be used in real-time, with real drones, in real environments. Through Augmented Reality (AR) web technologies, this solution adapts open areas into complex flight courses through printed markers, which enable the creation of training challenges with a decreased risk of crashing the drone, as the obstacles are virtual and presented to the pilot as an overlay in the display reproducing the drone stream. The results of the user tests demonstrate that this proposal can improve training sessions performed in the same space through the definition of variable virtual trajectories. This research work results in the detailed characterization of a real-time drone control solution that uses natural hand gestures to potentiate the engagement of users and their education for the control of drones, promoting the popularity of UAVs in our society as service providers. It also works as a scientific guide that provides the knowledge and methodology required to design, demonstrate, and evaluate real-time drone control solutions, supported by the characterization of several theoretical topics from HCI, UAVs, and HDI, which describe the scope. For instance, new solutions can benefit from the modularity of the defined architecture to adapt the proposed solution using a different control interface, or even to control different devices besides drones. On their turn, the proposed wearable interface can be integrated in different solutions for the control of drones or other devices, or consider a different set of hand gestures, whilst the gesture library itself can be reused with different hand gesture interfaces.
Descrição
Submetido à Universidade de Trás-os-Montes e Alto Douro para o preenchimento dos requisitos parciais para a obtenção do grau de Doutor em Informática
Palavras-chave
Treino de voo com Realidade Aumentada , interfaces de gestos para controlo de drones