Mehanizam za usmjeravanje pogleda virtualnog agenta temeljem zvučnog i vizualnog ulaza

master's thesis

• Title (english): A mechanism for directing the virtual agent's gaze based on sound and visual input
• Author: Slaven Bakula
• Mentor: Tomislav Stipančić (mentor)
• Granter: University of Zagreb; Faculty of Mechanical Engineering and Naval Architecture; Zagreb
• Defense date and country: 2022-07-21, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Mechanical Engineering

Abstract

U sklopu Laboratorija za projektiranje izradbenih i montažnih sustava implementiran je virtualni softverski agent PLEA koji koristi multi-modalni pristup prilikom zaključivanja o emocionalnom stanju osobe s kojom komunicira temeljen na zvučnim i vizualnim informacijama. Koristeći neverbalni oblik komunikacije, korisnici mogu tako ostvariti kontakt sa virtualnim agentom. Kako bi komunikacija virtualnog agenta bila što sličnija stvarnoj osobi potrebno je implementirati značajke svakodnevnog govora kao što je pogled u oči prilikom komuniciranja. U ovom radu opisano je stvaranje jednog takvog sustava koji može biti implementiran u postojeći PLEA virtualni agent. Rad opisuje kreiranje virtualnog agenta u Unreal Engine programu na udaljenom računalu. U programskom jeziku Python uz korištenje dostupnih biblioteka napisan je program za prepoznavanje lica te određivanje kuta njegovog položaja na području kamere. Preko socket komunikacije šalju se vrijednosti definirane pozicije u stvarnom vremenu i ostvaruje se reakcija virtualnog agenta. Uz vizualnu akviziciju, napravljen je i program koji koristi lokalizaciju izvora zvuka korištenjem LyraTD-MSC modula te se slanjem iznos kuta, također preko socket protokola, ostvaruje reakcija virtualnog agenta.

Abstract (english)

As part of the Laboratory for the Design of Manufactured and Assembled Systems, a virtual software agent PLEA has been implemented, which uses a multimodal approach when inferring the emotional state of the person with whom it communicates based on sound and visual information. Using a non-verbal form of communication, users can make contact with a virtual agent. In order for the virtual agent's communication to be as similar as possible to a real person, it is necessary to implement features of everyday speech such as eye contact when communicating. This paper describes the creation of one such system that can be implemented in the existing PLEA virtual agent. The paper describes the creation of a virtual agent in the Unreal Engine program on a remote computer. A program for face recognition and determination of the angle of its position in the camera area was written in the Python programming language with the use of available libraries. Through socket communication, the values of the defined positions are sent in real time and the reaction of the virtual agent is realized. In addition to visual acquisition, a program that uses the localization of the sound source was created using the LyraTD-MSC module, and by sending the value of the angle, also via the socket protocol, the reaction of the virtual agent is realized.

• Keywords: prepoznavanje lica; Unreal Engine; Python; socket protokol
• Keywords (english): face recognition; Unreal Engine; Python; socket protocol
• Language: croatian
• URN:NBN: urn:nbn:hr:235:801522
• Study programme: Title: Mechanical Engineering; specializations in: Design, Process and Energy Engineering, Production Engineering, Engineering Modelling and Computer Simulation, Marine Engineering, Industrial Engineering and Management, Materials Engineering, Mechatronics and Robotics, Autonomous Systems; Course: Mechatronics and Robotics; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra inženjer/inženjerka strojarstva (magistar/magistra inženjer/inženjerka strojarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Embargoed access; Embargo expiration date: 2024-07-07
• Created on: 2022-07-07 07:53:53