Unaprijedna troslojna perceptronska neuronska mreža s adaptivnom ReLu aktivacijskom funkcijom

undergraduate thesis

• Title (english): A feedforward three-layered perceptron neural network with the adaptive ReLU actiavtion function
• Author: Luka Penava
• Mentor: Danko Brezak (mentor)
• Granter: University of Zagreb; Faculty of Mechanical Engineering and Naval Architecture; Zagreb
• Defense date and country: 2025-02-24, Croatia
• Scientific / art field, discipline and subdiscipline: TECHNICAL SCIENCES; Mechanical Engineering; Production Mechanical Engineering

Abstract

U višeslojnim perceptronskim mrežama najčešće korištene aktivacijske funkcije su sigmoidalna i tangens-hiperbolna. Za razliku od toga, u dubokim mrežama dominira ReLU funkcija uvedena radi ublažavanja problema nestajućeg gradijenta, prisutnog kod sigmoidalnih i tangens-hiperbolnih funkcija. Iako ReLU funkcija rješava taj problem, njena primjena donosi i nova ograničenja poput pojave neaktivnih neurona. Stoga se, u ovom radu, istražuju učinkovitost primjene ReLU funkcije, konkretno ReLU s prilagodljivim nagibom i ReLU s prilagodljivim pragom aktivacije. Cilj rada je analizirati svojstva adaptacija, fokusirajući se na identifikaciju dinamičkih sustava i predviđanje kaotičnih vremenskih serija. Performanse različitih aktivacijskih funkcija se uspoređuju analizom mjere točnosti algoritma učenja (NRMS) s ciljem identificiranja optimalnih rješenja za navedene probleme.
Usporedbom svih triju testova perceptronske neuronske mreže, može se zaključiti da ReLU
aktivacijska funkcija s prilagodljivim pragom aktivacije pokazuje najbolja svojstva aproksimacije
i generalizacije. Njena jednostavnost implementacije čini ju idealnom čak i za kompleksnije sustave. Osim toga njezina uporaba ne zahtjeva preveliku računalnu snagu, memoriju, niti dugo vrijeme treniranja.

Abstract (english)

In multilayer perceptron networks, the most commonly used activation functions are
sigmoidal and hyperbolic tangent. In contrast, deep networks are dominated by the ReLU function,
which was introduced to mitigate the vanishing gradient problem present in sigmoidal and
hyperbolic tangent functions. Although the ReLU function solves this problem, its application also
introduces new limitations, such as the appearance of inactive neurons. Therefore, this paper
investigates the effectiveness of applying ReLU function adaptations, specifically ReLU with
adaptive slope and ReLU with adaptive activation threshold. The aim of the work is to analyze the
properties of these adaptations, focusing on the identification of dynamic systems and the
prediction of chaotic time series. The performance of different activation functions is compared
by analyzing the measure of accuracy of the learning algorithm (NRMS), with the aim of
identifying optimal solutions for the stated problems.
By comparing all three tests of the perceptron neural network, it can be concluded that the ReLU
activation function with adaptive activation threshold exhibits the best approximation and
generalization properties. Its simplicity of implementation makes it ideal even for more complex
systems. Furthermore, its use does not require excessive computing power, memory, or long
training times.

• Keywords: Neuronske mreže; ReLU; Adaptive ReLU; Identifikacija dinamičkih sustava; Identifikacija kaotičnih sustava
• Keywords (english): Neural Networks; ReLU; Adaptive ReLU; Dynamic Systems Identification; Chaotic Systems Identification
• Language: croatian
• URN:NBN: urn:nbn:hr:235:913047
• Study programme: Title: Mechanical Engineering; Study programme type: university; Study level: undergraduate; Academic / professional title: sveučilišni/a prvostupnik/prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka strojarstva (sveučilišni/a prvostupnik/prvostupnica (baccalaureus/baccalaurea) inženjer/inženjerka strojarstva)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Open access
• Created on: 2025-02-23 09:06:27