neural networks

bibliographie.bib

@@ -121,4 +121,22 @@
   file        = {:/home/lukas/Bachelorarbeit/papers/RBF.pdf:PDF},
 }
 
+@WWW{NN-math,
+  author       = {Piotr Skalski},
+  title        = {Deep Dive into Math Behind Deep Networks},
+  date         = {2018-08-17},
+  url          = {https://towardsdatascience.com/https-medium-com-piotr-skalski92-deep-dive-into-deep-networks-math-17660bc376ba},
+  organization = {Towards Data Science},
+  urldate      = {2019-07-16},
+}
+
+@WWW{NN-python,
+  author       = {James Loy},
+  title        = {How to build your own Neural Network from scratch in Python},
+  date         = {2018-05-14},
+  url          = {https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6},
+  organization = {Towards Data Science},
+  urldate      = {2019-07-16},
+}
+
 @Comment{jabref-meta: databaseType:biblatex;}

images/graphviz/general.dot

@@ -0,0 +1,41 @@
+digraph G {
+    rankdir=LR
+    splines=line
+    nodesep=.05;
+	bgcolor="transparent";
+    penwidth = 0
+    node[label=""]    
+    subgraph cluster_0 {
+		color=white;
+                node [style=solid,color=blue4, shape=circle];
+		x1 x2 x3 x4 ;
+        bgcolor="transparent";
+
+		label = "input layer";
+	}
+
+	subgraph cluster_1 {
+		color=white;
+		node [style=solid,color=red2, shape=circle];
+		a12 a22 a32 a42 a52;
+		label = "hidden layer";
+	}
+
+	subgraph cluster_2 {
+		color=white;
+		node [style=solid,color=red2, shape=circle];
+		a13 a23 a33 a43 a53;
+		label = "hidden layer";
+	}
+
+	subgraph cluster_3 {
+		color=white;
+		node [style=solid,color=seagreen2, shape=circle];
+		O1 O2 O3 O4;
+		label="output layer";
+	}
+
+    {x1; x2; x3; x4} -> {a12;a22;a32;a42;a52};
+    {a12; a22; a32; a42; a52} -> {a13;a23;a33;a43;a53}; 
+	{a13;a23;a33;a43;a53} -> {O1;O2;O3;O4}
+}

images/graphviz/graph.dot

@@ -0,0 +1,34 @@
+# https://gist.github.com/thigm85/5760134
+digraph G {
+    rankdir=LR
+	splines=line
+    nodesep=.05;
+    bgcolor="transparent";
+    penwidth = 0  
+          
+    node [label=""];
+        
+    subgraph cluster_0 {
+		color=white;
+        node [style=solid,color=blue4, shape=circle];
+		x1 x2 x3 x4 x5 x6;
+		label = "input layer";
+	}
+
+	subgraph cluster_1 {
+		color=white;
+		node [style=solid,color=red2, shape=circle];
+		a12 a22 a32 a42;
+		label = "hidden layer";
+	}
+
+	subgraph cluster_2 {
+		color=white;
+		node [style=solid,color=seagreen2, shape=circle];
+		O1;
+		label="output layer";
+	}
+
+    {x1; x2; x3; x4; x5; x6} -> {a12;a22;a32;a42};
+    {a12; a22; a32; a42} -> O1
+}

main.tex

@@ -88,7 +88,7 @@ In the first simulation run for every parameter combination from Table \ref{tab:
 
 \todo{spacing is ugly}
 
-\begin{lstlisting}[language=bash]
+\begin{lstlisting}[language=bash,flexiblecolumns=false]
 miluphcuda -N 20000 -I rk2_adaptive -Q 1e-4 -n 300  -a 0.5 -H -t 144.0 -f impact.0000 -m material.cfg -s -g
 \end{lstlisting}
 
@@ -268,13 +268,59 @@ Applying the same method to a list of random points allows to interpolate their
 
 The scipy function \texttt{scipy.interpolate.Rbf} allows directly interpolating a value similar to \texttt{griddata} in Section \ref{sec:griddata-implementation}. A difference in usage is that it only allows interpolating a single value, but as it is pretty quick it is possible to calculate multiple values sequentially.
 
+\subsection{Results}
+
+\begin{figure}[h] % TODO: h is temporary 
+	\centering
+	\includegraphics[width=0.4\linewidth]{images/graphviz/general.pdf}
+	\caption{}
+	\label{fig:neuralnetwork-general}
+\end{figure}
 
 \section{Neural Networks}
 
+Another method that is good at taking pairs of input and output values and then being able to predict output values for arbitrary input sets are \textit{Artificial neural networks} (\texttt{ANNs}).
+
 \subsection{Theory}
 
+The idea behind artificial neural networks is trying to emulate the functionality of neurons by having nodes that are connected to each others. The weights of theses connections are modified during the training to represent the training data and can then be used to predict new results for input values not seen in the training data.
+
+Every neural network needs an input layer with as many nodes as input parameters and an output layer with a node for every output value. In between there can be multiple hidden layers with an arbitrary amount of nodes. (Figure \ref{fig:neuralnetwork-general})
+
+If we first only consider a single neuron then on iteration it calculates the sum over all input values multiplied with their weight $w$. Afterwards an activation function $g$ is applied to the sum $z$ to get the prediction $\hat{y}$.
+
+\begin{equation}
+	z=\sum_{i}w_ix_i \qquad \hat{y}=g(z)
+\end{equation}
+
+The non-linear activation function allows the network to be able to approximate all functions instead of being just a linear function itself. Popular activation functions are the sigmoid function $\sigma(x)={\frac {1}{1+e^{-x}}}$ and the ReLU function (\textit{rectified linear unit}, $f(x)=\max(0,x)$).\footcite{NN-math}
+
+After this first step (the \textit{feedforward}) is done, the weights can be modified by comparing the prediction with the real output (the \textit{backpropagation}). The function that describes the error between those is called the Loss function and one of them is the mean squared error function:
+
+\begin{equation}
+	L(\hat{y},y)=\sum_{i}(\hat{y}_i-y_i)^2
+\end{equation}
+
+To update the weights the derivative of the Loss function over the weights is calculated and added to the existing weights.\todo{more details?}\footcite{NN-python}
+
+
+
 \subsection{Implementation}
 
+As building a neural network from scratch gets complex very quickly it is easier to use \texttt{Keras}\footnote{\url{https://keras.io}} which provides easy to use high-level functions over the calculations provided by \texttt{TensorFlow}\footnote{\url{https://www.tensorflow.org/}}. To build our network, we only need to specify the structure of the layers, take our input and let the network train 200 times.\todo{explain more once it is finished}
+
+\begin{lstlisting}[language=Python]
+model = Sequential()
+model.add(Dense(6, input_dim=6, kernel_initializer='normal', activation='relu'))
+model.add(Dense(5, kernel_initializer='normal', activation='relu'))
+model.add(Dense(4, kernel_initializer='normal', activation='relu'))
+model.add(Dense(1, kernel_initializer='normal'))
+model.compile(loss='mean_squared_error', optimizer='adam')
+
+model.fit(X, Y, epochs=200)
+\end{lstlisting}\todo{just a placeholder}
+
+
 \subsection{Results}
 
 \subsection{Issues}

template.tex

@@ -123,7 +123,8 @@ american, % language of the document
 	stepnumber=1,                    % the step between two line-numbers. If it's 1, each line will be numbered
 	stringstyle=\color{strings},     % string literal style
 	tabsize=2,	                   % sets default tabsize to 2 spaces
-	language=bash
+	language=bash,
+	flexiblecolumns=true
 }
 \lstset{literate=% support Umlauts in listings
 	{Ö}{{\"O}}1
@@ -136,6 +137,19 @@ american, % language of the document
 }
 
 
+\lstset{literate=%
+	*{0}{{{\color{red!20!violet}0}}}1
+	{1}{{{\color{red!20!violet}1}}}1
+	{2}{{{\color{red!20!violet}2}}}1
+	{3}{{{\color{red!20!violet}3}}}1
+	{4}{{{\color{red!20!violet}4}}}1
+	{5}{{{\color{red!20!violet}5}}}1
+	{6}{{{\color{red!20!violet}6}}}1
+	{7}{{{\color{red!20!violet}7}}}1
+	{8}{{{\color{red!20!violet}8}}}1
+	{9}{{{\color{red!20!violet}9}}}1
+}
+
 %--------------------------------------- PDF output and hyperlinks ----------------------------