From 07e9742ce5c376b70cde0bb88211a716d713c060 Mon Sep 17 00:00:00 2001 From: Lukas Winkler Date: Wed, 11 Sep 2019 12:52:47 +0200 Subject: [PATCH] move comparison to conclusion --- 40_interpolations.tex | 1 - 44_comparison.tex | 23 ----------------------- 50_conclusion.tex | 23 ++++++++++++++++++++++- 3 files changed, 22 insertions(+), 25 deletions(-) delete mode 100644 44_comparison.tex diff --git a/40_interpolations.tex b/40_interpolations.tex index abf7712..f530edd 100644 --- a/40_interpolations.tex +++ b/40_interpolations.tex @@ -4,4 +4,3 @@ \input{42_rbf.tex} \input{43_nn.tex} -\input{44_comparison.tex} \ No newline at end of file diff --git a/44_comparison.tex b/44_comparison.tex deleted file mode 100644 index 705d58b..0000000 --- a/44_comparison.tex +++ /dev/null @@ -1,23 +0,0 @@ -% !TeX spellcheck = en_US -\section{Comparison} -\label{sec:comparison} -\todo{To comparison} -To compare the three methods explained above and measure their accuracy an additional set of 100 simulations (with the same properties as the ones listed in Section \ref{sec:resimulation}) was created. These results are neither used to train or select the neural network, nor are in the dataset for griddata and RBF interpolation. Therefore, we can use them to generate predictions for their parameters and compare them with the real fraction of water that remained in those simulations. By taking the mean absolute difference and the mean squared error between the predictions and the real result, the accuracy of the different methods can be estimated (Table \ref{tab:comparison}). As one of these parameter sets is outside the convex hull of the training data and griddata can't extrapolate, this simulation is skipped and only the remaining 99 simulations are considered for the griddata accuracy calculation. - -Of the three methods, the trained neural network has the highest mean squared error. This seems to be at least partly caused by the fact that during training the neural network, the data is generalized, causing the final network to output the \enquote{smoothest} interpolations. While this causes the errors to be higher, it might be possible that the fine structured details in the simulation output is just a artifact of the simulation setup and doesn't represent real world collisions. -\todo{better wording} - -Another important aspect to compare is the interpolation speed. The neural network is able to give the 100 results in about \SI{4}{\milli\second} (after loading the trained model). RBF interpolation is still reasonably fast, taking about \SI{8.5}{\second} (\SI{85}{\milli\second} per interpolation). But as \texttt{griddata} expects a grid-based parameter space, it becomes really slow when adding the resimulation data with random parameters. A single interpolation takes about \SI{35}{\second} totaling to around an hour for all 99 test cases. Using only the original dataset brings the runtime down to around \SI{10}{\second}, but causes the results to be less accurate than all other methods. (first row in Table \ref{tab:comparison}) - -\begin{table}[h] - \centering - \begin{tabular}{rcc} - & {mean squared error} & {mean error} \\ - griddata (only original data) & 0.014 & 0.070 \\ - neural network & 0.010 & 0.069 \\ - RBF & 0.008 & 0.057 \\ - griddata & 0.005 & 0.046 - \end{tabular} - \caption{Prediction accuracy for the different interpolation methods} - \label{tab:comparison} -\end{table} \ No newline at end of file diff --git a/50_conclusion.tex b/50_conclusion.tex index 4d8a1ef..f992b9a 100644 --- a/50_conclusion.tex +++ b/50_conclusion.tex @@ -1,2 +1,23 @@ -\chapter{Conclusion} +% !TeX spellcheck = en_US +\chapter{Comparison and Conclusion} +\label{sec:comparison} +To compare the three methods explained above and measure their accuracy an additional set of 100 simulations (with the same properties as the ones listed in Section \ref{sec:resimulation}) was created. These results are neither used to train or select the neural network, nor are in the dataset for griddata and RBF interpolation. Therefore, we can use them to generate predictions for their parameters and compare them with the real fraction of water that remained in those simulations. By taking the mean absolute difference and the mean squared error between the predictions and the real result, the accuracy of the different methods can be estimated (Table \ref{tab:comparison}). As one of these parameter sets is outside the convex hull of the training data and griddata can't extrapolate, this simulation is skipped and only the remaining 99 simulations are considered for the griddata accuracy calculation. + +Of the three methods, the trained neural network has the highest mean squared error. This seems to be at least partly caused by the fact that during training the neural network, the data is generalized, causing the final network to output the \enquote{smoothest} interpolations. While this causes the errors to be higher, it might be possible that the fine structured details in the simulation output is just a artifact of the simulation setup and doesn't represent real world collisions. +\todo{better wording} + +Another important aspect to compare is the interpolation speed. The neural network is able to give the 100 results in about \SI{4}{\milli\second} (after loading the trained model). RBF interpolation is still reasonably fast, taking about \SI{8.5}{\second} (\SI{85}{\milli\second} per interpolation). But as \texttt{griddata} expects a grid-based parameter space, it becomes really slow when adding the resimulation data with random parameters. A single interpolation takes about \SI{35}{\second} totaling to around an hour for all 99 test cases. Using only the original dataset brings the runtime down to around \SI{10}{\second}, but causes the results to be less accurate than all other methods. (first row in Table \ref{tab:comparison}) + +\begin{table}[h] + \centering + \begin{tabular}{rcc} + & {mean squared error} & {mean error} \\ + griddata (only original data) & 0.014 & 0.070 \\ + neural network & 0.010 & 0.069 \\ + RBF & 0.008 & 0.057 \\ + griddata & 0.005 & 0.046 + \end{tabular} + \caption{Prediction accuracy for the different interpolation methods} + \label{tab:comparison} +\end{table} \ No newline at end of file