Neural Networks in CUDA from Scratch - 3. Multiple Layers

We are extending our CUDA Kernel so it can work through multiple layers in a single call. The code can be found here: https://github.com/ThoenigAdrian/NeuralNetworksCudaTutorial/blob/main/Part%203%20-%20Multiple%20Layers/kernel.cu What we did so far: Part 1 - Implement Cuda Kernel which can execute the sigmoid activation function for many values at once Part 2 - Add the Linear Layer (Dot Product) to our Kernel. So far we only had inputs and outputs but no hidden layers. This will be a multipart series: I am planning to cover the following topics 1. Implementing activation layer 2. Linear layer *3. Feedforward with multiple layers* 4. Forward pass with multiple inputs 5. Backpropagation 6. Applying gradient descent 7. Training loop with multiple iterations 8. Cost function 9. Scaling If you are interested in general CUDA Programming, I can recommend the following the following resources: Cuda Crash Course: @CoffeeBeforeArch https://www.youtube.com/watch?v=2NgpYFdsduY&list=PLxNPSjHT5qvtYRVdNN1yDcdSl39uHV_sU&index=1&t=0s Cuda Crash Cource v2: @CoffeeBeforeArch https://www.youtube.com/watch?v=fBK19n2x-Bg&list=PLxNPSjHT5qvtdK7OVxhbzLmYDnmV_yiYr&index=1&t=0s NVIDIA CUDA Tutorial @Creel https://www.youtube.com/watch?v=m0nhePeHwFs&list=PLKK11Ligqititws0ZOoGk3SW-TZCar4dK&index=1&t=0s Nvidia Documentation: https://docs.nvidia.com/cuda/cuda-c-p... (Yes the official documentation, it's actually really good)