Deep Learning and Neural Networks: From Gradient Calculus to Deployment

A feedforward network composes transformations. For one layer, z = Wx + b and h = φ(z), where x is the input, W contains weights, b is a bias, and φ is a nonlinear activation. Without nonlinearities, stacked affine layers collapse into one affine transformation, regardless of depth. ReLU, defined as max(0, z), creates piecewise-linear functions; GELU provides a smooth activation commonly used in Transformers. A network’s architecture determines its inductive biases: convolutions exploit spatial locality, while attention allows interactions between positions in a sequence.

Training minimizes an objective such as the mean loss over examples plus a regularization term. Regression might use squared error, while multiclass classification typically uses cross-entropy on logits. Logits are unconstrained scores, not probabilities; softmax converts them into a normalized distribution. Implementations generally combine softmax and cross-entropy in a numerically stable operation. Learning a useful function also requires generalization: success on training data is insufficient. Separate validation and test sets help measure performance on unseen examples, provided the split prevents leakage between related records, users, or time periods.

Backpropagation applies the chain rule efficiently to a computational graph. If the upstream derivative for a layer is g = ∂L/∂h, then δ = g ⊙ φ′(z), ∂L/∂W = δxᵀ, ∂L/∂b = δ, and ∂L/∂x = Wᵀδ for a single example using column vectors. Batched implementations sum or average parameter gradients according to the loss reduction. These operations are vector-Jacobian products: automatic differentiation need not construct every full Jacobian. Ordinary neural networks usually optimize parameters in Euclidean space; genuinely manifold-constrained parameters require additional geometric machinery rather than merely inserting nonlinear activations.

Consider a scalar network with x = 2, w = 0.5, b = 0, h = ReLU(wx + b), prediction ŷ = vh, and v = 3. With target y = 1 and loss L = ½(ŷ − y)², the forward pass gives h = 1, ŷ = 3, and L = 2. The derivatives are ∂L/∂v = 2, ∂L/∂w = 12, and ∂L/∂b = 6. One simultaneous gradient-descent update with learning rate 0.01 produces v = 2.98, w = 0.38, and b = −0.06. Recomputing gives ŷ = 2.086 and L ≈ 0.590: a concrete reduction, though larger steps would not guarantee improvement.

A typical PyTorch iteration clears gradients, computes predictions, evaluates the loss, calls backward(), and updates parameters through an optimizer. Gradients accumulate by default, so omitting optimizer.zero_grad() changes the algorithm unless accumulation is intentional. Use model.train() during training and model.eval() for validation, especially with dropout or batch normalization. Evaluation mode does not disable gradient recording; combine it with an appropriate no-gradient context. For multiclass CrossEntropyLoss with class-index targets, provide raw logits and integer labels rather than applying softmax first.

Neural network optimization depends on more than the optimizer name. AdamW separates weight decay from its adaptive gradient update, but learning rate, batch size, scheduling, and parameter grouping still matter. First overfit a tiny dataset: inability to do so often exposes incorrect labels, disconnected gradients, or loss-shape errors. Then track training and validation losses, gradient norms, throughput, and memory consumption. Mixed precision training can improve efficiency, but numerical behavior depends on hardware and dtype. FP16 commonly needs loss scaling; BF16 usually does not, although sensitive reductions may still benefit from FP32 computation.

Transformers represent tokens as vectors and mix information through attention mechanisms. For input matrix X, learned projections produce Q = XWQ, K = XWK, and V = XWV. Scaled dot-product attention computes softmax(QKᵀ/√dₖ + M)V, where M encodes restrictions such as causal masking. The scale helps control score magnitudes as key dimension grows. Multi-head attention learns several interaction patterns, while feedforward blocks transform each position separately. Residual connections and normalization support optimization, and positional information is necessary because attention alone does not encode token order.

For a concrete cost example, a sequence of 1,024 tokens has 1,048,576 query-key pairs per head; doubling sequence length quadruples that count. Memory-efficient attention implementations can avoid materializing the complete score matrix, but standard dense attention still has quadratic pairwise computation. During autoregressive inference, a key-value cache avoids recomputing earlier keys and values, trading additional memory for speed. Architectural choices therefore depend on context length, latency, and available memory—not only parameter count. For a small labeled text dataset, fine-tuning an existing encoder is often a better experiment than training a decoder-only language model from scratch.

In data-parallel distributed training, each worker holds a model replica and processes different examples. PyTorch DistributedDataParallel synchronizes gradients, typically through all-reduce operations, before each replica applies the same optimizer update. With eight workers, a local batch of 16, and four accumulation steps, the effective global batch is 512 examples. If each microbatch loss is a mean, divide it by four before backward() to preserve the intended accumulated gradient scale. This calculation assumes equal microbatch sizes and averaged cross-worker gradients; uneven batches require explicit weighting to retain the correct example-level objective.

Use a distributed sampler so workers do not unintentionally train on identical examples, and update its epoch setting when shuffling. During accumulation, DDP’s no_sync() context can skip synchronization on intermediate microbatches, with synchronization enabled on the final one. When model states exceed device memory, sharded approaches partition optimizer state, gradients, or parameters rather than replicating everything. Activation checkpointing saves memory by recomputing selected activations during backward. Diagnose bottlenecks with profiling: adding GPUs may expose slow storage, insufficient preprocessing, communication overhead, or small kernels instead of delivering proportional speedups.

Deployment begins with a stable inference contract: input names, shapes, dtypes, preprocessing, output interpretation, and supported batch sizes. Exporting a PyTorch model to ONNX represents its computation in an interchange format, but does not guarantee every operator or dynamic shape will work on every backend. Compare exported outputs against the original model on representative inputs, using explicit tolerances and task-level metrics. ONNX Runtime executes models through execution providers that target different hardware. TensorRT builds optimized inference engines for supported NVIDIA hardware, applying transformations such as fusion and precision selection.

Benchmark the complete service rather than only one model invocation. Include warm-up, accelerator synchronization where needed, preprocessing, transfers, and postprocessing; measure throughput and tail latency at realistic concurrency. Dynamic-shape TensorRT workloads need suitable optimization profiles, and compiled artifacts should be tested against the intended runtime and hardware environment. FP16 or INT8 execution may improve performance, but quantization requires accuracy validation and, depending on the method, representative calibration data or quantization-aware training. Keep a known-good model and a rollback path: faster inference is not an improvement if numerical changes break important predictions.

Consider building a support-ticket classifier. Start by defining the labels, ambiguous cases, and the cost of misclassification. Split related tickets by customer or thread when necessary to avoid near-duplicate leakage, and consider a chronological test set if future deployment is the goal. Build a simple baseline before fine-tuning a pretrained Transformer. Tokenize consistently, batch examples with appropriate padding, and mask padding positions. Evaluate per-class precision and recall as well as an aggregate measure such as macro-F1; overall accuracy alone can conceal poor performance on uncommon but operationally important categories.

Run controlled experiments that change one major factor at a time, recording data versions, configurations, checkpoints, and software dependencies. Select the model using validation results, then evaluate the chosen configuration on the held-out test set without tuning against it. Export the model, verify output parity, and load-test the service under realistic ticket lengths and traffic patterns. After release, monitor input changes and obtain labeled samples to assess actual quality. This workflow connects the course’s mathematics and systems topics: correct gradients make training possible, efficient infrastructure makes iteration practical, and disciplined evaluation establishes whether the system is useful.