DeepSeek: Introduction

This course explores DeepSeek, an open-source large language model family that achieves cost-effective, state-of-the-art performance through a Mixture-of-Experts (MoE) architecture. The material is organized into three parts: understanding the MoE architecture and how experts are selected and routed, examining DeepSeek's benchmark results and reasoning strengths in math, coding, and logic, and finally covering practical local deployment, cost analysis, and optimization.

To understand why the Mixture-of-Experts (MoE) architecture is powerful, it helps to first understand how traditional dense large language models work. In a dense model, every single part of the network is active for every piece of information (every token) it processes. This design gives consistent results, but because it uses all of its billions of parameters for every single token, the computational cost is very high, making it an expensive architecture to run.

Conceptually, the flow of a dense model looks like this: input flows into the model, which may have N total parameters, and all N parameters are active for every token. Every single part of the model has to work to process the data and produce an output....