Erudex sealErudex
Software & AI Engineering
All levels

Information Theory

Academic (University) preparation with worked examples, written practice and independent learning assessments.

Independent university-level study of Information Theory. This original curriculum draws on public academic subject descriptions; it is not an official faculty syllabus, university credit, accreditation or an Ivy League qualification. Topics covered: Self-information; Entropy; Conditional entropy; Mutual information. Kraft inequality; Prefix codes; Huffman coding; Source coding theorem. Typical sequences; Asymptotic equipartition; Compression limits; Universal coding. Discrete channels; Channel capacity; Symmetric channels; Channel coding theorem. Gaussian channels; Power constraints; Water filling; Bandwidth tradeoffs. Rate-distortion functions; Lossy compression; Quantization; Source-channel separation. Multiple-access channels; Broadcast channels; Distributed compression; Network information limits. Information inequalities; Data processing; Fano inequality; Information-theoretic inference. Projects use safe simulations, public datasets or textual analysis; specialist subject review is required. Includes lessons, supplementary YouTube tutorials, practical exercises, quizzes and free/paid practice tests. Erudex is independent, not affiliated with the source institution or awarding body. Its 80% learning checks are not official grades or university credit. Content requires qualified subject review before use as a complete teaching programme.

32h Certificate available
InternationalAcademic (University)Independent preparation

Curriculum

One complete track with its own final exam and certificate.

Independent university-level study of Information Theory. This original curriculum draws on public academic subject descriptions; it is not an official faculty syllabus, university credit, accreditation or an Ivy League qualification. Topics covered: Self-information; Entropy; Conditional entropy; Mutual information. Kraft inequality; Prefix codes; Huffman coding; Source coding theorem. Typical sequences; Asymptotic equipartition; Compression limits; Universal coding. Discrete channels; Channel capacity; Symmetric channels; Channel coding theorem. Gaussian channels; Power constraints; Water filling; Bandwidth tradeoffs. Rate-distortion functions; Lossy compression; Quantization; Source-channel separation. Multiple-access channels; Broadcast channels; Distributed compression; Network information limits. Information inequalities; Data processing; Fano inequality; Information-theoretic inference. Projects use safe simulations, public datasets or textual analysis; specialist subject review is required.

Self-information; Entropy; Conditional entropy; Mutual information

  • Self-Information and Shannon Entropy of Discrete Random Variables Lab60 min
  • Joint Entropy, Conditional Entropy, and the Chain Rule Lab60 min
  • Relative Entropy (KL Divergence) and Mutual Information Lab60 min
  • Fundamental Information Inequalities and Data Processing Lab60 min

Careers this prepares you for

    How you study

    Every lesson pairs an in-depth video with written notes, key takeaways, a hands-on lab, exercises and a quiz. Work at your own pace, then sit the final examination for your Erudex certificate.