Course Name: Information Theory and Coding

Course Code: S2293037

Semester: 5

Credit: 4

Program: Electronics

Course Module: Compulsory

Responsible: Jinghui Chu

E-mail:cjh@tju.edu.cn

Department:School of Electrical and Information Engineering, Tianjin University

TimeAllocation(1 credit hour = 45 minutes)

Course Description

• Information Quantity: Entropy, Joint Entropy, Conditional Entropy, Mutual Information, Differential Entropy.

• Data Compression: Kraft Inequality, Optimal Codes and Bounds on Code Length, Huffman Codes, Lossy Quantization, Rate Distortion Function.

• Channel Capacity: Channel, Symmetric Channels, Channel Capacity, Channel Coding Theorem.

Prerequisite

Signal Processing, Signal and Systems.

Course Objectives

This course presents the basic concepts of Information theory, and it covers the following topics: information quantities, sources and channels, channel capacity, lossless source coding, lossy source coding.

After taking this course, students will be able to gain knowledge and understanding of theories and concepts of information theory.

Course Syllabus

1. Introduction

2. Entropy

   1. Entropy, Property of Entropy

   2. Joint and Conditional Entropy

   3. Mutual Information

   4. Relationships between Entropy and Mutual Information

   5. Differential Entropy

   6. Entropy Rates of a Stochastic Process

3. Data Compression

   1. Examples of Codes

   2. Kraft Inequality

   3. Optimal Codes and Bounds on Code Length

   4. Huffman Codes

   5. Shannon-Fano-Elias Coding

4. Channel Capacity

   1. Examples

   2. Symmetric Channels

   3. Properties of Channel Capacity

   4. Channel Coding Theorem

   5. Gaussian Channel and Capacity

5. Rate Distortion Theory

   1. Quantization

   2. Definitions

   3. Calculation of the Rate Distortion Function

Textbooks & References

• T. M. Cover and J. A. Thomas.Elements of Information Theory. John Wiley, 2006.

We may also use readings from a few reference books:

• R. B. Ash.Information Theory. Dover, 1990.

• T. Berger.Rate Distortion Theory: A Mathematical Basis for Data Compression. Prentice Hall, 1971.

Grade Distribution

Attendance: 20% Final Exam:80%

Capability Tasks

CT1: To understand basic science, and to have analytical ability and the ability to integrate related knowledge.

CT2: To apply relevant professional knowledge to the field of science and technology: understanding of the basic concepts and its connotation, application of different methods and concepts which have been learned, capability of judging the scope and limitations of such applications.

CT3: To grasp methodologies and engineering tools: identifying, utilizing and solving problems. Even if the students are not familiar with the content, they can turn to computer tools for systematic analysis.

CT4: To carry out experiments in research environment with the abilities to utilize tools, especially for data collection and processing.

CT10: To have the capacity to work in international environment; the capability to master one or more foreign languages and be open to foreign cultures; be able to acclimatize themselves to the international language environment.

Achievements

• To be able to analyze the effects of a choice of an access method (e.g. frequency band) - Level: M

• To be able to evaluate a channel capacity and the feasibility of a digital transmission through this channel. - Level: M

• To be able to analyze source and channel coding. - Level: M

• To be able to implement a system for control and error detection and to implement basic lossless data compression techniques. - Level: M

Students: Electronics, Year 3