Skip to content
Hallway
0:00 / 0:00

#informationtheory#davidmackay#entropy

Lecture 1: Introduction to Information Theory

Jakob Foerster

9.7K subscribers • 1 video on Hallway

YouTube
  • Recommended by 3 people: 3 on Hacker News.
  • A classic. People kept recommending it from 2018 to 2021.
  • Presented by David MacKay, Professor of Natural Philosophy at University of Cambridge
  • May be out of date. Uploaded 12 years ago, in an area where tools change fast.

Intermediate. Assumes working knowledge.

411K views12 years ago3 recommendations

Computer scienceAI and machine learningComputation and complexityMath for machine learning

David MacKay introduces the fundamental concepts of information theory, data compression, and noisy-channel coding in this opening university lecture.

Lecture 1 of the Course on Information Theory, Pattern Recognition, and Neural Networks. Produced by: David MacKay (University of Cambridge) Author: David MacKay, University of Cambridge A series of sixteen lectures covering the core of the book "Information Theory, Inference, and Learning Algorithms" (Cambridge University Press, 2003, inference.eng.cam.ac.uk/mackay/itila/) which can be bought at Amazon (amazon.co.uk/exec/obidos/ASIN/0521642981/davidmackay0f-21), and is available free online (inference.eng.cam.ac.uk/mackay/itila/). A subset of these lectures used to constitute a Part III Physics course at the University of Cambridge. The high-resolution videos and all other course material can be downloaded from the Cambridge course website (inference.eng.cam.ac.uk/mackay/itprnn/). Snapshots of the lecture can be found here: inference.eng.cam.ac.uk/itprnn_lectures/ These lectures are also available at videolectures.net/course_information_theory_pattern_recognit… (synchronized with snapshots and slides)

Why it's here

3 people recommend this

What people say. Highly recommended as part of an exceptional, clear, and engaging lecture series on information theory and machine learning.

swframe2on Hacker News • Recommends

On “Probabilistic Programming and Bayesian Methods for Hackers (2013)”

I recommend David MacKay's Intro to Info Theory class: link to this video There is a free book for it at: link Daphne Koller's 3 coursara classes on Probabilistic Graphical Models are also really good.
View on Hacker News

maxovon Hacker News • Recommends

On “Ask HN: What, in your opinion, are the greatest and most useful textbooks?”

Had . Unfortunately Professor MacKay passed away from cancer in 2016. We lost a great soul. I concur that his textbook and sense of humor were fantastic. It was very enjoyable for me to learn the material from his lectures. They are a great supplement to the textbook (or maybe vice versa?) and you can find them here: link to this video
View on Hacker News

samch93on Hacker News • Recommends

On “Sustainable Energy without the Hot Air (Revised, Community Edition)”

It‘s such a tragedy that he died so young. Prof Mackay was a role model who impacted our world in many positive ways (e.g. his work in information theory, Bayesian statistics, environmental science). I recommend his lecture on information theory which is available on youtube 1. He also documented the progress of his cancer very detailed on his blog 2, which is a very interesting (and sad) read. 1 link to this video 2 link
View on Hacker News

noudon Hacker News • Mentions

On “David MacKay: Information Theory, Inference, and Learning Algorithms: The Book”

"Your browser does not support playback of available video formats. Please install Adobe Flash player or upgrade to a more modern browser." ...Okay? I found the videos on youtube as well: link to this video
View on Hacker News

scythmic_waveson Hacker News • Mentions

On “David MacKay: Information Theory, Inference, and Learning Algorithms: The Book”

From lecture 1 1: > Ok, so, we have an answer of 1000 plus or minus 30 thanks to "Mr. Binomial", the inventor of the distribution that bears his name. Lol, this is great. 1: link to this video
View on Hacker News

teleforceon Hacker News • Mentions

On “Reading list to join AI field from Hugging Face cofounder”

Thanks for the list. Interesting to see that the book by the late David MacKay made it to the list, it is available to download in several formats from the book's website. Information Theory, Inference, and Learning Algorithms: link The lectures based on the book by David MacKay himself: link to this video Probably it's not a surprise that the father of information theory Claude Shannon is the earliest one to propose stochastic LLM based on Markov chain.
View on Hacker News