Welcome to Deep Bottleneck’s documentation!

Contents

• Big Picture
  • Entropy & Mutual Information
  • What is this mysterious information bottleneck?
  • An Introduction into Neural Networks
  • Basic Maths
• Contributing
  • Extending the framework
  • Git workflow
  • Style Guide
  • Experiment workflow
  • Documentation
• Glossary
  • Information Theory Basics
  • Mathematical Terms in Tishby’s Experiments
• Literature
• Literature Summary
  • 1. THE INFORMATION BOTTLENECK METHOD (Tishby 1999)
  • 2. DEEP LEARNING AND THE INFORMATION BOTTLENECK PRINCIPLE (Tishby 2015)
  • 3. OPENING THE BLACK BOX OF DEEP NEURAL NETWORKS VIA INFORMATION (Tishby 2017)
  • 4. ON THE INFORMATION BOTTLENECK THEORY OF DEEP LEARNING (Saxe 2018)
  • 5. ON THE INFORMATION BOTTLENECK THEORY OF DEEP LEARNING
• User guide
  • Installation
  • How to use the framework
• Experiments
  • Description of cohorts
  • Comparing activation functions for a minimal model
  • Calculation of mutual information for different parts of the dataset
  • Experiment Evaluation of Activation Functions
  • Standard vs. Weighted Binning
  • Effect of weight renormalization on activity patterns
  • The data set provided by Tishby
  • Our attempt to generate the data set above
• Indices and tables
• API Documentation
  • deep_bottleneck package