Skip to content

04-09: RL Book Word Frequency Analyzer

Python: Data Analytics

View the live site — ijk37.com

Project 09

Home  |  All Projects  |  Notes  |  Exercises  |  Quiz Hub

An NLP-based PDF analytics system that dissects Reinforcement Learning: An Introduction by Sutton & Barto — word frequencies, bigrams, trigrams, chapter-wise breakdowns, word-cloud visualizations, and an interactive Streamlit dashboard.

Topics covered: pypdf · nltk · pandas · matplotlib · wordcloud · numpy · Pillow · streamlit · collections.Counter · re

Features

  • Full-book & chapter-wise analysis — processes all 17 chapters individually
  • Text preprocessing — lowercase, punctuation removal, stopword filtering, lemmatization
  • NLP analysis — unigram, bigram, and trigram frequency tables
  • Visualizations — bar charts, horizontal charts, standard and shaped word clouds (circle, ellipse, star, human)
  • Exported outputs — CSV files and PNG charts per chapter
  • Interactive Streamlit dashboard — select full book or any chapter, view tables and all visualizations

Libraries Used

Library Purpose
pypdf Extract text from the PDF
nltk Tokenization, stopword removal, lemmatization
pandas Tabular data handling and CSV export
matplotlib Chart generation
wordcloud Word cloud images
numpy Mask arrays for shaped word clouds
Pillow (PIL) Image processing
streamlit Interactive dashboard
collections.Counter Word and phrase frequency counting
re Regular-expression text cleaning

Project Structure

04-32-rl-book-word-frequency-analyzer/
├── app.py                    ← Streamlit dashboard entry point
├── src/
│   ├── phase_1.py            ← PDF text extraction
│   ├── phase_2.py            ← Text preprocessing & NLP
│   ├── phase_3.py            ← Chapter-wise analysis
│   ├── phase_4.py            ← Visualization generation
│   └── phase_5.py            ← Orchestrator (run all phases)
├── outputs/                  ← Generated CSV files
├── figures/                  ← Generated charts and word clouds
├── shapes/
│   ├── star.jpg
│   └── human.jpg
├── requirements.txt
└── .gitignore

How to Run

# 1. Clone and enter the folder
git clone <repository-url>
cd 04-32-rl-book-word-frequency-analyzer

# 2. Create and activate a virtual environment
python -m venv .venv
# Windows:
.venv\Scripts\Activate.ps1
# macOS / Linux:
source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Download NLTK data (first time only)
python -c "import nltk; nltk.download('stopwords'); nltk.download('wordnet')"

# 5. Place the PDF in the project root
#    Expected filename: RL_BartoSutton.pdf

# 6. Run the full analysis pipeline
python src/phase_5.py

# 7. Launch the dashboard
streamlit run app.py

Sample Outputs

Output type Location
Word / bigram / trigram CSV tables outputs/
Top-20 frequency bar charts figures/
Standard and shaped word clouds figures/
Interactive Streamlit dashboard localhost:8501

Possible Future Improvements

  • Automatic chapter detection directly from PDF structure
  • POS-tag-aware lemmatization
  • Sentiment analysis per chapter
  • Named Entity Recognition (NER)
  • Topic modelling (LDA)
  • Interactive Plotly charts
  • Multi-book comparison
  • Direct PDF upload from the dashboard
  • AI-generated chapter summaries
  • Export dashboard report as PDF