Google Scholar Scraper & Quality Filtering Dashboard

An enterprise-grade, web-based dashboard and automation engine to search, scrape, and filter Google Scholar articles. It features robust integration with the Tor Network for self-healing SOCKS5 proxy rotation, automatically bypassing Google's anti-bot blocks and pagination limits.

🚀 Key Features

Advanced Query Builder: Build complex boolean queries (supporting MUST, AND, OR operators) directly from the visual interface.
Tor Circuit Rotation: Automatically rotates Tor exit nodes using the Tor control port (9151) on detecting soft-blocks or captcha pages.
Resilient Pagination: Syncs search offset states (start_index) to resume scraping seamlessly exactly where it was interrupted after an IP rotation.
Duplicate Elimination: Automatically filters overlapping search results in real-time, ensuring unique outputs in final CSV records.
Academic Quality Filters:
- Filter by SCImago Journal Rank (SJR), Quartile rankings (Q1 to Q4), and H-index via local database matching.
- Filter by minimum citation counts.
- Filter by Open Access status.
- Whitelist specific journals or publishers.
Live Output Terminal: Simulates a live CLI output stream directly in the web browser using Server-Sent Events (SSE).
Live Data Table: Sortable tabular view of scraped papers with interactive abstract hover tooltips.

🛠️ System Architecture

graph TD
    A[Web UI Dashboard] -->|REST API & SSE /stream| B[Flask Backend]
    B -->|Threaded Runner| C[Scraper Engine]
    C -->|Construct Query| D[scholarly API]
    D -->|SOCKS5 Proxy 9150| E[Tor Browser Network]
    E -->|Fetch Page| F[Google Scholar]
    F -->|Soft-Block / Captcha| C
    C -->|NEWNYM Control Signal 9151| E
    C -->|Lookup metadata| G[SCImago Local Database]
    C -->|Fetch DOIs & Abstracts| H[Crossref / Semantic Scholar APIs]
    C -->|Save clean results| I[CSV Export]

📋 Installation & Setup

Prerequisites

Python 3.10+ (Ensure it is added to your environment PATH).
Tor Browser: Must be installed and running in the background to handle proxy routing and circuit rotation.

Installation Steps

Clone the repository (or navigate to the project directory):
```
cd "Google Scholar Data"
```
Verify or set up the Virtual Environment: If not already done, create a virtual environment:
```
python -m venv .venv
```
Install Locked Dependencies: Install all required packages precisely using the pinned requirements.txt:
```
.\.venv\Scripts\pip install -r requirements.txt
```

🚦 How to Run

Launch Tor Browser:
- Open the Tor Browser on your system.
- Keep it running in the background. (This opens the SOCKS5 proxy on port 9150 and the control port on 9151).
Start the Scraper Dashboard: Double-click the Start Scraper.bat file in the root directory. This script will:
- Kill any ghost servers on port 5000.
- Start the Flask backend in a minimized background window.
- Automatically launch http://127.0.0.1:5000 in your default browser.
Configure and Run:
- Enter your search keywords in the configuration panel.
- Set results limits, year ranges, or quality criteria.
- Click ▶ RUN. Watch the live terminal extract and verify papers in real-time.

📁 Repository Structure

Code File/ - Contains the Flask backend and core scraping scripts:
- app.py - Web server routes, SSE streaming handler, and UI controls.
- scholar_scraper.py - Core scraping logic, SCImago database matching, Tor IP rotation, and Crossref API metadata pipeline.
- templates/ - Frontend views (index.html and data_view.html).
- static/ - UI CSS styles and client-side JavaScript controllers.
Data Folder/ - Stores all downloaded and scraped CSV output results.
Assets/ - Local JSON metadata lists (e.g. scimagojr_2025.json).
requirements.txt - Python package pin definitions ensuring system compatibility.
Start Scraper.bat - Windows launcher script.

4.6 KiB Raw Blame History