92 lines
4.6 KiB
Markdown
92 lines
4.6 KiB
Markdown
# Google Scholar Scraper & Quality Filtering Dashboard (v1.0.2)
|
|
|
|
An enterprise-grade, web-based dashboard and automation engine to search, scrape, and filter Google Scholar articles. It features robust integration with the **Tor Network** for self-healing SOCKS5 proxy rotation, automatically bypassing Google's anti-bot blocks and pagination limits.
|
|
|
|
## 🚀 Key Features
|
|
|
|
* **Automated PDF Downloader Pipeline**: Downloads PDFs in 2 stages:
|
|
* **Direct Google Scholar Links**: Sourced directly from `eprint_url` if indexed.
|
|
* **Sci-Hub Fallback**: Automatically queries the `sci-hub.sidesgame.com` mirror using DOIs to scrape and download the PDF.
|
|
* **Query-Specific Run Directories**: Automatically structures output runs into `Data Folder/scrape_{query}_{timestamp}/` containing the CSV file and all downloaded PDF files.
|
|
* **Safe Filename Sanitization**: Automatically renames downloaded PDFs using cleaned paper titles. All special symbols (`.`, `-`, `:`, `/`, etc.) are stripped and replaced with underscores (`_`) to ensure perfect OS compatibility.
|
|
* **Hybrid Links Layout**: The preview page links column renders only the `PAGE` button and a toggle: either the green `PDF` button (if downloaded successfully) or a grey `RG SEARCH` button (pointing to a Google search on ResearchGate for manual human verification and 1-click download).
|
|
* **Advanced Query Builder**: Build complex boolean queries (supporting `MUST`, `AND`, `OR` operators) directly from the visual interface.
|
|
|
|
---
|
|
|
|
## 🛠️ System Architecture
|
|
|
|
```mermaid
|
|
graph TD
|
|
A[Web UI Dashboard] -->|REST API & SSE /stream| B[Flask Backend]
|
|
B -->|Threaded Runner| C[Scraper Engine]
|
|
C -->|Construct Query| D[scholarly API]
|
|
D -->|SOCKS5 Proxy 9150| E[Tor Browser Network]
|
|
E -->|Fetch Page| F[Google Scholar]
|
|
F -->|Soft-Block / Captcha| C
|
|
C -->|NEWNYM Control Signal 9151| E
|
|
C -->|Lookup metadata| G[SCImago Local Database]
|
|
C -->|Fetch DOIs & Abstracts| H[Crossref / Semantic Scholar APIs]
|
|
C -->|Save clean results| I[CSV Export]
|
|
```
|
|
|
|
---
|
|
|
|
## 📋 Installation & Setup
|
|
|
|
### Prerequisites
|
|
1. **Python 3.10+** (Ensure it is added to your environment `PATH`).
|
|
2. **Tor Browser**: Must be installed and running in the background to handle proxy routing and circuit rotation.
|
|
|
|
### Installation Steps
|
|
|
|
1. **Clone the repository** (or navigate to the project directory):
|
|
```bash
|
|
cd "Google Scholar Data"
|
|
```
|
|
|
|
2. **Verify or set up the Virtual Environment**:
|
|
If not already done, create a virtual environment:
|
|
```bash
|
|
python -m venv .venv
|
|
```
|
|
|
|
3. **Install Locked Dependencies**:
|
|
Install all required packages precisely using the pinned `requirements.txt`:
|
|
```bash
|
|
.\.venv\Scripts\pip install -r requirements.txt
|
|
```
|
|
|
|
---
|
|
|
|
## 🚦 How to Run
|
|
|
|
1. **Launch Tor Browser**:
|
|
* Open the Tor Browser on your system.
|
|
* Keep it running in the background. (This opens the SOCKS5 proxy on port `9150` and the control port on `9151`).
|
|
|
|
2. **Start the Scraper Dashboard**:
|
|
Double-click the **`Start Scraper.bat`** file in the root directory. This script will:
|
|
* Kill any ghost servers on port `5000`.
|
|
* Start the Flask backend in a minimized background window.
|
|
* Automatically launch `http://127.0.0.1:5000` in your default browser.
|
|
|
|
3. **Configure and Run**:
|
|
* Enter your search keywords in the configuration panel.
|
|
* Set results limits, year ranges, or quality criteria.
|
|
* Click **▶ RUN**. Watch the live terminal extract and verify papers in real-time.
|
|
|
|
---
|
|
|
|
## 📁 Repository Structure
|
|
|
|
* `Code File/` - Contains the Flask backend and core scraping scripts:
|
|
* [app.py](file:///d:/Website/Life%20OS/Oppurtunity/Google%20Scholar%20Data/Code%20File/app.py) - Web server routes, SSE streaming handler, and UI controls.
|
|
* [scholar_scraper.py](file:///d:/Website/Life%20OS/Oppurtunity/Google%20Scholar%20Data/Code%20File/scholar_scraper.py) - Core scraping logic, SCImago database matching, Tor IP rotation, and Crossref API metadata pipeline.
|
|
* `templates/` - Frontend views ([index.html](file:///d:/Website/Life%20OS/Oppurtunity/Google%20Scholar%20Data/Code%20File/templates/index.html) and [data_view.html](file:///d:/Website/Life%20OS/Oppurtunity/Google%20Scholar%20Data/Code%20File/templates/data_view.html)).
|
|
* `static/` - UI CSS styles and client-side JavaScript controllers.
|
|
* `Data Folder/` - Stores all downloaded and scraped CSV output results.
|
|
* `Assets/` - Local JSON metadata lists (e.g. `scimagojr_2025.json`).
|
|
* `requirements.txt` - Python package pin definitions ensuring system compatibility.
|
|
* `Start Scraper.bat` - Windows launcher script.
|