This commit is contained in:
ytshen 2023-10-19 21:35:29 +08:00
commit 92941b99ce
49 changed files with 1093 additions and 0 deletions

31
.github/workflows/main.yml vendored Normal file
View File

@ -0,0 +1,31 @@
name: Zotero Addons Scraping
on:
workflow_dispatch: # 手动触发事件
schedule:
- cron: '0 3 * * *' # 每天早上3点执行
jobs:
scrape_data:
runs-on: ubuntu-latest
steps:
- name: Checkout Repository
uses: actions/checkout@master
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.x
- name: Install Dependencies
run: |
pip install requests
pip install argparse
- name: Run Script
run: |
python main.py -i addons -o addon_infos.json --github_repository ${GITHUB_REPOSITORY} --github_token ${GITHUB_TOKEN}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

159
.gitignore vendored Normal file
View File

@ -0,0 +1,159 @@
addon_infos.json
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
.idea/

21
LICENSE Normal file
View File

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2023 syt
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

80
README.md Normal file
View File

@ -0,0 +1,80 @@
# Zotero Addons Scraper
This is a script repository for scraping Zotero addon collections for [Zotero7+](https://www.zotero.org), intended for use with [Zotero addons](https://github.com/syt2/zotero-addons).
*Switch to GitHub(backup) source in [Zotero addons](https://github.com/syt2/zotero-addons) to use this repository.*
The script utilizes the GitHub API and GitHub Actions to automatically scrape, parse, and publish corresponding addon information from the [addons](addons) file to the latest release.
## Contributing New Addons
If you have a new add-on to parse, add the json information of the add-on in the [addons](addons) folder with the format
``` json
{
"id": "zoteroAddons@ytshen.com",
"name": "Zotero 插件合集",
"repo": "syt2/zotero-addons",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
}
]
}
```
## Addon Information Format
The output format for add-on information follows the format specified in [zotero-chinese/zotero-plugins](https://github.com/zotero-chinese/zotero-plugins), which is as follows:
```ts
export interface PluginInfo {
/**
* 插件名称
*/
name: string;
/**
* 插件仓库
*
* 例如:northword/zotero-format-metadata
*
* 注意前后均无 `/`
*/
repo: string;
/**
* 插件的发布地址信息
*/
releases: Array<{
/**
* 当前发布版对应的 Zotero 版本
*/
targetZoteroVersion: string;
/**
* 当前发布版对应的下载通道
*
* `latest`:最新正式发布;
* `pre`:最新预发布;
* `string`:发布对应的 `git.tag_name`;
* 注意 `git.tag_name` 有的有 `v` 而有的没有,可以通过发布链接来判断
*/
tagName: "latest" | "pre" | string;
currentVersion?: string;
xpiDownloadUrl?: {
github: string;
gitee: string;
ghProxy: string;
jsdeliver: string;
kgithub: string;
};
releaseData?: string;
downloadCount?: number;
assetId?: number;
}>;
description?: string;
star?: number;
author?: {
name: string;
url: string;
avatar: string;
};
}
```

76
addon_keys.py Normal file
View File

@ -0,0 +1,76 @@
id = "id"
name = "name"
repo = "repo"
releases = "releases"
targetZoteroVersion = "targetZoteroVersion"
tagName = "tagName"
currentVersion = "currentVersion"
xpiDownloadUrl = "xpiDownloadUrl"
github = "github"
gitee = "gitee"
ghProxy = "ghProxy"
jsdeliver = "jsdeliver"
kgithub = "kgithub"
releaseData = "releaseData"
downloadCount = "downloadCount"
assetId = "assetId"
description = "description"
star = "star"
author = "author"
url = "url"
avatar = "avatar"
# from https://github.com/zotero-chinese/zotero-plugins/blob/main/src/plugins.ts
# export interface PluginInfo {
# /**
# * 插件名称
# */
# name: string;
# /**
# * 插件仓库
# *
# * 例如:northword/zotero-format-metadata
# *
# * 注意前后均无 `/`
# */
# repo: string;
# /**
# * 插件的发布地址信息
# */
# releases: Array<{
# /**
# * 当前发布版对应的 Zotero 版本
# */
# targetZoteroVersion: string;
# /**
# * 当前发布版对应的下载通道
# *
# * `latest`:最新正式发布;
# * `pre`:最新预发布;
# * `string`:发布对应的 `git.tag_name`;
# * 注意 `git.tag_name` 有的有 `v` 而有的没有,可以通过发布链接来判断
# */
# tagName: "latest" | "pre" | string;
#
# currentVersion?: string;
# xpiDownloadUrl?: {
# github: string;
# gitee: string;
# ghProxy: string;
# jsdeliver: string;
# kgithub: string;
# };
# releaseData?: string;
# downloadCount?: number;
# assetId?: number;
# }>;
#
# description?: string;
# star?: number;
# author?: {
# name: string;
# url: string;
# avatar: string;
# };
# }

15
addons/Chartero.json Normal file
View File

@ -0,0 +1,15 @@
{
"id": "chartero@volatile.static",
"name": "Chartero",
"repo": "volatile-static/Chartero",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "1.3.3"
},
{
"targetZoteroVersion": "7",
"tagName": "latest"
}
]
}

10
addons/KeepZotero.json Normal file
View File

@ -0,0 +1,10 @@
{
"name": "KeepZotero",
"repo": "yhmtsai/KeepZotero",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

11
addons/Zotero-TLDR.json Normal file
View File

@ -0,0 +1,11 @@
{
"id": "zoterotldr@syt.com",
"name": "Zotero TL;DR",
"repo": "syt2/Zotero-TLDR",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
}
]
}

15
addons/ZoteroStyle.json Normal file
View File

@ -0,0 +1,15 @@
{
"id": "zoterostyle@polygon.org",
"name": "Zotero Style",
"repo": "MuiseDestiny/ZoteroStyle",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "pre"
},
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

10
addons/Zutilo.json Normal file
View File

@ -0,0 +1,10 @@
{
"name": "Zutilo Utility for Zotero",
"repo": "wshanks/Zutilo",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,15 @@
{
"id": "delitemwithatt@redleaf.me",
"name": "Delitemwithatt",
"repo": "redleafnew/delitemwithatt",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
},
{
"targetZoteroVersion": "6",
"tagName": "0.1.06"
}
]
}

View File

@ -0,0 +1,15 @@
{
"id": "zoterocitation@polygon.org",
"name": "Zotero Citation",
"repo": "MuiseDestiny/eaiser-citation",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
},
{
"targetZoteroVersion": "6",
"tagName": "0.3.1"
}
]
}

15
addons/jasminum.json Normal file
View File

@ -0,0 +1,15 @@
{
"id": "jasminum@linxzh.com",
"name": "Jasminum",
"repo": "l0o0/jasminum",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "pre"
},
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

10
addons/lyz.json Normal file
View File

@ -0,0 +1,10 @@
{
"name": "LyZ",
"repo": "wshanks/lyz",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

15
addons/notero.json Normal file
View File

@ -0,0 +1,15 @@
{
"id": "notero@vanoni.dev",
"name": "Notero",
"repo": "dvanoni/notero",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
},
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,10 @@
{
"name": "scite Plugin for Zotero",
"repo": "scitedotai/scite-zotero-plugin",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

10
addons/tara.json Normal file
View File

@ -0,0 +1,10 @@
{
"name": "Tara",
"repo": "l0o0/tara",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

10
addons/zotcard.json Normal file
View File

@ -0,0 +1,10 @@
{
"name": "ZotCard",
"repo": "018/zotcard",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,15 @@
{
"id": "zoterotag@euclpts.com",
"name": "Zotero Tag",
"repo": "windingwind/zotero-actions-tags",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "pre"
},
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

11
addons/zotero-addons.json Normal file
View File

@ -0,0 +1,11 @@
{
"id": "zoteroAddons@ytshen.com",
"name": "Zotero 插件合集",
"repo": "syt2/zotero-addons",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,11 @@
{
"id": "betterauthors@github.com",
"name": "Zotero Better Authors",
"repo": "github-young/zotero-better-authors",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,15 @@
{
"id": "better-bibtex@iris-advies.com",
"name": "Better BibTex for Zotero",
"repo": "retorquere/zotero-better-bibtex",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
},
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,15 @@
{
"id": "Knowledge4Zotero@windingwind.com",
"name": "Zotero Better Notes",
"repo": "windingwind/zotero-better-notes",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "pre"
},
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,10 @@
{
"name": "Zotero Citation Counts Manager",
"repo": "eschnett/zotero-citationcounts",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

15
addons/zotero-figure.json Normal file
View File

@ -0,0 +1,15 @@
{
"id": "zoterofigure@polygon.org",
"name": "PDF Figure",
"repo": "MuiseDestiny/zotero-figure",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
},
{
"targetZoteroVersion": "6",
"tagName": "0.0.7"
}
]
}

11
addons/zotero-file.json Normal file
View File

@ -0,0 +1,11 @@
{
"id": "zoterofile@polygon.org",
"name": "Zotero File",
"repo": "MuiseDestiny/zotero-file",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,15 @@
{
"id": "zotero-format-metadata@northword.cn",
"name": "Zotero Format Metadata",
"repo": "northword/zotero-format-metadata",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
},
{
"targetZoteroVersion": "6",
"tagName": "0.4.4"
}
]
}

15
addons/zotero-gpt.json Normal file
View File

@ -0,0 +1,15 @@
{
"id": "zoterogpt@polygon.org",
"name": "Zotero GPT",
"repo": "MuiseDestiny/zotero-gpt",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "pre"
},
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,10 @@
{
"name": "Zotero Inspire",
"repo": "fkguo/zotero-inspire",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,10 @@
{
"name": "Mdnotes for Zotero",
"repo": "argenos/zotero-mdnotes",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

15
addons/zotero-night.json Normal file
View File

@ -0,0 +1,15 @@
{
"id": "night@tefkah.com",
"name": "Night for Zotero",
"repo": "tefkah/zotero-night",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "pre"
},
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,10 @@
{
"name": "MarkDBConnect",
"repo": "daeh/zotero-obsidian-citations",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

10
addons/zotero-ocr.json Normal file
View File

@ -0,0 +1,10 @@
{
"name": "Zotero OCR",
"repo": "UB-Mannheim/zotero-ocr",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,15 @@
{
"id": "open-pdf@iris-advies.com",
"name": "Open PDF",
"repo": "retorquere/zotero-open-pdf",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
},
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,15 @@
{
"id": "pdfpreview@windingwind.com",
"name": "Zotero PDF Preview",
"repo": "windingwind/zotero-pdf-preview",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
},
{
"targetZoteroVersion": "7",
"tagName": "pre"
}
]
}

View File

@ -0,0 +1,15 @@
{
"id": "zoteropdftranslate@euclpts.com",
"name": "Zotero PDF Translate",
"repo": "windingwind/zotero-pdf-translate",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "pre"
},
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,10 @@
{
"name": "Reading List for Zotero",
"repo": "Dominic-DallOsto/zotero-reading-list",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,15 @@
{
"id": "zoteroreference@polygon.org",
"name": "Zotero Reference",
"repo": "MuiseDestiny/zotero-reference",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "pre"
},
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

10
addons/zotero-scihub.json Normal file
View File

@ -0,0 +1,10 @@
{
"name": "Sci-Hub Plugin for Zotero",
"repo": "ethanwillis/zotero-scihub",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,10 @@
{
"name": "Zotero ShortDOI",
"repo": "bwiernik/zotero-shortdoi",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,10 @@
{
"name": "Storage Scanner for Zotero",
"repo": "retorquere/zotero-storage-scanner",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,10 @@
{
"name": "Zotero 更新影响因子",
"repo": "redleafnew/zotero-updateifs",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,15 @@
{
"id": "greenfrog@redleafnew.me",
"name": "Green Frog",
"repo": "redleafnew/zotero-updateifsE",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
},
{
"targetZoteroVersion": "6",
"tagName": "0.13.0"
}
]
}

10
addons/zotfile.json Normal file
View File

@ -0,0 +1,10 @@
{
"name": "ZotFile 汉化版",
"repo": "lychichem/zotfile",
"releases": [
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

197
main.py Normal file
View File

@ -0,0 +1,197 @@
import concurrent.futures
import json
import os
import time
import urllib.parse
import requests
import argparse
from addon_keys import *
# 输出格式参考 [zotero-chinese/zotero-plugins](https://github.com/zotero-chinese/zotero-plugins)
def parse(plugin, **kwargs):
if repo not in plugin or releases not in plugin or len(plugin[releases]) <= 0:
return
if name not in plugin:
plugin[name] = plugin[repo].split('/')[-1]
headers = {}
if github_token := kwargs.get('github_token'):
headers['Authorization'] = f'token {github_token}'
# fetch author info
author_url = f"https://api.github.com/users/{plugin[repo].split('/')[0]}"
try:
author_resp = requests.get(author_url, headers=headers)
author_resp_info = json.loads(author_resp.content)
plugin_author = {}
if 'name' in author_resp_info:
plugin_author[author] = author_resp_info['name']
if 'html_url' in author_resp_info:
plugin_author[url] = author_resp_info['html_url']
if 'avatar_url' in author_resp_info:
plugin_author[avatar] = author_resp_info['avatar_url']
if plugin_author.keys():
plugin[author] = plugin_author
except Exception as e:
print(f'request {author_url} failed: {e}')
# fetch repo info
repo_url = f'https://api.github.com/repos/{plugin[repo]}'
try:
repos_resp = requests.get(repo_url, headers=headers)
repos_info = json.loads(repos_resp.content)
if repos_info['description'] and description not in plugin:
plugin[description] = repos_info['description']
if repos_info['stargazers_count'] is not None:
plugin[star] = repos_info['stargazers_count']
except Exception as e:
print(f'request {repo_url} failed: {e}')
# fetch release info
release_infos = []
for release in plugin[releases]:
if tagName not in release:
continue
release_url = f'https://api.github.com/repos/{plugin[repo]}/releases'
if release[tagName] == 'latest':
release_url += '/latest'
elif release[tagName] != 'pre':
release_url += f'/tags/{release[tagName]}'
try:
release_resp = requests.get(release_url, headers=headers)
release_info = json.loads(release_resp.content)
if release[tagName] == 'pre':
release_info = [info for info in release_info if info['prerelease']]
if release_info:
release_info = release_info[0]
else:
continue
if 'tag_name' in release_info:
release[currentVersion] = release_info['tag_name']
if 'assets' not in release_info:
continue
release_assets = release_info['assets']
release_assets.sort(key=lambda item: item['updated_at'] if 'updated_at' in item else '', reverse=True)
release_assets = [asset for asset in release_assets if asset['content_type'] == 'application/x-xpinstall']
if not release_assets:
continue
release_asset = release_assets[0]
if 'browser_download_url' not in release_asset:
continue
if 'id' in release_asset:
release[assetId] = release_asset['id']
if 'download_count' in release_asset:
release[downloadCount] = release_asset['download_count']
release[releaseData] = release_asset['updated_at']
release[xpiDownloadUrl] = {
'github': release_asset['browser_download_url'],
'ghProxy': 'https://ghproxy.com/?q=' + urllib.parse.quote(release_asset['browser_download_url']),
'kgithub': release_asset['browser_download_url'].replace('github.com', 'kgithub.com'),
}
release_infos.append(release)
except Exception as e:
print(f'request {release_url} failed: {e}')
plugin[releases] = release_infos
return plugin
def parse_addon_infos(input_dir, output_filepath, **kwargs):
plugins = []
for addon_json_filename in os.listdir(input_dir):
if not addon_json_filename.endswith('.json'):
continue
addon_json_filepath = os.path.join(input_dir, addon_json_filename)
with open(addon_json_filepath, 'r') as file:
plugins.append(json.load(file))
addon_infos = []
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = [executor.submit(parse, plugin, github_token=kwargs.get('github_token')) for plugin in plugins]
for future in concurrent.futures.as_completed(futures):
if addon_info := future.result():
addon_infos.append(addon_info)
addon_infos.sort(key=lambda item: item[star] if star in item else 0, reverse=True)
with open(output_filepath, "w") as json_file:
json.dump(addon_infos, json_file, ensure_ascii=False)
return addon_infos
def upload_json_to_release(github_repository, release_id, upload_file_name, upload_file, **kwargs):
headers = {
'Accept': 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
"Content-Type": "application/octet-stream",
}
if github_token := kwargs.get('github_token'):
headers['Authorization'] = f'token {github_token}'
upload_url = f'https://uploads.github.com/repos/{github_repository}/releases/{release_id}/assets?name={upload_file_name}'
try:
with open(upload_file, "rb") as file:
upload_resp = requests.post(upload_url, data=file, headers=headers)
if upload_resp.status_code != 201:
print(f'upload release assets code: {upload_resp.status_code}')
else:
print('upload release assets succeed')
except Exception as e:
print(f'upload release assets failed: {e}')
def create_release(github_repository, **kwargs):
headers = {
'Accept': 'application/vnd.github+json',
'X-GitHub-Api-Version': '2022-11-28',
}
if github_token := kwargs.get('github_token'):
headers['Authorization'] = f'token {github_token}'
create_release_url = f'https://api.github.com/repos/{github_repository}/releases'
cur_time = int(time.time())
param = {
'tag_name': f'{cur_time}',
'target_commitish': 'master',
'name': f'{cur_time}',
'body': 'publish addon_infos.json',
'draft': False,
'prerelease': False,
'generate_release_notes': False,
}
try:
create_resp = requests.post(create_release_url, json=param, headers=headers)
if create_resp.status_code == 201:
create_release_info = json.loads(create_resp.content)
release_id = create_release_info['id']
return release_id
else:
print(f'create release code: {create_resp.status_code}')
except Exception as e:
print(f'create release failed: {e}')
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='params')
parser.add_argument('--github_repository', nargs='?', type=str, required=True, help='github repository')
parser.add_argument('--github_token', nargs='?', type=str, help='github token')
parser.add_argument('-i', '--input', nargs='?', type=str, default="addons", help='input addon dir')
parser.add_argument('-o', '--output', nargs='?', type=str, default="addon_infos.json", help='output addon filepath')
args = parser.parse_args()
if not args.github_repository:
raise 'Need specific github repository'
parse_addon_infos(args.input, args.output, github_token=args.github_token)
if release_id := create_release(args.github_repository, github_token=args.github_token):
upload_json_to_release(args.github_repository,
release_id,
upload_file_name='addon_infos.json',
upload_file=args.output,
github_token=args.github_token)

4
testAddons/README.md Normal file
View File

@ -0,0 +1,4 @@
# Zotero Addons Scraper
Just for test.
Do not modify this folder.

View File

@ -0,0 +1,15 @@
{
"id": "zoterostyle@polygon.org",
"name": "Zotero Style",
"repo": "MuiseDestiny/ZoteroStyle",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "pre"
},
{
"targetZoteroVersion": "6",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,11 @@
{
"id": "zoteroAddons@ytshen.com",
"name": "Zotero 插件合集",
"repo": "syt2/zotero-addons",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
}
]
}

View File

@ -0,0 +1,15 @@
{
"id": "zotero-format-metadata@northword.cn",
"name": "Zotero Format Metadata",
"repo": "northword/zotero-format-metadata",
"releases": [
{
"targetZoteroVersion": "7",
"tagName": "latest"
},
{
"targetZoteroVersion": "6",
"tagName": "0.4.4"
}
]
}