JosephFerano/auto-scraper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make output dir if it doesn't exist

Browse Source
This commit is contained in:
Joseph Ferano 2025-07-31 21:26:31 +07:00
parent 3a033c01f2
commit 0e581981d1
2 changed files with 10 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
README.org
View File

@ -1,6 +1,6 @@
#+OPTIONS: toc:nil #+OPTIONS: toc:nil
* SearchApi Automated Browser * Automated Browser
** Overview ** Overview
@ -21,12 +21,18 @@ python orchestrator.py <website> [options]
Make sure to have =docker= installed. For python best practices, create a virtual Make sure to have =docker= installed. For python best practices, create a virtual
environment and install dependencies: environment and install dependencies:
#+begin_src bash #+begin_src sh
python -m venv venv python -m venv venv
source venv/bin/activate source venv/bin/activate
pip install -r requirements.txt pip install -r requirements.txt
#+end_src #+end_src
Also build the container
#+begin_src sh
docker build -t search-api .
#+end_src
*** Arguments *** Arguments
- =website= (required): The website URL you want to scrape - =website= (required): The website URL you want to scrape

6
orchestrator.py
View File

@ -40,8 +40,8 @@ env_vars = {
output_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'output') output_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'output')
# os.makedirs(output_dir, exist_ok=True) os.makedirs(output_dir, exist_ok=True)
# os.chmod(output_dir, 0o777) os.chmod(output_dir, 0o777)
if args.browser_args: if args.browser_args:
env_vars['BROWSER_ARGS'] = ','.join(args.browser_args) env_vars['BROWSER_ARGS'] = ','.join(args.browser_args)
@ -51,7 +51,6 @@ if args.proxy_url:
try: try:
container_start_time = time.time() container_start_time = time.time()
print("Container start time:", container_start_time)
container = docker_client.containers.run( container = docker_client.containers.run(
args.image_name, args.image_name,
@ -143,7 +142,6 @@ try:
subprocess.run(['open', 'output/report.html']) subprocess.run(['open', 'output/report.html'])
print("Report opened in browser") print("Report opened in browser")
except (subprocess.CalledProcessError, FileNotFoundError): except (subprocess.CalledProcessError, FileNotFoundError):
print("what happened")
pass pass