Installing Selenium on a Mac to get started with Web Scraping

Installing Selenium on a Mac to get started with Web Scraping

In this tutorial, I will guide you through the process of installing Selenium on a Mac and using it in Python to search for smartwatches on Amazon.

🤖
What is Selenium?

Selenium is a powerful tool used for web scraping and browser automation. It's like a robot that can interact with websites, extract information, and perform actions just like a human would.

Selenium allows you to automate browser actions, such as clicking buttons, filling out forms, and navigating between pages. This makes it an excellent tool for tasks like data extraction (web scraping), web testing, and automating repetitive tasks on websites.

Tutorial

Install Chrome & Chromedriver

To get started with Web Scraping on Mac OS, we need to install Google Chrome and Chromedriver. You can do this by running the following Bash-Script in your terminal.

🤖
What is Chromedriver?

Chromedriver is a tool that helps you automate and control Google Chrome for tasks like web scraping and testing.

1) Copy Bash Script

First, copy the following code and create a new file called: mac_update_chromedriver_and_chrome.sh

👑
Direct Download
Registered members can directly download the Bash-Script-File for MacOS or Linux at the bottom of this blog post.
#!/bin/bash

# Check if Homebrew is installed, install if not
if ! command -v brew &> /dev/null; then
    echo "Homebrew not found. Installing Homebrew..."
    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
fi

# Update Homebrew
brew update

# Install Google Chrome
if ! command -v google-chrome &> /dev/null; then
    echo "Google Chrome not found. Installing Google Chrome..."
    brew install --cask google-chrome
else
    echo "Google Chrome already installed. Skipping..."
fi

# Install Chromedriver
if ! command -v chromedriver &> /dev/null; then
    echo "Chromedriver not found. Installing Chromedriver..."
    brew install chromedriver
else
    echo "Chromedriver already installed. Skipping..."
fi

echo "Installation complete!"
Save this script in a file called: mac_update_chromedriver_and_chrome.sh

2) Run Bash Script

Now run your Bash-Script it from your Terminal app on Mac OS.

  1. Open the Terminal App and navigate to the directory where the script is located using the "cd" command
cd /Users/nick/Desktop

2. Make the script executable by running the command:

chmod +x update_chromedriver_and_chrome.sh

What does the 'chmod' command do?

chmod is a command in MacOs and Linux that stands for "change mode". It is used to change the permissions of files and directories.

The chmod command allows you to set permissions for three different types of users: the owner of the file, members of the group that the file belongs to, and all other users.

The permissions that can be set using chmod are read (r), write (w), and execute (x). These permissions can be set or removed for each of the three types of users.

For example, the command "chmod +x myscript.sh" would add the execute permission to the file "myscript.sh", allowing the user to run the script.

3. Run the script by typing the command:

./update_chromedriver_and_chrome.sh

When running the script for the first time, you may encounter the following error message:

If you encounter this error, navigate to:

"System Preferences" --> "Security & Privacy"

and click "Allow Anyway"

Now run your bash script again using the command:

./update_chromedriver_and_chrome.sh

3) Check if Installation was Successful

Run the following code in your terminal to check if the installation of Chrome & Chromedriver was successful:

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --version && chromedriver --version 

4) Install Selenium in Python Environment

First, activate your Python-Environment. Replace 'nicks-env' with your own python environment you'd like to use:

source venv/nicks-env/bin/activate

And install selenium using

pip install selenium

5) Run a little test script

This script opens Chrome, goes to amazon.com and searches for 'smartwatch'.

Note that the script uses the 'time' library, so you might to 'pip install time' first.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import time

print("start")
driver = webdriver.Chrome()

url = 'https://www.amazon.com'

driver.get(url)

time.sleep(1)

driver.find_element(by=By.XPATH, value='//*[@id="twotabsearchtextbox"]').send_keys('smartwatch')

time.sleep(1)

driver.find_element(by=By.XPATH, value='//*[@id="nav-search-submit-button"]').click()

time.sleep(5)

driver.close()

print("end")

If you see 'start' and 'end' printed in the console, everything worked as expected.

Additional Resources

Subscribe to this Blog and my Free Newsletter to get Additional Resources

  • For Mac: File Update Chromedriver and Chrome
  • For Linux: File Update Chromedriver and Chrome