Introduction to Web Scraping

Web Scraping is the process of automatically extracting data from websites. Instead of manually copying information, we write a script that "visits" the page, downloads its content (HTML), and parses it to find the data we are interested in. It is an extremely powerful technique for data collection.

Warning: Always check a website's `robots.txt` file (e.g., `https://www.example.com/robots.txt`) and its terms of use to ensure that scraping is allowed. Always be "polite" to servers by making requests at reasonable intervals.

1. Virtual Environment and Installation

Open your terminal and create a new environment for the project:

mkdir job_scraper
cd job_scraper
python -m venv venv
# Activation...
pip install requests beautifulsoup4

2. Initial Script

Create a file `job_scraper.py`. We will start by importing the libraries and defining our constants:

• `requests`: To make the HTTP request and download the page.
• `BeautifulSoup` (from `bs4`): To parse the HTML.
• `csv`: To write the data to a CSV file.
• `URL`: The address of the website we will scrape.
• `HEADERS`: It is good practice to send a `User-Agent` header to make our request look like that of a regular browser.

The first step is to request the content of the webpage from the server. We will use the `requests.get()` function. It is crucial to wrap this call in a `try...except` block to handle potential network problems (e.g., timeout, connection error).

The `response.raise_for_status()` method is very useful, as it automatically checks if the response was successful (status code 2xx) and raises an exception if there was an error (e.g., 404 Not Found, 500 Server Error).

Now that we have the HTML, we need to parse it to find the information we are interested in. This is where BeautifulSoup shines. We create a `BeautifulSoup` object, giving it the HTML content and a "parser" (usually `"html.parser"`).

How do we find the right elements?

This is the most crucial and often the most difficult part of web scraping. It requires "inspecting" the source code of the webpage:

1. Open the URL in your browser (Chrome, Firefox).
2. Right-click on a job posting and select "Inspect" or "Inspect Element".
3. In the window that opens, observe the HTML structure. Look for the tags (e.g., `
   `, `

   `) and classes (`class="..."`) that surround the elements you want (title, company).

For kariera.gr, we observe that each job posting is inside a `

# (Update scrape_jobs and the main block)

def scrape_jobs():
    try:
        response = requests.get(URL, headers=HEADERS, timeout=10)
        response.raise_for_status() 
    except requests.exceptions.RequestException as e:
        print(f"Error during HTTP request: {e}")
        return

    # Create a BeautifulSoup object to parse the HTML
    soup = BeautifulSoup(response.content, "html.parser")
    
    # Find the container that holds all the job listings
    job_cards = soup.find_all("div", class_="col-sm-12 col-md-12 col-lg-6 col-xl-4 p-2")
    
    if not job_cards:
        print("No job listings found. The website structure may have changed.")
        return
    
    print(f"Found {len(job_cards)} job cards.")
    # (Data extraction will be done in the next step)

if __name__ == "__main__":
    scrape_jobs()