Automated AI Job Search Agent: System Architecture & Setup Guide

Automating the job search process eliminates the daily manual effort of browsing job boards and evaluating posting compatibility. By leveraging Large Language Models (LLMs) and targeting Applicant Tracking System (ATS) endpoints directly, job seekers can receive a personalized daily digest containing hyper-relevant job postings complete with direct application links.

1. The Core Problem & Debunking Job Scraper Myths

Many automated job-seeking tools claim to scrape major job marketplaces like LinkedIn or Naukri. However, relying on scrapers built for these platforms presents major architectural limitations:

• Walled Gardens & Bot Protection: Marketplaces like LinkedIn and Naukri operate similarly to closed social networks. They employ aggressive anti-bot protection, rate limits, and captcha checks that block web crawlers and IP addresses.
• Redirection to ATS: Marketplaces rarely host the native application logic. Clicking "Apply On Company Site" on LinkedIn invariably redirects candidates to an Applicant Tracking System (ATS) platform (e.g., Greenhouse, Lever, Ashby).
• The ATS Advantage: ATS platforms host the actual, structured job descriptions and application forms. Scraping job endpoints directly at the ATS level yields clean, unblocked job listings and direct application links.

Note: Automated job discovery should supplement—not replace—your broader job search strategy. Continue networking, cold emailing, and applying through traditional channels alongside daily automated digests.

2. High-Level Architecture & End-to-End Workflow

The automated job search agent processes job listings through an automated, decoupled pipeline:

3. Environment Setup & Project Initialization

To set up the automated job engine locally, execute the setup sequence in your environment.

3.1 Project Structure Setup

1. Create a workspace directory on your desktop (e.g., job_demo).
2. Open the directory within Visual Studio Code or your preferred editor.
3. Open a terminal instance and clone the repository.

# Clone the automation project repository
git clone <REPOSITORY_URL>

# Navigate into the project root directory
cd <PROJECT_FOLDER_NAME>

3.2 Virtual Environment Setup

Isolate your Python project dependencies using a virtual environment (venv).

# Check Python installation
python --version

# Create virtual environment
python -m venv venv

# Activate virtual environment (Windows PowerShell)
.\venv\Scripts\Activate.ps1

# Activate virtual environment (macOS/Linux)
source venv/bin/activate

4. Dependency Verification & Base Testing

Once the virtual environment is active, install the required dependencies and run a mock setup check.

4.1 Dependency Installation

pip install -r requirements.txt

4.2 Resume Placement & Mock Verification

1. Copy your master resume (e.g., resume.pdf) into the project root folder.
2. Run the preliminary verification script to confirm Python environment integrity and library resolution:

python test_setup.py

Expected Output:

[INFO] Scanning resume dependencies...
[INFO] Dependencies satisfied. Mock scanning completed successfully.

5. API Configuration: Gemini LLM & Gmail SMTP Setup

The automation engine requires two external service integrations: an LLM Provider (Google Gemini API) to analyze and score jobs, and an SMTP Server (Gmail) to send daily email alerts.

5.1 Google Gemini API Key

1. Access Google AI Studio.
2. Click Create API Key.
3. Copy the generated string for your .env configuration file.

5.2 Gmail SMTP Credentials Setup

Standard account passwords will not work for programmatic email dispatch via Gmail. You must generate a 16-digit App Password:

1. Log into your Google Account and navigate to Security Settings.
2. Enable 2-Step Verification (if not already active).
3. Search for App Passwords.
4. Create a new App Password named Job Automation Engine.
5. Save the generated 16-digit passkey.

5.3 Setting Up .env Environment File

Create a file named .env in your project root directory and populate it with your credentials:

# LLM Configuration
LLM_PROVIDER=gemini
GEMINI_API_KEY=your_gemini_api_key_here

# SMTP Configuration
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=your_email@gmail.com
MAIL_TO=your_email@gmail.com
SMTP_PASSWORD=your_16_digit_app_password

6. Target Company Discovery via ChatGPT Prompting

Rather than manually searching for target company ATS endpoints, use the following prompt with ChatGPT/Claude along with your resume to generate a list of target company URLs.

Target Company Extraction Prompt

I have uploaded my resume. I want to target software engineering / AI roles at companies that host their career boards on major Applicant Tracking Systems (Greenhouse, Lever, or Ashby).

Please perform the following:
1. Analyze my resume experience and skill set.
2. Identify 15 to 20 relevant tech companies matching my profile.
3. Verify if their job boards are hosted on Greenhouse, Lever, or Ashby.
4. Format the output strictly as python strings/list formatted for direct entry into a company configuration file matching the system syntax:

Example format:
"greenhouse/stripe",
"lever/datadog",
"ashby/linear"

Paste the resulting company paths directly into your project's companies.py or companies.json file.

7. Granular Filtering & System Configuration

Configure your application filter preferences inside config.py (or config.json). This file controls what jobs are included, excluded, or flagged based on match quality.

7.1 Configuration Structure Example

{
  "target_titles": [
    "Backend Engineer",
    "Software Development Engineer",
    "Server Engineer",
    "Infrastructure Engineer",
    "Site Reliability Engineer",
    "Member of Technical Staff"
  ],
  "excluded_titles": [
    "Director",
    "Vice President",
    "Intern",
    "Apprentice",
    "Frontend",
    "Mobile"
  ],
  "locations": [
    "Bangalore",
    "India"
  ],
  "allow_remote": true,
  "max_age_days": 30,
  "score_threshold": 7
}

7.2 Configuration Key Breakdown

8. State Management & Duplicate Prevention

To prevent sending duplicate alerts for jobs you have already reviewed, the engine maintains a state cache file named seen.json.

Initializing seen.json

When setting up the project for the first time or clearing history, initialize seen.json as an empty object:

{}

Warning: Do not delete seen.json during daily operation. If deleted, the automation engine will re-evaluate and resend all matching active roles that you already received previously.

9. System Execution & Daily Workflow

Step 1: Parse Resume to Profile JSON

Convert your unstructured PDF/Word resume into a structured profile.json format using your configured LLM API.

python parse_resume.py

Step 2: Test Evaluation (Dry Run)

Run the scraper and evaluation engine locally without dispatching emails. Output will save to a local digest.html file.

python main.py --dry-run

Step 3: Execute Live Job Search & Email Dispatch

Run the execution pipeline to scrape endpoints, score postings, update seen.json, and send the email digest.

python main.py --send-email

10. The Daily 3-Command Routine

Once configuration and initialization are complete, execute this 3-command routine every morning:

# 1. Navigate to project workspace
cd path/to/job_demo/project_folder

# 2. Activate virtual environment (Windows PowerShell)
.\venv\Scripts\Activate.ps1

# 3. Execute job discovery engine & send email digest
python main.py --send-email

11. Troubleshooting & Edge Cases

• LLM API Quota Limits (429 ResourceExhausted): If you run the script repeatedly during testing, you may hit the free tier quota limit for Gemini. Switch models in your .env file (e.g., from gemini-1.5-flash to gemini-1.5-pro) or wait for quota reset.
• No Emails Received:
  1. Check if score_threshold in config.json is set too high (e.g., 9). Lower it to 5 or 6 to test.
  2. Verify that seen.json isn't blocking listings. Reset seen.json to {} if troubleshooting.
  3. Ensure Gmail App Password (16 digits) is used, not your standard account password.
• Missing Target Roles: Verify that excluded_titles does not contain overly broad broad terms that might conflict with desired roles.

Summary and Key Takeaways

• Target ATS Platforms Directly: Scraping marketplaces like LinkedIn and Naukri fails due to anti-bot measures. The agent scrapes direct ATS boards (Greenhouse, Lever, Ashby) to get clean, accessible job links.
• LLM Engine Integration: Google Gemini parses unstructured resumes into structured JSON profiles (profile.json) and scores scrapings against relevant candidate skills.
• Secure SMTP Relay: Notifications are delivered directly to your inbox via Gmail SMTP utilizing a 16-digit Google App Password.
• ChatGPT Config Generation: ChatGPT prompts automate target company path generation for matching ATS platforms.
• Fine-Grained Match Rules: Control search precision using target titles, excluded keywords, geographic filters, and score thresholds (default: 7/10).
• State Preservation (seen.json): Deduplication mechanisms track evaluated job IDs to avoid repeat notifications.
• Daily Execution Efficiency: Running 3 simple terminal commands every morning provides an automated daily job search digest.