Web Search & Scraping with Make.com and HARPA GRID

# Features of this Scraping Workflow

In this guide, we’ll show you how to use a Make.com workflow to efficiently scrape data from websites. You can either search for resources or upload your own list of URLs, making it easy to gather the information you need.

Here are four ways this tool can help boost your organization’s profits:

1. Supplier and Partner Research: Quickly find and evaluate potential partners by scraping data from supplier websites, making your procurement process smoother and more cost-effective.

2. Lead Generation: Automatically gather contact information from targeted sites to fill your sales pipeline with qualified leads, increasing your chances of closing deals.

3. Market Trends: Stay updated on industry changes and customer preferences by collecting real-time data, allowing you to make informed decisions that drive higher sales.

4. Automated Blog Management: Keep your blog fresh with the latest news and insights by scraping relevant content, engaging your audience and establishing your brand as an industry leader.

By integrating HARPA AI with the Make.com workflow, you can further streamline your processes and achieve measurable results.

# Requirements for this Scenario:

1. Complete the basic HARPA & Make.com setup:

   • Sign up for Make.com and install the official HARPA AI Web Browser Agent app.

   • Set up an active HARPA GRID Node.

   • Configure the [HARPA API Connection - NEED LINK] in Make.com.

An alternative way to extract data from a webpage is by using AI processing. To start, import the Convert Page to JSON or Database Enrichment command into your HARPA. This will convert webpage content into a structured and customizable JSON format.

You can find an example of this alternative approach in this guide.

# Creating and Running a Scenario

# Search Web Resources for Scraping

Instead of Web Searching module, you can add a connection to another module (such as your CRM, Google Sheets, or another resource that contains a list of links where you need to scrape for information).

Let's say you don't have this data and want to find a list of links based on your search query - "Contact Email Car service Austin, Texas" to find contact information for selling your products.

TIP: When creating your search query, include the words "contact" or "email" right away. This helps find pages that contain contact information.

1. Create a new scenario in Make.com

2. Click "+" and add a HARPA: Search The Web module to your scenario

3. Choose your connection and type in what you want to search for. Remember, if you have other modules in your scenario that automatically create search queries, you can drag the relevant item into this field.

4. Test this Module, and it will return a data array containing the Search Page text content and 8 links to found resources, including page titles and descriptions.

We now have a list of links where we can search for contact emails using either the Scraping Module or AI Command.

# Iterator Mudule

This module will help us process links one by one, whether they are found or already in your possession, parsing information from each website.

1. Add the next module - Iterator. This will help process each link in the data array one by one.

2. In the Iterator settings, drag the array element containing our previous search results. Note: You need the full array containing all the data, not specific fields like link or description.

# Finding Information Using HARPA Scraping Module

The "Scrap Web Page" module lets you convert webpage content to Markdown format using your browser. You can select specific elements using CSS, XPath, or Text selectors.

If you don't specify elements to grab, you'll get the full webpage text. In this example, we'll focus on extracting contact emails.

Don't worry if you're not familiar with CSS, XPath, or Text selectors - you can always ask AI for help.

This is what we'll try to accomplish.

1. Click "+" and add a HARPA: Scrap Web Page module to your scenario

2. Select your connection and drag the URL element from the previous module (Iterator).

3. The most challenging part is finding the right data - you may need to experiment to find the correct selectors for emails. Usually AI provides helpful hints.

{
  "selector": "a[href^='mailto:']",
  "selectorType": "css",  // or "auto"
  "at": "all",
  "take": "innerText",
  "label": "emails"
}

4. Test your entire scenario to make sure the data is received before we move on to creating Google Sheets with all contacts.

It's fine if emails are not found on some websites - they might not exist there, or you may need to search using two different selectors where one of them will work. E.g.:

{
  "selector": "//*[contains(text(),'@')]",
  "selectorType": "xpath",
  "at": "all",
  "take": "innerText",
  "label": "emails2"
}

We have created a scenario that collects emails based on search terms targeted to your potential customers. You can send this information to your Database, CRM, Google Sheets or other storage systems. You can also use more HARPA AI Commands to create custom emails for each contact.

# Storing Parsing Results in Google Sheets

1. Add the Google Sheets: Add a Row module. If you have never used Google Sheets with Make.com before, you need to create a connection.

2. Open your Google Drive and create a spreadsheet in Google Sheets to store the extracted data. Of course, it's possible to set up automatic creation of Google Sheets, but that would only complicate this explanation.

Create a Google Sheets file and add these column headers in row 1: Title, Website, Description, Email, Page Content (optional - for creating personalized emails later)

11. Go back to Make.com and select your file by name in the Google Sheets module. Choose the Sheet Name and drag data from the previous Module into the appropriate columns for storage.

You can add a filter to the link between modules. For example, you can filter out websites where the email field is empty, if those sites don't interest you.

Your Scenario is ready - you can now save and test it.