Web scraping is a process of fetching data from various pages in Internet. If you want to know how to start scraping with Multilogin profiles – follow this guide, and you will learn how to do a simple script!

Step 1: prepare IDE or similar software

You'll need anything to write your script. It's up to you what to use, but we recommend to use IDE for that. Follow the first 4 steps from the following article: Getting started with automation scripting.

Step 2: create the script connecting to API and define functions

In this step, you'll need to make the script work with API. The script will include:

• API endpoints
• Variables for credentials
• Defined functions for sign in, opening and closing profile
• Imported modules, including requests, hashlib, time. Some modules related to Selenium will be included as well
• Sign in request

Use the following template for it:

import requests
import hashlib
import time
from selenium import webdriver
from selenium.webdriver.chromium.options import ChromiumOptions
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.by import By

MLX_BASE = "https://api.multilogin.com"
MLX_LAUNCHER = "https://launcher.mlx.yt:45001/api/v1"
MLX_LAUNCHER_V2 = (
    "https://launcher.mlx.yt:45001/api/v2"  # recommended for launching profiles
)
LOCALHOST = "http://127.0.0.1"
HEADERS = {"Accept": "application/json", "Content-Type": "application/json"}
# TODO: Insert your account information in both variables below
USERNAME = ""
PASSWORD = ""
# TODO: Insert the Folder ID and the Profile ID below
FOLDER_ID = ""
PROFILE_ID = ""


def signin() -> str:
    payload = {
        "email": USERNAME,
        "password": hashlib.md5(PASSWORD.encode()).hexdigest(),
    }
    r = requests.post(f"{MLX_BASE}/user/signin", json=payload)
    if r.status_code != 200:
        print(f"\nError during login: {r.text}\n")
    else:
        response = r.json()["data"]
    token = response["token"]
    return token


def start_profile() -> webdriver:
    r = requests.get(
        f"{MLX_LAUNCHER_V2}/profile/f/{FOLDER_ID}/p/{PROFILE_ID}/start?automation_type=selenium",
        headers=HEADERS,
    )
    response = r.json()
    if r.status_code != 200:
        print(f"\nError while starting profile: {r.text}\n")
    else:
        print(f"\nProfile {PROFILE_ID} started.\n")
    selenium_port = response["data"]["port"]
    driver = webdriver.Remote(
        command_executor=f"{LOCALHOST}:{selenium_port}", options=ChromiumOptions()
    )
    # For Stealthfox profiles use: options=Options()
    # For Mimic profiles use: options=ChromiumOptions()
    return driver


def stop_profile() -> None:
    r = requests.get(f"{MLX_LAUNCHER}/profile/stop/p/{PROFILE_ID}", headers=HEADERS)
    if r.status_code != 200:
        print(f"\nError while stopping profile: {r.text}\n")
    else:
        print(f"\nProfile {PROFILE_ID} stopped.\n")

token = signin()
HEADERS.update({"Authorization": f"Bearer {token}"})

Step 3: choose web page to scrape data from

You can use any website that contains text, but for this guide, we recommend trying this page – it’s great for practicing automation tasks: Large & Deep DOM.

Step 4: look for the target info

In our case, it will be the data from the table below:

We'll get all the values from the table. Here's what you can do:

1. Open DevTools in your browser. Here's how to do that for Chromium- and Firefox-based browsers:
   1. Windows and Linux: press Ctrl + Shift + I
   2. macOS: press Cmd + Option + I
2. Make sure that you are on the “Elements” tab
3. Use search hot key to find the target value
   1. Windows and Linux: CTRL + F
   2. macOS: Cmd + F
4. Type the text value you want to see. In our case, it is “Table”
5. Look for the value you will need to use for scraping. In our case, it will be the following: <table id="large-table"> 
6. Hover the element with the tags in the “Elements” tab
7. Right click and then left click “Copy” – “Copy selector”
8. Write down the value somewhere – you'll need it later

Step 5: get back to IDE and add new strings of code

1. Get back to the IDE of your choice (for example, VS Code)
2. Click the code field and add a variable for opening and performing actions in the profile: driver = start_profile()
3. Add driver.get(“<your website>”). In our case, it'll be the following command: 

   driver.get("https://the-internet.herokuapp.com/large")

4. Now we need some delay for the script, so it will try to do the other commands 5 seconds after opening the web page: time.sleep(5)

Step 6: make the script to find the element

Use this command to find the element: driver.find_element(By.<attribute on the page>, "<element>"). It tells the script exactly what to look for on the page. Since we copied the CSS selector in step 4, your actual command will look like this:

driver.find_element(By.ID, "large-table")

We'll need to get its value later, so we need to make a variable for the command, for example, fetch:

fetch = driver.find_element(By.ID, "large-table")

Step 7: print the end result and stop the profile

1. Use print() function to print the end result. As we need to extract text value, we need to get the text from our variable. The result will be the following:

   print(fetch.text)

2. Add the function to stop the profile at the end:

   stop_profile()

3. Save the .py script, you'll need to run after some extra steps

Step 8: prepare the script before running it

1. Install the following Python libraries (look for documentation of your IDE for more details):
   1. requests
   2. selenium
2. Insert your values into the below variables in the script:
   1. USERNAME: your Multilogin X account email
   2. PASSWORD: your Multilogin X account password (MD5 encryption is not required)
   3. FOLDER_ID, PROFILE_ID: find these values using our guides on DevTools or Postman

Step 9: run the script

1. Open the desktop app (or connect the agent if you are using the web interface)
2. By default, the script below works for Mimic. To use it for Stealthfox, replace options=ChromiumOptions() with options=Options() in the following line:
   driver = webdriver.Remote(command_executor=f'{LOCALHOST}:{selenium_port}', options=ChromiumOptions()) 
3. Run the .py file with your automation code 

If you've done everything correctly, you'll be able to see the result in the terminal.

Notes

Congratulations with your first scraping script! You are not restricted to those only options. Python and Selenium are quite flexible tools, and there is more potential to them. Here are a couple of tips:

• If you need to fetch several values by similar values (for example, IDs), you can use the following function:
  driver.find_elements(By.<attribute on the page>, "<element>")
• You can add several values to the print() function. You can find more info about it in Internet. For example, you can add the text before the fetch.text. It will make the printing result more readable, and it can be also useful for debugging the script. Here is the example which you can test out in the script:

  print("Your values: ", fetch.text)

• There are more ways of implementing Selenium. Check their help center for more details: Selenium Documentation 

Full script

import requests
import hashlib
import time
from selenium import webdriver
from selenium.webdriver.chromium.options import ChromiumOptions
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.by import By

MLX_BASE = "https://api.multilogin.com"
MLX_LAUNCHER = "https://launcher.mlx.yt:45001/api/v1"
MLX_LAUNCHER_V2 = (
    "https://launcher.mlx.yt:45001/api/v2"  # recommended for launching profiles
)
LOCALHOST = "http://127.0.0.1"
HEADERS = {"Accept": "application/json", "Content-Type": "application/json"}
# TODO: Insert your account information in both variables below
USERNAME = ""
PASSWORD = ""
# TODO: Insert the Folder ID and the Profile ID below
FOLDER_ID = ""
PROFILE_ID = ""


def signin() -> str:
    payload = {
        "email": USERNAME,
        "password": hashlib.md5(PASSWORD.encode()).hexdigest(),
    }
    r = requests.post(f"{MLX_BASE}/user/signin", json=payload)
    if r.status_code != 200:
        print(f"\nError during login: {r.text}\n")
    else:
        response = r.json()["data"]
    token = response["token"]
    return token


def start_profile() -> webdriver:
    r = requests.get(
        f"{MLX_LAUNCHER_V2}/profile/f/{FOLDER_ID}/p/{PROFILE_ID}/start?automation_type=selenium",
        headers=HEADERS,
    )
    response = r.json()
    if r.status_code != 200:
        print(f"\nError while starting profile: {r.text}\n")
    else:
        print(f"\nProfile {PROFILE_ID} started.\n")
    selenium_port = response["data"]["port"]
    driver = webdriver.Remote(
        command_executor=f"{LOCALHOST}:{selenium_port}", options=ChromiumOptions()
    )
    # For Stealthfox profiles use: options=Options()
    # For Mimic profiles use: options=ChromiumOptions()
    return driver


def stop_profile() -> None:
    r = requests.get(f"{MLX_LAUNCHER}/profile/stop/p/{PROFILE_ID}", headers=HEADERS)
    if r.status_code != 200:
        print(f"\nError while stopping profile: {r.text}\n")
    else:
        print(f"\nProfile {PROFILE_ID} stopped.\n")


token = signin()
HEADERS.update({"Authorization": f"Bearer {token}"})
driver = start_profile()
driver.get("https://the-internet.herokuapp.com/large")
time.sleep(5)
fetch = driver.find_element(By.ID, "large-table")
print(fetch.text)
stop_profile()