Introduction

Welcome to The Book for thirtyfour.

thirtyfour is a crate for automating Web Browsers in Rust using the WebDriver / Selenium ecosystem. It also provides some support for the Chrome DevTools Protocol, which is used by popular frameworks such as Cypress and Playwright.

Why is it called "thirtyfour" ?

Thirty-four (34) is the atomic number for the Selenium chemical element (Se) ⚛️.

Features

All W3C WebDriver V1 and WebElement methods are supported
Create new browser session directly via WebDriver (e.g. chromedriver)
Create new browser session via Selenium Standalone or Grid
Find elements (via all common selectors e.g. Id, Class, CSS, Tag, XPath)
Send keys to elements, including key-combinations
Execute Javascript
Action Chains
Get and set cookies
Switch to frame/window/element/alert
Shadow DOM support
Alert support
Capture / Save screenshot of browser or individual element as PNG
Chrome DevTools Protocol (CDP) support (limited)
Advanced query interface including explicit waits and various predicates
Component Wrappers (similar to Page Object Model)

Installation

To use the thirtyfour crate in your Rust project, you need to add it as a dependency in your Cargo.toml file:

[dependencies]
thirtyfour = "0.33.0-alpha.2"

To automate a web browser, thirtyfour needs to communicate with a webdriver server. You will need to download the appropriate webdriver server for your browser.

For Chrome, download chromedriver
For Firefox, download geckodriver

The webdriver may be zipped. Unzip it and place the webdriver binary somewhere in your PATH. Make sure it is executable and that you have permissions to run it.

You will also need the corresponding web browser to be installed in your Operating System. Make sure the webdriver version you download corresponds to the version of the browser you have installed.

If the webdriver is not the right version for your browser, it will show an error message when you try to start a new session using thirtyfour.

Writing Your First Browser Automation Code

Before we begin, you'll need to install Rust. You can do so by using the rustup tool.

Let's start a new project. Open your terminal application and navigate to the directory where you usually put your source code. Then run these commands:

cargo new --bin my-automation-project
cd my-automation-project

You will see a Cargo.toml file and a src/ directory there already.

First, let's edit the Cargo.toml file in your editor (e.g. Visual Studio Code) and add some dependencies:

[dependencies]
thirtyfour = "0.33.0-alpha.2"
tokio = { version = "1", features = ["full"] }

Great! Now let's open src/main.rs and add the following code.

NOTE: Make sure you remove any existing code from main.rs.

Don't worry, we'll go through what it does soon.

/src/main.rs

use std::error::Error;

use thirtyfour::prelude::*;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error + Send + Sync>> {
    let caps = DesiredCapabilities::chrome();
    let driver = WebDriver::new("http://localhost:9515", caps).await?;
    // Navigate to https://wikipedia.org.
    driver.goto("https://wikipedia.org").await?;
    let elem_form = driver.find(By::Id("search-form")).await?;

    // Find element from element.
    let elem_text = elem_form.find(By::Id("searchInput")).await?;

    // Type in the search terms.
    elem_text.send_keys("selenium").await?;

    // Click the search button.
    let elem_button = elem_form.find(By::Css("button[type='submit']")).await?;
    elem_button.click().await?;

    // Look for header to implicitly wait for the page to load.
    driver.query(By::ClassName("firstHeading")).first().await?;
    assert_eq!(driver.title().await?, "Selenium - Wikipedia");

    // Always explicitly close the browser.
    driver.quit().await?;

    Ok(())
}

Next we need to make sure our webdriver is running.

Open your terminal application and run:

chromedriver

NOTE: This tutorial is currently set up for Chrome. If you'd prefer to run on Firefox instead, this is explained below.

Now open a new tab in your terminal and run your code:

cargo run

If everything worked correctly you should have seen a Chrome browser window open up, navigate to the "Selenium" article on Wikipedia, and then close again.

Running on Firefox

To run the code using Firefox instead, we first need to tell thirtyfour to use the configuration for Firefox. To do this, change the first to lines of your main function to this:

#![allow(unused)]
fn main() {
    let caps = DesiredCapabilities::firefox();
    let driver = WebDriver::new("http://localhost:4444", caps).await?;
}

Now, instead of running chromedriver in your terminal, we'll run geckodriver instead:

geckodriver

And again, in the other tab, run your code again:

cargo run

If everything worked correctly, you should have seen the Wikipedia page open up on Firefox this time.

Congratulations! You successfully automated a web browser.

Understanding Web Browser Automation

So what did this code actually do?

NOTE: This section goes much deeper into how web browser automation works. Feel free to skip ahead and come back when you're ready to dig deeper.

How Thirtyfour Works

Well, the first thing to know is that thirtyfour doesn't talk directly to the Web Browser. It simply fires off commands to the webdriver server as HTTP Requests. The webdriver then talks to the Web Browser and tells it to execute each command, and then returns the response from the Web Browser to thirtyfour. As long as the webdriver is running, thirtyfour can do just about anything a human can do in a web browser.

Explaining The Code

So let's go through the code and see what is going on.

#![allow(unused)]
fn main() {
    let caps = DesiredCapabilities::chrome();
    let driver = WebDriver::new("http://localhost:9515", caps).await?;
}

This is where we actually make the initial connection to the webdriver and start a new session in the web browser. This will open the browser in a new profile (so it won't add anything to your history etc.) and navigate to the default start page.

Capabilities

The way we tell it what browser we want is by using DesiredCapabilities. In this case, we construct a new ChromeCapabilities instance. Each *Capabilities struct will have additional helper methods for setting options like headless mode, proxy, and so on. See the documentation for more details.

You may have heard of selenium. Selenium is simply a proxy that forwards requests to one or more webdriver servers. Using Selenium you can interact with multiple browsers at once. You simply tell Selenium where each of the webdrivers are, and some details of each browser, and then in your code you give thirtyfour the address of the selenium server, not the webdriver, and use the DesiredCapabilities to request a particular browser.

Element Queries

Next, we tell the browser to navigate to Wikipedia:

#![allow(unused)]
fn main() {
    driver.goto("https://wikipedia.org").await?;
}

And then we look for an element on the page:

#![allow(unused)]
fn main() {
    let elem_form = driver.find(By::Id("search-form")).await?;
}

We search for elements using what we call a selector or locator. In this case we are looking for an element with the id of search-form.

If you actually navigate to https://wikipedia.org and open your browser's devtools (F12 in most browsers), then go to the Inspector tab, you will see the raw HTML for the page. In the search box at the top of the inspector, if you type #search-form (the # is how we specify an id) you will see that it highlights the search form element.

This is the container element that contains both the input field and the button itself.

But we want to type into the field, so we need to do another query:

#![allow(unused)]
fn main() {
    let elem_text = elem_form.find(By::Id("searchInput")).await?;
}

And again, if you search in the inspector for #searchInput you get an <input /> element which is the one we want to type into.

Typing Text Into An Element

To type into a field, we just tell thirtyfour to send the keys we want to type:

#![allow(unused)]
fn main() {
    elem_text.send_keys("selenium").await?;
}

This will literally type the text into the input field.

Now we need to find the search button and click it. Finding the element means doing another query:

#![allow(unused)]
fn main() {
    let elem_button = elem_form.find(By::Css("button[type='submit']")).await?;
}

This time we use a CSS selector to locate a <button> element with an attribute type that has the value submit. To learn more about selectors, click here.

Clicking The Button

Next, the call to click() tells thirtyfour to simulate a click on that element:

#![allow(unused)]
fn main() {
    elem_button.click().await?;
}

Dealing With Page Loading Times

The page now starts loading the result of our search on Wikipedia. This brings us to our first issue when automating a web browser. How does our code know when the page has finished loading? If we had a slow internet connection, we might try to find an element on the page only for it to fail because that element hasn't loaded yet.

How do we solve this?

Well, one option is to simply tell our code to wait for a few seconds and then try to find the element we are looking for. But this is brittle and likely to fail. Don't do this. There are cases where you are forced to use this approach, but it should be a last resort. Incidentally, from a website testing perspective, it probably also means a human is going to be unsure of when the page has actually finished loading, and this is an indicator of a poor user experience.

Instead, we usually want to look for an element on the page and wait until that element is visible.

#![allow(unused)]
fn main() {
driver.query(By::ClassName("firstHeading")).first().await?;
assert_eq!(driver.title().await?, "Selenium - Wikipedia");
}

Better Element Queries

To have thirtyfour "wait" until an element has loaded, we use the query() method which provides a "builder" interface for constructing more flexible queries. Again, this uses one of the By selectors, and we tell it to return only the first matching element.

The query() method will poll every half-second until it finds the element. If it cannot find the element within 30 seconds, it will timeout and return an error. The polling time and timeout duration can be changed using WebDriverConfig, or you can also override them for a given query.

The query() method is the recommended way to search for elements. The find() and find_all() methods exist only for compatibility with the webdriver spec. The query() method uses these under the hood, but adds polling and other niceties on top, including giving you more details about your query if an element was not found.

See the ElementQuery documentation for more details on the kinds of queries it supports.

Closing The Browser

Finally we need to close the browser window. If we don't, it will remain open after our application has exited.

#![allow(unused)]
fn main() {
    driver.quit().await?;
}

And this concludes the introduction.

Happy Browser Automation!

Feature Flags

rustls-tls: (Default) Use rustls to provide TLS support (via reqwest).
native-tls: Use native TLS (via reqwest).
component: (Default) Enable the Component derive macro (via thirtyfour_macros).

Element Queries

Basic queries

The find() and find_all() methods in both WebDriver and WebElement provide a simple way to perform direct element queries, returning a result instantly.

However, for many types of queries, these methods are inadequate. For example, there is no polling, and no way to wait for something to show up on a page. If an element doesn't exist at the instant you look for it, you'll get an error.

This isn't helpful for the majority of element queries, so thirtyfour provides a more advanced query interface, called ElementQuery.

ElementQuery

The WebDriver::query() and WebElement::query() methods return an ElementQuery struct.

Using ElementQuery, you can do things like:

#![allow(unused)]
fn main() {
let elem_text =
    driver.query(By::Css("match.this")).or(By::Id("orThis")).first().await?;
}

This will execute both queries once per poll iteration and return the first one that matches.

See ElementQuery for more details.

Waiting For Element Changes

Sometimes you already have a reference to an element, but you want to perform an action that might change the element in some way. One way to do this would be to call WebDriver::query() and poll for the element with its new attributes. But it might be challenging to get the right query, and the query might return a different element that already has the attribute you specified.

If you already have a reference to the element, why not use that to poll for the changes you expect?

The thirtyfour crate provides a way to wait for virtually any desired change on an existing element. It's called ElementWaiter.

ElementWaiter

The WebElement::wait_until() method returns an ElementWaiter struct.

Using ElementWaiter you can do things like this:

#![allow(unused)]
fn main() {
elem.wait_until().displayed().await?;
// You can optionally provide a nicer error message like this.
elem.wait_until().error("Timed out waiting for element to disappear").not_displayed().await?;

elem.wait_until().enabled().await?;
elem.wait_until().clickable().await?;
}

And so on. See the ElementWaiter docs for more details.

Components

Components are a more structured way to automate an element or page.

This approach may seem familiar to anyone who has used a Page Object Model before. However, a Component can wrap any node in the DOM, not just "pages".

It uses smart element resolvers that can lazily resolve elements within the component and cache them for further use. You can also nest components, making them an extremely powerful feature for automating any modern web app.

Example

Given the following HTML structure:

<div id="checkbox-section">
    <label>
        <input type="checkbox" id="checkbox-option-1" />
        Option 1
    </label>

    <label>
        <input type="checkbox" id="checkbox-disabled" disabled />
        Option 2
    </label>

    <label>
        <input type="checkbox" id="checkbox-hidden" style="display: none;" />
        Option 3
    </label>
</div>

#![allow(unused)]
fn main() {
/// This component shows how to wrap a simple web component.
#[derive(Debug, Clone, Component)]
pub struct CheckboxComponent {
    base: WebElement, // This is the <label> element
    #[by(css = "input[type='checkbox']", first)]
    input: ElementResolver<WebElement>, // This is the <input /> element
}

impl CheckboxComponent {
    /// Return true if the checkbox is ticked.
    pub async fn is_ticked(&self) -> WebDriverResult<bool> {
        let elem = self.input.resolve().await?;
        let prop = elem.prop("checked").await?;
        Ok(prop.unwrap_or_default() == "true")
    }

    /// Tick the checkbox if it is clickable and isn't already ticked.
    pub async fn tick(&self) -> WebDriverResult<()> {
        // This checks that the element is present before returning the element.
        // If the element had become stale, this would implicitly re-query the element.
        let elem = self.input.resolve_present().await?;
        if elem.is_clickable().await? && !self.is_ticked().await? {
            elem.click().await?;
            // Now make sure it's ticked.
            assert!(self.is_ticked().await?);
        }

        Ok(())
    }
}

/// This component shows how to nest components inside others.
#[derive(Debug, Clone, Component)]
pub struct CheckboxSectionComponent {
    base: WebElement, // This is the outer <div>
    #[by(tag = "label", allow_empty)]
    boxes: ElementResolver<Vec<CheckboxComponent>>, // ElementResolver works with Components too.
    // Other fields will be initialized with Default::default().
    my_field: bool,
}
}

So how do you construct a Component?

Simple. The Component derive automatically implements From<WebElement>.

#![allow(unused)]
fn main() {
let elem = driver.query(By::Id("checkbox-section")).await?;
let component = CheckboxSectionComponent::from(elem);

// Now you can get the checkbox components easily like this.
let checkboxes = component.boxes.resolve().await?;
for checkbox in checkboxes {
    checkbox.tick().await?;
}
}

This allows you to wrap any component using ElementResolver to resolve elements and nested components easily.

For more details see the documentation for Component and the Component derive macro.

Running against selenium

NOTE: This documentation assumes you are already familiar with Selenium. To learn more about Selenium, visit the documentation

NOTE: To run the selenium example, start selenium server and then run:

cargo run --example selenium_example

Below, you can find my recommended development environment for running selenium tests.

Essentially, you need three main things set up as a minimum:

Selenium standalone running on some server, usually localhost at port 4444.

For example, http://localhost:4444
The webdriver for your browser somewhere in your PATH, e.g., chromedriver (Chrome) or geckodriver (Firefox)
Your code that imports this library

If you want you can download selenium and the webdriver manually, copy the webdriver to somewhere in your path, then run selenium manually using java -jar selenium.jar.

However, this is a lot of messing around, and you'll need to do it all again any time either selenium or the webdriver gets updated. A better solution is to run both selenium and webdriver in a docker container, following the instructions below.

Setting up Docker and Selenium

To install docker, see https://docs.docker.com/install/ (follow the SERVER section if you're on Linux, then look for the Community Edition)

Once you have docker installed, you can start the selenium server, as follows:

docker run --rm -d -p 4444:4444 -p 5900:5900 --name selenium-server -v /dev/shm:/dev/shm selenium/standalone-chrome:4.1.0-20211123

For more information on running selenium in docker, visit docker-selenium

Using Selenium Manager

The selenium team is working on a project called "Selenium Manager", which is similar to bonigarcia's WebDriverManager but as a CLI. It's written in Rust as a Clap CLI, so we have the benefit of using it as a library as well. To add it to your project, you can add the selenium project as a git dependency in your Cargo.toml. Be sure to specify the branch is "trunk", like so.

[dependencies]
selenium-manager = { git = "https://github.com/SeleniumHQ/selenium", branch = "trunk" }

NOTE: Due to the way the selenium repository is structured, it is quite large. Better integration with selenium-manager, or equivalent functionality, is planned.

Running the tests for `thirtyfour`

You only need to run the tests if you plan on contributing to the development of thirtyfour. If you just want to use the crate in your own project, you can skip this section.

Make sure selenium is not still running (or anything else that might use port 4444 or port 9515).

To run the tests, you need to have an instance of geckodriver and an instance of chromedriver running in the background, perhaps in separate tabs in your terminal.

Download links for these are here:

chromedriver: https://chromedriver.chromium.org/downloads
geckodriver: https://github.com/mozilla/geckodriver/releases

In separate terminal tabs, run the following:

Tab 1:
```
chromedriver
```
Tab 2:
```
geckodriver
```
Tab 3 (navigate to the root of this repository):
```
cargo test
```
NOTE: By default the tests will run in chrome only. If you want to run in firefox, do:
```
THIRTYFOUR_BROWSER=firefox cargo test
```

Frequently Asked Questions

If you have a question not answered here, head over to the discussion page and ask your question there.

Remember to search through existing discussions and issues to check that your question hasn't already been asked/answered.

Why Doesn't The Browser Close On Exit?

Rust does not have async destructors, which means there is no reliable way to execute an async HTTP request on Drop and wait for it to complete. This means you are in charge of closing the browser at the end of your code, via a call to WebDriver::quit() as in the above example.

If you do not call WebDriver::quit() then the browser will stay open until it is either explicitly closed later outside your code, or the session times out.

Thirtyfour: Web Browser Automation In Rust

Introduction

Why is it called "thirtyfour" ?

Features

Installation

Writing Your First Browser Automation Code

Running on Firefox

Understanding Web Browser Automation

How Thirtyfour Works

Explaining The Code

Capabilities

Element Queries

Typing Text Into An Element

Clicking The Button

Dealing With Page Loading Times

Better Element Queries

Closing The Browser

Feature Flags

Element Queries

Basic queries

ElementQuery

Waiting For Element Changes

ElementWaiter

Components

Example

Running against selenium

Setting up Docker and Selenium

Using Selenium Manager

Running the tests for `thirtyfour`

Frequently Asked Questions

Why Doesn't The Browser Close On Exit?

Further Reading

Thirtyfour Reference Documentation

Selenium Documentation

The W3C WebDriver V1 Specification