Selenium Setup
This article contains notes on installing, coding, and running Selenium.
-
Run in Google Cloud a Docker image contaning Selenium and associated software. built using Ansible scripts that created the image.
Add Tesseract to extract text from pictures (OCR = Optical Character Recognition)
Some misconceptions
When we mention “Selenium” and “performance testing” in the same sentence, the first thing that most people say is “you can only run a few users on a machine”.
But I’m not talking about running several browser instances on each single machine.
I’m talking about running Selenium once while another program converts what goes back and forth over the network into a script.
The secret sauce is …
Running example
Selenium has no GUI. It runs as a console
However, reports are produced by TestNG, a plug-in to Selenium.
Docker images
Docker images containing Selenium server:
Ansible
Ansible task files to establish Selenium server:
- https://github.com/arknoll/ansible-role-selenium
- https://github.com/quarkslab/ansible-selenium-server
- https://mtlynch.io/testing-ansible-selenium/
Run using Maven after GitHub
- Install Maven.
- Install Selenium.
- Install the various browsers Selenium will control (Chrome, Firefox, etc.).
- Navigate to or create a folder to hold a new folder to be created by Git.
Clone from GitHub a repository containing sample tests:
git clone https://github.com/wilsonmar/Selenium-samples
cd Selenium-samples
Select a folder:
cd Python-soup
Look at the root layer of the repository. These files are there for use with Eclipse IDE:
- .classpath
- .project
- .settings
- .metadata
Some prefer to add them in .gitignore so they are not in the repo.
.gitignore .DS_Store
.DS_Store files should be ignored. They are created by MacOS. There is an entry for it in the .gitignore file so they are not stored in GitHub.
Maven
Invoke Maven to download dependencies and run Selenium:
mvn clean verify -Pbrowser-phantomjs
Maven creates a folder named target to receive downloads before starting Selenium.
verify -Pbrowser-phantomjs specifies use of the PhantomJS headless browser. Alternately, other browsers:
mvn clean verify -Pbrowser-chrome
mvn clean verify -Pbrowser-firefox
mvn clean verify -Pbrowser-edge
mvn clean verify -Pbrowser-internet-explorer
mvn clean verify -Pbrowser-opera
PROTIP: Several separate runs are needed to test on several browsers.
At the end of the run, you should see:
View Sample Selenium scripts
If you’re not using an IDE, use the Atom text editor to open a folder list:
atom .
View the pom.xml file.
The browsers handled by Selenium are identifined by a profile with a property within the pom.xml file which also specifies to Maven what dependencies to download and how to run Selenium.
PROTIP: If you work within enterprise firewalls, change the external URLs to internal ones, which may be managed within Nexus or Artifactory servers.
Dive into folder to view:
src/test/resources/webdrivermanager.properties .
Here is where Maven knows to download drivers.
PROTIP: The LATEST is specified. But a specific version would ensure that all drivers downloaded are the ones previously tested to work with each other. Specific versions can be specified with java invocation:
Dive into the src folder.
Notice there is a main and a test folder. Under each is a folder path:
java/selenium/utils
At the end of that path under main contains a TestUtils.java file which defines generic Java utility functions such as randomBetween, isDuplicatePresent, isAllEquals.
Selenium is all about testing, so the end of the path under test contains many more java files to control the browser.
test/java/selenium/configurations/TestConfig.java makes use of properties browser.name and base_url retrieved by variables in
main/java/selenium/configurations/TypedProperties.java .
PROTIP: Properties controlling a specific test are defined in properties files rather than hard-coded into code so that different properties can be used during a run by temporarily replacing the file.
SeleniumTestWrapper
Tests are driven by a wrapper which App-specific test code extend:
Look in SeleniumTestWrapper.java defined as a abstract class which are extended by other code.
Code in the file controls agent strings and cookies that browsers automatically send back to servers.
The code also manages the screen dimensions of the browser window.
These enable app-specific test code to focus on business. Under the Annotations folder are files that define code generation by the Java compiler.
Annotations
Each Java compiler annotation: @Before, @Rule, and @After is defined within code imported:
The annotation code imported add additional functionality such as logging. These decorators also provide metadata (data about data).
Instead of using Java inheritance, Java frameworks Spring and Hibernate use AOP (Aspect oriented programming) to provide a mechanism to inject code for preProcessing and postProcessing for an event. A “hook” in code before and after a method execution for consumer code in those places.
App-specific Tests
Edit the file defined to test an app:
The file contains annotations:
These annotations are defined by imports:
The earlier version, 2.53 and below, used these libraries:
The earlier 2.x code was:
QUESTION: How can we read the code in these annotation libraries?
Page Objects
The core driving code refers to definitions within the pageobjects folder:
The Page Object design pattern separates locator definitions in a separate java file so that various tests only need to refer to a reference rather than reduntantly specifying the way to locate objects on each page. This reduces maintenance over time as the website HTML changes. Change the locator technique in one place and all tests are good again.
The basic rule is that tests don’t declare variables (to manage state on their own), manipulate the DOM directly, nor create objects (using the “new” constructor keyword).
Webdriver Install
There are Web Driver programs for each combination of operating system (macOS, Windows, Linux, etc.), internet browser (Chrome, Firefox, etc.) and each version (78, 79, etc.). This means the Web Driver you install today would likely be obsolete when a new version of the browser is automatically installed on your machine.
PROTIP: There are two ways to install Selenium Web Driver. One is to manually install for whatever is your current version. The other is to use a Webdriver package manager.
Webdriver Manager
- Checks for the latest version of the WebDriver binary
- Downloads the WebDriver binary if it’s not present on your system
- Exports the required WebDriver Java environment variables needed by Selenium
Its automation makes use of Maven utility commonly used by Java.
The WebDriver can download files from an open source repository:
Selenium3Hello1
Obtain Selenium to test a sample call to Google Search:
Selenium3Hello1 is used to verify whether the Selenium core install works. It doesn’t use any browser driver.
Selenium3GoogleSearch1 works on multiple browsers.
Look into the folder:
NOTE: It is not best practice, but the sample scripts in my GitHub contains binary files copied from binary repositories.
File Sea.jpg is required only by the sample program from Neotys.
PROTIP: For team/production coding, place photos in a folder such as pics .
The file is placed at the project’s root folder to make it easy to specify its path.
File chromedriver.exe and other browser driver files are at the root of each script folder.
The more correct way is to specify the files (and their specific versions) in a pom.xml file that point to the location of those external dependencies, and then have each user of the repository to run Maven to obtain those files.
Nevertheless, the drivers are included so you can get going quickly.
Invoke sample command
On Windows:
It is assumed that the Java program is within the PATH which the operating system looks for executables.
View the sample command file Selenium3Chrome1.bat
PROTIP: Windows command prompt (cmd.exe) allows the umlaut ^ (Shift + 6) character to indicate line continuation.
Run the sample Selenium Java program on a Windows machine:
Selenium3Usahidi1NeoloadChrome1.bat
On Macs & Linux
View the sample command file Selenium3Chrome1.sh
Notice there is no “.exe” in the driver.
PROTIP: Bash shell scripts use the back-slash character (above the Enter/return key) to indicate line continuation.
chmod +X *.sh
Run the sample Selenium Java program:
Selenium3Usahidi1NeoloadChrome1.sh
Coding
Drivers for browsers go into the Reference folder.
Implicit waits. Don’t use them. Especially with explicit waits.
Cross platform
Windows vs. Mac vs. Linux
Cross browser
Obtain jars and drivers
There are the ways to assemble what Selenium needs:
a). Copy a working Selenium project (as shown above) that already has the files needed. Then edit it for your own uses. This is the quickest and simplest way.
b). Copy set of jars from an in-house binary respository (such as Nexus or Artifactory).
d). Manual download from websites, which you’ll need to do when a new version comes along.
e). Build from source.
PROTIP: Download all the latest install files at one sitting to test compatibility of the set. Assemble them together in a single folder for copying into each Selenium project folder so each can stand alone when distributed.
This explains the driver operation.
Maven
Manual Download
If you’re using a sample script, skip this.
PROTIP: Download files for all operating systems so the scripts begin as cross-system capable (works on Windows, Mac, Linux).
Under the “Selenium Standalone Server” section heading, click the link next to “Download version”:
Click Save of file named with the same version number, such as:
selenium-server-standalone-3.5.0.jar
to the Downloads folder.
Java bindings
Under the “Selenium Client & WebDriver Language Bindings” section heading, click the link click the Download link associated with the programming language you use, such as Java.
Click Save of file named with the same version number:
selenium-java-3.5.0.zip
to the Downloads folder.
Windows Internet Explorer driver
On Windows only, under “The Internet Explorer Driver Server” section heading, click the link 64-bit Windows IE
Click Save of file named with the same version number, such as:
IEDriverServer_x64_3.5.0.zip
to the Downloads folder.
Mac Safari WebDriver
This section is only applicable on an Apple Mac.
Under the “Safari” section heading in the Selenium webpage, the SafariDriver.safariextz file is deprecated. So no need to download it.
Starting with Safari 10 that comes with OS X El Capitan and macOS Sierra, in 2016, WebKit running Java 1.8+ supports the new W3C WebDriver browser automation API. So Selenium automatically launches the driver without further configuration.
However, it needs to be enabled because it’s off by default.
Click “Develop” on the menu bar and select “Allow Remote Automation”.
Authorize safaridriver to launch the webdriverd service which hosts the local web server.
Open a Terminal window to run:
cd /usr/bin
./safaridriver -p 5678
BLAH: The response asks for a port number.
Complete the authentication prompt.
To verify, run the Selenium script:
Eclipse IDE
-
Within Eclipse, right-click on your project name to select New, Folder.
Type in name “lib” (for library). Finish.
Server into lib folder
Click OK to the pop-up dialog for Copy files.
Selenium Java into lib folder
Within the Downloads folder, unzip by double-clicking on file
selenium-java-3.5.0.zip.
Dive into the folders to drag
selenium-java-3.5.0.jar
to the lib folder.
Click OK to the pop-up dialog for Copy files.
Eclipse folder
Click on the door icon and a Finder window opens up.
new References folder for Browser drivers
Select all the files.
NOTE: Previous versions did not include JUnit and others.
Choose Copy Files. OK.
Establish lib as References
To make Eclipse recognize the jar files:
Highlight the two files in the lib folder.
Click first file. Hold Cntrl while clicking the second file.
Right-click for Build Path. Add to Build Path.
A Referenced Libraries item should appear under Package Explorer.
Download Chrome driver
Open Chrome browser, finish work on all windows as you’ll need to relaunch by the time this is done.
Notice “Version 79.0.3945.79” or whatever it is what you do this.
Right-Click the file for your operating system, such as:
Click “Save” and wait for the download to finish.
Use your mouse to drag to move the chromedriver file to that folder.
Try your Selenium WebDriver code.
Use Gecko Firefox driver
The “Latest release” is shown at the top. Later releases are below.
Mozilla also calls it the “Marionette Proxy”.
Scroll down to the Downloads section and click on the file for your operating system.
geckodriver-v0.18.0-macos.tar.gz for Mac (1.31 MB)
geckodriver-v0.18.0-win64.zip for Windows
Unzip the macos file for the geckodriver executable.
Unzip the win64 file for the geckodriver.exe executable.
Restart your system and try your Selenium WebDriver code as shown in video.
Headless drivers
Ghost Driver or PhantomJS turns Selenium “headless”, accessing the DOM.
HTML Unit from https://selenium-release.storage.googleapis.com/index.html selenium-html-runner-3.5.0.jar
Junit into lib folder
Click the text to the right of “Looking for the latest version?”, such as Download junit-4.10.jar (253.2 kB).
This downloads file junit-4.10.jar (from 2011).
Drag from within the Downloads folder file
junit-4.10.jar
and drop it within the lib folder when the mouse turns into a “+” sign.
New Class
IntelliJ IDE
Java Coding
## TODO: Random number
Add-on functionality
TestNG has more in-built annotations than JUnit, making testing easier.
TestNG requires a download from http://testng.org/
Its @DataProvider and parameters enables data-driven testing.
JUnit does not generate a HTML reports. But TestNG generates an XSLT report.
- https://www.youtube.com/watch?v=OTtFSnZY4f8
VIDEO on emitting industry-standard logs.
VIDEO on Advanced topics.
- https://www.youtube.com/watch?v=0UQ9pAlY3qg
Read CSV files
To get your Selenium Java code to read CSV files:
Specify the dependency in Maven, Ant, etc. or
Download from https://mvnrepository.com/artifact/net.sf.opencsv/opencsv/
Add in your Java code:
Read Excel files
To get your Selenium Java code to read Excel files:
Download Apache POI — the Java library for Microsoft documents from
https://poi.apache.org/download.html
the binary distribution for the latest stable version, such as
poi-bin-3.16-20170419.tar.gz
Unzip using RAR so you don’t extract
Select the file under the HTTP heading.
Rock Stars
Simon Stewart (@sha98c, blog.rocketpoweredjetpants.com) invented WebDriver, now Lead Committer
- State of the Union at SeleniumConf Austin 5 April 2017. says the original vision was:
FIT Test > FIT Fixture > Page Objects > WebDriver
“Definitely do not use FIT”
“We test so that when we release software, we are confident it works, as early as possible.”
- Automation Best Practices 14 Jul 2016
Video Tutorials
- Selenium WebDriver Eclipse Java Project Setup: For the absolute beginner on Windows with Java 1.6 [13:58]
https://www.dropbox.com/s/inuirqwhlr3w7zf/slides.pdf?dl=0 Dave Hoeffer
CA DevTest
SauceLab Cloud
Open an account at SauceLab.com.
set environment variables SAUCE_USERNAME SAUCE_ACCESS_KEY
SELENIUM_BROWSER SELENIUM_VERSION SELENIUM_PLATFORM
Python Web scraping
This is based on Pluralsight’s 1h 7m video course “Scraping Dynamic Web Pages with Python and Selenium” by Pratheerth Padman covers use of Python invoked by Jupyter Notebook. Python has library Beautiful Soup (to scrape HTML and XML from web pages) and Selenium 2.0 WebDriver (to emulate keyboard and mouse movements based on JSON commands).
-
Install Selenium for Python3 (covered above)
Install Python with virtualvenv and Anaconda.
Download my repository containing Jupyter Notebooks:
Open a Terminal and navigate to that folder. See https://wilsonmar.github.io/jupyter
Call Selenium from Python
In Jupyter Notebook, run file demo_mod2.ipynb file which opens a hard-coded web page, then quit.
To open browser options and set arguments (as if manually in Settings”) before opening:
Run file demo_mod3 -2.ipynb file which manipulates iframes:
To handle popups:
Run file demo_mod4.ipynb in Jupyter to use the soup class to scrape quotes attributed to Soccer player Wayne Rooney from premierleague.com.
NOTE: There is an error in this script.
Other info
NOTE: All “Selenium3” require Java 1.8+.
PROTIP: A number is included with each component to provide for version control, since everything changes all the time in IT.
selenium 4.10.0
The selenium package is used to automate web browser interaction from Python.
Several browsers/drivers are supported (Firefox, Chrome, Internet Explorer), as well as the Remote protocol.
Supported Python Versions
Installing
If you have pip on your system, you can simply install or upgrade the Python bindings:
Alternately, you can download the source distribution from PyPI (e.g. selenium-4.10.0.tar.gz), unarchive it, and run:
Note: You may want to consider using virtualenv to create isolated Python environments.
Drivers
Selenium requires a driver to interface with the chosen browser. Firefox, for example, requires geckodriver, which needs to be installed before the below examples can be run. Make sure it’s in your PATH , e. g., place it in /usr/bin or /usr/local/bin .
Failure to observe this step will give you an error selenium.common.exceptions.WebDriverException: Message: ‘geckodriver’ executable needs to be in PATH.
Other supported browsers will have their own drivers available. Links to some of the more popular browser drivers follow.
Современная Веб-Автоматизация при Помощи Python и Selenium
В данной статье вы изучите продвинутую технику веб-автоматизации в Python. Мы используем Selenium с браузером без графического интерфейса, экспортируем отобранные данные в CSV файлы и завернем ваш отобранный код в класс Python.
Содержание
1. Мотивация: отслеживаем музыкальные привычки
Предположим, что вы время от времени слушаете музыку на bandcamp.com или soundcloud и вам хочется вспомнить название песни, которую вы услышали несколько месяцев назад.
Есть вопросы по Python?
На нашем форуме вы можете задать любой вопрос и получить ответ от всего нашего сообщества!
Telegram Чат & Канал
Вступите в наш дружный чат по Python и начните общение с единомышленниками! Станьте частью большого сообщества!
Паблик VK
Одно из самых больших сообществ по Python в социальной сети ВК. Видео уроки и книги для вас!
Конечно, вы можете покопаться в истории вашего браузера и проверить каждую песню, но это весьма болезненная затея… Все, что вы помните, это то, что вы услышали песню несколько месяцев назад и она в жанре электроника.
«Было бы классно», думаете вы «Если бы у меня запись моей истории прослушиваний. Я мог бы просто взглянуть на электронную музыку, которую я слушал пару месяцев назад и найти эту песню!»
Сегодня мы создадим простой класс Python под названием BandLeader, который подключается к bandcamp.com, стримит музыку из раздела «Найденное» на главной странице, и отслеживает вашу историю прослушиваний.
История прослушиваний будет сохранена на диске в CSV файле. Далее, вы можете в любой момент просматривать CSV файл в вашей любимой программе для работы с таблицами, или даже в Python.
Если у вас есть опыт в веб-парсинга в Python, то вы знакомы с созданием HTTP запросами и использованием API Python для навигации в DOM. Сегодня мы затронем все эти пункты, за одним исключением.
Сегодня вы используете браузер в режиме без графического интерфейса (режим «командной строки») для выполнения запросов HTTP.
Консольный браузер – это обычный веб браузер, который работает без видимого пользовательского интерфейса. Как вы могли догадаться, он может делать больше, чем выполнять запросы: проводить рендер HTML (правда, вы этого не будете видеть), хранить информацию о сессии, даже проводить асинхронные сетевые связи на коде JavaScript.
Если вы хотите автоматизировать современную сеть, консольные браузеры – неотъемлемая часть.
Бесплатный бонус: Скачайте основу проекта Python+Selenium с полным исходным кодом, который вы можете использовать как основу для вашего веб-парсинга в Python и автоматических приложениях.
2. Установка и Настройка Selenium
Первый шаг, перед тем как написать первую строчку кода – это установка Selenium с поддержкой WebDriver для вашего любимого браузера. Далее в статье мы будем работать с Firefox Selenium, но Chrome также будет отлично.
По выше указанным ссылкам имеется полное описание процесса установки драйверов для Selenium.
Далее, нужно установить Selenium при помощи pip, или как вам удобнее. Если вы создали виртуальное пространство для этого проекта, просто введите:
Setup Selenium with Python and Chrome Driver on Ubuntu & Debian
Selenium is a versatile tool that can be used for automating browser-based tests. It has a wide range of features that make it an ideal choice for automating tests. Selenium can be used to automate tests for web applications and web services. Selenium supports a number of programming languages, including Java, C#, Python, and Ruby.
This makes it possible to write tests in the language that you are most comfortable with. In addition, Selenium has a large user community that provides support and help when needed.
In this blog post, you will learn to set up a Selenium environment on an Ubuntu system. Also provides you with a few examples of Selenium scripts written in Python.
Prerequisites
You must have Sudo privileged account access to the Ubuntu system.
One of the examples also required a desktop environment to be installed.
Step 1: Installing Google Chrome
Use the below steps to install the latest Google Chrome browser on Ubuntu and Debian systems.
- First of all, download the latest Gooogle Chrome Debian package on your system.
- Now, execute the following commands to install Google Chrome from the locally downloaded file.
This will complete the Google Chrome on your Ubuntu or Debian system. This will also create an Apt PPA file for further upgrades.
Step 2: Installing Selenium and Webdriver for Python
We will use a virtual environment for running Python scripts. Follow the below steps to create Python virtual environment and install the required python modules.
- Create a directory to store Python scripts. Then switch to the newly-created directory.
- Set up the Python virtual environment and activate it.
Once the environment is activated, You will find the updated prompt as shown below screenshot:
Example 1: Selenium Python Script with Headless Chrome
Your system is ready to run Selenium scripts written in Python. Now, create a sample selenium script in Python that fetches the title of a website.
This script will run headless, So you can run it without an X desktop environment. You can simply SSH to your system and run the below example:
- Create a Python script and edit it in your favorite text editor:
- Copy-paste the following Selenium Python script to the file.