How to Create Webkit Browser with Python
In this tutorial we’ll create simple web browser using Python PyQt framework. As you may know PyQt is a set of Python bindings for Qt framework, and Qt (pronounced cute) is C++ framework used to create GUI-s. To be strict you can use Qt to develop programs without GUI too, but developing user interfaces is probably most common thing people do with this framework. Main benefit of Qt is that it allows you to create GUI-s that are cross platform, your apps can run on various devices using native capabilities of each platform without changing your codebase.
Qt comes with a port of webkit, which means that you can create webkit-based browser in PyQt.
Our browser will do following things:
- load urls entered by user into input box
- show all requests performed while rendering the page
- allow you to execute custom JavaScript in page context
Hello Webkit
Let’s start with simplest possible use case of PyQt Webkit: loading some url, opening window and rendering page in this window.
This is trivial to do, and requires around 13 lines of code (with imports and whitespace):
If you pass url to script from command line it should load this url and show rendered page in window.
At this point you maybe have something looking like command line browser, which is already better than python-requests or even Lynx because it renders JavaScript. But it’s not much better than Lynx because you can only pass urls from command line when you invoke it. We definitely need some way of passing urls to load to our browser.
Add address bar
To do this we’ll just add input box at the top of the window, user will type url into text box, browser will load this url. We will use QLineEdit widget for input box. Since we will have two elements (text input and browser frame), we’ll need to add some grid layout to our app.
At this point you have bare-bones browser that shows some resembrance to Google Chrome and it uses same rendering engine. You can enter url into input box and your app will load url into browser frame and render all HTML and JavaScript.
Add dev tools
Of course the most interesting and important part of every browser are its dev tools. Every browser worth its name should have its developer console. Our Python browser should have some developer tools too.
Let’s add something similar to Chrome “network” tab in dev tools. We will simply keep track of all requests performed by browser engine while rendering page. Requests will be shown in table below main browser frame, for simplicity we will only log url, status code and content type of responses.
Do do this we will need to create a table first, we’ll use QTableWidget for that, header will contain field names, it will auto-resize each time new row is added to table.
To keep track of all requests we’ll need to get bit deeper into PyQt internals. Turns out that Qt exposes NetworkAccessManager class as an API allowing you to perform and monitor requests performed by application. We will need to subclass NetworkAccessManager, add event listeners we need, and tell our webkit view to use this manager to perform its requests.
First let’s create our network access manager:
I have to say that some things in Qt are not as easy and quick as they should be. Note how awkward it is to get status code from response. You have to use response method .attribute() and pass reference to class property of request. This returns QVariant not int and when you convert to int it returns tuple.
Now finally we have a table and a network access manager. We just need to wire all this together.
Now fire up your browser, enter url into input box and enjoy the view of all requests filling up table below webframe.
If you have some spare time you could add lots of new functionality here:
- add filters by content-type
- add sorting to table
- add timings
- highlight requests with errors (e.g. show them in red)
- show more info about each request — all headers, response content, method
- add option to replay requests and load them into browser frame, e.g. user clicks on request in table and this url is loaded into browser.
This is long TODO list and it would be probably interesting learning exercise to do all these things, but describing all of them would probably require to write quite a long book.
Add way to evaluate custom JavaScript
Finally let’s add one last feature to our experimental browser — ability to execute custom JavaScipt in page context.
After everything we’ve done earlier this one comes rather easily, we just add another QLineEdit widget, connect it to web page object, and call evaluateJavaScript method of page frame.
then we instantiate it in our main clause and voila our dev tools are ready.
Now the only thing missing is ability to execute Python in page context. You could probably develop your browser and add support for Python along JavaScript so that devs writing apps targeting your browser could.
Moving back and forth, other page actions
Since we already connected our browser to QWebPage object we can also add other actions important for end users. Qt web page object supports lots of different actions and you can add them all to your app.
For now let’s just add support for “back”, “forward” and “reload”. You could add those actions to our GUI by adding buttons, but it will be easier to just add another text input box.
just as before you also need to create instance of ActionInputBox, pass reference to page object and add it to our GUI grid.
Mozzerella Ashbadger
The first steps building the browser with PyQt5
So far we’ve learned the basics of building Python GUI applications with Qt. In this tutorial we’ll take what we’ve learned and apply it to creating a custom web browser — Mozzerella Ashbadger — in Python.
Mozzerella Ashbadger is the latest revolution in web browsing! Go back and forward! Save files! Get help! (you’ll need it). Any similarity to other browsers is entirely coincidental.
QtWebEngineWidgets is not included in the main PyQt5 repository. If you see errors when running this relating to this module, you can install it using pip install PyQtWebEngine
The full source code for MooseAche is available in the 15 minute apps repository. You can download/clone to get a working copy, then install requirements using:
You can then run MooseAche with:
Read on for a walkthrough of how the code works.
The browser widget
The core of our browser is the QWebView which we import from PyQt5. QtWebEngineWidgets . This provides a complete browser window, which handles the rendering of the downloaded pages.
Below is the bare-minimum of code required to use web browser widget in PyQt.
If you click around a bit you’ll discover that the browser behaves as expected — links work correctly, and you can interact with the pages. However, you’ll also notice things you take for granted are missing — like an URL bar, controls or any sort of interface whatsoever. This makes it a little tricky to use. In the next part we’ll extend this basic skeleton of a browser and add some navigation controls.
PyQt5
PyQt5 is a Python binding fo the cross-platform GUI toolkit QT, implemented as a toolkit plug-in. PyQt us free software developed by the British firm Riverbank Coumputing.
PytQt5 may also be embedded in C++ based application to allow of those applications to configure or enhance the functionality of those applications.
Installation
The GPL version of PyQt5 can be installed from PyPi:
The wheels include a code of the required parts of the LGPL version of Qt.
pip will also build and install the binding from the sdist package but Qt’s qmake tool must be on PATH.
Creating your own browser using Python
Hey there! In this tutorial, we will be learning to create a simple browser in Python using PyQt5 in PyCharm.
Qt is a set of cross-platform C++ libraries that implement high-level APIs for accessing various aspects of modern desktop and mobile systems such as location and positioning services, Bluetooth connectivity, etc.
PyQt5 is a comprehensive set of Python bindings for Qt v5, that enables Python to be used as an alternative application development language to C++ on all supported platforms including iOS and Android.
Implementation
- Firstly, open PyCharm and create a project titled Browser. Then open the terminal and type the below-listed commands to install the respective libraries.
- Then, within the main.py file in this project, type the below-specified code. Refer to the code’s comments regarding various functionalities supported by the browser and their implementations.
To access the arrowhead and reload symbols used in the above code, click here.
Output
In the video below, you can see the output of our project that we have just built: