Python requests. 403 Forbidden
I needed to parse a site, but i got an error 403 Forbidden. Here is a code:
Please, say what the problem is.
3 Answers 3
It seems the page rejects GET requests that do not identify a User-Agent . I visited the page with a browser (Chrome) and copied the User-Agent header of the GET request (look in the Network tab of the developer tools):
Just add to Alberto’s answer:
If you still get a 403 Forbidden after adding a user-agent , you may need to add more headers, such as referer :
The headers can be found in the Network > Headers > Request Headers of the Developer Tools. (Press F12 to toggle it.)
403 error while using github API #5801
Was initially discovered in the ‘ghexport’ project; I did a reasonable amount of debugging and created this repo before submitting this issue to PyGithub, but thats a lot to look through, just leaving it here as context.
It’s been hard to reproduce, the creator of ghexport (where this was initially discovered) didn’t have the same issue, so I’m unsure of the exact reason
Expected Result
requests succeeds for the authenticated request
Actual Result
Request fails, with:
Reproduction Steps
Apologies if this is a bit too specific, but otherwise requests works great on my system and I can’t find any other way to reproduce this — Is a bit long as it requires an auth token
Go here and create a token with scopes like:
I’ve compared this to httpx, where it doesn’t fail:
Another thing that may be useful as context is the pdb trace I did here, which was me stepping into where the request was made in PyGithub , and making all the requests manually using the computed url / headers . Fails when I use requests.get but httpx.get works fine:
System Information
The text was updated successfully, but these errors were encountered:
Yeah, I was expecting that might be the problem with reporting the issue, is there anything else I can do to diagnose this further?
You could look at what’s present in your response.request object. I printed the headers as there shouldn’t be anything else differing. You may also look into whether you have any proxies or other intermediaries that Requests might be detecting and using for your traffic — which httpx might not be using.
I don’t want to hijack this issue, and will happily file my own if my issue is unrelated, but I’m having a problem with making an OPTIONS request. If I run the following code in python 2.7 with requests 2.7.0 it works fine and I get a 204 back. If I run it with python 3.8.5 I get a 403 :
Works fine with cURL too:
Will see if I can try and find similarities/switch to python2.7 and see if that makes any difference when I debug this further, probably this weekend
Cant tell if an immediate switch to python2.7 does anything — cant test with httpx since that requires 3.6. But running on 2.7 I get the same error as on 3.9.4
Modified the script to work on both versions:
On 2.7, installed using the AUR requests package:
So I’d guess your error is something different?
Will try and inspect the request info to see if theres anything different there.
@seanbreckenridge try downgrading your urllib to 1.24. I was able to work around my 403 issue using Python 3.x that way.
Created a pipenv with 1.24; ran the same script above, doesn’t seem to fix my issue. So the issues we had were probably separate
I had the same issue in the gpodder app which uses requests, in this case a podcast CDN was issuing 403 for some HTTP requests, which are normally CDN redirects. I am not sure why, but as a workaround I found that using the underlying PreparedRequest object and session send() it works.
I’m on version: 2.24.0-lp152.3.3.1
To reproduce (at least for a week or two while this podcast URL is valid):
Once again, this is not reproducible for me.
@davedeaven you did make me wonder though, what happens if you do s.trust_env = False first? Also do you have a .netrc that’s supplying your authorization headers here or did you leave out code? Finally, when you see the 403, what does r.history show you?
@sigmavirus24, I tried the s.trust_env, it works.
And you are correct, I do have a
/.netrc of the form
default login anonymous password deaven@deaven.net
If I remove this, then the request is successful even without the s.trust_env setting. So that is the cause. this solves my issue because I do not need the
/.netrc file, it was left over from an earlier era. Although it does seem to be somewhat unexpected behavior that the prepared request works differently, and I will note that in the same environment, tools like wget (and browsers) work fine to retrieve this URL
Really appreciate the fast response on this, thank you!
Seems to be the same issue I had — I also had a .netrc file which had information to login to github like
Python-сообщество
Попробовал, то же самое. Пока удалось продвинутся только со ссылками на сторонние сайты благодаря удалению слэша в адресе. Member area почему-то для меня по-прежнему закрыта.
#2 Фев. 15, 2017 15:21:02
Requests: 403 в питоне, OK в браузере
Слэш в конце пришлось удалить по рекомендациям отсюда http://stackoverflow.com/questions/19909127/python-requests-403-on-post
Говорят, нужно скармливать именно файл, а не каталог.
А ещё там же говорят, что 403 — это не 401, и сделано именно для защиты:
Отредактировано m0rtal (Фев. 15, 2017 15:23:07)
#3 Фев. 15, 2017 16:27:30
Requests: 403 в питоне, OK в браузере
Ещё немного поправили: User-Agent большими буквами. Но всё равно не помогает.
#4 Фев. 15, 2017 16:42:29
Requests: 403 в питоне, OK в браузере
У меня вот так получилось картинку слить —
#5 Фев. 15, 2017 16:43:53
Requests: 403 в питоне, OK в браузере
Вместо “Mozilla Firefox” пишите просто “Mozilla”
#6 Фев. 15, 2017 16:45:54
Requests: 403 в питоне, OK в браузере
Добавил полную строку User-Agent отсюда, вроде пошло. Редко ещё бывает 403, но пока не системно, буду разбираться.
How to fix problem http error 403 in python 3 web scraping?
HTTP Error 403 is a common error encountered while web scraping using Python 3. It indicates that the server is refusing to fulfill the request made by the client, as the request lacks sufficient authorization or the server considers the request to be invalid. This error can be encountered for a variety of reasons, including the presence of IP blocking, CAPTCHAs, or rate limiting restrictions. In order to resolve the issue, there are several methods that can be implemented, including changing the User Agent, using proxies, and implementing wait time between requests.
Method 1: Changing the User Agent
If you encounter HTTP error 403 while web scraping with Python 3, it means that the server is denying you access to the webpage. One common solution to this problem is to change the user agent of your web scraper. The user agent is a string that identifies the web scraper to the server. By changing the user agent, you can make your web scraper appear as a regular web browser to the server.
Here is an example code that shows how to change the user agent of your web scraper using the requests library:
In this example, we set the User-Agent header to a string that mimics the user agent of the Google Chrome web browser. You can find the user agent string of your favorite web browser by searching for "my user agent" on Google.
By setting the User-Agent header, we can make our web scraper appear as a regular web browser to the server. This can help us bypass HTTP error 403 and access the webpage we want to scrape.
That's it! By changing the user agent of your web scraper, you should be able to fix the problem of HTTP error 403 in Python 3 web scraping.
Method 2: Using Proxies
If you are encountering HTTP error 403 while web scraping with Python 3, it is likely that the website is blocking your IP address due to frequent requests. One way to solve this problem is by using proxies. Proxies allow you to make requests to the website from different IP addresses, making it difficult for the website to block your requests. Here is how you can fix HTTP error 403 in Python 3 web scraping with proxies:
Step 1: Install Required Libraries
You need to install the requests and bs4 libraries to make HTTP requests and parse HTML respectively. You can install them using pip:
Step 2: Get a List of Proxies
You need to get a list of proxies that you can use to make requests to the website. There are many websites that provide free proxies, such as https://free-proxy-list.net/ . You can scrape the website to get a list of proxies:
This code scrapes the website and gets a list of HTTP proxies that support HTTPS. The proxies are stored in the proxies list.
Step 3: Make Requests with Proxies
You can use the requests library to make requests to the website with a proxy. Here is an example code that makes a request to https://www.example.com with a random proxy from the proxies list:
This code selects a random proxy from the proxies list and makes a request to https://www.example.com with the proxy. If the request is successful, it prints the response text. Otherwise, it prints the status code of the failed request.
Step 4: Handle Exceptions
You need to handle exceptions that may occur while making requests with proxies. Here is an example code that handles exceptions and retries the request with a different proxy:
This code uses a while loop to keep retrying the request with a different proxy until it succeeds. It handles ProxyError , ConnectionError , and Timeout exceptions that may occur while making requests with proxies.
Method 3: Implementing Wait Time between Requests
When you are scraping a website, you might encounter an HTTP error 403, which means that the server is denying your request. This can happen when the server detects that you are sending too many requests in a short period of time, and it wants to protect itself from being overloaded.
One way to fix this problem is to implement wait time between requests. This means that you will wait a certain amount of time before sending the next request, which will give the server time to process the previous request and prevent it from being overloaded.
Here is an example code that shows how to implement wait time between requests using the time module:
In this example, we are sending a request to https://example.com with headers that mimic a browser request. We are then using a for loop to send 5 requests with a 1 second delay between requests using the time.sleep() function.
Another way to implement wait time between requests is to use a random delay. This will make your requests less predictable and less likely to be detected as automated. Here is an example code that shows how to implement a random delay using the random module:
In this example, we are sending a request to https://example.com with headers that mimic a browser request. We are then using a for loop to send 5 requests with a random delay between 1 and 5 seconds using the random.randint() function.
By implementing wait time between requests, you can prevent HTTP error 403 and ensure that your web scraping code runs smoothly.