Visit website using proxy server, and with different IP addresses 

Python can be used to automate tasks. One of the tasks that can be automated easily is web-scrapping and clicking on button on web pages.


This is an example program for writing a simple python code. We will follow steps below to find a proxy server (since we don't want to web server to find out we are clicking thousand times a second), and then we will use beautifulSoup to scrap the web page, find the button and then push that button.


1) Find the proxy
To find the proxy, we will do:


url = "https://free-proxy-list.net"
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")
# result = soup.findAll('tr')
result = soup.find('table')
result = result.findAll('tr')
for _ in result:
    print(_)
    root = ET.fromstring(str(_))
    proxy_ip = root[0].text
    proxy_port = root[1].text
    if proxy_ip != "IP Address":
        proxy = 'https://' + proxy_ip + ':' + proxy_port
        os.environ['https_proxy'] = proxy






This code above, will use a list of proxies to find IP and Port of proxy server. 


2) Visit the website






driver = webdriver.Safari()
driver.get("https://youtu.be/blahblahblah")
elem = driver.find_element_by_class_name("ytp-large-play-button-bg")
print(elem.get_attribute("fill"))
driver.close()




Here we are using web driver to browse a youtube channel. Each time the youtube channel will be browsed with a different IP address
































Comments











Post a Comment































Popular posts from this blog









PI Web API and Python





















The OsiSoft PI has an html based application system called PIWebAPI to extract data. There are a couple of steps to follow to set up the system, which I will cover in my posts later, but for now, let's see how we can access the data.   We will search for all tags in our database called "DTBASE" and show the results.   Assumptions:  IP address: 192.168.1.44  Username to piwebapi: username  Password to piwebapi: password   import base64  import requests  from django.shortcuts import render, redirect   First we need to create the query url. We can use api url creator programs such as Postman . It is much easier to check if API is actually working.   url = "https://192.168.1.44/piwebapi/search/query?q=*   Then we will create the header string for html before browsing it using GET.   usrPass = username + ":" + password  b64Val = base64.b64encode(usrPass.encode()).decode("ascii")  payload = ""  headers = {          "Authorization": ...








Read more










Python Basics





















   Eclipse and PyDev     In preparation for installing eclipse, we need a version of Java JDK installed on the system. The current Java versions include 8, 9 and 10; however version 8 is the most safe on our systems as other installations may depend on that version. Install JDK package by:   $ sudo add-apt-repository ppa:webupd8team/java   $ sudo apt update   $ sudo apt install oracle-java8-installer   You have to acknowledge the code license.      You can actually have multiple JDK versions on your system and switch between them by:   $ update-java-alternatives -l              // see the alternatives   $ sudo update-alternatives --config java   // choose one     Install Eclipse     Ubuntu provides Eclipse and PyDev as packages, but in the past I have not been able to get them to provide the full PyDev functionality, so I use the packages available from the eclipse home site:   http://www.eclipse.org   Within the Eclipse downloads available from t...








Read more