How to Log in to Almost Any Websites
Tags: java, webscraping
Published: 2019-12-30 11:03:10

---
title: How to Log in to Almost Any Websites
published: true
date: 2019-01-02 09:52:27 UTC
tags: ["Java", "Web Scraping"]
series: "Java Web Scraping"
cover_image: https://www.scrapingbee.com/images/post/java-login/cover-1.png
canonical_url: https://www.scrapingbee.com/blog/how-to-log-in-to-almost-any-websites/
---

In the first [article about java web scraping](https://dev.to/scrapingbee/introduction-to-web-scraping-with-java-5i8) I showed how to extract data from CraigList website. 
But what about the data you want or if the action you want to carry out on a website requires authentication ?

In this short tutorial I will show you how to make a generic method that can handle most authentication forms. 

### Authentication mechanism

There are many different authentication mechanisms, the most frequent being a login form , sometimes with a [CSRF token](http://https://en.wikipedia.org/wiki/Cross-site_request_forgery#Forging_login_requests) as a hidden input. 

To auto-magically log into a website with your scrapers, the idea is : 

* GET /loginPage

* Select the first `<input type="password">` tag

* Select the first `<input>` before it that is not hidden

* Set the value attribute for both inputs

* Select the enclosing form, and submit it. 

### Hacker News Authentication

Let's say you want to create a bot that logs into hacker news (to submit a link or perform an action that requires being authenticated) :

Here is the login form and the associated DOM : 

![](https://www.scrapingbee.com/images/post/java-login/screenshot_hn_login_form.png)

Now we can implement the login algorithm

```java	
	public static WebClient autoLogin(String loginUrl, String login, String password) throws FailingHttpStatusCodeException, MalformedURLException, IOException{
		WebClient client = new WebClient();
		client.getOptions().setCssEnabled(false);
		client.getOptions().setJavaScriptEnabled(false);
		
		HtmlPage page = client.getPage(loginUrl);
		
		HtmlInput inputPassword = page.getFirstByXPath("//input[@type='password']");
		//The first preceding input that is not hidden
		HtmlInput inputLogin = inputPassword.getFirstByXPath(".//preceding::input[not(@type='hidden')]");
		
		inputLogin.setValueAttribute(login);
		inputPassword.setValueAttribute(password);
		
		//get the enclosing form
		HtmlForm loginForm = inputPassword.getEnclosingForm() ;
		
		//submit the form
		page = client.getPage(loginForm.getWebRequest(null));
		
		//returns the cookie filled client :)
		return client;
	}
```

Then the main method, which :

* calls `autoLogin` with the right parameters

* Go to `https://news.ycombinator.com`

* Check the logout link presence to verify we're logged

* Prints the cookie to the console

```java
	public static void main(String[] args) {
		
		String baseUrl = "https://news.ycombinator.com" ;
		String loginUrl = baseUrl + "/login?goto=news" ; 
		String login = "login";
		String password = "password" ;
		
		try {
			System.out.println("Starting autoLogin on " + loginUrl);
			WebClient client = autoLogin(loginUrl, login, password);
			HtmlPage page = client.getPage(baseUrl) ;
			
			HtmlAnchor logoutLink = page.getFirstByXPath(String.format("//a[@href='user?id=%s']", login)) ;
			if(logoutLink != null ){
				System.out.println("Successfuly logged in !");
				// printing the cookies
				for(Cookie cookie : client.getCookieManager().getCookies()){
					System.out.println(cookie.toString());
				}
			}else{
				System.err.println("Wrong credentials");
			}
			
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
```
You can find the code in this [Github repo](https://github.com/ksahin/introWebScraping)
### Go further

There are many cases where this method will not work : Amazon, DropBox... and all other two-steps/captcha protected login forms. 

Things that can be improved with this code : 

* Handle the check for the logout link inside `autoLogin`

* Check for `null` inputs/form and throw an appropriate exception

In a next post I will show you how to deal with captchas or virtual numeric keyboards with OCR and captchas breaking APIs !

If you like web scraping and are tired taking care of proxies, JS rendering and captchas, you can check our new [web scraping API](https://www.scrapingbee.com), the first 1000 API calls are on us.