-
August 24th, 2010, 10:46 AM
#1
Attempting to log into website from java
Hi!
I'm attempting to log into a website from java, but it doesn't seem to be working. I get:
Code:
Running the program...
Problems while reading the response
Problems encounterd.
java.io.IOException: Server returned HTTP response code: 403 for URL: <url of form>
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at KingdomScraper.readResponse(KingdomScraper.java:195)
at KingdomScraper.httpPostLogin(KingdomScraper.java:48)
at KingdomScraper.main(KingdomScraper.java:23)
Here's the code:
Code:
import java.net.*;
import java.io.*;
public class KingdomScraper {
private static final String POST_CONTENT_TYPE = "application/x-www-form-urlencoded";
private static final String LOGIN_USER_NAME_PARAMETER_NAME = "username";
private static final String LOGIN_PASSWORD_PARAMETER_NAME = "password";
private static final String LOGIN_USER_NAME = "myusername";
private static final String LOGIN_PASSWORD = "mypass";
private static final String TARGET_URL = "<url of form>";
public static void main(String[] args)
{
System.out.println("Running the program...\n");
KingdomScraper httpUrlBasicAuthentication = new KingdomScraper();
httpUrlBasicAuthentication.httpPostLogin();
}
/**
* The single public method of this class that
* 1. Prepares a login message
* 2. Makes the HTTP POST to the target URL
* 3. Reads and returns the response
*
* @throws IOException
* Any problems while doing the above.
*
*/
public void httpPostLogin ()
{
try
{
// Prepare the content to be written
// throws UnsupportedEncodingException
String urlEncodedContent = preparePostContent(LOGIN_USER_NAME, LOGIN_PASSWORD);
System.out.println(urlEncodedContent);
HttpURLConnection urlConnection = doHttpPost(TARGET_URL, urlEncodedContent);
String response = readResponse(urlConnection);
System.out.println("Successfully made the HTPP POST.");
System.out.println("Recevied response is: '/n" + response + "'");
}
catch(IOException ioException)
{
System.out.println("Problems encounterd.");
}
}
/**
* Using the given username and password, and using the static string variables, prepare
* the login message. Note that the username and password will encoded to the
* UTF-8 standard.
*
* @param loginUserName
* The user name for login
*
* @param loginPassword
* The password for login
*
* @return
* The complete login message that can be HTTP Posted
*
* @throws UnsupportedEncodingException
* Any problems during URL encoding
*/
private String preparePostContent(String loginUserName, String loginPassword) throws UnsupportedEncodingException
{
// Encode the user name and password to UTF-8 encoding standard
// throws UnsupportedEncodingException
String encodedLoginUserName = URLEncoder.encode(loginUserName, "UTF-8");
String encodedLoginPassword = URLEncoder.encode(loginPassword, "UTF-8");
/* String content = "login=" + LOGIN_ACTION_NAME +" &" + LOGIN_USER_NAME_PARAMETER_NAME +"="
+ encodedLoginUserName + "&" + LOGIN_PASSWORD_PARAMETER_NAME + "=" + encodedLoginPassword;*/
String content = LOGIN_USER_NAME_PARAMETER_NAME +"="
+ encodedLoginUserName + "&" + LOGIN_PASSWORD_PARAMETER_NAME + "=" + encodedLoginPassword;
return content;
}
/**
* Makes a HTTP POST to the target URL by using an HttpURLConnection.
*
* @param targetUrl
* The URL to which the HTTP POST is made.
*
* @param content
* The contents which will be POSTed to the target URL.
*
* @return
* The open URLConnection which can be used to read any response.
*
* @throws IOException
*/
public HttpURLConnection doHttpPost(String targetUrl, String content) throws IOException
{
HttpURLConnection urlConnection = null;
DataOutputStream dataOutputStream = null;
try
{
// Open a connection to the target URL
// throws IOException
urlConnection = (HttpURLConnection)(new URL(targetUrl).openConnection());
// Specifying that we intend to use this connection for input
urlConnection.setDoInput(true);
// Specifying that we intend to use this connection for output
urlConnection.setDoOutput(true);
// Specifying the content type of our post
urlConnection.setRequestProperty("Content-Type", POST_CONTENT_TYPE);
urlConnection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8");
HttpURLConnection.setFollowRedirects(true);
// Specifying the method of HTTP request which is POST
// throws ProtocolException
urlConnection.setRequestMethod("POST");
// Prepare an output stream for writing data to the HTTP connection
// throws IOException
dataOutputStream = new DataOutputStream(urlConnection.getOutputStream());
// throws IOException
dataOutputStream.writeBytes(content);
dataOutputStream.flush();
dataOutputStream.close();
return urlConnection;
}
catch(IOException ioException)
{
System.out.println("I/O problems while trying to do a HTTP post.");
ioException.printStackTrace();
// Good practice: clean up the connections and streams
// to free up any resources if possible
if (dataOutputStream != null)
{
try
{
dataOutputStream.close();
}
catch(Throwable ignore)
{
// Cannot do anything about problems while
// trying to clean up. Just ignore
}
}
if (urlConnection != null)
{
urlConnection.disconnect();
}
// throw the exception so that the caller is aware that
// there was some problems
throw ioException;
}
}
/**
* Read response from the URL connection
*
* @param urlConnection
* The URLConncetion from which the response will be read
*
* @return
* The response read from the URLConnection
*
* @throws IOException
* When problems encountered during reading the response from the
* URLConnection.
*/
private String readResponse(HttpURLConnection urlConnection) throws IOException
{
BufferedReader bufferedReader = null;
try
{
// Prepare a reader to read the response from the URLConnection
// throws IOException
bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String responeLine;
// Good Practice: Use StringBuilder in this case
StringBuilder response = new StringBuilder();
// Read until there is nothing left in the stream
// throws IOException
while ((responeLine = bufferedReader.readLine()) != null)
{
response.append(responeLine);
}
return response.toString();
}
catch(IOException ioException)
{
System.out.println("Problems while reading the response");
ioException.printStackTrace();
// throw the exception so that the caller is aware that
// there was some problems
throw ioException;
}
finally
{
// Good practice: clean up the connections and streams
// to free up any resources if possible
if (bufferedReader != null)
{
try
{
// throws IOException
bufferedReader.close();
}
catch(Throwable ignore)
{
// Cannot do much with exceptions doing clean up
// Ignoring all exceptions
}
}
}
}
}
Now, the form itself is odd. It seems to do a 302 redirect from what I can tell. It also seems to have a csrfmiddlewaretoken (cross site request forgery) being sent along with the username and pass in the POST header. Would that cause me problems?
Code:
<form method="post" action="">
<div style='display:none'><input type='hidden' name='csrfmiddlewaretoken' value='82859e97af9452a2a1d3bf3731eb7fb6' /></div> <!--Whatever this is?Looks to be sent along with the username and password as POST data.-->
<table>
<tbody>
<tr><th><label for="id_username">Username:</label></th><td><input id="id_username" type="text" name="username" maxlength="30" /></td></tr>
<tr><th><label for="id_password">Password:</label></th><td><input type="password" name="password" id="id_password" /></td></tr>
</tbody>
</table>
<input type="submit" class="button" value="Log In" />
</form>
Any suggestions ? I'm rather stuck. Thanks!
Last edited by Lucky75; August 24th, 2010 at 10:49 AM.
-
August 24th, 2010, 11:09 AM
#2
Re: Attempting to log into website from java
Have you looked at HTML Error 403 ?
Computers are good at following instructions, but not at reading your mind...
Donald Knuth
Please use [CODE]...your code here...[/CODE] tags when posting code. If you get an error, please post the full error message and stack trace, if present.
-
August 24th, 2010, 11:24 AM
#3
Re: Attempting to log into website from java
Im aware of what a 403 is, but I don't know why I'm getting one. It doesn't even return a 403 when I enter the form data manually (with a wrong password), it just says "incorrect password" on the page. And I'm logging in with the correct username and password, so I don't know what's wrong.
-
August 24th, 2010, 10:48 PM
#4
Re: Attempting to log into website from java
-
August 25th, 2010, 08:35 AM
#5
Re: Attempting to log into website from java
For starters you're doing the post wrong. You need to find out how the username and password are sent across the wire. They're not posted in plain text for all the world to see. Most common is a base 64 of
Code:
username + ":" + password
The http header portion when you're posting will contain this field if you inspect the standard post through firebug I believe:
Code:
Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==
Where the character string that followed 'Basic' is what I described above.
------
If you are satisfied with the responses, add to the user's rep!
-
August 25th, 2010, 09:43 AM
#6
Re: Attempting to log into website from java
Do you have a proxy server that you can use for viewing what the browser is sending to a server?
Look at what the browser sends to the server when you submit the HTML form.
Then look at what your program sends.
Norm
-
August 25th, 2010, 10:06 AM
#7
Re: Attempting to log into website from java
I was using HTTPFox to check what was being sent. The post data is simply:
csrfmiddlewaretoken 82859e53de1351b2a1d2bg3735eb7fb6
username <myusername>
password <mypass>
Here are the headers being sent:
Code:
(Request-Line) POST /mainarea/login/ HTTP/1.1
Host <website>
User-Agent Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-gb,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 115
Connection keep-alive
Referer http://<website>/shared/
Cookie clientid=84c4g8ed09bdad579b4789387a00a9f67aa2b19b; csrftoken=82859e53de1351b2a1d2bg3735eb7fb6; __utma=238191684.645653426.1268514417.1282736373.1282747727.315; __utmz=238191684.1268514417.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __gads=ID=672d76eb7da6f90b:T=1275291231:S=ALNI_MZeUQst6f35XiDSeG2HQogX60STsQ; __utmc=238191684; sessionid=b7b97f114649dra2ce33ece21b7ea970; __utmb=238191684.17.10.1282747727
Content-Type application/x-www-form-urlencoded
Content-Length 89
Which then 302 redirects to the main entry page. It doesn't look like it's sending the username and password as anything other than plaintext? Or am I not following
Even Wireshark shows its being sent like this:
csrfmiddlewaretoken=82859e53de1351b2a1d2bg3735eb7fb6&username=<myusername>&password=<mypass>
Perhaps I need to send a content length? Edit: Didn't seem to help sending a content length. Still 403 error.
Last edited by Lucky75; August 25th, 2010 at 10:33 AM.
-
August 26th, 2010, 01:20 AM
#8
Re: Attempting to log into website from java
-
August 26th, 2010, 06:50 AM
#9
Re: Attempting to log into website from java
What was the difference between what your program sends and what a browser sends?
Norm
-
August 26th, 2010, 08:31 AM
#10
Re: Attempting to log into website from java
Not sure I follow what you're asking. You can see what is being sent above in the code. I'm sending the content type, user agent and content length. I'm not sending the other header options, but I don't think I should need to.
It might have something to do with the session ID/client ID or cookies or something, but I shouldn't need to send any of that if I'm logging in for the "first time", should I?
-
August 26th, 2010, 08:44 AM
#11
Re: Attempting to log into website from java
Not sure I follow what you're asking. You can see what is being sent above
I'm assuming that the HTML form is correctly sent by a browser and it gets a response. So if you look at what the browser sends to the server and compare that against what your program sends to the server, perhaps you'll see a difference.
You've only posted what your code sends. What does the browser send?
Norm
-
August 26th, 2010, 09:20 AM
#12
Re: Attempting to log into website from java
Originally Posted by Norm
I'm assuming that the HTML form is correctly sent by a browser and it gets a response. So if you look at what the browser sends to the server and compare that against what your program sends to the server, perhaps you'll see a difference.
You've only posted what your code sends. What does the browser send?
No, I posted both. Post #7 is captured from httpFox through the browser. The code is what I'm intending to send through the program in post #1.
It looks like it's sending the same POST data (aside from that middlewaretoken thing, although I did add that on as well, hardcoded). I'm also sending what I think are the necessary headers.
Thanks for the help
-
August 26th, 2010, 09:36 AM
#13
Re: Attempting to log into website from java
Is the middleware token some kind of nonce? Does it change when you re-negotiate a session or it becomes invalidated?
------
If you are satisfied with the responses, add to the user's rep!
-
August 26th, 2010, 10:04 AM
#14
Re: Attempting to log into website from java
The code is what I'm intending to send through the program in post #1.
You don't show what the code actually sends to the server to be able to compare it to what the browser sent shown in post#7
Norm
-
August 26th, 2010, 10:30 AM
#15
Re: Attempting to log into website from java
The middleware token looks like it expires in a year, so I don't think it's changing frequently. Doesn't seem to, anyway.
Here is the data that the program is sending, captured from wireshark:
Code:
POST /mainarea/login/ HTTP/1.1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
Content-Length: 89
Host: <website>
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
csrfmiddlewaretoken=82859e53de1351b2a1d2bg3735eb7fb6&username=<myusername>&password=<mypass>
Looks pretty similar...
---------------------------------------------------
EDIT: AHA! Apparently I needed to send the cookies as well. I went through firefox and deleted all the cookies, and then I got a 403 in my browser also. I refreshed the page and tried again, and the csrfmiddlewaretoken had changed, whereas before it was always the same.
Now that I can log in, how can I get data from the other pages? Do I need to look at the response manually and save any session IDs, and then send them off with new requests again or something? It's weird, because I'm trying to do the same thing except use GET to fetch a different page, and I'm sending the sessionID and csrfmiddlewaretoken, but I get a 403 for the second page now. First page responds fine.
Thanks!
Last edited by Lucky75; August 26th, 2010 at 12:15 PM.
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|