Attempting to log into website from java

Hi!

I'm attempting to log into a website from java, but it doesn't seem to be working. I get:

Code:

---

Running the program...
Problems while reading the response
Problems encounterd.
java.io.IOException: Server returned HTTP response code: 403 for URL: <url of form>
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
        at KingdomScraper.readResponse(KingdomScraper.java:195)
        at KingdomScraper.httpPostLogin(KingdomScraper.java:48)
        at KingdomScraper.main(KingdomScraper.java:23)

---

Here's the code:

Code:

---

import java.net.*;
import java.io.*;



public class KingdomScraper {

        private static final String POST_CONTENT_TYPE = "application/x-www-form-urlencoded";
            private static final String LOGIN_USER_NAME_PARAMETER_NAME = "username";
            private static final String LOGIN_PASSWORD_PARAMETER_NAME = "password";
       
            private static final String LOGIN_USER_NAME = "myusername";
            private static final String LOGIN_PASSWORD = "mypass";
       
            private static final String TARGET_URL = "<url of form>";

        public static void main(String[] args)
        {
        System.out.println("Running the program...\n");
        KingdomScraper httpUrlBasicAuthentication = new KingdomScraper();
        httpUrlBasicAuthentication.httpPostLogin();

        }
       
          /**
    * The single public method of this class that
    * 1. Prepares a login message
    * 2. Makes the HTTP POST to the target URL
    * 3. Reads and returns the response
    *
    * @throws IOException
    * Any problems while doing the above.
    *
    */
    public void httpPostLogin ()
    {
        try
        {
            // Prepare the content to be written
            // throws UnsupportedEncodingException
            String urlEncodedContent = preparePostContent(LOGIN_USER_NAME, LOGIN_PASSWORD);
 
            System.out.println(urlEncodedContent);
            HttpURLConnection urlConnection = doHttpPost(TARGET_URL, urlEncodedContent);
 
            String response = readResponse(urlConnection);
 
            System.out.println("Successfully made the HTPP POST.");
            System.out.println("Recevied response is: '/n" + response + "'");
 
        }
        catch(IOException ioException)
        {
            System.out.println("Problems encounterd.");
        }
    }

    /**
    * Using the given username and password, and using the static string variables, prepare
    * the login message. Note that the username and password will encoded to the
    * UTF-8 standard.
    *
    * @param loginUserName
    * The user name for login
    *
    * @param loginPassword
    * The password for login
    *
    * @return
    * The complete login message that can be HTTP Posted
    *
    * @throws UnsupportedEncodingException
    * Any problems during URL encoding
    */
    private String preparePostContent(String loginUserName, String loginPassword) throws UnsupportedEncodingException
    {
        // Encode the user name and password to UTF-8 encoding standard
        // throws UnsupportedEncodingException
        String encodedLoginUserName = URLEncoder.encode(loginUserName, "UTF-8");
        String encodedLoginPassword = URLEncoder.encode(loginPassword, "UTF-8");
 
      /* String content = "login=" + LOGIN_ACTION_NAME +" &" + LOGIN_USER_NAME_PARAMETER_NAME +"="
        + encodedLoginUserName + "&" + LOGIN_PASSWORD_PARAMETER_NAME + "=" + encodedLoginPassword;*/
 
        String content = LOGIN_USER_NAME_PARAMETER_NAME +"="
        + encodedLoginUserName + "&" + LOGIN_PASSWORD_PARAMETER_NAME + "=" + encodedLoginPassword;
       
        return content;
 
    }
   
    /**
    * Makes a HTTP POST to the target URL by using an HttpURLConnection.
    *
    * @param targetUrl
    * The URL to which the HTTP POST is made.
    *
    * @param content
    * The contents which will be POSTed to the target URL.
    *
    * @return
    * The open URLConnection which can be used to read any response.
    *
    * @throws IOException
    */
    public HttpURLConnection doHttpPost(String targetUrl, String content) throws IOException
    {
        HttpURLConnection urlConnection = null;
        DataOutputStream dataOutputStream = null;
        try
        {
            // Open a connection to the target URL
            // throws IOException
            urlConnection = (HttpURLConnection)(new URL(targetUrl).openConnection());
 
            // Specifying that we intend to use this connection for input
            urlConnection.setDoInput(true);
 
            // Specifying that we intend to use this connection for output
            urlConnection.setDoOutput(true);
 
            // Specifying the content type of our post
            urlConnection.setRequestProperty("Content-Type", POST_CONTENT_TYPE);
            urlConnection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8");
 
            HttpURLConnection.setFollowRedirects(true);
            // Specifying the method of HTTP request which is POST
            // throws ProtocolException
            urlConnection.setRequestMethod("POST");
 
            // Prepare an output stream for writing data to the HTTP connection
            // throws IOException
            dataOutputStream = new DataOutputStream(urlConnection.getOutputStream());
 
            // throws IOException
            dataOutputStream.writeBytes(content);
            dataOutputStream.flush();
            dataOutputStream.close();
 
            return urlConnection;
        }
        catch(IOException ioException)
        {
            System.out.println("I/O problems while trying to do a HTTP post.");
            ioException.printStackTrace();
 
            // Good practice: clean up the connections and streams
            // to free up any resources if possible
            if (dataOutputStream != null)
            {
                try
                {
                    dataOutputStream.close();
                }
                catch(Throwable ignore)
                {
                    // Cannot do anything about problems while
                    // trying to clean up. Just ignore
                }
            }
            if (urlConnection != null)
            {
                urlConnection.disconnect();
            }
 
            // throw the exception so that the caller is aware that
            // there was some problems
            throw ioException;
        }
    }

    /**
    * Read response from the URL connection
    *
    * @param urlConnection
    * The URLConncetion from which the response will be read
    *
    * @return
    * The response read from the URLConnection
    *
    * @throws IOException
    * When problems encountered during reading the response from the
    * URLConnection.
    */
    private String readResponse(HttpURLConnection urlConnection) throws IOException
    {
 
        BufferedReader bufferedReader = null;
        try
        {
            // Prepare a reader to read the response from the URLConnection
            // throws IOException
            bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
            String responeLine;
 
            // Good Practice: Use StringBuilder in this case
            StringBuilder response = new StringBuilder();
 
            // Read until there is nothing left in the stream
            // throws IOException
            while ((responeLine = bufferedReader.readLine()) != null)
            {
                response.append(responeLine);
            }
 
            return response.toString();
        }
        catch(IOException ioException)
        {
            System.out.println("Problems while reading the response");
            ioException.printStackTrace();
 
            // throw the exception so that the caller is aware that
            // there was some problems
            throw ioException;
 
        }
        finally
        {
            // Good practice: clean up the connections and streams
            // to free up any resources if possible
            if (bufferedReader != null)
            {
                try
                {
                    // throws IOException
                    bufferedReader.close();
                }
                catch(Throwable ignore)
                {
                    // Cannot do much with exceptions doing clean up
                    // Ignoring all exceptions
                }
            }
 
        }
    }

   

}

---

Now, the form itself is odd. It seems to do a 302 redirect from what I can tell. It also seems to have a csrfmiddlewaretoken (cross site request forgery) being sent along with the username and pass in the POST header. Would that cause me problems?

Code:

---

<form method="post" action="">
    <div style='display:none'><input type='hidden' name='csrfmiddlewaretoken' value='82859e97af9452a2a1d3bf3731eb7fb6' /></div> <!--Whatever this is?Looks to be sent along with the username and password as POST data.-->
    <table>
        <tbody>
            <tr><th><label for="id_username">Username:</label></th><td><input id="id_username" type="text" name="username" maxlength="30" /></td></tr>
<tr><th><label for="id_password">Password:</label></th><td><input type="password" name="password" id="id_password" /></td></tr>
        </tbody>
    </table>
   
    <input type="submit" class="button" value="Log In" />

</form>

---

Any suggestions ? I'm rather stuck. Thanks!

Have you looked at HTML Error 403 ?

Computers are good at following instructions, but not at reading your mind...
Donald Knuth

Im aware of what a 403 is, but I don't know why I'm getting one. It doesn't even return a 403 when I enter the form data manually (with a wrong password), it just says "incorrect password" on the page. And I'm logging in with the correct username and password, so I don't know what's wrong.

bump

For starters you're doing the post wrong. You need to find out how the username and password are sent across the wire. They're not posted in plain text for all the world to see. Most common is a base 64 of

Code:

---

username + ":" + password

---

The http header portion when you're posting will contain this field if you inspect the standard post through firebug I believe:

Code:

---

Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==

---

Where the character string that followed 'Basic' is what I described above.

Do you have a proxy server that you can use for viewing what the browser is sending to a server?
Look at what the browser sends to the server when you submit the HTML form.

Then look at what your program sends.

I was using HTTPFox to check what was being sent. The post data is simply:

csrfmiddlewaretoken 82859e53de1351b2a1d2bg3735eb7fb6
username <myusername>
password <mypass>

Here are the headers being sent:

Code:

---

(Request-Line)        POST /mainarea/login/ HTTP/1.1
Host        <website>
User-Agent        Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
Accept        text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language        en-gb,en;q=0.5
Accept-Encoding        gzip,deflate
Accept-Charset        ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive        115
Connection        keep-alive
Referer        http://<website>/shared/
Cookie        clientid=84c4g8ed09bdad579b4789387a00a9f67aa2b19b; csrftoken=82859e53de1351b2a1d2bg3735eb7fb6; __utma=238191684.645653426.1268514417.1282736373.1282747727.315; __utmz=238191684.1268514417.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __gads=ID=672d76eb7da6f90b:T=1275291231:S=ALNI_MZeUQst6f35XiDSeG2HQogX60STsQ; __utmc=238191684; sessionid=b7b97f114649dra2ce33ece21b7ea970; __utmb=238191684.17.10.1282747727
Content-Type        application/x-www-form-urlencoded
Content-Length        89

---

Which then 302 redirects to the main entry page. It doesn't look like it's sending the username and password as anything other than plaintext? Or am I not following :)

Even Wireshark shows its being sent like this:
csrfmiddlewaretoken=82859e53de1351b2a1d2bg3735eb7fb6&username=<myusername>&password=<mypass>

Perhaps I need to send a content length? Edit: Didn't seem to help sending a content length. Still 403 error.

bump

What was the difference between what your program sends and what a browser sends?

Not sure I follow what you're asking. You can see what is being sent above in the code. I'm sending the content type, user agent and content length. I'm not sending the other header options, but I don't think I should need to.

It might have something to do with the session ID/client ID or cookies or something, but I shouldn't need to send any of that if I'm logging in for the "first time", should I?

Quote:

---

Not sure I follow what you're asking. You can see what is being sent above

---

I'm assuming that the HTML form is correctly sent by a browser and it gets a response. So if you look at what the browser sends to the server and compare that against what your program sends to the server, perhaps you'll see a difference.

You've only posted what your code sends. What does the browser send?

Quote:

---

Originally Posted by Norm

I'm assuming that the HTML form is correctly sent by a browser and it gets a response. So if you look at what the browser sends to the server and compare that against what your program sends to the server, perhaps you'll see a difference.

You've only posted what your code sends. What does the browser send?

---

No, I posted both. Post #7 is captured from httpFox through the browser. The code is what I'm intending to send through the program in post #1.

It looks like it's sending the same POST data (aside from that middlewaretoken thing, although I did add that on as well, hardcoded). I'm also sending what I think are the necessary headers.

Thanks for the help :)

Is the middleware token some kind of nonce? Does it change when you re-negotiate a session or it becomes invalidated?

Quote:

---

The code is what I'm intending to send through the program in post #1.

---

You don't show what the code actually sends to the server to be able to compare it to what the browser sent shown in post#7

The middleware token looks like it expires in a year, so I don't think it's changing frequently. Doesn't seem to, anyway.

Here is the data that the program is sending, captured from wireshark:

Code:

---

POST /mainarea/login/ HTTP/1.1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
Content-Length: 89
Host: <website>
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

csrfmiddlewaretoken=82859e53de1351b2a1d2bg3735eb7fb6&username=<myusername>&password=<mypass>

---

Looks pretty similar...

---------------------------------------------------
EDIT: AHA! Apparently I needed to send the cookies as well. I went through firefox and deleted all the cookies, and then I got a 403 in my browser also. I refreshed the page and tried again, and the csrfmiddlewaretoken had changed, whereas before it was always the same.

Now that I can log in, how can I get data from the other pages? Do I need to look at the response manually and save any session IDs, and then send them off with new requests again or something? It's weird, because I'm trying to do the same thing except use GET to fetch a different page, and I'm sending the sessionID and csrfmiddlewaretoken, but I get a 403 for the second page now. First page responds fine.

Thanks!

up

Not much I can recommend. Your code is trying to mimic a browser's actions. Do it in the browser, watch what happens and try to do the same in your code.

Yeah, that's what I've been doing. I've been comparing everything, but they look almost identical to me now :S

How do sessions work in general? So when I log in, the website will send me back a session as a cookie? And from then on, all I would need to do is send that cookie as a part of the header when making GET requests? Or is there something more to it?

That's all I know about it. Return the cookie the server gives you.

In general you should be passing the headers untouched back to you as they may have added things in there (such as your session id). As long as it's passed the next time, you should still be able to maintain authentication and shouldn't be receiving a 403.

Also, double check that the implementation of your http connection can handle passing cookies back. I got burned by this once when I was running unit/integration testing against a project of mine where I need to startup an embedded application server and verify that I can hit some rest methods, which must pass authentication first.

Using the jersey-apache-client did the trick (jersey-client.jar http connection does not do session management for you).

The way I had to use my implementations was something like this:

Code:

---

// ...
import com.sun.jersey.client.apache.ApacheHttpClient;
import com.sun.jersey.client.apache.config.DefaultApacheHttpClientConfig;
import com.sun.jersey.api.client.WebResource;
import com.sun.jersey.api.representation.Form;
import javax.ws.rs.core.MediaType;

public abstract class BaseTest {
    protected DefaultApacheHttpClientConfig config = new DefaultApacheHttpClientConfig();
    protected ApacheHttpClient client;

    // in some method
    public void init() {
      config.getProperties().put(DefaultApacheHttpClientConfig.PROPERTY_HANDLE_COOKIES, true);

        client = ApacheHttpClient.create(config);
        resource = client.resource("http://my.server.com/auth");
       
        Form form = new Form();

        // construct my form object and post ... in my case with this media type.
        String result = resource.accept(MediaType.APPLICATION_JSON).post(String.class, form);
    }
}

---