How to have c# application go to url and submit information returning string

[excalaber]

Hello everyone, I just want to take the time to thank all of you for your help over the past year. I have been lurking these forums from time to time, but only now have I become a registered poster

In any event, does anyone know how I can have my c# application go to a url (through a control or object possibly) and submit data into two textboxes and submit the information on the page. Then, the information requested would be returned to my application in the form of a string ready for parsing. I'm unsure of which control or object to use, possibly Http request object?

Thank you in advance, and I apologize if any of this was answered already in a different form. ~MikE~

[creatorul]

Well you have to work with HttpWebRequest and HttpWebResponse . The "string" ( more accurate the stream) will be in the HttpWebResponse :

StreamReader reader = newStreamReader(wresponse1.GetResponseStream());

From here on you can read each line with StreamReader's ReadLine() method.
Filling the boxes from the webpage is up to the page's format so it's up to you to find how to fill those ones.
For example google is sending the url request like this:

http://www.google.ro/search?hl=ro&q=cars&meta=

where "cars" is the string filled by me in the search box.

[excalaber]

Thank you very much for your hasty response! Currently I have a console application that returns all the html of the website that I am trying to work with. The only problem I am actually having is that the form on this website is using the POST method with the action name ending in no extension at all. I'm not sure how I can find what url request the site is using and/or the javascript used. The only javascript in the source I can see is for errors and loading...I don't see anything relevant to where I can find the routine submitting this information.

If it makes any difference, the Action after POST (for example) = Home if the url of the website was say http://www.mytesturl.com/Home

I can understand if it was Action = "myscript.cgi" or something of that nature.

Thanks again in advance! ~MikE~

[jss3426]

1. do you have the url of the website for which you do not know where it is posting to?

2. also you can use http request or you can use AJAX, atlas.net. That would do what you want to also. Check the AJAX programming thread

[excalaber]

Thanks again for another reply! Yes, I absolutely have the url of the website with the textboxes x 2 and submit button. After more research I find that it's using the post method. The only thing I think I could do is use an example such as this: http://www.netomatix.com/HttpPostData.aspx . Now the only problem is I am only able to use the version of .NET previous to 2.0 and it does have to be in c# for the time being. Is the aforementioned approach only available in .NET 2.0?

I'm a little confused as to the procedure I would use for this, but so far I had created an example that sent the data using method post to the website and then reading the data seems to be where I'm having trouble. If I get an Internal Error (500) in the response data, does that necessarily mean I successfully sent information to the site and submitted it (even though it might have been sent incorrectly) ?

The following is what I had after I sent the data to receive whatever comes back after submitting as a stream:

HttpWebResponse HttpWResp = (HttpWebResponse)myRequest.GetResponse();

//This is where I would think I'd put the code to work with httpWResp object.

StreamReader reader = new StreamReader (HttpWResp.GetResponseStream());

//My attempt at writing at least one line of the http response.

Console.Write(reader.ReadLine());

HttpWResp.Close();

newStream.Close();


Thanks again everyone.

[creatorul]

For HttWebRequest and HttpWebResponse

 
.NET Framework

Supported in: 3.0, 2.0, 1.1, 1.0

.NET Compact Framework

Supported in: 2.0, 1.0

All the other classes from that example are from .NET 1.0 or 1.1 .

So you don't have GET you have POST. Does the webpage you are trying to communicate with use cookies ?
There is a software that you should use in order to sniff all the client/server communication before you should proceed with your current development.
First you should sniff the usual communication using a web browser. Then use your application and see if it behaves the same ( see the header and check if all the variables are sent in the same way, also check if it is using coockies ).
You can get Ethereal from here
When you use it try to use a filter to display only the communication with your server.

[excalaber]

For HttWebRequest and HttpWebResponse

 
.NET Framework

Supported in: 3.0, 2.0, 1.1, 1.0

.NET Compact Framework

Supported in: 2.0, 1.0

All the other classes from that example are from .NET 1.0 or 1.1 .

So you don't have GET you have POST. Does the webpage you are trying to communicate with use cookies ?
There is a software that you should use in order to sniff all the client/server communication before you should proceed with your current development.
First you should sniff the usual communication using a web browser. Then use your application and see if it behaves the same ( see the header and check if all the variables are sent in the same way, also check if it is using coockies ).
You can get Ethereal from here
When you use it try to use a filter to display only the communication with your server.

Wow you guys are great, thank you so much! Ok so I obtained the correct post data now and I sent the variables the same way in my code as if I'd typed it in the website myself. My problem must be cookies, because when I'm capturing the packets I notice cookie sessions. What would be the next step from there?
The part with the cookie I notice says something along the lines of:

Cookie: JSESSIONID=00017WuOXwRhvWAf9Uqv_MF_NIe:10amel9h5...

Thanks again in advance! I'm really getting closer!

*EDIT* When I go to the website and enter the info, I notice the variables being sent after the POST HTTP/1.1 have (application/x-www-form-urlencoded) in the line. When I send the variables through the application (although it has application/x-www-form-urlencoded under content type on the drop down menus), there is no parenthesis with that after the POST HTTP/1.1. Also, when doing it through the website I can click the expandable and see the text being sent, wheras in my application there is no text being sent apparently.

Another difference I notice is that mine says expect: 100-continue after teh content length - 49, and the working version doesn't have that. Is there a reason it's waiting for a continue? I'm learning a lot in this thread, thanks again.

[creatorul]

This is an example of how to use cookies. I requested the website and sent the user and pass, then I kept all the cookies from the response and then requested the page I wanted from the website with the same CookieContainer.

CookieContainer cookies = newCookieContainer();

ServicePointManager.Expect100Continue = false;

mystr = "bla bla bla POST data gets here.. you should copy it from request header by using the browser and ethereal to sniff it";

data = Encoding.Default.GetBytes(mystr);

HttpWebRequest wr = (HttpWebRequest)WebRequest.Create("http://www.com/login.php");

wr.Accept = "*/*";

wr.Referer = "http://www.com/lalala/";

wr.ContentLength = data.Length;

wr.ContentType = "application/x-www-form-urlencoded";

wr.Method = "POST";

wr.CookieContainer = cookies;

wr.KeepAlive = true;

wr.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";

Stream newStream;

newStream = wr.GetRequestStream();

newStream.Write(data, 0, data.Length);

newStream.Close();

HttpWebResponse wresponse = (HttpWebResponse)wr.GetResponse();

CookieCollection ck = cookies.GetCookies(newUri("http://www.com/login.php"));

foreach (Cookie co in ck)

{

co.Path = "/";

cookies.Add(co);

}

 

HttpWebRequest wr1 = (HttpWebRequest)WebRequest.Create("http://www.com/lalala/");

wr1.Accept = "*/*";

wr1.Referer = "http://www.com/lalala/";

wr1.CookieContainer = cookies;

wr1.KeepAlive = true;

wr1.ContentType = "application/x-www-form-urlencoded";

wr.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";

HttpWebResponse wresponse1 = (HttpWebResponse)wr1.GetResponse();

StreamReader reader = newStreamReader(wresponse1.GetResponseStream());

 

string line;

while ((line = reader.ReadLine()) != null)
 
//read line by line ...
 

You can disable the 100-continue anyway.

ServicePointManager.Expect100Continue = false;

You should adapt the code above to your request/response session. You should modify the urls above and the rest of the datas by just sniffing the normal behavior of a login session using the browser.

[excalaber]

Thanks again creatorul! I modified the above code to work with my application and now I have the values successfully sending to the website (not sure if this had anything to do with cookies, I just noticed that the accept and some others were different as well). My problem now is that I notice upon going to the website and clicking submit, when sniffed the HTTP with the POST has cookie: JSESSIONID=00017WuOXwRhvWAf9Uqv_MF_NIe:10amel9h5\r\n

When I run it using the previous code (substituting the urls and the like) I don't have the cookie in the sniffer at all at the POST line.

Aside from a few checksum errors, other things that differ (not sure if this would be a problem):

Accept-Language: en-us\r\n
Accept-Encoding: gzip, deflate\r\n
Cache-Control: no-cache\r\n

Also, there is a GET line while in ethereal using the "cookie code". I'm assuming that is to obtain the cookie?

In addittion, if it helps at all, the response I'm getting back is the HTTP for the original site before I hit submit or anything.

Thanks again for your help, I've already come a long way (especially in learning a few things alone the way)

[creatorul]

Well it depends how you handle the request/response. I can't tell you exactly how to do because it is different from one site to another . If the site is using cookies .. one way is to request the page first and in the response you will get some cookies. ( basically the same like in the code above). Then you send another request ( now you have to send the datas from the fields that you want to fill) keeping the old cookies and the response will be what you want.
Accept-Language and the others don't matter you can leave them defaults.
I suppose you want to log into a website. Then you send the user and pass in the second request as described above.
You should follow the way the browser handles it and see what types of actions he does ( GET, POST).

[poochi]

One minor suggestion. You can use WebClient class instead of HttpWebRequest. WebClient class is a wrapper to the HttpWebRequest. It exposes most of the properties/settings the HttpWebRequest class wanted (ConnectionTimeOut is not exposed though), but it is very easy to use.

Sample Code:

WebClient webClient = new WebClient();
webClient.Headers.Add("Content-Type", "text/xml");
byte[] responseBytes = webClient.UpdateValues(Url, "POST", namevalueStringCollection);

// responseBytes = webClient.UploadData(Url, "POST", Encoding.ASCII.GetBytes(Data));

string response = Encoding.ASCII.GetString(responseBytes);

[excalaber]

Nice! Thank you everyone, I figured out what the problem was! The example that you gave me creatorul just needed to have the POST sent the second time after the cookies were obtained. Originally I left it how it was and I never sent back the POST method after I obtained the aformentioned cookies. I understand the whole process now and I see why it is necessary. Thanks again, I will be creating the parser and I'm done with this section. Take care.