A http GET request with cookies and referer won't send out?
My program needs to login to a website, and then get web page which only logined members can get.
First, I GET a login page for initial cookies (asp sessionId);
Then, I send a POST with proper data (to login) and cookies I got in step 1.
Last, I send a GET with the cookies I got in previous step to fetch the page which I really need.
The first and second step works pretty fine, but the last step failed. It Never send out a http GET message (oh, to be honest, sometimes, it sends out). I used HttpLook to make sure this.
here is the trace message:
Making Http Request: url=http://www.akaoyan.com/index_shiti_more.asp?action=&c_id=42&s_id=133&page=1, info==null:True. namespace Deerchao.Utility public static string Get(string url, GetPageInfo info) private static string DecodeData(HttpWebResponse w) public static string Get(string url) public static CookieCollection GetCookies(string url, GetPageInfo info) public static HttpWebRequest MakeRequest(string url, GetPageInfo info) HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url); if (info.PostData != null && info.PostData != string.Empty) public class GetPageInfo What could be wrong?
System.Net Verbose: 0 : [1772] WebRequest::Create(http://www.akaoyan.com/index_shiti_more.asp?action=&c_id=42&s_id=133&page=1)
System.Net Verbose: 0 : [1772] HttpWebRequest#10722382::HttpWebRequest(http://www.akaoyan.com/index_shiti_more.asp?action=&c_id=42&s_id=133&page=1#-940297018)
System.Net Verbose: 0 : [1772] Exiting HttpWebRequest#10722382::HttpWebRequest()
System.Net Verbose: 0 : [1772] Exiting WebRequest::Create() -> HttpWebRequest#10722382
quiting MakeRequest...
System.Net Verbose: 0 : [1772] HttpWebRequest#10722382::GetResponse()
System.Net Information: 0 : [1772] Associating HttpWebRequest#10722382 with ServicePoint#55634702
System.Net Information: 0 : [1772] Associating Connection#41745883 with HttpWebRequest#10722382
System.Net Information: 0 : [1772] Associating HttpWebRequest#10722382 with ConnectStream#32282393
System.Net Information: 0 : [1772] HttpWebRequest#10722382 - Request: GET /index_shiti_more.asp?action=&c_id=42&s_id=133&page=1 HTTP/1.1
“PickManager.vshost.exe”(?): ?“C:\WINDOWS\assembly\GAC_MSIL\System.resources\2.0.0.0_zh-CHS_b77a5c561934e089\System.resources.dll”,?
System.Net Information: 0 : [1772] ConnectStream#32282393 - ?
{
Accept-Language: zh-cn
Accept: */*
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
Host: www.akaoyan.com
Connection: Keep-Alive
}?
System.Net Information: 0 : [1772] Connection#41745883 - ?: Version=1.1,StatusCode=200,StatusDescription=OK?
System.Net Information: 0 : [1772] Connection#41745883 - ?
{
Content-Length: 5999
Content-Type: text/html
Date: Wed, 02 Aug 2006 16:01:51 GMT
Set-Cookie: LET%5Fonline=cookies%5Ftrue=yes; path=/,ASPSESSIONIDQQBTTTRD=MKLDPPCAMEBCKNBANJPGLJNM; path=/
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Cache-control: private
}?
System.Net Information: 0 : [1772] ConnectStream#38333420::ConnectStream(? 5999 ?)
System.Net Information: 0 : [1772] Associating HttpWebRequest#10722382 with ConnectStream#38333420
System.Net Information: 0 : [1772] Associating HttpWebRequest#10722382 with HttpWebResponse#66398263
System.Net Verbose: 0 : [1772] Exiting HttpWebRequest#10722382::GetResponse() -> HttpWebResponse#66398263
Cookies fetched.
Making Http Request: url=http://www.akaoyan.com/login.asp?action=login_chk, info==null:False.
System.Net Verbose: 0 : [1772] WebRequest::Create(http://www.akaoyan.com/login.asp?action=login_chk)
System.Net Verbose: 0 : [1772] HttpWebRequest#32289439::HttpWebRequest(http://www.akaoyan.com/login.asp?action=login_chk#1907320142)
System.Net Verbose: 0 : [1772] Exiting HttpWebRequest#32289439::HttpWebRequest()
System.Net Verbose: 0 : [1772] Exiting WebRequest::Create() -> HttpWebRequest#32289439
System.Net Verbose: 0 : [1772] HttpWebRequest#32289439::GetRequestStream()
System.Net Information: 0 : [1772] Associating HttpWebRequest#32289439 with ServicePoint#55634702
System.Net Information: 0 : [1772] Associating Connection#38678664 with HttpWebRequest#32289439
System.Net Information: 0 : [1772] Associating HttpWebRequest#32289439 with ConnectStream#16206357
System.Net Information: 0 : [1772] HttpWebRequest#32289439 - Request: POST /login.asp?action=login_chk HTTP/1.1
System.Net Information: 0 : [1772] ConnectStream#16206357 - ?
{
Accept-Language: zh-cn
Accept: */*
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
Referer: http://www.akaoyan.com/index_shiti_more.asp?action=&c_id=42&s_id=133&page=1
Content-Type: application/x-www-form-urlencoded
Host: www.akaoyan.com
Cookie: LET%5Fonline=cookies%5Ftrue=yes; ASPSESSIONIDQQBTTTRD=MKLDPPCAMEBCKNBANJPGLJNM
Content-Length: 65
Expect: 100-continue
}?
System.Net Information: 0 : [1772] Connection#38678664 - ?: Version=1.1,StatusCode=100,StatusDescription=Continue?
System.Net Information: 0 : [1772] Connection#38678664 - ?
{
}?
System.Net Verbose: 0 : [1772] Exiting HttpWebRequest#32289439::GetRequestStream() -> ConnectStream#16206357
System.Net Verbose: 0 : [1772] ConnectStream#16206357::Write()
System.Net Verbose: 0 : [1772] Data from ConnectStream#16206357::Write
System.Net Verbose: 0 : [1772] 00000000 : 75 73 65 72 6E 61 6D 65-3D 75 6E 69 62 65 74 74 : username=unibett
System.Net Verbose: 0 : [1772] 00000010 : 65 72 26 70 61 73 73 77-6F 72 64 3D 31 39 38 32 : er&password=1982
System.Net Verbose: 0 : [1772] 00000020 : 30 35 30 35 26 69 6D 67-5F 6C 6F 67 69 6E 2E 78 : 0505&img_login.x
System.Net Verbose: 0 : [1772] 00000030 : 3D 31 34 26 69 6D 67 5F-6C 6F 67 69 6E 2E 79 3D : =14&img_login.y=
System.Net Verbose: 0 : [1772] 00000040 : 34 : 4
System.Net Verbose: 0 : [1772] Exiting ConnectStream#16206357::Write()
System.Net Verbose: 0 : [1772] ConnectStream#16206357::Close()
System.Net Verbose: 0 : [1772] Exiting ConnectStream#16206357::Close()
System.Net Verbose: 0 : [1772] ConnectStream#16206357::Close()
System.Net Verbose: 0 : [1772] Exiting ConnectStream#16206357::Close()
quiting MakeRequest...
System.Net Verbose: 0 : [1772] HttpWebRequest#32289439::GetResponse()
System.Net Information: 0 : [1772] Connection#38678664 - ?: Version=1.1,StatusCode=200,StatusDescription=OK?
System.Net Information: 0 : [1772] Connection#38678664 - ?
{
Content-Length: 3074
Cache-Control: private
Content-Type: text/html
Date: Wed, 02 Aug 2006 16:01:51 GMT
Set-Cookie: LET%5Fonline=login%5Fpassword=11fdcf5dc03e6c0a&login%5Fid=&login%5Fusername=unibetter&time%5Fload=2006%2D8%2D3+0%3A01%3A46&guest%5Fname=%D3%CE%BF%CD50305687&cookies%5Ftrue=yes; path=/
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
}?
System.Net Information: 0 : [1772] ConnectStream#55914028::ConnectStream(? 3074 ?)
System.Net Information: 0 : [1772] Associating HttpWebRequest#32289439 with ConnectStream#55914028
System.Net Information: 0 : [1772] Associating HttpWebRequest#32289439 with HttpWebResponse#55432815
System.Net Verbose: 0 : [1772] Exiting HttpWebRequest#32289439::GetResponse() -> HttpWebResponse#55432815
Cookies fetched.
Making Http Request: url=http://www.akaoyan.com/index_shiti_more.asp?action=&c_id=42&s_id=133&page=1, info==null:False.
System.Net Verbose: 0 : [1772] WebRequest::Create(http://www.akaoyan.com/index_shiti_more.asp?action=&c_id=42&s_id=133&page=1)
System.Net Verbose: 0 : [1772] HttpWebRequest#31853378::HttpWebRequest(http://www.akaoyan.com/index_shiti_more.asp?action=&c_id=42&s_id=133&page=1#-940297018)
System.Net Verbose: 0 : [1772] Exiting HttpWebRequest#31853378::HttpWebRequest()
System.Net Verbose: 0 : [1772] Exiting WebRequest::Create() -> HttpWebRequest#31853378
quiting MakeRequest...
System.Net Verbose: 0 : [1772] HttpWebRequest#31853378::GetResponse()
System.Net Information: 0 : [1772] Associating HttpWebRequest#31853378 with ServicePoint#55634702
System.Net Information: 0 : [1772] Associating Connection#41745883 with HttpWebRequest#31853378
System.Net Verbose: 0 : [3816] HttpWebRequest#31853378::Abort(?)
System.Net Error: 0 : [3816] Exception in the HttpWebRequest#31853378:: - ?
System.Net Error: 0 : [1772] Exception in the HttpWebRequest#31853378::EndGetResponse - ?
The blue texts are frommy own code, the green ones are fromstep1, brown onesstep2, red onesstep3.
("
[code=c#]
using System;
using System.Net;
using System.IO;
using System.Text;
using System.Diagnostics;
{
public class WebPageGetter
{
private static string defaultEncoding = "gb2312";
private static string defaultLanguage = "zh-cn";
{
int timesTried = 0;
for(timesTried=0; timesTried<1; timesTried++)
{
//try
//{
timesTried++;
HttpWebRequest req = MakeRequest(url, info);
HttpWebResponse res = (HttpWebResponse)req.GetResponse();
return DecodeData(res);
//}
//catch
//{
// continue;
//}
}
return null;
}
{
Trace.Write("Entering DecodeData...\r\n");
string strEncoding = null;
string s = w.Headers["content-type"];
if (s != null)
{
int i = s.IndexOf("charset=");
if (i != -1)
{
strEncoding = s.Substring(i + 8);
}
}
MemoryStream ms = new MemoryStream();
byte[] buffer = new byte[0x400];
Stream rs = w.GetResponseStream();
for (int i = rs.Read(buffer, 0, buffer.Length); i > 0; i = rs.Read(buffer, 0, buffer.Length))
{
ms.Write(buffer, 0, i);
}
rs.Close();
if (strEncoding == null)
{
MemoryStream stream = ms;
stream.Seek((long)0, SeekOrigin.Begin);
StreamReader reader = new StreamReader(stream, Encoding.ASCII);
string all = reader.ReadToEnd();
if (all != null)
{
int i = all.IndexOf("charset=");
int j = -1;
if (i != -1)
{
j = all.IndexOf("\"", i);
if (j != -1)
{
int k = i + 8;
strEncoding = all.Substring(k, (j - k) + 1);
char[] chArray = new char[2] { '>', '"' };
strEncoding = strEncoding.TrimEnd(chArray);
}
}
}
}
Encoding encoding = null;
if (strEncoding == null)
{
encoding = Encoding.GetEncoding(defaultEncoding);
}
else
{
try
{
encoding = Encoding.GetEncoding(strEncoding);
}
catch
{
encoding = Encoding.GetEncoding(defaultEncoding);
}
}
ms.Seek((long)0, SeekOrigin.Begin);
StreamReader reader2 = new StreamReader(ms, encoding);
Trace.Write("Content fetched.\r\n");
return reader2.ReadToEnd();
}
{
return Get(url, null);
}
{
HttpWebRequest req = MakeRequest(url, info);
HttpWebResponse res = (HttpWebResponse)req.GetResponse();
Trace.Write("Cookies fetched.\r\n");
return res.Cookies;
}
{
Trace.Write(string.Format("Making Http Request: url={0}, info==null:{1}.\r\n", url, info==null));
if (url == null || url == string.Empty)
return null;
req.AllowAutoRedirect = true;
req.CookieContainer = new CookieContainer();
req.Headers.Add("Accept-Language", defaultLanguage);
req.Accept = "*/*";
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
req.KeepAlive = true;
if (info != null)
{
if (info.Cookies != null)
{
req.CookieContainer.Add(info.Cookies);
}
if (info.Referer != null && info.Referer != string.Empty)
req.Referer = info.Referer;
{
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.AllowWriteStreamBuffering = true;
req.Timeout = 8000;
//write post data
byte[] bt = System.Text.Encoding.ASCII.GetBytes(info.PostData);
req.ContentLength = bt.Length;
Stream rs = req.GetRequestStream();
StreamWriter sw = new StreamWriter(rs, Encoding.GetEncoding("ISO-8859-1"));
sw.Write(info.PostData);
sw.Close();
rs.Close();
}
}
Trace.Write("quiting MakeRequest...\r\n");
return req;
}
}
{
public CookieCollection Cookies = null;
public string PostData = "";
public string Referer = "";
}
}
[/code].

