1. Home

Product groups

Support

Can you scrape content with WodHttpDLX in Delphi - WeOnlyDo Discussion board

Can you scrape content with WodHttpDLX in Delphi (wodHttpDLX)

by buddy, Sunday, October 11, 2009, 22:17 (5317 days ago)

Hey,

I have just recently started messing with WodHttp and so far love what I see although im still unsure what capabilities it can do... Right now Im working on a project where I need to post items to a web site then scrape the information that is give back... I was curious if WodHttpDLX could do this or not?

I know it can post the info for as i have that part coded but was not sure if it could then scrape the content that is returned. I know that there is a .Get property however I think that is just for the HTML source but could be wrong...

Here is a small example of what I want it to do....
Go To tinyurl.com
Then post link...
then scrape the new url that tiny url has gathered...

Very simple example but again curious if wodhttp has the skills to get it done lol...

I am coding in Delphi 7 and Delphi 2009 either one is fine as I have both.

locked
  4117 views

Re: Can you scrape content with WodHttpDLX in Delp

by wodDamir, Sunday, October 11, 2009, 22:41 (5317 days ago) @ buddy

Hi Buddy,

When you do a POST, the component automatically retrieves the response returned by server.

However, since tinyurl.com returns the URL in a webpage, the response will contain whole HTML source of the page. You will need to parse response body in order to find the URL.

Hope this helps.

Regards,
Damba

locked
  3959 views

Re: Can you scrape content with WodHttpDLX in Delp

by buddy, Sunday, October 11, 2009, 23:33 (5317 days ago) @ wodDamir

Ah I see....

Im not sure if I quite understand when u say it automatically retrieves the response by the server? Could u explain or give an example of this?

As for the tinyurl i see what you mean... It would be best if i just take the source of the page and then trim it down to what i need exactly.

Lets say if I want to do the following...
Go to Craigslist.org and search for all Mazdas for sale and then scrape all ads for that... Would the Get function be able to do that or any other function of WodHttp?

Thanks all ur help is much appreciative :D

locked
  3949 views

Re: Can you scrape content with WodHttpDLX in Delp

by wodDamir, Sunday, October 11, 2009, 23:50 (5317 days ago) @ buddy

Buddy,

Post method works the same as Get. The only difference is that Post actually sends provided data, while Get only retrieves the requested page.

What I mean is when you call Post method, it's the same as if you pressed the Submit Button ( Make TinyURL! ). The url you provided is submitted to page, and the request is processed. After that, the server provides response, which you receive as HttpResponse object.

If you check Response at that point, you should have the page source in Response.Body property.

The same applies to the other sites. You need to submit your search criteria, so I assume you will need to use Post method. However, extracting the data you require from the response is up to you.

Regards,
Damba

locked
  3930 views

Re: Can you scrape content with WodHttpDLX in Delp

by buddy, Monday, October 12, 2009, 00:00 (5317 days ago) @ wodDamir

Ah now i fully understand.... LOL sorry im still new to coding in delphi as well... So sorry for so many questions and etc. But again I do appreciate how much u have helped me and being so patient with me as well.

Im glad if i have any questions I can come to u guys lol...

Thanks again

locked
  3965 views

Thank you very much for the rapid responses. I was a little nervous about dealing with a company that is on a different continent from me. You have proven my concerns to be unfounded.

Robert Pacheco
Surebridge, Inc.

I really like these products - they are consistently first rate!

David Witten
WWR Development, Inc.

Thank you very much for the rapid responses. I was a little nervous about dealing with a company that is on a different continent from me. You have proven my concerns to be unfounded.

Robert Pacheco
Surebridge, Inc.

I can only hope I will have the pleasure to work with other products by "We Only Do" in the future.

Tomer Spivak
Verint Systems Ltd

Your professionalism is reflected into your email, your product and your web site. I am confident that these will make the difference.

Sylvain Paquette
R&D Analyst at InfoPharm Inc.

The people at WeOnlyDo are amazing! ... I always get a super fast response from customer service, and the products are great too. Thanks WeOnlyDo

Jeff Hall
Occupational Health Research

Your .NET components allow us to blend .NET technology seamlessly with secure communication, and with excellent technical support.

Bil Bragg
Dionach

Not only wodSFTPdll is excellent, the service you have provided is outstanding and second to none!

Ian D. Mead, author of UltraEdit-32
IDM Computer Solutions

Thank you for the great customer service ... I am really impressed with the wodSSH ActiveX control.

Andrew Ross
Kiwi Enterprises

Thanks! We love your software and your great support!

Sandy Fliderman
Primary Development Corporation