PHP-based Parser and mySQL-Database

Hello! Here a short abstract about the project. For detailed informations please have a look below on this project description website: - php based crawler (PHP 5.2.5) which can run directly by calling up the website (no ssh please!) - the crawler should parse the informations from the crawled websites and write them into a mySQL database - there are some rules if the crawler should write informations to the datase and? if not (but nothing complicate) Below you find a detailed project description. The space here is too small to do it here. :) ## Deliverables Detailed project description: Starting point for the crawler is the website <[login to view URL]> (it is in german language but this should not bother you) By the way: I think that you will see the same like me when you call up the link but if not you can find attached three screenshots which display it. In the first step all 100 displayed news have to be parsed and written into the db: For each news block: - headline: For the first news block the headline is "DAX-Vorschau: Konjunkturskepsis und Übernahmephantasien" -link: For the first news block the link is "<[login to view URL]>" -source: For the first news block the source is "Reuters Deutschland" -timepoint: To get the timepoint of a news block you must be a bit tricky. You can see the text "?Vor 9 Stunden?" in the first news block. This is german and? is translated "9 hours ago". To get the timestamp? just take the actual timestamp and substract 9 * 3600. Note: If it is only 1 hour it is "Vor 1 Stunde", so the last "n" is missing. With "n" it is the plural. Another german time unit is "Minuten", for example "Vor 34 Minuten". Yes, you are right: "Minuten" is minutes. ;) So for this example just take the actual timepoint and substract 34 * 60. Note: If it is only 1? minute it is "Vor 1 Minute", so the last "n" is missing. With "n" it is the plural. The last possible german time unit is a date (if the news is older than 24 hours, only a date is displayed). For example "?20.08.2010?" as you can find it at the second news block. Note: In german language the? date format is dd.mm.yyyy. So here it is august, 20th 2010.? So if there is a date displayed at a news block please find out the timestamp from that day, 12 o'clock? miday. In this example the timestamp (timepoint) would be 1261306800 -> mktime(12,0,0,08,20,2010) -art_text: The? art_text is the whole text you can find when crawling the site <[login to view URL]> Please remove html tags and, if possible, java scripts and else -amount: This is the number of news to the same topic. The first news block from our example has no amound value displayed. So this news is the only one to this topic. So here amount is 1. But have a look to the second news block. There you can find in green letters "Alle 340 Artikel". So here the amount is 340. If your crawler cannot find this part for a news block, amount is 1. -news_id: Every news block has an only id. You find it in the source code of the site. For the first news block the id is? ? 8797404495131. The tag your crawler must look for is "story cid-8797404495131 l-de". Note: Every news block has only one single id (cid). Undepending on the fact if it is one single news in a news block or 340 news in a news block like the second news block. Please crawl not only the main news of a news block, crawl all blue links (not the green ones). So here your crawler would follow only one link from the first news block because it contains only one news but it would follow 4 links in the second news block, 3 links in the third news block and so on. The news_id is for all news from the same news block similar. If all variables are parsed together, insert it into the database: insert into boerse (headline,link,source,timepoint,art_text,amount,news_id) values ('$headline','$link','$source','$timepoint','$art_text','$amount','$news_id') Now follows the next step: Follow the link in green that displays the amount. So, here in the example the first news block has not such a link. So let's take the second one. If you follow the link "Alle 340 Artikel" please add "&scoring=n" to the link. Now the complete link is <[login to view URL]> From all news on this site I need only the timepoint (you can get it as described above) and the id. You find the id in the source code. For example for the first news the id is "MAA4AEgAUABgAWoCZGU". Look in the source code for id="MAA4AEgAUABgAWoCZGU". Every news has a different id. Note: It may happen that there are more then one page to crawl. In this example all news find place on this site because it are less than 100. But it may be more that 100. To show what I mean have a look to this link: <[login to view URL]> At the end of the site you can find some more links to go on. So please follow all links. So, if you have put your twi variables together, just insert them into the database:? insert into entries_kum (id,timepoint) values ('$id','$timepoint') Now follows the last step which is similar to the last step. Please only change the last letter from the last link from "n" to "d". So you have <[login to view URL]> Now do the same like in the last step. The only difference is that here are more news listed. And here you can also see that at the end of the site your crawler can follow further links. Note: This also may happen on the site where there is a "n" and the end of the link and not a "d". Only in this example the site is large enough. And the database table has a bit different name: insert into entries_all (id,timepoint) values ('$id','$timepoint') If a news_id is already present in the database table, mySQL will not overwrite this entry so just drop an error message from mySql or PHP if one should appear. Okay, that's it! If you have further questions, just ask!