Eliminate package file save errors


When one request of download site is processing the package file is created and crawled site is downloaded to temporary files. When download site is raised (by queue or button) the mechanism have to do:

1/ Checks insufficient disk space on temporary directory and raise errors [TODO by developer]
2/ Checks correct directory structure and create it when is needed. [TODO by developer]
3/ Eliminate deprecated functions and remove it from ProcessPage() function [TODO by developer]

Before the work: please fork source code and make changes there.
Job range: only (P_HTTP.cs) class file
Afer the work: Clean unnecesarry comments and make a documentation of procedure inside your new function.

I'm waiting for you reply.
Good luck
Closed Mar 14, 2013 at 6:13 AM by altego


jackielhong wrote Feb 21, 2013 at 3:50 AM

I have installed VS 2012 successfully and looking into the code now. I will probably have some questions for you soon. Thanks, Jackie

jackielhong wrote Feb 22, 2013 at 5:49 AM

For #1 to check insufficient space, wouldn't the system automatically throw an exception when file exceeds the amount of temporary space allocated ? If so, couldn't we catch the exception and output the error ? Thanks, Jackie

altego wrote Feb 22, 2013 at 9:47 AM

Answer for question #1(jackielhong). I thing the exception could be at function CrawlerAfterDownload function. This is a good way to resolve it.

jackielhong wrote Feb 26, 2013 at 4:59 AM

Thanks for your reply. I will look into #2 and #3 as well before creating a fork for all the changes. Jackie

jackielhong wrote Feb 28, 2013 at 3:08 AM

I was thinking about #1 some more and have some more questions for you.
  1. Wanted to confirm that when you say temporary directory, you don't mean the one as specified in the System variable ? In looking at the code, the temporary directory is one specified by your app.config ?
  2. If we do throw the exception if out of disk space, any idea how we can test this ?
  3. For queue mechanism and if exception is thrown for out of disk space, did you want the program to end and output error to the log ? Will the program need to clean up the older temporary files and restart ?
  4. As I see the program remove the older temporary files at the start, do you anticipate the files to be quite large that the system would run out of disk space often? If so, as an alternate solution, we could perhaps check that current amount of temporary files will not exceed certain amount of disk space before creating new one ?
Thank you.

altego wrote Feb 28, 2013 at 7:19 AM

Answers below:
1/ Exactly configuration of temporary directory is located at app.config at section
<clear />
<add key="idc_access_dbcon_file" value="idc.accdb" />
<add key="idc_cache_dir" value="D:\tmp\idc" />
Another localization doesn't work yet :-).

2/ We have to make event named of out disk space. When the event will be raised then some function at the main form show messagebox with question "It will be not sufficient disk space. Do you want to clean older packages?". If clicked Yes the 80% container of packages will be deleted.

3/ At queue mechanism we will be configuring standard behavior of event out of disk space at control panel.

4/ The program at the start of the package removes only items that are not in the database. So for the example mechanism. It can serve as a base for another functions,

jackielhong wrote Mar 1, 2013 at 7:11 AM

I am now trying to add a site and see if I can check the site so that I can eventually test out the changes later on. When I do this nothing happens ? I see here is currently no code to call to doSelectedSiteTest ? Also, when I add a site, I don't see a record created in the SITES table. Should I ?

BTW: upon running the app, if we discover issues, do you want us to create an entry in Codeplex. Or are you in the middle of still fixing things ? For example, when I run the app, and do File-> Control Panel, it errors that "string was not recognized as a valid Boolean". Thanks.

altego wrote Mar 1, 2013 at 12:15 PM

I'll be grateful for the creation of new issues. In each one I will explain at what stage and what was happening. It will be a knowledge base.

altego wrote Mar 14, 2013 at 6:13 AM

This functionality is already redesigned.