Things that make you go "Hmmmm"

Wordpress php.ini settings to load external url

2011-11-22T10:33:00.000-08:00

A friend of mine had an issue with a vanilla wordpress install that he had that was using a premium theme.

Upon loading the theme, the page showed many errors relating to servers security setting would not allow the loading of data from external urls.

The information we were given was to set the php.ini in the directory of the calling page, so that the setting could be overridden - so we did, no change.

The solution is that the php.ini file in the wp-admin folder needs setting, not in the calling page, because the calling page in this case is a settings page itself that is being imported into the wp-admin/index.php file.

The settings in php.ini are then:

allow_url_fopen = 1
allow_url_include = 1

PHP .htaccess redirect

2011-05-04T06:35:00.000-07:00

I have been helping a neighbour of mine who is an Optician in Spalding, and he was trying to make sure that when people were searching for "Opticians in Spalding" they would make sure that the old pages on his website that were already listed in Google, would still link through to his new site that he's spent ages on.

Being a windows server man, I know how to do this in IIS, but as his new website is hosted on a linux server, it needed to be compatible with that. So anyway, I found out that you can do 301 redirects on Apache using a .htaccess file.

In this case, it was real simple, create a blank file called .htaccess and the place an entry for each for each file that needs redirecting, i.e:

Redirect 301 /HTML/Consultations.htm http://www.molsom.co.uk/eye-tests.htm

Excellent, and I now know that visitors to my local Optician in Spalding will still get there from any old bookmarks!

EventType clr20r3 system.io.filenotfoundexception

2011-03-01T08:27:00.000-08:00

I had this error today when trying to launch a .net application on a remote host machine. The file io exception threw me somewhat as it made me think that the issue was with loading a config file.

In fact the culprit of this error was not the config files, it was in fact missing dlls that are normally referenced from the GAC.

As I had no idea which dll it didn't know about, I just set each dlls reference to say copy to local = true, and then deployed the whole contents.

List.Sort() alphabetically

2010-12-22T04:47:00.000-08:00

I was faced with the challenge today of sorting a list of objects, where one of the properties was a string. My objects were lists of keywords, and I wanted to sort the keywords alphabetically.

I came across some example of lambda expressions, and found that rather than having to write complicated sort methods for my class, I could just do the following:


keywords.Sort((a,b) => String.Compare(a.keyword, b.keyword)):

Awesome, works a treat!

SQL Server could not spawn FRunCM thread. Check the SQL Server error log and the Windows event logs for information about possible related problems.

2010-11-24T01:08:00.000-08:00

I encountered this error this morning when my SQL Server instance wouldn't start after I made some connection changes yesterday. I had to make the changes because a needed an ODBC connection, and that would only seem to work if I configured it to use named pipes.

After checking the logs, I received the error "SQL Server could not spawn FRunCM thread. Check the SQL Server error log and the Windows event logs for information about possible related problems.".

So, I went back to undo each change I had made, and noticed that the last change I made was to enable VIA as a protocol for my SQL Server instance. Disabled that, and hey presto, SQL Server now starts fine.

The strange thing is that enabling VIA in my SQL Express instance didn't seem to cause the same issue on that server though.

asp.net checkbox modalpopupextender checkbox doesn't check or uncheck

2010-04-27T04:45:00.001-07:00

There is an apparent bug in asp.net that when you have a checkbox inside a panel that is acting as a modal popup with the modalpopupextender, that when you try to check or uncheck it, it doesn't work!

I found the following thread, and it seems that quite a few people have had this very same problem.

From reading many posts, it seems that this happens if you do not set the TargetControlID property of the extender correctly.

However, this becomes an issue if you have multiple links, all that show the same extender. To get round this, you can simply create a fake button:

asp.net button not posting back

2010-04-27T03:33:00.000-07:00

I have a modalpopup extender attached to a panel, inside which resides two asp.net buttons. Each of these had the normal "Onclick" attribute assigned to them. However, when they were clicked, no postback was occuring.

From some reading around, it appears that this is by design, and that the buttons are there for this to be closed.

So how do you get round this?

You can attach a javscript client side event that mimmicks the behaviour of what happens with asp anyway, by forcing the postback to take place.

Firstly, in the head of your page, define a function that ensures that the page is valid first, and then causes the postback to take place (the validation part is crucial for this to work.

function doPostback(sender,e)
{
if(Page_IsValid)
{
__doPostback(sender,e);
}
else
{
return false;
}

}

Now in your code behind, tell your buttons to call this:

btnSaveUserDetails.OnClientClick = "javascript:return doPostback('" + btnSaveUserDetails.UniqueID + "');";

It is important here that the uniqueID is used, as this ensures that any controls that are nested pass the right details to the postback function.

And ta da, they now work!

modalpopupextender event fires but does not show

2010-04-26T06:35:00.000-07:00

I lost half a day of my life to this silly mistake today.

I had on my page a panel, with a modal popupextender. When I clicked a link that ran a serverside mdlPopup.show() or a client side $('mdlPopup').show(), the events fired, but no modal popup showed.

The reason for this was because I had set the "visible" attribute of my panel to "false", rather than setting the style of it to "display:none"!

Silly....

The process cannot access the file because it is being used by another process. (Exception from HRESULT: 0x80070020)

2010-03-30T04:37:00.001-07:00

When starting IIS 7 under Vista or Windows 7, you receive the error from the IIS Manager saying "The process cannot access the file because it is being used by another process. (Exception from HRESULT: 0x80070020)"

The reason for this is well documented, and it is because you have another site setup with the same binding - in english, this means you have two sites setup that are configured to use the same port (probably 80).

In my circumstances, this was confusing because I had only one IIS instance and this was bound to port 80, but wouldn't start.

So, to find out what was conflicting, I went to the command prompt and typed:

NETSTAT -ano

This showed an entry of address 0.0.0.0 using port 80 up!

This also showed the process ID of 4400, so I looked in task manager and found that this was Skype.exe

After digging through the advanced settings in Skype, it appears that by default, Skype decides to listen on ports 80 and 443 for incoming connections (probably to appeas firewalls), so unticking this allowed my IIS to start.

Useful to find out, but it took a while!

ie don't print filename

2010-01-05T05:53:00.000-08:00

I needed to convert a webpage into a pdf today, and using PDFCreator, this is normally a doddle.

However, whenever I did, it printed the address to the footer of the page.

After much faffing around, I found that under the page setup are "headers" and "footers". If you delete what is in those fields, it stop ie and firefox printing the filenames and page numbers.

XPath select attributes containing a string

2010-01-04T05:25:00.001-08:00

I needed to find a way to use XPath to select items from within a html document, that would return back to me all UL tags that had an id that contained the word "menu".

To find all the UL's, it is simple enough to do:


//UL

However, to select the id of "menu", I then found:


//ul[@id='menu']

What I ended up with, to find any ULs that contained the word "menu" in the id attribute was:


//ul[contains(@id,'menu')]

Reset AutoIncrement Number in MySQL

2009-12-31T03:01:00.000-08:00

To reset your autoincrement number, you can simply run the following SQL:

ALTER TABLE 'table_name' AUTO_INCREMENT = X

Where X is the ID number you want to set the increment start point to, so running

ALTER TABLE 'table_name' AUTO_INCREMENT = 0

resets the ID increment process, and

ALTER TABLE 'table_name' AUTO_INCREMENT = 1023

Would set the start seed to 1023 - sometimes useful for making things looked "used", for example with order systems.

Making changes to overall_header.php in phpBB do not take effect

2009-12-31T02:40:00.001-08:00

I have been, for some time, trying to make changes to the code in the header file for phpBB. These were not changes for functionality, rather links required to other parts of the site.

Anyway, I had made the changes locally in my local copy of the overall_header.php file of the prosilver theme, but I ftp'd them up, the changes didn't happen.

Frustrated, I spent some time checking what phpBBs admin area said was in the file using the built in editor, and it said my changes were done.

However, what it doesn't say is that once you've made any changes to page files or style documents, you have to run the PURGE ALL feature from the admin control panel to flush out cached copies!

Ran that, and hey presto, fixed.

Person [user@example.com] on behalf of Another Person [user@otherexample.com]

2009-12-17T07:00:00.000-08:00

Today I had a query whereby a user was sending emails after outlook had been reinstalled, and was getting this in the emails he was sending, even if it was from his normal account.

The problem seemed to be that the default email being sent from was incorrect, so setting the correct mailbox cured it.

PHP C# WCF Rijndael

2009-11-17T07:58:00.000-08:00

I have been working with .NET and webservices for some time, but a colleague of mine has now moved over to using WCF instead of basic webservices.

After many, many problems dealing with simple soap message exchanges, we finally got our WCF service to talk to PHP (using NuSOAP) by setting the encoding to UTF8 and the binding to type basicHttpBinding.

My next problem was why when I was encoding data using Rijndael, was the encoded data returned from WCF not being decoded by mcrypt in PHP correctly.

Eventually, after some digging around, I found a really good article at http://benvanmol.blogspot.com/2009/10/secure-communication-between-net-and.html that summises the following points:

• You need to enable padding in the .net rijndael class. This ensures any blocks are padded with 0’s to fill them. This is done with Padding.Zeros
• Specifically set Cipher Block Chaining in the .Net class.
• The Initialisation Vector (IV) needs to be 16 bytes.
• You need to specifically set the encryption to the MD5 Hash algorithm, and change this to return the 32 character hash.
• Setup base64 encoding either end.

So, from this article by Ben van Mol I was able to produce a class from his code as follows:


public class PhpRijndael
    {
         System.Security.Cryptography.Rijndael r = null;

        public void InitializePhpRijndael(string iv, string key,int blockSize)
        {
            byte[] keyBytes = Encoding.ASCII.GetBytes(EncodeTo64(key));
            byte[] hash = MD5.Create().ComputeHash(keyBytes);
            string ret = "";
            foreach (byte a in hash)
            {
                ret += a.ToString("x2");
            }
            string iv64 = EncodeTo64(iv);
            byte[] ivBytes = Convert.FromBase64String(iv64);

            r = System.Security.Cryptography.Rijndael.Create();

            r.Padding = PaddingMode.Zeros;
            r.BlockSize = blockSize;
            r.Key = Encoding.ASCII.GetBytes(ret);
            r.IV = ivBytes;

        }

        public string Decrypt(string str)
        {
            byte[] encryptedBytes = Convert.FromBase64String(str);
            byte[] decryptedBytes = transformBytes(
                r.CreateDecryptor(), encryptedBytes);

            string plaintext = Encoding.ASCII.GetString(decryptedBytes);
            int idx = plaintext.IndexOf("\0");
            if (idx > -1)
                plaintext = plaintext.Substring(0, idx);

            return plaintext;
        }

        public string Encrypt(string plaintext)
        {
            byte[] plainBytes = Encoding.ASCII.GetBytes(plaintext);
            byte[] encryptedBytes = transformBytes(
                r.CreateEncryptor(), plainBytes);

            return Convert.ToBase64String(encryptedBytes);
        }

        private byte[] transformBytes(ICryptoTransform transform,
            byte[] plainBytes)
        {
            MemoryStream memStream = new MemoryStream();
            CryptoStream cryptStream =
                new CryptoStream(memStream, transform,
                CryptoStreamMode.Write);
            cryptStream.Write(plainBytes, 0, plainBytes.Length);
            cryptStream.Close();
            byte[] encryptedBytes = memStream.ToArray();
            memStream.Close();

            return encryptedBytes;
        }

        private string EncodeTo64(string toEncode)
        {
            byte[] toEncodeAsBytes = System.Text.ASCIIEncoding.ASCII.GetBytes(toEncode);
            string returnValue = System.Convert.ToBase64String(toEncodeAsBytes);

            return returnValue;

        }
    }

In WCF I expose this in my endpoint as:


private string passPhrase = "secret";                
private string initVector = "ujkrtadxcfrzpj1Bs5fpM18doZQDGYS4";        
private int keySize = 256;

public string EncryptData(string value)
        {
            
           PhpRijndael r = new PhpRijndael();
           r.InitializePhpRijndael(initVector, passPhrase, keySize);
           string ret = "";
           ret = r.Encrypt(value);
           return ret;
        }
        public string ReturnTestData()
        {
            return EncryptData("Test Encrypted Data");
        }

Note: Both of these are expose on the interface as OperationContracts inside the ServiceContract.

Now, in the PHP world, we need to create a page to call this, so inside a normal html form, we do the following:


if( isset($_POST['Submit']) )
{
//Show me all errors please
error_reporting(E_ALL);
ini_set('display_errors', '1');
//load nusoap
require_once('../lib/nusoap.php');
//client config to CZ
$client = new nusoap_client('http://example/Service?wsdl',true);
$client->setEndpoint("http://example/Service/EndpointName");
$client->setUseCurl(0);
//set utf-8, or else it breaks.
$client->soap_defencoding='UTF-8';
$client->setDebugLevel( 1 );
//any errors?
$err = $client->getError();
if ($err) {
 echo 'Constructor error
' . $err . '
';
 echo 'Debug
' . htmlspecialchars($client->getDebug(), ENT_QUOTES) . '
';
 exit();
}
// This is a paramter list that can be used later, but not used at the moment
$params = array(
    'value' => "666"
);
//Call dummy method for some test info
$result = $client->call('ReturnTestData',$params);
//is there a fault?
if ($client->fault) {
 echo 'Fault
'; print_r($result); echo '
';
} else {
 $err = $client->getError();
 if ($err) {
  echo 'Error
' . $err . '
';
 } else {
  echo 'Result
'; print_r($result); echo '
';
 }
}

//decode the base sixty four encoded data returned 
$bsf = base64_decode(trim($result["ReturnTestDataResult"]));

//setup auth
$key = 'secret';   
$iv='ujkrtadxcfrzpj1Bs5fpM18doZQDGYS4';  

//a function to configure Rijndael from the afforementioned article 
function init_rijndael ($key,$iv) {   
 $rj = mcrypt_module_open('rijndael-256', '', 'cbc', '');   
 if ($rj !== FALSE)   
 {   
   $expected_key_size = mcrypt_enc_get_key_size($rj);   
     
   $key = substr(md5(base64_encode($key)), 0, $expected_key_size);   
      
   mcrypt_generic_init($rj, $key, $iv);   
 }   
 return $rj;   
}
//setup Rijndael
$rj = init_rijndael($key,$iv);   
  
//decrypt result
$decryptedResult = mdecrypt_generic($rj, $bsf);
//print out.
echo 'Result
'; echo($decryptedResult); echo '
';  
}

And hey presto - finally! a working php to WCF communication in Rijndael!

203.117.91.73 sdra64.exe

2009-11-06T04:34:00.000-08:00

My eset firewall started complaining last night about connections going outbound to the IP address 203.117.91.73

After a call to ESET, I found out that that I had a trojan virus called sdra64.exe on my machine!

How do you get rid of it?

The guy from ESET firstly check for the virus using a program called RootAlyzer. This confirmed the infection. Next he downloaded procexp.exe from Microsoft to see what dependancies sdra64.exe had. Using another program called Process Unloader, he then closed the handles to sdra64.exe, and all files located in c:\windows\system32\lowsec. Once this was done, he created blank executable from notepad, and overwrote sdra64.exe, then deleted the lowsec folder.

Finally, in the registry under windowsnt\winlogon, he removed the reference to sdra64.exe in the userinit key.

A nasty little beast of a virus, and apparently acquired randomly from visiting a website!

Error: CryptCertStrToNameW failed => 0x80092023 (-2146885597)

2009-10-22T05:14:00.000-07:00

I was following a guide today on WCF and using makecert to create self signed certificates e.g. makecert –n "CN-FakeRootCA” –r –sv FakeRootCA.pvk FakeRootCA.cer

Whenever I tried to create the certificate, I got the error
Error: CryptCertStrToNameW failed => 0x80092023 (-2146885597)

The reason for this was the version of makecert being used was an old version. Even though I was running it from the visual studio 2008 command prompt, the environment mappings were causing it to run version 5.1, instead of version 6.

Using windows search I found that the latest version is at:
C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin

So you can either makecert from this directory, or alter you path variables.

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.FormatException: Invalid character

2009-10-16T06:46:00.000-07:00

I started receiving these errors a couple of weeks ago, very randomly, and always reporting a fault with ScriptResource.axd and a strange parameter being passed as "d" as a querystring to it.

After some reading around, I eventually came across the following thread that was dicussing the issue.

https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=434997

The end of the thread, dated 12/10/09 states that this is a bug in IE8 that is hopefully going to be fixed soon, and that it doesn't affect the end users view of the site.

ASP.NET Ajax document.getElementById is null or not an object

2009-06-29T01:12:00.000-07:00

This has taken me three days to figure out. For what seemed like absolutely no reason, my ajax enabled application just stopped working. I went back through all my changes to see what it could have possibly have been to make the application start throwing the javascript document.getElementById('') is null or not an object error, and specifically, why it was doing this on document.getElementById('HEAD[0]').

I eventually found a secton of html that I had commented out with html comments, and I had included in this an asp.net ajax tag.

Removing these comments fixed the problem!

SQL Server Cannot resolve collation conflict for equal to operation.

2009-06-24T08:21:00.000-07:00

Today I migrated an exisiting set of tables from one database, to a new database, to isolate their work.

After copying all the stored procs, views, tables and data, one of my stored procs wouldn't run.

Everytime I tried to execute it, I got the error:

"Cannot resolve collation conflict for equal to operation."

It transpires that this error can be caused by many different problems, butt he main one being that the collation types for the columns being joined in the sql statement is different.

In my instance this was not the case, but by telling the columns to use the databases default collation on either side of the join fixed the problem - e.g.

SELECT table1.*, table2.* FROM table1
INNER JOIN table2 ON table1.column1 COLLATE DATABASE_DEFAULT = table2.column1 COLLATE DATABASE_DEFAULT

CompareValidator doesn't fire

2009-06-11T04:38:00.000-07:00

I have an application where I am using a CompareValidator on a field that is acting as a CAPTCHA, therefore I am comparing the value that is entered.

When testing the page, I was clicking the submit button on the form, before entering any text, but when I did this, it was submitting the form! Why, when the value couldn't possibly be the same?

The reason for it is the depths of the AJAX js files:


function CompareValidatorEvaluateIsValid(val) {
    var value = ValidatorGetValue(val.controltovalidate);
    if (ValidatorTrim(value).length == 0)
        return true;
    var compareTo = "";
    if ((typeof(val.controltocompare) != "string") ||
        (typeof(document.getElementById(val.controltocompare)) == "undefined") ||
        (null == document.getElementById(val.controltocompare))) {
        if (typeof(val.valuetocompare) == "string") {
            compareTo = val.valuetocompare;
        }
    }
    else {
        compareTo = ValidatorGetValue(val.controltocompare);
    }
    var operator = "Equal";
    if (typeof(val.operator) == "string") {
        operator = val.operator;
    }
    return ValidatorCompare(value, compareTo, operator, val);
}

The following line is the culprit:


if (ValidatorTrim(value).length == 0)
        return true;

If the length of the information input into the box being validated is 0, i.e. nothing entered, then this validates as true. This means that if you compare something to nothing, that is valid.... stupid, but valid.

The solution to this is to add a RequiredFieldValidator as well, which then forces the length to be greater than 0, an thus makes it all work

Compiler Error Message: CS0115: 'ASP.pagename_aspx.GetTypeHashCode()': no suitable method found to override

2009-06-04T07:20:00.001-07:00

I came accross this problem today when converting a set of application pages into master pages.

My initial problem was that I had set CodeFile attribute of the @Page directive, rather than the CodeBehind attribute.

The second problem was that my page was inheriting from the Masterpage, rather than from System.Web.Ui.Page.

When I fixed these it worked.

It essence, if you get this error, check out what, and where your page is inheriting from.

Outlook 2007 Email Headers

2009-06-02T06:49:00.000-07:00

I keep forgetting where to find this one....

When you wish to see the header, right click on the message in your inbox and choose "Message Options". The window that pops up has the headers showing in it.

TinyMCE ListBox Plugin for Merge Tags

2009-05-19T09:14:00.000-07:00

Well, today I finally managed to cobble together the following solution for a problem I had whereby I wanted a plugin for TinyMCE whereby I could select a value from a drop down / listbox, and have it inserted into TinyMCE.

So here's how I did it:

First define a javascript array BEFORE initialising the TinyMCE editor. This allows you to create this array server side if you want (by writing a response stream declaring it if necessary), or allows you to control it per page. E.g:

var arrUserTags = new Array('{{Name}}','{{Address}}','{{Company}}');

I created a new plugin in tiny_mce/plugins/usertags/editor_plugin.js as follows:


(function() {
tinymce.create('tinymce.plugins.usertags', {  
init : function(ed, url) {
},  
createControl: function(n, cm) {      
switch (n) {          
case 'usertags':              
var mlb = cm.createListBox('usertags', {                   
title : 'Insert details    ',                   
onselect : function(v) {                        
tinyMCE.activeEditor.selection.setContent(v);                   
}              
});
        
//The array arrUserTags, must be defined before the editor is added!!! Or if you prefer, just ignore the array idea, and just do multiple mlb.add() statements.                             
for(i=0;i

This you can then register as a plugin in tinymce.

CompareValidator doesn't compare numeric amounts correctly

2009-05-19T09:12:00.000-07:00

Well at least that is what I thought today!

I had a compare validator set on two text boxes that were used to enter monetary amounts. The validator was to ensure that box B was greater than Box A.

In this scenario, it worked:

Box A = 100, Box B = 200 = Fine

But...

Box A = 500, Box B = 1000 = Invalid

Why? Because I hadn't set the TYPE attribute of the validator to CURRENCY

D'oh!

TinyMCE UpdatePanel Javascript errors

2009-05-14T05:06:00.001-07:00

I have spent days on this one, and it took myself and a colleague both working on it to find a solution that actually worked!

Here's the scenario.... you want to use TinyMCE, and have it working in an UpdatePanel. It sounds simple enough.

What I did was add the TinyMCE editor script, setup my tinyMCE.init() method, and hey presto, I got my editors. I set my code behind for my save button, and it fired fine when pressed, however, the editor disappeared! So, I moved my init function into the code behind, and registered it with the scriptmanager.registerclientscript method. This was fine, and when svaing my content, the editor did not disappear anymore - however, when I clicked the save button for a second time, I got a javascript error on the editor trying to access it's content.

The specific part of the javascript from tiny_mce.js was erroring on "return this.bodyElementthis.getDoc().body"

Anyway, I searched through many examples of how to solve this, including remove the editors before performing the postback, and eventually found a downloadable example that worked:

http://codeodyssey.com/archive/2007/7/18/updatepanel-tinymce-demo-with-project-zip-file

However, when I tried to use this in my application, with TinyMCE version 3.2.3, it didn't work. I copied over the version from this download, and then it seemed to fix the problem - but I lost some of the TinyMCE functionality!

Eventually, my colleague stumbled upon this absoluetly brilliant .NET extender, that handles everything for you, and allows you customise each editor independantly:

http://weblogs.asp.net/cjdevos/archive/2008/06/19/ajax-extender-for-tinymce-continued.aspx

So, if your having problems with TinyMCE and updatepanels - download it and use it!

Math.Log in VBScript

2009-04-15T03:12:00.000-07:00

Today I was working on an old Classic ASP project, and I was converting some JavaScript code to ASP.

Part of the code used the JavaScript Math.Pow function, e.g. Math.Pow(4,3), and I needed to do this in classic ASP.

I found that the ^ operator in VBScript does this, but it wasn't easy to transpose into my code, so I just added the following function to the page:

FUNCTION Pow(val, val2)
Pow = val ^ val2
END FUNCTION

Added called it as needed, e.g. Pow(4,3)

SWFObject and LightBox2 - IE Errors

2009-02-12T06:34:00.000-08:00

Today a colleague of mine had a problem whereby he was trying to use LightBox 2 in a webpage, but on that page, we received the "Internet explorer cannot open the internet site XXX Operation Aborted" message, and the page doesn't load.

After a lot of debugging, nothing was apparent, and it didn't happen in Firefox.

After reading around a bit, it seems that IE has a few issues with DOM commands, and as such, the SWFObject and Lightbox2 scripts both try to access the DOM at the same time.

So, all I did was move the SWFObject script to the foot of the page, and keep the div that it is written into in its normal place.

Sorted.

Wireless stops working after XP SP3 installation

2009-01-05T08:58:00.000-08:00

A friend brought her laptop round to mine yesterday, as after xp had done an update and installed xp sp3, the wireless was showing as connected, but no page would load. When wired in, all was fine.

I tried connecting it to my WPA enabled network, and all was fine. After googlin the problem, I found that there were some issues that some people had had with WPA after xp sp3, but not with WEP - in fact, they were recommending switching to WEP (Not a good idea).

Anyway, I did the alternative, changed their laptop and belkin router over to wpa2 psk, and it's fine now!

Javascript parseInt, parseFloat, decide on data type

2008-12-17T08:13:00.000-08:00

I was facing an issue today whereby I need to accept a value from an input control, but not knowing what it contained - i.e. a number, a floating point number (decimal) or a string, but I needed to perform maths on it.

With some help of some friends over at forums.asp.net, I came up with the following. You pass it your value, and it will send back the right data type to use on mathematical operations like "+" which normally concatenates strings instead of adding numbers as strings.

function parseValues(aValue)
{
if(isNaN(aValue))
{
return aValue;
}
else
{
if(aValue.indexOf(".")>0)
{
return parseFloat(aValue);
}
else
{
return parseInt(aValue);
}
}
}

ScriptResource.axd - if(!a._events) error

2008-12-10T03:56:00.000-08:00

I was getting this error today, for some unknown reason, and obviously debugging it is of no use, as the brightspark that declared everything as "a", "b" or "c" obviously didn't want us to!

Anyway, after some searching around on my page, I found that a modalpopupextender that I have added to my page, had an invalidad cancelcontrolid, I had mis-spelled it, and therefore this error must be because "a" in this instance was the control that was to be used to close (cancel) the popup.

The joys of using other peoples code....

ASP Buttons not firing events

2008-11-06T07:03:00.000-08:00

How stupid did I feel today? Very, that's what.

In my perfectly working application, I add a new tab to the tabcontainer, and add some link buttons in it, but when running, the buttons would not work - they wouldn't run the code behind - I was completely perplexed as to why this was happening.

Eventually a colleague said to me "are you using the CausesValidation" - AH!!! Somewhere else on the application there is a requiredfieldvalidator, and that was telling the page that navigation couldn't take place, but the field was required - but in this case, it was allowed to happen. So, setting the property of CausesValidation = false, solved the problem.

Sothink DHTML fails to install on DHTMLmenu.dll

2008-11-05T03:39:00.000-08:00

I've had this problem twice now, and here's how I got round it to make it work under win xp sp2.

Start the installation, when it fails to register the file, click ignore. The installation will complete, but the program won't work.

Start the uninstall for it, again it will hang. Go into task manager, and kill the uninstall process. A temp file is produced, so kill that process also (it's the only one beginning with _tmp).

Do the install again, and it works!

AJAX NumericUpDown extender hides textbox

2008-10-20T02:50:00.000-07:00

I couldn't for the life of me figure out why my nicely styled textbox was being hidden when I added an numeric up down extender to it.

The reason, you have to explicitly add a width attribute to the extender so that it knows how big the parent box is!

Comparing two files, one on a hard drive, and one from a web site

2008-09-19T02:11:00.001-07:00

In my quest to create an effective and lean search engine service for the company I work for, I used to crawl all of our websites and download every file from them, and write them to a local cache. When the cache was detected as updated, my indexing system (based on Lucene.Net) detects the changes, and reindexes those sites.

You may well be able to imagine, that with over 800 sites, this is an awfully large amount of data to writing to the cache, and then an even bigger job for the index to reindex all these files!

So, what I wanted was a mechanism whereby I could compare the local copy, to a streamed copy of the file from the server into the local memory, and see there were any changes.

To do this, I immediate thought of looking for something like a checksum check, but a colleague recommended looking at doing a MD5 hash on them, using a tool like CipherLite by Obivex. Looking into this, i found an easier way, by using a webrespone to download a memory stream, opening the local file using a filestream, and performing hash on them to see whether the contents were the same:

//get a memory stream to hold the data that is downloaded
MemoryStream msFile = new MemoryStream();
writer = new BinaryWriter(msFile);
byte[] RecvBuffer = new byte[10240];
int nBytes, nTotalBytes = 0;
// loop to receive response buffer
while((nBytes = response.socket.Receive(RecvBuffer, 0, 10240, SocketFlags.None)) > 0)
{
// increment total received bytes
nTotalBytes += nBytes;
// write received buffer to file
writer.Write(RecvBuffer, 0, nBytes);
// check if the uri type not binary to can be parsed for refs
if(bParse == true)
// add received buffer to response string
strResponse += Encoding.ASCII.GetString(RecvBuffer, 0, nBytes);
// update view text
// check if connection Keep-Alive to can break the loop if response completed
if(response.KeepAlive && nTotalBytes >= response.ContentLength && response.ContentLength > 0)
break;
}
bool bContinue = false;
FileStream fStream = null;
try
{
//check to see if the file exists on the local file system
if(File.Exists(PathName))
{
//open the file, and read in the stream
fStream = File.Open(PathName, FileMode.Open,FileAccess.Read,FileShare.Read);

//compare the two streams, to see if they are the same (see later)
bContinue = compareFiles(msFile,fStream);
}
else
{
//file doesn't exist, download anyway
bContinue = true;
}
}
catch(Exception ex)
{
bContinue = true;
LogError(ex.Message,"");
}
finally
{
if(fStream!=null)
fStream.Close();
fStream = null;
}
if(bContinue)
{
//create a stream to create the new file
streamOut = File.Open(PathName, FileMode.Create, FileAccess.Write, FileShare.ReadWrite);
//create the new copy
msFile.WriteTo(streamOut);
//close up
streamOut.Close();
}
msFile.Close();

So, to how the streams are compared. I adapted my solution to one I found via a google search on hashing:

bool compareFiles(MemoryStream file1, FileStream file2)
{
using (HashAlgorithm hashAlg = HashAlgorithm.Create())
{
// Calculate the hash for the files.
byte[] hashBytesA = hashAlg.ComputeHash(file1);
byte[] hashBytesB = hashAlg.ComputeHash(file2);
// Compare the hashes.
if (BitConverter.ToString(hashBytesA) == BitConverter.ToString(hashBytesB))
{
//they are the same
return true;
}
else
{
//they are different
return false;
}
}
}

Hope this helps.

Detecting an enter key press on an asp.net form (but when using AJAX as well!)

2008-09-19T02:11:00.000-07:00

I've been more than familiar with detecting key presses in javascript for quite some time, and used to do it all the time in classic asp, but this time when I came to do it in the .net world, in was a little more involved.

This setup was also the first time I had to cope with Masterpages using AJAX inside them.

So, I have a simple content placeholder, inside this a textbox and these themselves are inside an updatepanel. I also have a modalpopup extender that shows whenever the page causes an update (asyncpostback).

So, to get this to work, I had to tie up my javascript with the ScriptManager.RegisterClientScriptBlock() method and placing the script into a LiteralControl that I had to place on the form also:

if(!Page.IsPostBack)
{
this.textBoxQuery.Attributes.Add("onkeypress", "javascript:return checkEnter(event);");
Page.RegisterHiddenField("__EVENTTARGET", textBoxQuery.ClientID);
string strScript = "function checkEnter(e){ " +
"var characterCode; " +
"if(e && e.which){ " +
"e = e; " +
"characterCode = e.which; " +
"} " +
"else { " +
"e = event; " +
"characterCode = e.keyCode; " +
"} " +
"if (characterCode == 13) " +
"{ " +
" " +
" document.getElementById('" + ButtonSearch.UniqueID + "').click(); " +
" return false; " +
"} " +
"else " +
"{ " +
" return true; " +
"}" +
"}";

ScriptManager.RegisterClientScriptBlock(LitScript, this.GetType(), "regScripts", strScript, true);
}

This now captures the enter key, and handles it as a normal button click event.

Error messages "There is an error in XML document (1, 1994)." or inner exception of "DataTable does not match to any DataTable in source."

2008-09-11T06:33:00.000-07:00

Today I have spent nearly 5 hours trying to get to the bottom of why I was receiving these errors.

The first error made me think that this was caused by an invalid character in my data, which was being returned from a webservice to my data in my program.

I took a copy of what was being returned using "fiddler" to see what was going on in my browser - it showed the xml being returned was fine. I even validated the XML to check.

The interesting thing was the inner exception of "DataTable does not match to any DataTable in source." thrown in Visual Studio. Looking at this, it was telling me that the datatable being returned from the webservice, was not the same as the datatable I had in Visual Studio. The strange this was, they were both typed datatables! so how could they be wrong.... well, they weren't.

The source of the problem was the following which was at position 1994 in the XML:

"diffgr:diffgram msdata="urn:schemas-microsoft-com:xml-msdata" "

This was telling me that there was a difference in the schema of the dataset... <- Note here, a difference in the DATASET. I was not returning a dataset, I was returning a datatable! Anyway, a quick google brought up a nice little link from the good boys at Microsoft:

http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=105642

It turns out, you can't return a datatable from a webservice, you have to return a DATASET. What a load of pants.

So, to solve this, I had to alter the webservice method to return a fake dataset, that contained the datatable that I wanted, e.g:

[WebMethod]
public MyTypedDataSet(string someparams)
{
MyTypedDataSet ds = new MyTypedDataSet();
MyTypedDataTable dt = aFunctionThatGetsTheData(someparams);
ds.Tables.Add(dt.Copy());
return ds;
}
private MyTypedDataSet.MyTypedDataTable(string someinfo)
{
MyTypedDataSet.MyTypedDataTable dt;
//The work that gets the data is here
return dt;
}

I hope this helps someone if they have the same problem too!!!

Jamie.

IsNumeric in C#

2008-09-08T08:49:00.000-07:00

IsNumeric doesn't exist in c#, it's VB specific, but i've found to work today is the following:

int result;
Int32.TryParse(value_to_try, out result);

If it can parse an Int, it returns the value to the result variable, if it fails, it returns 0.

Brilliant!

Regex to find images in HTML

2008-07-29T07:52:00.000-07:00

The other day I wanted to find a way to extract the image urls from some HTML, so I found the following to work quite nicely:


Regex rgxFiles = new Regex(@"[\w/:.]+\.(?:jpg|bmp|gif|png)");
MatchCollection mtcFiles = Regex.Matches(strSourceString,@"[\w/:.]+\.(?:jpg|bmp|gif|png)",RegexOptions.IgnoreCase);

foreach (Match mFile in mtcFiles)
{
     
  //mFile.Value will contain the image address that we can work with

}

The HTTP verb POST used to access path ''Path' is not allowed

2008-07-29T07:49:00.000-07:00

This is probably one of the most stupid mistakes I have made in months. I kept this message when transferring data from one website page to another, and I couldn't understand why.....

Hmmmm.....

It helped when I gave the webpage an aspx extension!!!

One problem here, if you try and post data to a page that is not asp or aspx, IIS and .NET does NOT like it!

Hope this helps!

REGEX to remove HTML formatting

2008-07-24T00:56:00.001-07:00

I've seen a few ways of using Regex to remove HTML formatting, but the following seems to work fine for me!:

public string Strip(string text)
{
string sReturn = System.Text.RegularExpressions.Regex.Replace(text, @"<(.|\n)*?>", string.Empty);
sReturn = System.Text.RegularExpressions.Regex.Replace(text, @"(&\w*;)", string.Empty);
return sReturn;
}

ModalPopupExtender for deleting DataGridView rows

2008-07-11T05:59:00.000-07:00

In the words of Little Britain, "What a caffuffle"!

How difficult does it have to be to delete a row from a datagrid, and use an AJAX ModalPopupExtender to provide a nice UI for it? The answer, very!

For all those who are trying to do it, i'm not going to re-invent the wheel, simply take yourself over to Matt Berseths' blog, where he has managed to do it (and now have I!)

http://mattberseth.com/blog/2007/07/confirm_gridview_deletes_with.html

Seekafile Server - not removing indexed documents

2008-07-10T05:27:00.001-07:00

I've noticed that on a particular Windows XP machine, that when seekafile server is running, it does detect that a file has changed, but it does not remove the old one from the index.

So, what I am having to do, is modify the crawler application to delete the folder containing the site files, to force seekafile to remove the folder (and it's contents) from the index.

When the crawler then downloads the files from the site again, it create a new index - in fact, this way is better as it ensures that the site data is correct!

Chaning the ASP.NET Name attribute

2008-07-10T04:56:00.000-07:00

This had be frustrated for days - you can change the ID property of a dynamically created control, but you can't change the NAME attribute!

Why do I care? Simple, I wanted to use the ID property for use in JavaScript calls, and the NAME attribute for when I was posting the form data - for example, I wanted two textboxes, one with an ID of 12, but a name of "Name", and another with an ID of 13, but a name of "Age", so that when I submitted the form data, I could get a friendly name for it.

(Before anyone harps on about "the .NET framework uses the NAME property in postbacks" - I do know this, but it is not required in this scenario.)

So, how did I do it? I created a class that inherits from TextBox, and then created an property within it called Name. I then overrided the render method to render the box as I needed it to be done in HTML:


public class MTextBox : TextBox
{
private string _mName;
public string Name
{
get { return _mName; }
set { _mName = value; }
}
//overriden to print name attribute correctly
protected override void Render(HtmlTextWriter writer)
{
string sOutput = "";
//Here you can squirt out your HTML in whatever format you want, e.g. you can
//squirt out the HTML of Name=\" + __mName + "\" etc
writer.Write(sOutput);
}
}

Sorry I can't give a more complete example, but blogger treats the code sample as HTML!!! Contact me if you want the code.

In my application, I can then just do this:

MTextBox textBox = new MTextBox();
textBox.Name = "Something I want it to be!!!";

Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints.

2008-06-23T07:29:00.000-07:00

Well, this was a fun error I received today, and after banging my head against a brick wall, the solution was simple.

The typed dataset that I had created in Visual Studio, was not a direct match to the table that I was accessing on the development SQL DB. I updated the dataset, and hey-presto, we are all fine again now.

FileInfo FullName returns case insensitive results

2008-05-14T06:44:00.000-07:00

I had a problem the other day whereby I was trying to record urls that contained encoded information as a querystring parameter, e.g. file.asp?param=4H4tTy7u

The problem was that even though I could see the files in case sensitive manner, when I saved the file to the filesystem, they were being read as case insensitive, e.g. file.asp?param=4h4tty7u

This caused major problems, as when I then unencoded the information in the querystring, I was getting completely the wrong information.

Upon tracing, it became apparent that although I could see the mixed cases in windows explorer, the underlying filesystem was case insensitive, and returned all results in lowercase - something which was causing the major problem.

To get round this, I first found an MS article that said it was caused by a .NET 2 framework bug from 2006 that set a registry value to indicate case insensitivity to the filesystem. After I changed this value and restarted, still it did not work.

After a bit of googling, I found the following article, which proved to solve the problem for me eventually - quite why such an elaborate solution is required is beyond me, but hey! of you have the same problem, at least here is the solution!

//within your code....
string sFileToProcess = fi.FullName;

string dir = Path.GetDirectoryName(sFileToProcess);
dir = ReplaceDirsWithExactCase(
dir,
Path.GetPathRoot(dir).ToUpper());
string filename =
Path.GetFileName(GetExactCaseForFilename(sFileToProcess));
Console.WriteLine(dir + "\\" + filename);

//methods needed for the above to work.
public static string ReplaceDirsWithExactCase(string fullpath, string parent)
{
if (fullpath.LastIndexOf(@"\") != fullpath.Length - 1)
fullpath += @"\";
if (parent.LastIndexOf(@"\") != parent.Length - 1)
parent += @"\";
string lookfor =
fullpath.ToLower().Replace(parent.ToLower(), "");
lookfor =
(parent + lookfor.Substring(0,
lookfor.IndexOf(@"\"))).ToLower();
string[] dirs =
Directory.GetDirectories(parent);
foreach (string dir in dirs)
if (dir.ToLower() == lookfor)
{
if (lookfor + @"\" == fullpath.ToLower())
return dir;
else
return ReplaceDirsWithExactCase(fullpath, dir);
}
return null;
}

public static string GetExactCaseForFilename(string file)
{
string[] files =
Directory.GetFiles(Path.GetDirectoryName(file));
foreach (string f in files)
if (f.ToLower() == file.ToLower())
return f;
return null;
}

Automatically resizing an IFRAME based on it's content size

2008-04-24T07:17:00.000-07:00

I had a case today when I must have spent 3 or 4 hours trying various elaborate exaples to try and get an IFrame to resize based upon the size of the target page of that IFrame. Many examples were suggesting have a method in the parent frame, that was to be called by the child frame, and in all instances it would just not work for me.

I found the following to work, and fortunately, it required no ammendments to the target page, just changes to the page that contained the IFRAME.

Firstly, a little snippet of JavaScript was need within the HEAD tags, as shown below:

function resizeMe(obj)
{
var docHeight = dynFrame.document.body.scrollHeight;
obj.style.height = docHeight + 'px';
}

Note that in < IE5.5, you should use offsetHeight rather than scrollHeight.

Then in an IFrame, add the attribute to trigger the resize when the iframe has loaded, as such:

onload="resizeMe(this);"

You should now have a resizing IFrame based on its content.

Visual Studio .Net 2005 Hangs on Startup

2008-04-17T02:35:00.000-07:00

Well, the subject of this post if what has been happening to me today. For no apparent reason, VS.NET 2005 would just hang at the splash screen. After trying to reinstall, it still would not load.

Here are some suggestions, collated from a morning of googling the forums:

1) Check that your harddrive has at least 1GB of space free
2) Try holding the shift key down when starting VS, so that you disable any addons that may have been corrupt.
3) Try installing the lastest .NET runtime, i.e. at present 3.5
4) As a last resort, trying clearing out the user environment settings - WARNING! This will erase all settings you have made in VS, so make a backup if you really need them, and only do this as a last resort. Go to the command prompt, and type "devenv.exe /resetuserdata"

If none of those work, you're pretty much stuffed I think! However, number 2 worked for me!

C# Lucene.NET - Sorting

2008-04-17T02:18:00.000-07:00

I needed to perform a non-standard search in Lucene, basically a search that allowed me to order by something other than relevance - why? well, because the Boss wanted me to!

So, first of all, if you've been using Seekafile, you'll need to be aware that "out of the box", it comes with version 1.4 of Lucene. Verion 1.4 does not have the overloaded search method you need to supply a search to Lucene.

Firstly then, download the latest version of Lucene, which can be accessed by going the long way round of visiting the old www.dotlucene.net website, and then following the links to version 2.0 or greater. You will then be able to download a newer version of Lucene.

You will find then, that the search() method of the IndexSearcher will then expect a sort of some description. If you run search(query, Sort.RELEVANCE) that will perform a search the same as in old version 1.4 days. However, if you do a search such as search(query, new Sort("Fullname")) this will perform an alphabetic search on the fullname field!

There are loads more options, you can do other than just that - check out the Sort.cs class for options.

C# Lucene.NET - Part 3

2008-04-11T01:57:00.000-07:00

Part 3 - I need a book, dagnamit!

This is a really short post.

Basically, I needed to know more about Lucene, so I found out that there was a book about Lucene, called "Lucene in Action". The books author really knows his stuff, and if you are new to Lucene, it is an absolute must for you to buy it.

The book is well structured, and explains Lucene from a ground up approach, starting with the basics. The book does not cover, however, the theories behind search algorithms, proximity, Tf-idf or the vector space model, so if you are after that information, use the links I have provided.

C# Lucene.NET - Part 2

2008-04-11T01:08:00.000-07:00

Part 2 - Revenge of the HTML IFilter

So here's me, sat at my desk, happily indexing away using my nice little indexing server, and the Boss says - "all our html page titles are the same, and I don't want the file name to be shown as the title in the search results, but we do use the meta descripions a lot to detail what the page is about, can we use that?", "of course we can" I replied.

So, I go back into my indexing program, and implement a HTML Parser, and I add a new field to my Lucene document indexes when they are created, and I want to save the Meta tag called "content". I set my indexer off again, and great, no errors. I perform a search, but results come back for that field. I run it debug mode, and sure enough, when indexing, my HTML Parser couldn't pickup the HTML tag. Looking at the raw body of what the IFilters were doing, the HTML files were being passed into nlhtml.dll for filtering. Having read previously about IFilters and especially nlhtml.dll being used predominently for indexing HTML pages, I was really bemused as to why I couldn't gain access to the HTML tags - why it wouldn't let me get at them.

After two days, it dawned on me - that is exactly what an IFilter was supposed to do! the nlhtml.dll IFilter removed the "unnecessary" formatting from around the HTML Page, and only returned back to me what it thought was necessary, the content from within the tags - damn it! I need the META tags!

How did I overcome this you may ask (if you're interested that is) - well, I decided that I would supply a list to my search engine of files that I wanted to be considered as Plaintext - i.e. they needed to some "other" processing done on them in addition to being passed to a body content filtering process (in hindsight, I wish i'd called them html documents, rather than Plaintext, as it is more correct).

So here is what I cam up with - maybe not the most elegant, but by god it works! When it finds a file with the extension of a known Plaintext file, it still does the normal content body search, but it also does some specific searches for known html tags:

public Document LuceneDocument
{
get
{
Document doc = new Document();
doc.Add(Field.Keyword("name", fi.Name));
doc.Add(Field.Keyword("fullname", fi.FullName));
DirectoryInfo di = fi.Directory;
doc.Add(Field.Keyword("directparent", di.FullName));
while (di != null)
{
doc.Add(Field.Keyword("parent", di.FullName));
di = di.Parent;
}
doc.Add(Field.Keyword("created", DateField.DateToString(fi.CreationTime)));
doc.Add(Field.Keyword("modified", DateField.DateToString(fi.LastWriteTime)));
doc.Add(Field.Keyword("accessed", DateField.DateToString(fi.LastAccessTime)));
doc.Add(Field.Keyword("length", fi.Length.ToString()));
doc.Add(Field.UnIndexed("extension", fi.Extension));

//We need to know if this is a plaintextfile search - looking for meta info etc
//mainly used for htm, html, and asp files. This is set in config.xml
ArrayList sPlainTextFiles = new ArrayList(0);
SearchConfiguration cfg = null;
try
{
cfg = SearchConfiguration.Load(Directory.GetParent(Assembly.GetExecutingAssembly().Location) + "/config.xml");
if (cfg == null)
{
Log.Debug("Config file not found.");
}
else
{
sPlainTextFiles = cfg.PlainTextFiles;
}
}
catch (Exception e)
{
Log.Debug("Error loading the config file: " + e);
}
bool bPlainTextRequired = false;
for (int i = 0; i < extension ="=" bplaintextrequired =" true;" rawtext =" getBody();" plaintext = "" metacontent = "" metadescription = "" htmltitle = "" htmlh1 = "" plaintext =" getPlainBody();" metacontent =" getMetaContent(plainText);" metadescription =" getMetaDescription(plainText);" htmltitle =" getHtmlTitle(plainText);" htmlh1 =" getHtmlH1(plainText);" ret = ""> metaList = MetaParser.Parse(sText);
foreach (HtmlMeta meta in metaList)
{
if ((meta.Name.ToLower() == "title") && (meta.Content.Length > 0))
{
ret = meta.Content;
}
}
return ret;
}
private string getMetaDescription(string sText)
{
string ret = "";
List metaList = MetaParser.Parse(sText);
foreach (HtmlMeta meta in metaList)
{
if ((meta.Name.ToLower() == "description") && (meta.Content.Length > 0))
{
ret = meta.Content;
}
}
return ret;
}
private string getHtmlTitle(string sText)
{
string strOut = Regex.Match(sText,"(?<=title).*?(?=/title)", RegexOptions.IgnoreCase).Groups[0].Value; return strOut; } private string getHtmlH1(string sText) { string strOut = Regex.Match(sText, "(?<= h1).*?(?=/h1)", RegexOptions.IgnoreCase).Groups[0].Value;
Regex rx = new Regex(@"<[^\>]+\>");
return rx.Replace(strOut, "");
}

Please note that in the above regex, i've had to remove the <> from around the h1 and title tags, as it made the document go skew-wiff when published!

Did I miss something? Yes - getPlainBody() - well again, not very elegant, but after all, all we need if the plain text isn't it? So here is getPlainBody():

private string getPlainBody()
{
//If we need plain text, i.e. No IFiltering, then do this.
Log.Echo("Calling default parser for " + fi.FullName);
return PlainTextParser.Extract(fi.FullName);
}

And all the PlainTextParser is, is:

public class PlainTextParser
{
public PlainTextParser()
{
}

public static string Extract(string path)
{
string strRet = "";
try
{
StreamReader sr = new StreamReader(path);
strRet = sr.ReadToEnd();
//Remove carriage returns...
strRet.Replace("\r\n", "");
sr.Close();
}
catch (Exception ex)
{
Log.Echo("Plain Text Parser failed - " + ex.Message + ". Invoking default parser.");
strRet = DefaultParser.Extract(path);
}
return strRet;
}
}

Simple huh?

Well, if you have had the same problems as me, I hope this helps.

C# Lucene.NET - Part 1

2008-04-11T00:46:00.000-07:00

A few weeks ago I was asked to implement a new website search system, as our current Microsoft Indexing Server was becoming a little unreliable, and was frequently not updating documents when they had been saved. After a little searching around, I came accross Lucene.

Lucene was originally designed in Java, but since has been ported to several other langauges including perl, php, c++ and of course, .NET - however, the support and documentation for it in .NET is dire to say the least.

Lucene is purely the engine to allow you to Index files, and then peform searches against it, however, you will have to write all that yourself. Whilst looking for help, I came accross another open source project, called Seekafile, but again, all support for this has been revoked. I perservered with Seekafile though, as it was a useful tool. If you are looking for the basics of Lucene, you are probably going to want to look here, at the most .NET documentation for Lucene I was able to find (and prepare to be horrified).

My intention is to use this Blog to detail my trails and tribulations with Seekafile and Lucene, so if you are just starting out with it, I really hope that some of this will be of assistance to you!

So, part 1!

What is an IFilter...... and why should I care?

Seekafile is really useful, don't get me wrong, but I had problems from day 1 getting my head around what an IFilter was, and why the hell Seekafile needed them. Well, in simple terms, an IFilter is an interface to a component that has been pre-written for you, that is able to read a file of a certain type, and return back to you the text content that has been stripped of all the formatting and file nonsense that the host application requires. For example, a MS Word document has plenty of formatting around it, that tells word where, how, and why you have created a pretty looking table in the document for example. When indexing, and searching for that matter, are you interested in that pretty table? no, you just want to know what is inside it, so that is where the IFilter comes in.

You must know this, Indexing Service uses IFilters for everything, so I assuming, that pretty much all Indexing technology does as well.

So, IFilters are great huh? Yes, and no. Indeed they have many benefits, such as the fact that Windows natively only supports IFiltering of several key documents (effectively), and if you want to Index something more obscure you simply get another IFilter to help you do this. However, the disadvantage is that you have no control over how the IFilter does this, unless you write your own :-

If you are looking at IFilters, go here first http://msdn2.microsoft.com/en-us/library/ms692488(VS.85).aspx where you can find a really useful insight into how Windows decides what IFilter to use for what file type. Obviously, if you follow this, it simply shows you where the IFilter is located, and that then tells you what DLL is going to be used. A more useful tool is from Citeknet, which allows you to take an abstract view of IFilters, as it shows you what you have installed on your system, and which will filter which type of file - a useful overview if you ask me. Microsoft, specifically aimed at Indexing Service, do provide a little info on IFilters in articles http://msdn2.microsoft.com/en-us/library/ms692540(VS.85).aspx and http://msdn2.microsoft.com/en-us/library/ms692582(VS.85).aspx and from this, you will see that the good folks at Microsoft help you filter pretty much all their own stuff. Checkout the Citeknet site for some more obscure IFilters.

C# Web Crawling - Knowing when to download a new page

2008-04-10T02:11:00.001-07:00

I have been using a c# web crawler written by Hatem Mostafa, which is available on code project, and I have found extremely useful.

I have altered this now to work with our database of websites, and hey-presto, I can now crawl all our websites as if I were a user in the external world.

However, the limitation is this - it is all well and good crawling thousands of pages, but downloading them is costly. So how can you overcome this? use the HTTP Header Entity Tag which is a hash to identify changes in the file. Check the ETag against a record of files already downloaded, and see if the Etag has changed, if it has, proceed to download. Simple really, and reduces a lot of external traffic on the server as well.

Of course, this presumes that the web server is configured to send ETag headers, and that the pages aren't so dynamic, the the ETag becomes irrelevant, or is ommitted!