Tag Archives: duplicate

CRM 2011 Duplicate Checking Enhancements in Rollup 5

Microsoft CRM’s duplicate checking feature has always had a ‘quirk’ that bugged me.  Good news though, rollup 5 introduces a solution.   I’m not seeing this enhancement getting much mention so thought I would explain the enhancement and why you need to make use of it.

I talk about the limitation in this older post.  In a nutshell pre-rollup 5 any duplicate detection rules you defined would match blank fields to blank fields.  For example if you defined a rule that matched on first name + email address and you had a record with a first name of “Adrian” that had a blank email address then any future “Adrian” records who also had a blank email address would be considered a potential duplicate:

Existing record:


New record being entered:


Potential duplicate detected:


What Rollup 5 gives us is nice little check box that addresses this issue:


If you’re making use of CRM’s duplicate detection features you will definitely want to revisit your rules and consider checking this option.   Do remember as you test this that anytime you re-publish your rules CRM has to rebuild its match codes, so the change is not immediate.  There is a system job you can see attached to the Duplicate rule, that might be the one to monitor.

Now, if Microsoft would just tidy up the Duplicate Detection window and make that a bit more user friendly (or give us the ability to customise it) we would have a really nice solution.


Overcoming the Limitations of CRM’s Duplicate Checking Functionality – Part 2

In my earlier post I described how CRM’s duplicate checker will happily find matches on partial matches due to its approach of matching blanks.  A rule like:

… will match on just the First Name and Last Name fields when the E-mail field is blank, effectively opening your matching rule up to be just a name match, which can be far too broad.

The solution requires a bit of coding.  Here’s the approach:

1. Create a new attribute on the Contact entity to hold a shadow copy of the Email address field, we don’t need it to be visible to our users so make it non-searchable and don’t bother placing it on the form.

2. Create a Plug-In or Custom Workflow Assembly that will either copy the value of the Email field into this new attribute, or, in the absence of a value, will paste the GUID of the Contact record into our new Email Match Field.

3. Change your duplicate check rule to refer to the new Email Match Field instead of the standard Email field.

And you’re done.  A new Contact’s first name, last name and email address will be compared to the existing contacts, each of which will either have a proper email address or a GUID, so a match will require an existing contact to have the same email address.

Note: You’ll need to run your customisation across your existing data so keep that in mind when deciding on your approach.  

If you want to match on name + phone number you can follow the same approach but in your custom code consider stripping the formatting out of your phone number to ensure formatting variations don’t hide any duplicates.

Overcoming the Limitations of CRM’s Duplicate Checking Functionality

Microsoft CRM comes with a real time duplicate checking engine that warns users of potential duplicates as they enter new records.  Sounds good doesn’t it?  However, it’s not a comprehensive solution and has a few weaknesses especially when you start dealing with large numbers of customer records where the likes of good old John Smith starts to pop up a lot.   But with a bit of targeted customisation you can get it working for you.

Let’s start with some requirements.  Say we want a duplicate check to run as new Contacts are entered which looks for any existing contacts with the same name and email address.   Sounds useful.   And on the surface it looks like you can configure this.   Here’s what you would logically configure in CRM to try and address this requirement:


Now if you publish this rule and then push through a test you will find the duplicate checker doesn’t report a suspected duplicate when you expect it to.

The first thing to check is whether CRM has completed the creation of match code for this rule.  This shows up under System Jobs in the Settings area, look to see if a MatchCode job has run since you made the change, and check it has a status of Completed:

What CRM is doing here is its creating a match code for each Contact in the database based on the rule you defined.  In this case the match code string will be “<contact.fullname><contact.emailaddress>”.   The duplicate checker is then able to check against this one field to find duplicates.  CRM actually creates a new SQL table to house this matching data.

So, you need to let CRM complete that step, then you can retry your test.   But, again you will find the duplicate checker doesn’t appear to be working.

The problem appears to be an issue of timing.  The duplicate checker appears to be reading the ‘”Full Name” of the Contact being entered before CRM has actually calculated what that is (based on the First Name and Last Name fields that were entered). 

The workaround to this is to have the duplicate rule refer to the First Name and Last Name fields instead, e.g..:

Again, wait for CRM to rebuild it’s match code table, and then retry your test. 

Now, you should see the duplicate warning starting to appear:

I mentioned customisation earlier and we haven’t had to do any yet, and yet it looks like we have this working now.   One problem.   Try entering a new Contact with the same name as an existing contact where neither of the Contacts has an email address.   In this instance the duplicate checking engine will consider this to be a match.   So, in the absence of email addresses we are now matching on name alone.   If dealing with a lot of customers then this will result in a lot of false positives being reported, which frustrates users, who start to pay less attention to the duplicate detection dialog box.

The reason CRM is getting this wrong is because of it’s match code approach.   Imagine these contacts exist in your CRM:

Full Name Email Address Match Code
John Smith johnsmith@abc.com JohnSmithjohnsmith@abc.com
John Smith johns@xyz.co.nz JohnSmithjohns@xyz.co.nz
John Smith null JohnSmith

And you enter a new John Smith without an email address.   CRM determines its match code as:

John Smith null JohnSmith


And it then looks up that match code in its match code table, and find record # 3.

Stink.  😦

Next post, I’ll show you to address this.