Confused about how NullLevel works

[squalidbus]

Hello!

I am working on a linkage project using splink. With the data that I have, I can use fields for different levels of a person's location (street address, postcode, city, state, and country). Because information at a lower level will be found at a higher level (if two people share the same address, they also share the same postcode), I want to avoid over-inflating the match results. To do this, I am creating a custom comparison for location fields that resembles the following:

comparison_location = cl.CustomComparison(
    output_column_name="comparison_location",
    comparison_levels=[
        cll.ExactMatch('street'),
        cll.ExactMatch('postcode'),
        cll.ExactMatch('city'),
        cll.ExactMatch('state'),
        cll.ExactMatch('country'),
    ],
)

In the documentation for custom comparisons, the function NullLevel is referenced for situations whre one or both of the matching fields are null, but I am confused as to how this function works. Since the missingness of my location data is not consistent (a person may have a street value but not a postcode value, and vice versa), it would be useful for me to be able to do a null check for each field in the comparison, but I cannot tell if this is or is not supported.

Could someone please give a basic overview of how the NullLevel function works? I would like to specifically know:

1. If I put NullLevel('street') at the top of my custom comparison list and the street field is null, will that end the comparison without checking for similarity between other fields?
2. Is there a way to do multiple null checks for different fields in a custom comparison?

Thank you!

[ADBond]

The key things to know/remember with comparison levels are:

• a pair of records will always fall into the first level whose condition it satisfies, as specified by the order you supply them (when it comes to the SQL, these levels end up in a CASE WHEN... statement)
• any comparison level that is designated as a 'null level' will give a record pair that it captures the special comparison vector value '-1'. You are allowed more than one null level in the same comparison. All null levels are treated the same. Any non-null level will be given unique comparison vector values
• any record pair designated as falling in a null level (comparison vector value = -1) will contribute no match weight to the match prediction

So in answer to your first question, yes, if you have a first level of NullLevel('street'), then the remaining comparison levels will only capture pairs of records that both have 'street' present.

You can combine levels using cll.And(). If you are combining null levels this way, the result should still be categorised as a null level.

For your use-case you would probably want a null level at the top like:

comparison_levels=[
    cll.And(
        cll.NullLevel("street"),
        cll.NullLevel("postcode"),
        cll.NullLevel("city"),
        cll.NullLevel("state"),
        cll.NullLevel("country"),
    ),
    ...
]

This captures any pair of records that do not share any of the five fields being present. In other words, it captures all instances where:

• one or both 'street' values are NULL
• AND one or both 'postcode' values are NULL
• AND one or both 'city' values are NULL
• etc.

More complex configurations are possible, but unless you are doing any cross-field comparison levels that shouldn't be necessary.

A couple of other points - apologies if you are aware of these and it's just an omission in the example:

• In your code it should be cll.ExactMatchLevel('street') etc - may just be a typo, but want to make sure you aren't mixing up cl.ExactMatch (a comparison) and cll.ExactMatchLevel (a comparison level)
• You should also have an ElseLevel at the end to capture pairs that aren't captured by any of the 'matchy' levels

One thing to consider is that your levels do not distinguish between whether preceding fields are NULL or they mismatch. So for instance your level cll.ExactMatchLevel('postcode') will capture record pairs where:

• the postcode matches and one or both 'street' values were NULL
• OR the postcode matches and both street values were present but were different

and your 'else' level will capture record pairs where:

• one field mismatched but the other four were NULL in one or other record
• two fields mismatched but the other three were NULL in one or other record
• ...
• all five fields mismatched

and similarly for the further levels. I'm not sure what your data is like, but you may want to consider adding more levels to be able to distinguish some of these cases. You probably wouldn't need every possible case, and it may well be that what you have works well enough for your data - just something to bear in mind.

[ADBond]

and just a couple of additional points for reference:

• cll.And (and the related cll.Or) will configure a comparison level to be a null level when ALL of the levels it combines are null levels
• you can manually set a level to be a null level using .configure(), e.g. cll.CustomLevel("street_l = '' OR street_r = ''").configure(is_null_level=True)