Generating random lists of names with errors to explore fuzzy word matching

Generating names

The master list can easily be generated using the randomNames function in the R package randomNames. A call to this function provides samples of names from a large scale database. (It provides gender and ethnic variation if you need it.)

library(data.table)
library(randomNames)

set.seed(6251)
randomNames(4)

## [1] "Hale, James"      "el-Qazi, Najeema" "Sourn, Raj"       "Jensen, Tia"

Generating errors

To facilitate the generation of spelling errors, I’ve created a function that takes a string, a specified number of substitutions, a number of insertions (if negative then these are deletions), and an indicator that flips the order of the names (typically “Last Name, First Name”):

mis_string <- function(name, subs = 1, ins = 0, flip = FALSE) {
  
  names <- trimws(unlist(strsplit(name, split = ",")))
  
  if (subs) {
    for (i in 1 : subs) {
      change <- sample(1:2, 1)
      ii <- sample(nchar(names[change]), 1)
      l <- substr(names[change], ii, ii)
      s <- sample(letters[letters != l], 1)
      names[change] <- sub(l, s, names[change]) 
    }
  }
  
  if (ins > 0) {
    for (i in 1 : ins) {
      change <- sample(c(1, 2), 1)
      ii <- sample(nchar(names[change]), 1)
      stringi::stri_sub(names[change], ii, ii-1) <- sample(letters, 1)
    }
  }
  
  if (ins < 0) {
    for (i in 1 : -ins) {
      change <- sample(c(1, 2), 1)
      ii <- sample(nchar(names[change]), 1)
      stringi::stri_sub(names[change], ii, ii) <- ""
    }
  }
  
  paste(names[flip + 1], names[2 - flip], sep = ", ")
  
}

Here are two applications of mis_string on the name “Vazquez Topete, Deyanira”:

mis_string("Vazquez Topete, Deyanira", subs = 2, ins = 2)

## [1] "Vazhquiez Topete, Dmyanika"

mis_string("Vazquez Topete, Deyanira", subs = 1, ins = -2, flip = TRUE)

## [1] "Deynira, uazquez Topet"

Master list definitions

To generate the master list we define (using simstudy) a set of key parameters: an indicator pcr identifying whether the person has at least one test (70% will have a test), an indicator resident identifying whether the person is part of our resident cohort (20% of those on the master list will be residents), and a variable for the number of tests an individual has (conditional on having at least 1 test). There will be names on the master list that do not have any tests nor are they a resident; these patients are removed from the master list.

library(simstudy)

def_n <- defDataAdd(varname = "pcr", formula = 0.7, dist="binary")
def_n <- defDataAdd(def_n, varname = "resident", formula = 0.2, dist="binary")
def_n <- defDataAdd(def_n, varname = "n_tests", formula = 3, dist="noZeroPoisson")

PCR list error definitions

Each person with a PCR test will have one or more records in the PCR list. The following set of definitions indicates the number of substitutions and insertions (both specified as categorical integer variables) as well as whether the first and last names should be flipped.

def_p <- defDataAdd(varname = "subs", formula = ".8;.15;.04;.01", dist="categorical")
def_p <- defDataAdd(def_p, varname = "ins", 
  formula = "0.02;0.05;0.86;0.05;0.02", dist="categorical")
def_p <- defDataAdd(def_p, varname = "flip", formula = 0.10, dist="binary")

Generating the data

In this simulation I am generating 50 possible names:

set.seed(3695)

n <- 50

d_master <- data.table(id = 1:n, name = randomNames(n))
d_master <- addColumns(def_n, d_master)
d_master <- d_master[(pcr | resident)]

head(d_master)

##    id           name pcr resident n_tests
## 1:  1  Maas, Synneva   1        0       3
## 2:  2   Rock, Alyssa   1        0       3
## 3:  3    Lee, August   1        0       1
## 4:  4   Keefe, Dylan   1        0       1
## 5:  5       Yang, An   1        0       1
## 6:  6 Andrew, Crysta   1        0       3

In this case, there are be 7 individuals in the resident cohort and 40 individuals have at least one PCR test. 5 residents were tested:

d_master[, .(
    num_res = sum(resident), 
    num_pcr = sum(pcr), 
    num_both = sum( (resident & pcr) )
  )
]

##    num_res num_pcr num_both
## 1:       7      40        5

The PCR list will have 110 total tests for the 40 people with tests.

d_pcr <- genCluster(d_master[pcr == 1], "id", "n_tests", "pcr_id")
d_pcr <- addColumns(def_p, d_pcr)
d_pcr[, subs := subs - 1]
d_pcr[, ins := ins - 3]
d_pcr[, obs_name := mapply(mis_string, name, subs, ins, flip)]

d_pcr[, .(pcr_id, id, name, obs_name, subs, ins, flip)]

##      pcr_id id            name         obs_name subs ins flip
##   1:      1  1   Maas, Synneva    Sycnfvq, Maas    3   0    1
##   2:      2  1   Maas, Synneva    Synneva, Maas    0   0    1
##   3:      3  1   Maas, Synneva    Maas, Sznneva    1   0    0
##   4:      4  2    Rock, Alyssa     Rock, Alyssa    0   0    0
##   5:      5  2    Rock, Alyssa     Ropk, Alyssa    1   0    0
##  ---                                                         
## 106:    106 48 Wall, Sebastian Wall, Sebtastian    0   1    0
## 107:    107 48 Wall, Sebastian  Wall, kebastian    1   0    0
## 108:    108 48 Wall, Sebastian  wall, Sebastian    1   0    0
## 109:    109 49   Tafoya, April    Tafoya, April    0   0    0
## 110:    110 49   Tafoya, April    Tafoya, April    0   0    0

We end up with two lists - one with just residents only and one with a list of PCR tests. This is mimicking the actual data we might get from our flawed health data systems.

d_res <- d_master[resident == 1, .(id, name)]
d_pcr <- d_pcr[, .(pcr_id, id, name = obs_name, pcr)]

The truth

Before proceeding to the matching, here is a PCR test records for the residents. This is the correct match that we hope to recover.

d_pcr[ id %in% d_res$id]

##     pcr_id id               name pcr
##  1:     57 26       Diaper, nody   1
##  2:     58 26       Draper, Cody   1
##  3:     59 26      Cody, Drapeyr   1
##  4:     60 27  al-Naqvi, Qamraaa   1
##  5:     61 27  al-Naqvi, Qamraaa   1
##  6:     64 29 el-Hallal, Zahraaa   1
##  7:     65 29 Zahraaa, el-Hallal   1
##  8:     86 42       Allen, Jalyn   1
##  9:     87 42        llen, Jalyn   1
## 10:     88 42       Allen, Jalyn   1
## 11:    102 47 Sanandres, Bzandon   1
## 12:    103 47 Sananores, Brandon   1
## 13:    104 47 Sanandues, Brandon   1
## 14:    105 47 Sanandres, Brandon   1