**4.3 Propensity score method**

The patient characteristics at the baseline, which are known to impact FEV outcomes, are considered into the multivariable logistic regression model (Eq. (2)) for estimating propensity scores. **Figure 3** presented the histograms of propensity score for the Tobi treated and not-treated patient groups, showing different but overlapping propensity scores between the two groups. Propensity scores are grouped into five groups by quintiles. The distribution of propensity scores are compared between the Tobi treated and not treated patients within each of the five PS categories; as one could see from **Figure 4**, within each quintile categories, the two patient groups present comparable patterns in their likelihood of receiving

#### **Figure 3.**

*Histogram of the propensity score distributions by Tobi use (red) and not-group groups (blue). Related the measured confounders; therefore no arrow is drawn for this relationship.*

#### **Figure 4.**

*Box-Whisker plots of the distribution of propensity scores by Tobi use (red) and not use (blue) groups stratified by the quintiles.*


*Abbreviations: CF, cystic fibrosis; FEV1, percentage predicted of forced expiratory volume in 1 s; PS, propensity score. Calculations for standardized differences are described in Section 4.3.*

#### **Table 3.**

*Standardized difference (T-val) between Tobi treated and untreated patients.*

Tobi. To check for propensity score balance, we compared the Tobi treated and not treated patients on their baseline covariates, the standardized differences between the treated and not treated groups are presented in **Table 3**. The results show that there is a significant difference between the treated and not treated patients groups according to their gender, baseline FEV1, CF-related diabetes, pancreatic insufficiency, insurance status, prior hospitalization and dornase alfa use. After matching patients on their PS categories, as well as after adjusting by inverse propensity score weighting, we are able to achieve balance between the Tobi treated and not treated groups. Subsequently, we proceed with the propensity score analyses using the inverse propensity score weighted approach. The results are presented in **Table 4**, which can be contrasted with the results from the multivariable regression analyses in **Table 2**. The results from these two approaches are very similar; both are

suggesting negative Tobi treatment effect on the improvement of FEV. The results from randomized clinical trials, however, all suggest a positive Tobi treatment effect. Such differences might be explained by unmeasured confounding that is

**Stage 1 (predicts patient tobramycin use)***<sup>a</sup>*

Not treated — 0

Sex — —

Male 0 0

CF-related diabetes — —

No 0 0

No 0 0 Insurance — —

Other 0 0

None 0.598 (0.064), (<0.0001) 5.44 (0.69), (<0.0001) 1 0.251 (0.068) 2.89 (0.74) 2 0.148 (0.080) 0.48 (0.87) 3 or more 0 0 Dornase alfa use — —

Yes 0.224 (0.036), (<0.0001) 0.28 (0.40), (0.48)

*Multivariable analysis with standard adjustment for confounding by including characteristics as covariates.*

*\*Each model is adjusted for measured confounders by including each listed variable as a covariate. For each categorical variable, the coefficient is the difference in FEV1 decline between the indicated category and the reference category (labeled as coefficient = 0). For each continuous variable, it is the change in FEV1 decline when the variable is*

No 0 0 *Abbreviations: CF, cystic fibrosis; FEV1, percentage predicted of forced expiratory volume in 1 s.*

*increased by 1 unit. A negative value implies greater FEV1 decline. <sup>a</sup>*

*Instrumental variable analysis to predict lung function decline\*.*

*Multivariable analysis weighted using propensity scores.*

Patient tobramycin

*Evaluating Clinical Effectiveness with CF Registries DOI: http://dx.doi.org/10.5772/intechopen.84269*

Weight-for-age percentile

Pancreatic insufficiency

None or State/ Federal

Baseline hospitalizations<sup>+</sup>

*b*

**69**

**Table 4.**

use

**Covariates Coefficient (SE), (***P***-value) Coefficient (SE), (***P***-value)**

Treated — 2.55 (1.22), (0.0366)

Age 0.013 (0.003), (0.0002) 0.86 (0.04), (<0.0001) Baseline FEV1 0.010 (0.001), (<0.0001) 0.27 (0.01), (<0.0001)

Female 0.112 (0.027), (<0.0001) 1.23 (0.30), (<0.0001)

Yes 0.112 (1.27), (0.38) 1.93 (1.44), (0.18)

Yes 0.064 (0.074), (0.39) 0.52 (0.83), (0.54)

— —

0.000 (0.001), (0.74) 0.06 (0.01), (<0.0001)

— —

0.128 (0.030), (<0.0001) 1.58 (0.34), (<0.0001)

— —

**Stage 2 (predicts change in lung function)***<sup>b</sup>*

#### *Evaluating Clinical Effectiveness with CF Registries DOI: http://dx.doi.org/10.5772/intechopen.84269*


*Abbreviations: CF, cystic fibrosis; FEV1, percentage predicted of forced expiratory volume in 1 s. \*Each model is adjusted for measured confounders by including each listed variable as a covariate. For each categorical variable, the coefficient is the difference in FEV1 decline between the indicated category and the reference category (labeled as coefficient = 0). For each continuous variable, it is the change in FEV1 decline when the variable is*

*increased by 1 unit. A negative value implies greater FEV1 decline. <sup>a</sup>*

*Multivariable analysis with standard adjustment for confounding by including characteristics as covariates. b Multivariable analysis weighted using propensity scores.*

#### **Table 4.**

*Instrumental variable analysis to predict lung function decline\*.*

suggesting negative Tobi treatment effect on the improvement of FEV. The results from randomized clinical trials, however, all suggest a positive Tobi treatment effect. Such differences might be explained by unmeasured confounding that is

related to treatment selection bias but not recorded in the registry. We further proceed with IV analyses to examine the Tobi treatment effect.

are essential. In this chapter, we have described a step-by-step approach to formulating and implementing a registry data analysis. Understanding the research question, selecting the appropriate data source and identifying potential sources of bias are necessary before beginning to construct an analytic plan. The statistical considerations should include data quality assessments and descriptive analyses, and it is critically important to address selection bias due to both measured and unmeasured confounding. This is because selection bias is ubiquitous; failure to adequately address selection bias will lead to biased conclusions. Multivariable regression has been the primary means to combat selection bias. While this technique can help to minimize differences between groups, it is limited to relatively fewer covariates in the adjustment process. Propensity scores, which correspond to the probability of treatment assignment given pre-treatment characteristics, provide a way to summarize multiple covariates into a single score for each individual. Therefore, this approach is capable of handling a large dimension of confounders, which is particularly useful in registry studies when confounders are measured. Another advantage of PS is that it allows one to check between the treatment groups when conditioning on propensity score whether the confounding factors is balanced out. However, when important confounders are not measured, the PS method is limited. One solution is to perform sensitivity analyses by evaluating how estimated treatment effectiveness might change if there exists an unmeasured confounder with varying levels of prevalence. Such practice will allow one to gauge the impact of

In this example, the likelihood of tobramycin use depends on unmeasured char-

When designing and analyzing registry data, it is critically important to address biases and confounding that are inherent in this type of study. Although we have focused, in this chapter, on describing methods for controlling selection biases, registry data are often subject to other types of biases related to measurement and miss-classification error, immortal time bias, loss to follow up, and missing data. We encourage use of sensitivity analyses to understand the impacts of these potential biases to the study conclusions. There are rich literature sources and several guidelines for design and analysis of registry data. In addition to the literature referenced in this chapter, a very useful resource is the recent report on standards in the conduct of registry studies for patient centered outcomes research and the

In addressing selection bias, most often, treatment effects are examined using multiple linear regression with measured confounders included as covariates [34]. Increasingly, PS methods are employed. However, existing statistical methods to address unmeasured confounding may be underutilized in registry settings. The models that we have presented are by no means exhaustive. There is room to develop more methodology, particularly to combat time-varying treatment effects and utilize time-varying instruments [12]. It is possible that preference-based

acteristics at the patient, family or care level. The adjustment of unmeasured confounding that is possible through IV analysis may have led to more intuitive conclusions regarding treatment effect. Since CF care is organized by care center, it was reasonable to examine the validity of a preference-based instrument to combat treatment-selection bias. Thorough sensitivity analyses are necessary to examine the robustness of the IV. We limit our illustrative application to a single instrument. It is possible to include multiple instruments and gain more formal properties to

unmeasured confounders to the treatment effect.

*Evaluating Clinical Effectiveness with CF Registries DOI: http://dx.doi.org/10.5772/intechopen.84269*

testing assumption (ii).

references therein [33].

**71**

**5. Conclusions**

#### **4.4 Instrumental variables analysis**

It is possible that the discrepancy between the previously described registry analysis and clinical trial findings of the treatment effect are due to unmeasured confounding. It is common in observational settings to encounter confounding by indication bias that is not recorded in registries. In this application, we selected a preference-based instrument, center-level prescribing patterns, to combat this bias. The CFFPR includes more than 240 centers. For each center, we calculated the tobramycin-prescribing rate during the time frame of the study. This rate was calculated as the number of times the center prescribed tobramycin to the patient when eligible divided by the total number of times the center should have prescribed tobramycin. We considered a patient to be eligible for the treatment once he met the CFF guidelines for its use.

We had to determine whether the IV met the previously mentioned criteria to be a valid instrument. We began by performing the first-stage analysis outlined in Model (4). We include all potential confounders as explanatory variables, and we include the IV. The response variable in this equation is the tobramycin use. The first-stage results are presented in **Table 4** and reflect what we found in the exploratory analysis from **Table 1**. The IV included in this regression was a highly significant predictor of tobramycin use. The corresponding *t*-statistic was 28.2, *P* < 0.0001. These results indicate that we have met assumption (i) for center-level prescribing to be a valid instrument. We also note that **Table 4** shows that dornase alfa use is strongly associated with tobramycin use. We will revisit this finding in sensitivity analysis of our instrument. We performed the multiple linear regression specified in Model (5) to determine the association between tobramycin and lung function decline. This regression accounts for observed patient characteristics and provides an instrumented version of tobramycin use. The last column in **Table 4** shows that tobramycin was associated with less FEV1 decline, suggesting the existence of a positive treatment effect.

Assumption (ii) is not directly testable, but we examine it through sensitivity analyses of heterogeneous treatment effects. These effects may be caused by confounding from other medication use or differences in quality of care received across centers. We performed three different types of sensitivity analyses. First, we extracted quality of care markers through the CF Foundation Annual Report (1) and calculated them for each center. We correlated each marker with our IV and found no significant association. Second, we used subgroup analyses to determine the impact of dornase alfa use on tobramycin effectiveness. We divided the cohort into two distinct groups according to whether they reportedly used dornase alfa. We performed the IV analysis separately on each group. The two sets of results were similar with regard to first- and second-stage analyses. Third, we performed a secondary analysis of patients with *B. cepacia*. Although these patients are traditionally excluded from clinical trials and other effectiveness assessments because of their significantly poorer outcome, they often receive tobramycin in clinical practice. The first-stage analysis of this cohort was similar to the primary results; however, their second-stage analysis showed no significant treatment effect.

#### **4.5 Concluding remarks**

Registry data plays an increasingly important role in health care research. Appropriate design and careful statistical approaches to the analyses of registry data

#### *Evaluating Clinical Effectiveness with CF Registries DOI: http://dx.doi.org/10.5772/intechopen.84269*

are essential. In this chapter, we have described a step-by-step approach to formulating and implementing a registry data analysis. Understanding the research question, selecting the appropriate data source and identifying potential sources of bias are necessary before beginning to construct an analytic plan. The statistical considerations should include data quality assessments and descriptive analyses, and it is critically important to address selection bias due to both measured and unmeasured confounding. This is because selection bias is ubiquitous; failure to adequately address selection bias will lead to biased conclusions. Multivariable regression has been the primary means to combat selection bias. While this technique can help to minimize differences between groups, it is limited to relatively fewer covariates in the adjustment process. Propensity scores, which correspond to the probability of treatment assignment given pre-treatment characteristics, provide a way to summarize multiple covariates into a single score for each individual. Therefore, this approach is capable of handling a large dimension of confounders, which is particularly useful in registry studies when confounders are measured. Another advantage of PS is that it allows one to check between the treatment groups when conditioning on propensity score whether the confounding factors is balanced out. However, when important confounders are not measured, the PS method is limited. One solution is to perform sensitivity analyses by evaluating how estimated treatment effectiveness might change if there exists an unmeasured confounder with varying levels of prevalence. Such practice will allow one to gauge the impact of unmeasured confounders to the treatment effect.

In this example, the likelihood of tobramycin use depends on unmeasured characteristics at the patient, family or care level. The adjustment of unmeasured confounding that is possible through IV analysis may have led to more intuitive conclusions regarding treatment effect. Since CF care is organized by care center, it was reasonable to examine the validity of a preference-based instrument to combat treatment-selection bias. Thorough sensitivity analyses are necessary to examine the robustness of the IV. We limit our illustrative application to a single instrument. It is possible to include multiple instruments and gain more formal properties to testing assumption (ii).

## **5. Conclusions**

When designing and analyzing registry data, it is critically important to address biases and confounding that are inherent in this type of study. Although we have focused, in this chapter, on describing methods for controlling selection biases, registry data are often subject to other types of biases related to measurement and miss-classification error, immortal time bias, loss to follow up, and missing data. We encourage use of sensitivity analyses to understand the impacts of these potential biases to the study conclusions. There are rich literature sources and several guidelines for design and analysis of registry data. In addition to the literature referenced in this chapter, a very useful resource is the recent report on standards in the conduct of registry studies for patient centered outcomes research and the references therein [33].

In addressing selection bias, most often, treatment effects are examined using multiple linear regression with measured confounders included as covariates [34]. Increasingly, PS methods are employed. However, existing statistical methods to address unmeasured confounding may be underutilized in registry settings. The models that we have presented are by no means exhaustive. There is room to develop more methodology, particularly to combat time-varying treatment effects and utilize time-varying instruments [12]. It is possible that preference-based

instruments will provide a feasible approach to interrogating registries [14]. Admittedly, there are some situations, such as the IV regression specified in Model (3), where the sample size/power analysis calculation is not straightforward. There are approaches to simulate power for this model, but additional assumptions are necessary. Furthermore, in most controlled studies, we can follow up with subjects who drop out. We rarely have this capability in registry settings, which further limits our ability to diagnose the missing data mechanism.

/\*Next, we implement the propensity score regression model previously described. First, we use logistic regression to estimate propensity scores for each

model Tobi=base\_fev1 wtpct age inscat cfrd dnase pancr numhosp gender/

/\*We use the commands below to assign a subject-specific weight that corresponds to his or her propensity score from the logistic regression above. Since the propensity score, denoted *ps* below, corresponds to predicted probability of receiving the treatment, each subject who received the treatment will have weight *1*/*ps*, while each subject who did not receive the treatment will have weight *1*/(*1-ps*). The resulting dataset, *props2*, will consist of the *analysis\_data*, propensity scores that were previously created and stored in *props*, and the *ps\_weight* corresponding to

/\*We now implement the weighted multivariable regression. The commands are similar to our previous regression, except for our use here of the *weight* statement. By using this statement, we request computation of weighted means and variance estimates that are inversely proportional to the corresponding sum of weights.\*/

model dfev1=Tobi base\_fev1 wtpct age inscat cfrd dnase pancr numhosp gender/

/\*Finally, we present commands for the instrumental variables regression. The first model statement performs the first-stage regression of the treatment indicator *Tobi* on the instrument (*cid\_iv*) and all measured confounders. The result is a probit model with predicted probabilities of tobramycin use for each subject. The second model statement performs multiple linear regression with the instrumented version

model Tobi=cid\_iv base\_fev1 wtpct age inscat cfrd dnase pancr numhosp gender

model dfev1=base\_fev1 wtpct age inscat cfrd dnase pancr numhosp gender /

title 'Model (2): Propensity Score Regression';

class inscat cfrd dnase pancr numhosp gender;

each subject's weighting derived from the propensity score.\*/

class Tobi inscat cfrd dnase pancr numhosp gender;

of the tobramycin variable from the first model statement. title 'Model (3): Instrumental Variables Regression';

class inscat cfrd dnase pancr numhosp gender;

output out=Tobi prob proball predicted residual;

proc logistic data=analysis\_data;

*Evaluating Clinical Effectiveness with CF Registries DOI: http://dx.doi.org/10.5772/intechopen.84269*

output out=props pred=ps;

if Tobi=1 then ps\_weight=1/ps; if Tobi=0 then ps\_weight=1/(1-ps);

proc glm data=props2;

lsmeans Tobi/pdiff cl; weight ps\_weight;

proc qlim data=analysis\_data;

subject.\*/

link=logit;

run;

data props2; set props;

run;

cl solution;

run;

/discrete;

run;

**73**

select(Tobi=1);
