Domain shift is a frequent issue in machine learning problems, particularly in biomedical imaging contexts due to differences in imaging equipment. These differences induce systematic bias in imaging features, preventing generalization of most machine learning models across different imaging modalities. In this work, I analyze the presence of domain shift in the ADNI MRI dataset for an Alzheimer’s Disease prediction task and develop a novel hyperparameter optimization technique to weight certain target domains optimally.

The ADNI dataset contains multiple different “phases” of study: ADNI-1, ADNI-2, and ADNI-3. Apart from obvious differences in the time period of the studies and the subjects used in each, there are also clear differences in MRI imaging. ADNI-1 used a 1.5 Tesla MRI machine, whereas ADNI-2 and ADNI-3 used 3 Tesla machines. Comparing the extracted MUSE features from raw 3D MRI reveals significant distributional difference, with a Kolmogorov-Smirnov p-value of less than 1e-10. ADNI-1 is one domain, whereas ADNI-2/3 are another.

I benchmarked the performance of an ensemble model, trained using the AutoGluon AutoML framework, when trained on different domains. For this analysis, I restricted the feature set to only include MUSE MRI features; other features were sparse or lacked insight. AutoGluon TabularPredictors were separately trained on just ADNI-1, just ADNI-2/3, and ADNI-1 and ADNI-2/3 combined, with 20% of the data held out for testing, for the same amount of time and the same presets. The TabularPredictors were then evaluated on test sets of just ADNI-1, just ADNI-2/3, and ADNI-1 and ADNI-2/3 combined. The results showed a clear skew, where the domain of the training set led to improved performance on that domain’s test set, although it was never the case that ADNI-1 had higher accuracy than ADNI-2. This indicates further domain adaptation is necessary to boost performance across domains.

One such domain adaptation approach is so-called “domain weighting” — one can weight the samples of a certain domain to increase its importance in a model. For example, for a neural network, a sample’s contribution to the loss function is weighted.

The main contribution of my work is the development of software that can tune for the optimal weight ratio between different domains in a dataset. AutoGluon contains native support for advanced, continuous search space hyperparameter optimization. I created a “wrapper” model that takes in a certain weighting profile as a hyperparameter, and fits an ensemble mode within it to perform the actual classification task. Using this approach, I was able to tune the weight ratio “alpha” to down-weight ADNI-2/3 to 0.74, which led to 98% accuracy on both ADNI-1 and ADNI-2/3 test sets. I further experimented with decreased sample sizes for ADNI-1 in training, which led to improved results.

This research provides an initial approach for domain adaptation using domain weights in an AutoML framework. In the future, I would like to investigate the relationship between the optimal weight ratio and the frequency ratio between domains. In my research, this relationship was nonlinear, which seemed counterintuitive. I would also like to experiment with other domain adaptation techniques, including more representational learning approaches.