Top ten steps to get your genomics data AI-Ready

1. Clean up your data

Real-world genomics data can be messy, which can lead to messy AI predictions.

Begin by backing up raw data and then assessing data quality. Then, clean the data by correcting any errors, removing duplicate records, and fixing missing values – perhaps through estimation or further research. Follow this up by checking for any anomalies or inconsistencies.²

Detecting these issues early can prevent misleading results.

2. Keep it consistent

Standardise the data formats used and address any batch effects, where technical variations can creep in from different sample processing conditions.³ This is a common problem in large-scale biological data analyses. Use batch effect correction techniques such as ComBat, which is for removing technical variability.

Be sure to standardise metadata to provide the AI model with consistent inputs.

3. Make it relevant

Consider how the AI model will be used.

The data need to relate directly to the AI model’s tasks and reflect the real-world biological scenarios the model will encounter. Any irrelevant data will add noise and may lead to unreliable results.

4. Structure your data

AI models rely on well-organised, machine-readable data.

Convert raw sequence reads and other unstructured data into standardised formats such as tables, FASTA files for biological sequences, or BAM files for DNA sequence alignments.

5. Label everything

AI models rely on labelled and annotated data to provide context and ensure reliable predictions. Genomic features such as genes and regulatory elements need to be clearly annotated and linked to relevant biological traits and health outcomes.

Combine computational and manual curation for accurate annotations and more reliable AI results.⁴

6. Ensure dataset diversity

AI models perform best and provide more generalisable results when trained on diverse datasets. This is especially true for orthogonal data, where the variables do not correlate with each other.

Training on broad datasets avoids the AI model ‘overfitting’ to the data, which occurs when it trains too specifically to the target dataset, meaning it will not perform well on new, unfamiliar data.⁵

Solving biological problems often relies on combining diverse data types, so avoid training on a narrow subset of biology.

7. Balance your dataset

Be sure to balance data across different categories, such as healthy versus diseased samples, to avoid skewed or biased results.

Within genomics, underrepresented populations present a significant problem, which can impact the reliability of AI models.

Correct any imbalances by adding data from external sources, generating synthetic data, upweighting any underrepresented samples to ensure fairer learning, or using data resampling techniques to adjust the data distribution.⁶

8. Make it accessible

Take steps to ensure your data are easily accessible to others by storing the data in platforms that allow secure sharing and collaboration. This is central to the FAIR principles (Findable, Accessible, Interoperable, Reusable) of scientific data management.⁷

The FAIR principles ensure data are easily reusable for both machines and individuals, which enhances reproducibility.

9. Track its history

Keep a clear record of the data’s provenance, which is where the data came from and how they are processed, alongside relevant metadata.

Tracking data provenance ensures reproducibility and provides information on data quality because multiple processing steps can introduce errors or bias. It also creates transparency.

Use version control systems, such as Git, and provenance tracking tools like the Open Provenance Model.

10. Scale up for robust AI training

AI models perform best on large, varied datasets.

For large-scale data handling, cloud-based storage solutions, such as AWS for Genomics, and High-Performance Computing (HPC)⁸ can provide appropriate storage and compute resources.

For pre-processing of data, consider automated data pipelines such as Snakemake, Nextflow, or Apache Spark.

Sensitive data, such as patient information, can be kept private and secure using federated learning, in which models train across data nodes without directly accessing raw data.⁹