Skip to main content
Version: 🚀 Next

ETL Pipeline

When do I need to run the ETL pipeline?

You only need to run the ETL pipeline if you would like to reproduce results, collect information beyond the resources we have shared (such as data from blocks created after our cutoff point), or host your own solution.

You do not need to run the ETL pipeline if you only intend to sample new graphs or develop models on the sampled communities we provide.

This section details the Extract, Transform, and Load (ETL) pipeline for collecting Bitcoin data, formatting it as a graph, and importing it into a Neo4j graph database.

You can run the pipeline on the entire Bitcoin history or on a subset of blocks. Since running the pipeline end-to-end is highly resource-intensive, we made two key design decisions: first, we developed it from the ground up to run on a single machine, relying on on-disk processing for resource-heavy operations. Second, given its weeks-long runtime on the entire Bitcoin history, the pipeline is divided into distinct steps, and we provide checkpoint data for each one, allowing you to start the process at any point depending on your application's needs.

Please use the following guide to select the right entry point into the pipeline based on your specific goals and available resources.

  • Step 1: Sync a Bitcoin Node

    If you need to reproduce the entire pipeline, include historical data not in our current graph, or add data from the new blocks created after our cutoff date.

  • Step 2: Extract Block Data

    After completing Step 1, to generate the Bitcoin graph in TSV files.

  • Step 3: Import Data into Neo4j

    This step imports the formatted TSV files from Step 2 into a Neo4j database to enable efficient graph exploration and custom graph sampling.