Syncing Blockfilters
The chain
module has the ability to store BIP157 block filters locally. Generally these filters are useful
for doing wallet rescans. The idea is you can generate a list of script pubkeys you are interested in and see if
the block filter matches the scriptPubKey.
As we demonstrated in the chain docs with block headers, you can sync block filters from an external data source
as well. We are going to use bitcoind as an example of an external data source to sync filters against. It is important
that the bitcoind version you are using is >= v19
as the getblockfilter
rpc is implemented there. You need to make sure bitcoind is started with the -blockfilterindex
flag. This makes it
so we can query filters.
It is important to remember that you need fully synced block headers before you can sync filter headers and filters. Please see the chain docs for syncing block headers.
Abstract idea of syncing filters.
Our internal infrastructure depends on one function to be implemented to be able to sync filters.
val getFilterFunc: BlockHeader => Future[FilterWithHeaderHash] = ???
With getFilterFunc
given a BlockHeader
we can find it's associated GolombFilter
-- which is our internal repesentation
of a BIP157 block filter.
The basic idea for FilterSync.syncFilters()
is to look at our current best block header inside of our ChainApi.getBestBlockHeader()
and then check what our best block filter's block hash is with ChainApi.getBestFilterHeader()
. If the blockfilter returned from our internal
data store is NOT associated with our best block header, we attempt to sync our filter headers to catch up to our best block header.
Syncing block filters against bitcoind
We are going to implement getFilterFunc
with bitcoind and then sync a few filter headers.
implicit val system: ActorSystem = ActorSystem(s"filter-sync-example")
implicit val ec: ExecutionContext = system.dispatcher
implicit val chainAppConfig: ChainAppConfig = BitcoinSTestAppConfig.getNeutrinoTestConfig().chainConf
val instance = BitcoindInstanceLocal.fromConfigFile(BitcoindConfig.DEFAULT_CONF_FILE)
val bitcoind = BitcoindRpcClient(instance)
val bitcoindWithChainApiF: Future[BitcoindBaseVersionChainHandlerViaRpc] = {
ChainUnitTest.createChainApiWithBitcoindRpc(bitcoind)
}
val bitcoindF = bitcoindWithChainApiF.map(_.bitcoindRpc)
val chainApiF = bitcoindWithChainApiF.map(_.chainHandler)
val filterType = FilterType.Basic
val addressF = bitcoindF.flatMap(_.getNewAddress)
//this is the function that we are going to use to sync
//our internal filters against. We use this function to query
//for each block filter associated with a blockheader
val getFilterFunc: BlockHeader => Future[FilterWithHeaderHash] = { blockHeader =>
val prevFilterResultF =
bitcoindF.flatMap(_.getBlockFilter(blockHeader.hashBE, filterType))
prevFilterResultF.map { filterResult =>
FilterWithHeaderHash(filterResult.filter, filterResult.header)
}
}
//ok enough setup, let's generate a block that we need to sync the filter for in bitcoind
val block1F = for {
bitcoind <- bitcoindF
address <- addressF
hashes <- bitcoind.generateToAddress(1,address)
} yield hashes
//to be able to sync filters, we need to make sure our block headers are synced first
//so let's sync our block headers to our internal chainstate
val chainApiSyncedHeadersF = for {
bitcoind <- bitcoindF
handler <- chainApiF
getBestBlockHash = SyncUtil.getBestBlockHashFunc(bitcoind)
getBlockHeader = SyncUtil.getBlockHeaderFunc(bitcoind)
syncedChainApiHeaders <- ChainSync.sync(handler, getBlockHeader, getBestBlockHash)
} yield syncedChainApiHeaders
//now that we have synced our 1 block header, we can now sync the 1 block filter
//associated with that header.
val chainApiSyncedFiltersF = for {
syncedHeadersChainApi <- chainApiSyncedHeadersF
syncedFilters <- FilterSync.syncFilters(syncedHeadersChainApi,getFilterFunc)
} yield syncedFilters
//now we should have synced our one filter, let's make sure we have it
val resultF = for {
chainApi <- chainApiSyncedFiltersF
filterHeaderCount <- chainApi.getFilterHeaderCount()
filterCount <- chainApi.getFilterCount()
} yield {
println(s"filterHeaderCount=$filterHeaderCount filterCount=$filterCount")
}
//cleanup
resultF.onComplete { _ =>
for {
c <- bitcoindWithChainApiF
_ <- ChainUnitTest.destroyBitcoindChainApiViaRpc(c)
_ <- system.terminate()
} yield ()
}
Yay! Now we have synced block filters from an external data source. If you want to repeatedly sync you can just call
FilterSync.syncFilters(syncedFiltersChainApi,getFilterFunc)
every time you would like to sync.
Again, you need to ensure
your headers are synced before you can sync filters, so make sure that you are calling ChainSync.sync()
before syncing
filters.