Configuration
Indexer node configuration
Configuration File
The storetheindex config file is a JSON document located at $STORETHEINDEX_PATH/config. It is read once at indexer instantiation, either for an offline command, or when starting the daemon. Commands that execute on a running daemon do not read the config file at runtime.
Sample Config file
Version
Version
0
Current version of configuration file. This is used to defermine if an existing configuration should be upgraded. No default. This is written during initialization.
Identity
PeerID
12D3KooWJZSPZN7cudwBJ6UdTm8V5FmgWhAd2oVa8qFagU7bZSm1
PeerID is the peer ID of the indexer that must match the given PrivKey. If unspecified, it is automatically generated from the PrivKey. There is no default value. This is generated during initialization.
PrivKey
CAESQKP70L69Q7An97xPH6g3PgSypws6mHYcAZ77t9pC2a0/geY+Kh3gaSzfPqHpSyjy7Fd3hQPweU+BNbhhKHSHrJw=
PrivKey represents the peer identity of the indexer. If unset, the key is loaded from the file at the path specified via STORETHEINDEX_PRIV_KEY_PATH environment variable. There is no default value. This is generated during initialization.
Addresses
Admin
ip4/127.0.0.1/tcp/3002
Admin is the admin http listen address. Set to "none" to disable this server for both http and libp2p.
Finder
/ip4/0.0.0.0/tcp/3000
Finder is the finder http listen address. Set to "none" to disable this server for both http and libp2p.
Ingest
/ip4/0.0.0.0/tcp/3001
Ingest is the index data ingestion http listen address. Set to "none" to disable this server for both http and libp2p.
P2PAddr
/ip4/0.0.0.0/tcp/3003
P2PMaddr is the libp2p host multiaddr for all servers. Set to "none" to disable libp2p hosting.
NoResourceManager
false
NoResourceManager disables the libp2p resource manager when true.
Bootstrap
Peers
[/"dns4/bootstrap-1.mainnet.filops.net/tcp/1347/p2p/12D3KooWCwevHg1yLCvktf2nvLu7L9894mcrJR4MsBCcm4syShVc", ""]
It is an array of multiaddr strings. It defaults to the multiaddrs of the filecoin mainnet bootstrap nodes.
MinimumPeers
4
MinimumPeers governs whether to bootstrap more connections. If the node has less open connections than this number, it will open connections to the bootstrap nodes. Set to 0 to disable bootstrapping.
Datastore
Dir
datastore
Dir is the directory where the datastore is kept. If this is not an absolute path then the location is relative to the indexer repo directory.
Type
levelds
Type is the type of datastore.
Discovery
LotusGateway
LotusGateway is the host or host:port for a lotus gateway used to verify providers on the blockchain.
Policy
Policy configures which providers are allowed and blocked, rate-limited, and allow to publish on behalf of others.
PollInterval
24h0m0s
PollInterval is the amount of time to wait without getting any updates for a provider, before sending a request for the latest advertisement. Values are a number ending in "s", "m", "h" for seconds. minutes, hours.
PollRetryAfter
5h0m0s
PollRetryAfter is the amount of time from one poll attempt, without a response, to the next poll attempt, and is also the time between checks for providers to poll. This value must be smaller than PollStopAfter for there to be more than one poll attempt for a provider.
PollStopAfter
168h0m0s
PollStopAfter is the amount of time, from the start of polling, to continuing polling for the latest advertisment without getting a response.
PollOverrides
PollOverrides configures polling for specific providers.
RediscoverWait
5m0s
RediscoverWait is the amount of time that must pass before a provider can be discovered following a previous discovery attempt. A value of 0 means there is no wait time.
Timeout
2m0s
Timeout is the maximum amount of time that the indexer will spend trying to discover and verify a new provider.
Discovery.Policy
Allow
true
Allow is either false or true, and determines whether a peer is allowed (true) or is blocked (false), by default.
Except
["12D3KooWCwevHg1yLCvktf2nvLu7L9894mcrJR4MsBCcm4syShVc", "12D3KooWKhgq8c7NQ9iGjbyK7v7phXvG6492HQfiDaGHLHLQjk7R"]
Except is a list of peer IDs that are exceptions to the Allow policy. If Allow is true, then all peers are allowed except those listed in Except. If Allow is false, then no peers are allowed except those listed in Except. in other words, Allow=true means that Except is a deny-list and Allow=false means that Except is an allow-list.
Publish
true
Publish determines whether or not peers are allowed to publish advertisements for a provider with a different peer ID.
PublishExcept
["12D3KooWCwevHg1yLCvktf2nvLu7L9894mcrJR4MsBCcm4syShVc", "12D3KooWKhgq8c7NQ9iGjbyK7v7phXvG6492HQfiDaGHLHLQjk7R"]
PublisherExcept is a list of peer IDs that are exceptions to the Publish policy. If Publish is false, then all allowed peers cannot publish advertisements for providers with a different peer ID, unless listed in PublishExcept. If Publish is true, then all allowed peers can publish advertisements for any provider, unless listed in PublishExcept.
Discovery.PollOverrides
There are no default Discovery.PollOverrides elements. These are created manually.
PollOverrides
"PollOverrides": [ { "ProviderID": "12D3KooWRYLtcVBtDpBZDt5zkAVFceEHyozoQxr4giccF7fquHR2", "Interval": "12h0m0s", "RetryAfter": "45m0s", "StopAfter": "3h0m0s" } ],
Polling policy override for specific peers
Indexer
CacheSize
300000
Maximum number of CIDs that cache can hold. Setting to -1 disables the cache.
ConfigCheckInterval
30s
ConfigCheckInterval is the time between config file update checks.
GCInterval
30m0s
GCInterval configures the garbage collection interval for valuestores that support it.
ShutdownTimeout
10s
ShutdownTimeout is the duration that a graceful shutdown has to complete before the daemon process is terminated.
ValueStoreDir
valuestore
Directory where value store is kept. If this is not an absolute path then the location is relative to the indexer repo directory. ValueStoreDir string
ValueStoreType
sth
Type of valuestore to use, such as "sth" or "pogreb".
Ingest
AdvertisementDepthLimit
33554432
AdvertisementDepthLimit is the total maximum recursion depth limit when syncing advertisements. The value -1 means no limit and zero means use the default value. Limiting the depth of advertisements can be done if there is a need to prevent an indexer from ingesting long chains of advertisements. A chain of advertisements is synced by separate requests to the provider for each advertisement. These requests are done in groups (segments) of size set by SyncSegmentDepthLimit. AdvertisementDepthLimit sets the limit on the total number of advertisements across all segments.
EntriesDepthLimit
65536
EntriesDepthLimit is the total maximum recursion depth limit when syncing advertisement entries. The value -1 means no limit and zero means use the default value. The purpose is to prevent overload from extremely long entry chains resulting from publisher misconfiguration. A chain of multihash entries chunks is synced by separate requests to the provider for each chunk. These requests are done in groups (segments) of size set by SyncSegmentDepthLimit. EntriesDepthLimit sets the limit on the total number of entries chunks across all segments.
HttpSyncRetryMax
4
HttpSyncRetryMax sets the maximum number of times HTTP sync requests should be retried.
HttpSyncRetryWaitMax
30s
HttpSyncRetryWaitMax sets the maximum time to wait before retrying a failed HTTP sync.
HttpSyncRetryWaitMin
1s
HttpSyncRetryWaitMin sets the minimum time to wait before retrying a failed HTTP sync.
HttpSyncTimeout
10s
HttpSyncTimeout sets the time limit for HTTP sync requests.
IngestWorkerCount
10
IngestWorkerCount sets how many ingest worker goroutines to spawn. This controls how many concurrent ingest from different providers we can handle.
PubSubTopic
/indexer/ingest/mainnet
PubSubTopic sets the topic name to which to subscribe for ingestion announcements.
RateLimit
RateLimit contains rate-limiting configuration.
ResendDirectAnnounce
false
StoreBatchSize
4096
StoreBatchSize is the number of entries in each write to the value store. Specifying a value less than 2 disables batching. This should be smaller than the maximum number of multihashes in an entry block to write concurrently to the value store.
SyncSegmentDepthLimit
2000
SyncSegmentDepthLimit is the depth limit of a single sync in a series of calls that collectively sync advertisements or their entries. The value -1 disables the segmentation where the sync will be done in a single call and zero means use the default value.
SyncTimeout
2h0m0s
SyncTimeout is the maximum amount of time allowed for a sync to complete before it is canceled. This can be a sync of a chain of advertisements or a chain of advertisement entries. The value is an integer string ending in "s", "m", "h" for seconds. minutes, hours.
Ingest.RateLimit
Apply
false
Apply is either false or true, and determines whether a peer is subject to rate limiting (true) or not (false), by default.
Except
["12D3KooWCwevHg1yLCvktf2nvLu7L9894mcrJR4MsBCcm4syShVc", "12D3KooWKhgq8c7NQ9iGjbyK7v7phXvG6492HQfiDaGHLHLQjk7R"]
Except is a list of peer IDs that are exceptions to the Apply rule. If Apply is false then peers are not rate-limited unless they appear in the Except list. If Apply is true, then only the peers listed in Except are not rate-limited.
BlocksPerSecond
100
BlocksPerSecond is the number of blocks allowed to be transferred per second. An advertisement and a block of multihashes are both represented as a block, so this limit applies to both. Setting a value of 0 disables rate limiting, meaning that the rate is infinite.
BurstSize
500
BurstSize is the maximum number of blocks that can be received at once. After this, BlocksPerSecond additional blocks maybe received each second, and any more results in rate limiting. With HTTP ingestion, rate limiting waits until more blocks are allowed to be received. With graphsync, rate limiting terminates the session and resumes it when sufficient time has passed to be able to receive BurstSize blocks. A value of 0 results in 10 times BlocksPerSecond.
Logging
Level
info
Level sets the log level for all loggers that do not have a setting in // Loggers. The default value is "info".
Loggers
"Loggers": { "basichost": "warn", "bootstrap": "warn", "dt-impl": "warn", "dt_graphsync": "warn", "graphsync": "warn" }
Loggers sets log levels for individual loggers.
Peering
Peers
null
Runtime Reloadable Items
The storetheindex daemon can reload some portions of its config without restarting the entire daemon. This is done by editing the config file and then using the admin sub-command reload-config or sending the daemon process a SIGHUP signal. The daemon will automatically reload the edited config after 30 seconds when the daemon is run with the --watch-config flag or with the environ variable STORETHEINDEX_WATCH_CONFIG=true. The reloadable portions of the config files are:
Discovery.Policy
Indexer.ConfigCheckInterval
Indexer.ShutdownTimeout
Ingest.IngestWorkerCount
Ingest.RateLimit
Ingest.StoreBatchSize
Logging
Peering
Last updated