Configuration
Indexer node configuration
Configuration File
The storetheindex config file is a JSON document located at $STORETHEINDEX_PATH/config
. It is read once at indexer instantiation, either for an offline command, or when starting the daemon. Commands that execute on a running daemon do not read the config file at runtime.
Sample Config file
Version
Parameter | Example | Description |
---|---|---|
Version | 0 | Current version of configuration file. This is used to defermine if an existing configuration should be upgraded. No default. This is written during initialization. |
Identity
Parameter | Example | Description |
---|---|---|
PeerID | 12D3KooWJZSPZN7cudwBJ6UdTm8V5FmgWhAd2oVa8qFagU7bZSm1 | PeerID is the peer ID of the indexer that must match the given PrivKey. If unspecified, it is automatically generated from the PrivKey. There is no default value. This is generated during initialization. |
PrivKey | CAESQKP70L69Q7An97xPH6g3PgSypws6mHYcAZ77t9pC2a0/geY+Kh3gaSzfPqHpSyjy7Fd3hQPweU+BNbhhKHSHrJw= | PrivKey represents the peer identity of the indexer. If unset, the key is loaded from the file at the path specified via STORETHEINDEX_PRIV_KEY_PATH environment variable. There is no default value. This is generated during initialization. |
Addresses
Parameter | Example | Description |
---|---|---|
Admin | ip4/127.0.0.1/tcp/3002 | Admin is the admin http listen address. Set to "none" to disable this server for both http and libp2p. |
Finder | /ip4/0.0.0.0/tcp/3000 | Finder is the finder http listen address. Set to "none" to disable this server for both http and libp2p. |
Ingest | /ip4/0.0.0.0/tcp/3001 | Ingest is the index data ingestion http listen address. Set to "none" to disable this server for both http and libp2p. |
P2PAddr | /ip4/0.0.0.0/tcp/3003 | P2PMaddr is the libp2p host multiaddr for all servers. Set to "none" to disable libp2p hosting. |
NoResourceManager | false | NoResourceManager disables the libp2p resource manager when true. |
Bootstrap
Parameter | Example | Description |
---|---|---|
Peers | [/"dns4/bootstrap-1.mainnet.filops.net/tcp/1347/p2p/12D3KooWCwevHg1yLCvktf2nvLu7L9894mcrJR4MsBCcm4syShVc", ""] | It is an array of multiaddr strings. It defaults to the multiaddrs of the filecoin mainnet bootstrap nodes. |
MinimumPeers | 4 | MinimumPeers governs whether to bootstrap more connections. If the node has less open connections than this number, it will open connections to the bootstrap nodes. Set to 0 to disable bootstrapping. |
Datastore
Parameter | Example | Description |
---|---|---|
Dir | datastore | Dir is the directory where the datastore is kept. If this is not an absolute path then the location is relative to the indexer repo directory. |
Type | levelds | Type is the type of datastore. |
Discovery
Parameter | Example | Description |
---|---|---|
LotusGateway | LotusGateway is the host or host:port for a lotus gateway used to verify providers on the blockchain. | |
Policy | Policy configures which providers are allowed and blocked, rate-limited, and allow to publish on behalf of others. | |
PollInterval | 24h0m0s | PollInterval is the amount of time to wait without getting any updates for a provider, before sending a request for the latest advertisement. Values are a number ending in "s", "m", "h" for seconds. minutes, hours. |
PollRetryAfter | 5h0m0s | PollRetryAfter is the amount of time from one poll attempt, without a response, to the next poll attempt, and is also the time between checks for providers to poll. This value must be smaller than PollStopAfter for there to be more than one poll attempt for a provider. |
PollStopAfter | 168h0m0s | PollStopAfter is the amount of time, from the start of polling, to continuing polling for the latest advertisment without getting a response. |
PollOverrides | PollOverrides configures polling for specific providers. | |
RediscoverWait | 5m0s | RediscoverWait is the amount of time that must pass before a provider can be discovered following a previous discovery attempt. A value of 0 means there is no wait time. |
Timeout | 2m0s | Timeout is the maximum amount of time that the indexer will spend trying to discover and verify a new provider. |
Discovery.Policy
Parameter | Example | Description |
---|---|---|
Allow | true | Allow is either false or true, and determines whether a peer is allowed (true) or is blocked (false), by default. |
Except | ["12D3KooWCwevHg1yLCvktf2nvLu7L9894mcrJR4MsBCcm4syShVc", "12D3KooWKhgq8c7NQ9iGjbyK7v7phXvG6492HQfiDaGHLHLQjk7R"] | Except is a list of peer IDs that are exceptions to the Allow policy. If Allow is true, then all peers are allowed except those listed in Except. If Allow is false, then no peers are allowed except those listed in Except. in other words, Allow=true means that Except is a deny-list and Allow=false means that Except is an allow-list. |
Publish | true | Publish determines whether or not peers are allowed to publish advertisements for a provider with a different peer ID. |
PublishExcept | ["12D3KooWCwevHg1yLCvktf2nvLu7L9894mcrJR4MsBCcm4syShVc", "12D3KooWKhgq8c7NQ9iGjbyK7v7phXvG6492HQfiDaGHLHLQjk7R"] | PublisherExcept is a list of peer IDs that are exceptions to the Publish policy. If Publish is false, then all allowed peers cannot publish advertisements for providers with a different peer ID, unless listed in PublishExcept. If Publish is true, then all allowed peers can publish advertisements for any provider, unless listed in PublishExcept. |
Discovery.PollOverrides
There are no default Discovery.PollOverrides
elements. These are created manually.
Parameter | Example | Description |
---|---|---|
PollOverrides | "PollOverrides": [ { "ProviderID": "12D3KooWRYLtcVBtDpBZDt5zkAVFceEHyozoQxr4giccF7fquHR2", "Interval": "12h0m0s", "RetryAfter": "45m0s", "StopAfter": "3h0m0s" } ], | Polling policy override for specific peers |
Indexer
Parameter | Example | Description |
---|---|---|
CacheSize | 300000 | Maximum number of CIDs that cache can hold. Setting to -1 disables the cache. |
ConfigCheckInterval | 30s | ConfigCheckInterval is the time between config file update checks. |
GCInterval | 30m0s | GCInterval configures the garbage collection interval for valuestores that support it. |
ShutdownTimeout | 10s | ShutdownTimeout is the duration that a graceful shutdown has to complete before the daemon process is terminated. |
ValueStoreDir | valuestore | Directory where value store is kept. If this is not an absolute path then the location is relative to the indexer repo directory. ValueStoreDir string |
ValueStoreType | sth | Type of valuestore to use, such as "sth" or "pogreb". |
Ingest
Parameter | Example | Description |
---|---|---|
AdvertisementDepthLimit | 33554432 | AdvertisementDepthLimit is the total maximum recursion depth limit when syncing advertisements. The value -1 means no limit and zero means use the default value. Limiting the depth of advertisements can be done if there is a need to prevent an indexer from ingesting long chains of advertisements. A chain of advertisements is synced by separate requests to the provider for each advertisement. These requests are done in groups (segments) of size set by SyncSegmentDepthLimit. AdvertisementDepthLimit sets the limit on the total number of advertisements across all segments. |
EntriesDepthLimit | 65536 | EntriesDepthLimit is the total maximum recursion depth limit when syncing advertisement entries. The value -1 means no limit and zero means use the default value. The purpose is to prevent overload from extremely long entry chains resulting from publisher misconfiguration. A chain of multihash entries chunks is synced by separate requests to the provider for each chunk. These requests are done in groups (segments) of size set by SyncSegmentDepthLimit. EntriesDepthLimit sets the limit on the total number of entries chunks across all segments. |
HttpSyncRetryMax | 4 | HttpSyncRetryMax sets the maximum number of times HTTP sync requests should be retried. |
HttpSyncRetryWaitMax | 30s | HttpSyncRetryWaitMax sets the maximum time to wait before retrying a failed HTTP sync. |
HttpSyncRetryWaitMin | 1s | HttpSyncRetryWaitMin sets the minimum time to wait before retrying a failed HTTP sync. |
HttpSyncTimeout | 10s | HttpSyncTimeout sets the time limit for HTTP sync requests. |
IngestWorkerCount | 10 | IngestWorkerCount sets how many ingest worker goroutines to spawn. This controls how many concurrent ingest from different providers we can handle. |
PubSubTopic | /indexer/ingest/mainnet | PubSubTopic sets the topic name to which to subscribe for ingestion announcements. |
RateLimit | RateLimit contains rate-limiting configuration. | |
ResendDirectAnnounce | false | |
StoreBatchSize | 4096 | StoreBatchSize is the number of entries in each write to the value store. Specifying a value less than 2 disables batching. This should be smaller than the maximum number of multihashes in an entry block to write concurrently to the value store. |
SyncSegmentDepthLimit | 2000 | SyncSegmentDepthLimit is the depth limit of a single sync in a series of calls that collectively sync advertisements or their entries. The value -1 disables the segmentation where the sync will be done in a single call and zero means use the default value. |
SyncTimeout | 2h0m0s | SyncTimeout is the maximum amount of time allowed for a sync to complete before it is canceled. This can be a sync of a chain of advertisements or a chain of advertisement entries. The value is an integer string ending in "s", "m", "h" for seconds. minutes, hours. |
Ingest.RateLimit
Parameter | Example | Description |
---|---|---|
Apply | false | Apply is either false or true, and determines whether a peer is subject to rate limiting (true) or not (false), by default. |
Except | ["12D3KooWCwevHg1yLCvktf2nvLu7L9894mcrJR4MsBCcm4syShVc", "12D3KooWKhgq8c7NQ9iGjbyK7v7phXvG6492HQfiDaGHLHLQjk7R"] | Except is a list of peer IDs that are exceptions to the Apply rule. If Apply is false then peers are not rate-limited unless they appear in the Except list. If Apply is true, then only the peers listed in Except are not rate-limited. |
BlocksPerSecond | 100 | BlocksPerSecond is the number of blocks allowed to be transferred per second. An advertisement and a block of multihashes are both represented as a block, so this limit applies to both. Setting a value of 0 disables rate limiting, meaning that the rate is infinite. |
BurstSize | 500 | BurstSize is the maximum number of blocks that can be received at once. After this, BlocksPerSecond additional blocks maybe received each second, and any more results in rate limiting. With HTTP ingestion, rate limiting waits until more blocks are allowed to be received. With graphsync, rate limiting terminates the session and resumes it when sufficient time has passed to be able to receive BurstSize blocks. A value of 0 results in 10 times BlocksPerSecond. |
Logging
Parameter | Example | Description |
---|---|---|
Level | info | Level sets the log level for all loggers that do not have a setting in // Loggers. The default value is "info". |
Loggers | "Loggers": { "basichost": "warn", "bootstrap": "warn", "dt-impl": "warn", "dt_graphsync": "warn", "graphsync": "warn" } | Loggers sets log levels for individual loggers. |
Peering
Parameter | Example | Description |
---|---|---|
Peers | null |
Runtime Reloadable Items
The storetheindex daemon can reload some portions of its config without restarting the entire daemon. This is done by editing the config file and then using the admin sub-command reload-config
or sending the daemon process a SIGHUP
signal. The daemon will automatically reload the edited config after 30 seconds when the daemon is run with the --watch-config
flag or with the environ variable STORETHEINDEX_WATCH_CONFIG=true
. The reloadable portions of the config files are:
Discovery.Policy
Indexer.ConfigCheckInterval
Indexer.ShutdownTimeout
Ingest.IngestWorkerCount
Ingest.RateLimit
Ingest.StoreBatchSize
Logging
Peering
Last updated