Linux Metadata Guide

This guide will help DIY Linux users to understand what ‘metadata’ is, why it’s so important to a provider, why you should be backing it up, common mistakes made etc.

When we refer to a provider’s metadata, we are referring to a folder with multiple components in it. Together they are your metadata and are probably the most vital part of a DIY provider.

By default on Linux, metadata is stored in the user’s home folder in a hidden folder called ‘.scprime’. If you’re running as root this would be /root/.scprime or if running as a normal user it would be /home/<USER>/.scprime. In this folder we can see above we have 6 folders plus a file. While you don’t really need to fully understand all the detail as to what all the files in the metadata do, it’s useful to have a high level understanding as that can help in troubleshooting issues relates to your metadata.

apipassword – this is a file with an API password which spc uses when using the spd API. Common errors we see related to this file are caused by performing some actions with a root account and some with a user account. Errors like:

[POST request error; API authentication failed.]

are caused by running spc commands with a different user than is running spd.

consensus – this folder contains the blockchain. When you first start up a provider node you either let spd download the blockchain automatically or you can download the bootstrap file from the software download page, unzip it into the consensus folder and then start up spd and it will sync the last few remaining blocks.

gateway – this folder contains a few files pertaining to other nodes in the network

host – this folder contains various files and folders specific to your provider like the provider settings which you would have setup like pricing, collateral, upload/download pricing etc., information about contracts

siamux – this is a legacy folder no longer used so we can safely ignore this one

transactionpool – this folder contains a database of all the transactions that have taken place for the provider

wallet – this folder contains a database of all your wallet transactions

Starting SPD and your Metadata

The most important concept to take away from this guide is that each time you startup spd, it needs to be told where the metadata folder is located that it’s going to use. If you use a default location then you don’t need to worry about specifying it (and this is the best approach as it has the least chance of going wrong). If you choose to use a non-default location then you must start spd every time with the -d flag and give it the location of your metadata. So for someone using the default location, the following command would work fine:

./spd -M gctwh

However if you’ve chosen to store metadata in the /Metadata folder, then you must always start spd with the following command:

./spd -d /Metadata -M gctwh

Now another important thing to note about Linux is paths are case sensitive so the following 3 paths are all different:

/Metadata

/MetaData

/metadata

So if you specify /MetaData when you were using /Metadata previously, you will create a new provider effectively. The best way to avoid this is to setup the autostart script as documented in the Linux Setup Guide here. Once you have this setup then the easiest way to start up spd correctly every time is to just reboot your Linux PC and let the script start spd in the same way every time.

Using an Incorrect or Non-existent Metadata Folder

If you do happen to start spd with either an incorrect path to your metadata or forget to specify it at all when using a non-default location, spd will startup but will start downloading a new consensus file and so you might see your Consensus not sync’d and a lower block height than expected as below:

You might also find when you try to unlock your wallet, you get an error like below:

Could not unlock wallet: [POST request error; error when calling /wallet/unlock: [wallet has not been encrypted yet; wallet has not been encrypted yet]]

This is another indication you have created a new provider as your correct wallet would be encrypted. Also looking at the output of the following command to see your Provider ID doesn’t match what it was before.

./spc host -v

Or that you see no storage folders at the bottom of the output. These are all signs you have started spd with a new or different set of metadata. At this point you need to stop spc using:

./spc stop

Now you’ll have figure out where you have gone wrong as the situation is recoverable. If you for example add your storage folders back in, then you will lose any data in them and that won’t be recoverable.

Once spd has been stopped and you’ve worked out where your real metadata is, startup spd again pointing at the correct location and you should see your original Provider ID, be able to unlock the wallet with your password or seed phrase and see your original storage folders. At this point it’s worth then deleting the incorrect set of metadata to avoid using it by mistake in the future.

If you are unsure of where your metadata might be or you’ve created a 2nd copy and don’t know where the right one is, use the following command to help find it:

sudo find / -name consensus.db

This command will search your whole file system looking for consensus.db files, if you’ve got multiple sets of metadata then you will find multiple copies of the consensus.db file. For example:

scprime@ScPrime:~/ScPrime/ScPrime-v1.6.0-linux-amd64$ sudo find / -name consensus.db

[sudo] password for scprime:

find: ‘/run/user/1000/gvfs’: Permission denied

/mnt/USB_Backup/metadata_backup/consensus/consensus.db

/home/scprime/.scprime/consensus/consensus.db

Here we have one set of metadata in /mnt/USB_Backup/metadata_backup and another in /home/scprime/.scprime.

Where Should I Store My Metadata

As already mentioned above, the easiest thing to do is just leave it to be in its default location. This assumes that you’ve installed Linux onto an SSD and that the home folders are on that or another SSD. If you choose a non-default location, just ensure that it’s on an SDD drive. We do not recommend storing the metadata folder on a HDD, especially not the HDD used for your storage folder. The metadata is being written to a lot and needs fast read and write speeds for good performance overall of your provider. If anything goes wrong, for example running out of disk space on the drive you are storing the metadata on, then it can get corrupted and lead to spd not being able to load up and worse case you may have to start over with a new provider.