Linux Metadata Guide

This guide will help DIY Linux users to understand what ‘metadata’ is, why it’s so important to a provider, why you should be backing it up, common mistakes made etc.

When we refer to a provider’s metadata, we are referring to a folder with multiple components in it. Together they are your metadata and are probably the most vital part of a DIY provider.

By default on Linux, metadata is stored in the user’s home folder in a hidden folder called ‘.scprime’. If you’re running as root this would be /root/.scprime or if running as a normal user it would be /home/<USER>/.scprime. In this folder we can see above we have 6 folders plus a file. While you don’t really need to fully understand all the detail as to what all the files in the metadata do, it’s useful to have a high level understanding as that can help in troubleshooting issues relates to your metadata.

apipassword – this is a file with an API password which spc uses when using the spd API. Common errors we see related to this file are caused by performing some actions with a root account and some with a user account. Errors like:

[POST request error; API authentication failed.]

are caused by running spc commands with a different user than is running spd.

consensus – this folder contains the blockchain. When you first start up a provider node you either let spd download the blockchain automatically or you can download the bootstrap file from the software download page, unzip it into the consensus folder and then start up spd and it will sync the last few remaining blocks.

gateway – this folder contains a few files pertaining to other nodes in the network

host – this folder contains various files and folders specific to your provider like the provider settings which you would have setup like pricing, collateral, upload/download pricing etc., information about contracts

siamux – this is a legacy folder no longer used so we can safely ignore this one

transactionpool – this folder contains a database of all the transactions that have taken place for the provider

wallet – this folder contains a database of all your wallet transactions

Starting SPD and your Metadata

The most important concept to take away from this guide is that each time you startup spd, it needs to be told where the metadata folder is located that it’s going to use. If you use a default location then you don’t need to worry about specifying it (and this is the best approach as it has the least chance of going wrong). If you choose to use a non-default location then you must start spd every time with the -d flag and give it the location of your metadata. So for someone using the default location, the following command would work fine:

./spd -M gctwh

However if you’ve chosen to store metadata in the /Metadata folder, then you must always start spd with the following command:

./spd -d /Metadata -M gctwh

Now another important thing to note about Linux is paths are case sensitive so the following 3 paths are all different:

/Metadata

/MetaData

/metadata

So if you specify /MetaData when you were using /Metadata previously, you will create a new provider effectively. The best way to avoid this is to setup the autostart script as documented in the Linux Setup Guide here. Once you have this setup then the easiest way to start up spd correctly every time is to just reboot your Linux PC and let the script start spd in the same way every time.

Using an Incorrect or Non-existent Metadata Folder

If you do happen to start spd with either an incorrect path to your metadata or forget to specify it at all when using a non-default location, spd will startup but will start downloading a new consensus file and so you might see your Consensus not sync’d and a lower block height than expected as below:

You might also find when you try to unlock your wallet, you get an error like below:

Could not unlock wallet: [POST request error; error when calling /wallet/unlock: [wallet has not been encrypted yet; wallet has not been encrypted yet]]

This is another indication you have created a new provider as your correct wallet would be encrypted. Also looking at the output of the following command to see your Provider ID doesn’t match what it was before.

./spc host -v

Or that you see no storage folders at the bottom of the output. These are all signs you have started spd with a new or different set of metadata. At this point you need to stop spc using:

./spc stop

Now you’ll have figure out where you have gone wrong as the situation is recoverable. If you for example add your storage folders back in, then you will lose any data in them and that won’t be recoverable.

Once spd has been stopped and you’ve worked out where your real metadata is, startup spd again pointing at the correct location and you should see your original Provider ID, be able to unlock the wallet with your password or seed phrase and see your original storage folders. At this point it’s worth then deleting the incorrect set of metadata to avoid using it by mistake in the future.

If you are unsure of where your metadata might be or you’ve created a 2nd copy and don’t know where the right one is, use the following command to help find it:

sudo find / -name consensus.db

This command will search your whole file system looking for consensus.db files, if you’ve got multiple sets of metadata then you will find multiple copies of the consensus.db file. For example:

scprime@ScPrime:~/ScPrime/ScPrime-v1.6.0-linux-amd64$ sudo find / -name consensus.db

[sudo] password for scprime:

find: ‘/run/user/1000/gvfs’: Permission denied

/mnt/USB_Backup/metadata_backup/consensus/consensus.db

/home/scprime/.scprime/consensus/consensus.db

Here we have one set of metadata in /mnt/USB_Backup/metadata_backup and another in /home/scprime/.scprime.

Where Should I Store My Metadata

As already mentioned above, the easiest thing to do is just leave it to be in its default location. This assumes that you’ve installed Linux onto an SSD and that the home folders are on that or another SSD. If you choose a non-default location, just ensure that it’s on an SDD drive. We do not recommend storing the metadata folder on a HDD, especially not the HDD used for your storage folder. The metadata is being written to a lot and needs fast read and write speeds for good performance overall of your provider.

Backing Up Your Metadata

It is highly recommended to backup your metadata folder. If anything goes wrong for example running out of disk space on the drive you are storing the metadata on then it can get corrupted and lead to spd not being able to load up and worse case you may have to start over with a new provider. A backup of your metadata can save this scenario from happening which in the future could result in a lot of lost data and therefore earnings.

It is recommended to backup at least once a day and setup an automated backup script to backup the metadata folder off the machine to either a server you have access to or at least a USB drive.

Assuming you have setup the auto startup script to start spd every time your PC reboots, it’s relatively easy to use that script as the basis for a backup script. If you’ve not setup the auto startup script it’s highly recommended to do that, documented in the Linux Setup Guide here.

For the backup script, we’ll create a file called metadata_backup.sh in the folder where spd and spc are located but it can be saved anywhere. Edit the file with Nano or Vi and paste in the following:

#!/bin/bash

SPD_DATA=$HOME/.scprime

SCPRIME=$HOME/ScPrime/ScPrime-v1.6.0-linux-amd64

# first stop spd

$SCPRIME/spc stop

# copy metadata folder to backup location

cp -r $SPD_DATA/* /mnt/USB_Backup/metadata_backup/

# start spd in the same way as normal

nohup $SCPRIME/spd --host-addr :44282 --host-api-addr :44283 -d $SPD_DATA -M gctwh &

# unlock wallet

sleep 5s

SCPRIME_WALLET_PASSWORD=`cat $HOME/ScPrime/.seed`

export SCPRIME_WALLET_PASSWORD

$SCPRIME/spc wallet unlock

Note that your backup location, command to startup spd, metadata location and spd location will very likely be different, you will need to customize this script to your setup.

Save the file and then make it executable with the following command:

chmod 755 metadata_backup.sh

The script basically stops spd from running, copies the metadata folder to our backup location, starts up spd and unlocks the wallet.

It’s important now to test the script so run it manually using the following:

./metadata_backup.sh

Once it’s completed, check the backup folder and ensure you can see a copy of your metadata there and check to ensure spd is started up and your wallet is unlocked. If so, the final step is to create a cronjob to run the script on a regular basis. To do this, enter the following command:

crontab -e

If this brings a prompt to choose an editor, select your favorite one, that will open a file in your editor which is likely just some commented text with #’s at the start of each line. Go to the bottom of the file and enter the following:

@daily /home/scprime/ScPrime/ScPrime-v1.6.0-linux-amd64/metadata_backup.sh

Save the file and exit the editor. This sets the script to run daily. The path to where your script is located will likely differ so ensure you know the full path to your backup script. For the next few days, check your backups to ensure the backups are being taken and that spd is being restarted correctly.

Another option to backup your metadata is the following script written by discord community member Tacom Ogahiehe – https://github.com/Tacombel/scprime_metadata_backup