The MAJA does not produce any L2A products

This might be due to two reasons, if MAJA was installed manually:

1. The wrong version of MAJA was downloaded and installed on the system. The /opt/maja directory should be checked if the version 3.2.2 was installed and not another version like 3.3.x version.

2. The MAJA installs by default with no read rights for other users (only root user can read the MAJA installation directory). In this case, the following command should be executed:

			sudo chmod -R a+rx /opt/maja
		

3. Check that the value in the config table is the correct one for the key “demmaccs.maccs-launcher” that is “/opt/maja/3.2.2/bin/maja“. Also check that the gipp path is the correct one for the parameter “demmaccs.gips-path” that is “/mnt/archive/gipp_maja“:

			psql -U admin sen4cap -c "select * from config where key = 'demmaccs.maccs-launcher'"

			psql -U admin sen4cap -c "select * from config where key = 'demmaccs.gips-path'"
		

4. Check that the gipp_maja directory was copied in the /mnt/archive.

5. Check that srtm and swbd directories were copied in the /mnt/archive.

When a custom job is submitted it remains in the Submitted or Running state but no processor usage is indicated in the “System overview” tab. Also, steps in the job output from “Monitoring” tab do not indicate any progress in execution.

Also, executing the following commands result in a timeout from Slurm srun command:

			sudo su -l sen2agri-service

			srun ls -al
		

which returns something like:

			srun: Required node not available (down, drained or reserved)

			srun: job 3257 queued and waiting for resources
		

This usually happens when during high level products processing (L3X, L4X), the root partition (/) became full and no disk space was available on this partition.

In this case, the Slurm does not execute any commands and it should be re-initiated.

			sudo -u sen2agri-service scontrol update NodeName=localhost State=RESUME

			sudo systemctl restart slurmd slurmdbd slurmctld mariadb
		

The succes of the operation can be checked again by:

			sudo su -l sen2agri-service

			srun ls -al
		

which should display now the list of files in the current directory.

After the upgrade of the system to version 1.1 on some machines was observed that during login from the website, the system displays an error page and in the navigation bar is present the verifyLogin.php script. This results usually from an incorrect removal of the dependency package php-pgsql during the upgrade.

The problem can be verified also by performing the following command:

			sudo tail -f /var/log/httpd/error_log
		

and check for and error related to the pg_connect function (that is not found during the execution of login).

Normally, this will be corrected in the future versions and should be present only from upgrades from 1.0 to 1.1.

In order to solve the issue, the following commands should be executed:

			sudo yum -y install php-pgsql

			sudo systemctl restart httpd
		

After this, the login from the system web page should work now.

Normally, the system is querying for new products on SciHub in an incremental manner meaning that the services are not querying all the time from the beginning of the season but actually the queries are performed starting from the date of the last downloaded product. This is implemented in this way as for large sites the querying is very slow (for sites that consist for more polygons or more tiles and also large seasons, resulting thus into a lot of pages to be extracted, the limit being usually 50 products per page).Due to this mechanism, if some products are processed by SciHub with delays of several days (maybe due to some maintenance activities), some of the products can be missed from downloading.

A force re-check for new products from the begining of the season can be reinitiated by executing the following commands:

psql -U admin sen4cap -c "insert into config (key, value) values ('downloader.S2.forcestart', 'true') on conflict do update set value = 'true'"

psql -U admin sen4cap -c "insert into config (key, value) values ('downloader.S1.forcestart', 'true') on conflict do update set value = 'true'"

psql -U admin sen4cap -c "insert into config (key, value) values ('downloader.L8.forcestart', 'true') on conflict do update set value = 'true'

psql -U admin sen4cap -c "delete from downloader_history where status_id in (3, 4)"

psql -U admin sen4cap -c "update config set value = 'true' where key like '%.forcestart'"

sudo systemctl restart sen2agri-services

Additional filters can be added for a certain site or satellite, during the delete from downloader_history table, for example if only S2 needs to be reset, the above command could be modified in the following way:

psql -U admin sen4cap -c "delete from downloader_history where status_id in (3, 4) and satellite_id = 1"

Also, if the reset is intended only for a specific site, the above command can be modified in the following manner:

psql -U admin sen4cap -c "delete from downloader_history where status_id in (3, 4) and satellite_id = 1 and site_id = "

Where is the id from the database of your site. The list of sites can be viewed using the following command:

psql -U admin sen4cap -c "select id, name, short_name from site"

Sometimes, due to various problems like disk free space not available or failure of the share where /mnt/archive is mounted or failure of reading the input products, MAJA could fail and will mark the product as not available. Although a retry mechanism is implemented (in such cases, the product will be retried every day, for 3 days), still such products can occur in the database and they should be reprocessed once the external problem is solved.

In order to force reprocessing of the failed L2A products the following commands could be executed in the command line:

psql -U admin sen4cap -c "delete from l1_tile_history where downloader_history_id in (select id from downloader_history where status_id in (6,7))"

psql -U admin sen4cap -c "update downloader_history set status_id = 2 where status_id in (6,7)"

Please note that this will reset all the products that failed, even if the failure was due to high percentage of clouds or number of invalid pixels. This means that the products that previously failed from these reasons, will be marked as failed also at the next execution.

If ALL the products need to be re-processed (for example, due to a replacing the wrong version of MAJA that creates products in native format with a good one that creates products in MUSCATE format), in the list of status_id should be added also value 5 i.e replacing:

where status_id in (6,7)

by

where status_id in (5, 6,7)

and additionally, delete also the already processed L2A products from the product table with:

psql -U admin sen4cap -c "delete from product where product_type_id = 1 and satellite_id = 1"

IMPORTANT NOTE: The removal of all S2 L2A products in the database should be made with care and only in the cases when this is justified (like changing MAJA installation) otherwise an unnecessary reprocessing will be performed.

If the reset is desired only for a certain site or satellite, additional filters can be added in the queries above. For example, if only the products from the satellite S2 need to be reset the following queries can be used:

psql -U admin sen4cap -c "delete from l1_tile_history where downloader_history_id in (select id from downloader_history where status_id in (6,7) and satellite_id = 1)"

psql -U admin sen4cap -c "update downloader_history set status_id = 2 where status_id in (6,7) and satellite_id = 1"

Similarly, a filter on site_id can be added after at the end by appending:

" and site_id = <YOUR_SITE_ID>"

where <YOUR_SITE_ID> is the id of your site that you want to be reset.

The list of sites can be viewed using the following command:

psql -U admin sen4cap -c "select id, name, short_name from site"