Add run script and config template

[mary-cleaton]

Description

Add run script and config template. Minor changes done to run script to allow it to work on Spark 3.5.1 and with S3 buckets instead of HDFS, namely:

• Added if __name__ = "__main__": idiom to end of main.py so it can be run directly. Thus, run.py (which called it) is no longer needed for encapsulation purposes. This also ensures everything is within the scalelink folder.
• Added import to the __init__.py file within the scalelink folder. This allows the run_scalelink function in main.py to be imported directly in a script using from scalelink import run_scalelink once the code is packaged.
• Switched HDFS code checkpoint clean-up using subprocess to a checkpoint clean-up method for s3 bucket files using raz_client and boto3.
• Updated the required packages list, config template and read_configs function to facilitate this new checkpoint clean-up method.
• Updated create_spark_session so it works with Spark 3.5.1. Also updated its test.
• Added a configs template and updated it so it will work with the updated main.py.
• Updated test_get_input_variables to reflect changes in configs.

As this is a main script, there are no unit tests associated with it. However, I have tested that it runs using the synthetic test datasets from the Scalelink R package. Whoever picks this up for review, please contact me so that you can carry out the same checks.

Type of change

• Bug fix - non-breaking change
• New feature - non-breaking change
• Breaking change - backwards incompatible change, changes expected behaviour
• Non-user facing change, structural change, dev functionality, docs ...

Checklist:

• I have performed a self-review of my own code.
• I have commented my code appropriately, focusing on explaining my design decisions (explain why, not how).
• I have made corresponding changes to the documentation (comments, docstring, etc.. )
• I have added tests that prove my fix is effective or that my feature works.
• New and existing unit tests pass locally with my changes.
• I have updated the change log.

Peer review

Any new code includes all the following:

• Documentation: docstrings, comments have been added/ updated.
• Style guidelines: New code conforms to the project's contribution guidelines.
• Functionality: The code works as expected, handles expected edge cases and exceptions are handled appropriately.
• Complexity: The code is not overly complex, logic has been split into appropriately sized functions, etc..
• Test coverage: Unit tests cover essential functions for a reasonable range of inputs and conditions. Added and existing tests pass on my machine.

Review comments

Suggestions should be tailored to the code that you are reviewing. Provide context.
Be critical and clear, but not mean. Ask questions and set actions.

These might include:

• bugs that need fixing (does it work as expected? and does it work with other code
  that it is likely to interact with?)
• alternative methods (could it be written more efficiently or with more clarity?)
• documentation improvements (does the documentation reflect how the code actually works?)
• additional tests that should be implemented
  • Do the tests effectively assure that it
    works correctly? Are there additional edge cases/ negative tests to be considered?
• code style improvements (could the code be written more clearly?)

Further reading: code review best practices

[mary-cleaton]

@ONSdigital/scalelink-maintainers and @ONSdigital/scalelink-developers
See comment in description - contact me for files to run this on synthetic data as a test.

[ayomide2021]

I'm happy with all the changes that have made to the run script and config templete.
Is it possible to carryout the checks using the synthetic test datasets from the Scalelink R package?

[mary-cleaton]

I'm happy with all the changes that have made to the run script and config templete. Is it possible to carryout the checks using the synthetic test datasets from the Scalelink R package?

Have put the relevant files in your Dev workspace. :)

[mary-cleaton]

Two minor updates made since last QA.

[mary-cleaton]

After a lot of back-and-forth, I think the reason the most recent commits have had failing GitHub Actions for running the pytests is that Python 3.9 has been discontinued whilst this branch was waiting for QA.

I'm going to revert this to draft and open up the branch that bumps the Python versions. Once that's updated, this branch can be rebased to it and then we can try again to get the GitHub Actions to pass.