Administrator Manual

Getting Started

Introduction

Enterprises today are inundated with data, yet extracting meaningful insights remains a challenge that can impede growth and strategic agility. Our recommendation engine is at the forefront of addressing this challenge, offering a sophisticated solution that transforms raw, scrapped data into actionable intelligence.

Overview

Our system is a convergence of advanced analytics and user-centric design. It functions by tapping into a comprehensive database of scrapped data, applying complex algorithms, and delivering precise answers to user inquiries. This engine is not just a tool but a strategic asset that can decode the intricacies of vast datasets to inform decision-making processes.

Purpose

The engine is crafted to empower businesses by:

  • Generating Deep Insights: Leveraging multidimensional data correlations, it provides users with exhaustive answers that encompass all facets of their queries.
  • Enhancing Answer Accuracy: By utilizing cutting-edge analytics, the engine ensures that the answers are not only relevant but also precise.
  • Tailoring User Responses: It excels in discerning crucial data attributes, enabling customized responses to user-specific questions.
  • Facilitating Real-time Analysis: The engine delivers immediate answers and allows for on-the-fly scenario analysis, adapting to the dynamic context of user questions.
  • Demystifying Data Complexity: With its intuitive interface, the engine simplifies the complex data landscape, making sophisticated data-driven recommendations accessible to all tiers of users.

Running the Engine Locally

Cloning the Kinetik Simulation Project

  1. Open your GitHub repository in a web browser and locate the green “Code” button in the top right corner of the page.
  2. Click on the “Code” button to open a dropdown menu.
  3. From the dropdown menu, select the “HTTPS” option to obtain the repository URL in HTTPS format.
  4. Copy the repository URL to the clipboard.
  5. Open a terminal or command prompt on your local machine.
  6. Navigate to the directory where you want to clone the repository using the command cd followed by the directory path.
  7. Type the following command to clone the repository to your local machine:
git clone <https://github.com/TaoHuang0/Kinetik.git>
  1. Press Enter to execute the command. Git will download the repository to your local machine.
  2. After the cloning process is complete, navigate to the cloned repository directory using the command cd.
  3. Now, you can install the project dependencies using the appropriate package manager such as npm or yarn.

Installation

  1. Open a terminal or command prompt on your local machine.
  2. Navigate to the cloned repository directory using the command cd.
  3. Install the project dependencies using npm and yarn.npm install yarn install
  4. Wait for the installation process to complete. Once completed, you can run the project locally on your machine.

How to Run this Engine Locally Using AWS Access Keys:

Prerequisites:

Before you can run this Engine locally using AWS Access Keys, you will need to have the following:

An AWS account with appropriate permissions to access the lambda and API resources this app uses

AWS Access Keys for the account you will be using

Node.js and npm installed on your computer

The Amplify CLI installed your computer

Setting up Your AWS Access Keys

To set up your AWS Access Keys, follow these steps:

Log in to your AWS account.

Navigate to the IAM Dashboard.

Click on Users in the left-hand menu.

Click on your user name to open your user details.

Click on the Security credentials tab.

Click on Create access key to create a new access key.

Download the access key file or copy the access key ID and secret access key.

Running This Amplify App Locally

To run your Engine locally using AWS Access Keys, follow these steps:

  1. Clone your Amplify app’s repository to your local machine.
  2. Open a terminal window and navigate to the root directory of your app.
  3. Run
amplify init

to initialize the Amplify project.

  1. Run
amplify configure

to configure your AWS Access Keys. Enter the access key ID and secret access key that you obtained earlier, and choose the appropriate region for your app.

Run amplify push to deploy your app’s backend resources to AWS.

Start with Development Server

  1. Open a terminal or command prompt on your local machine.
  2. Navigate to the cloned repository directory using the command cd.
  3. Start the development server using the appropriate command.
npm start
  1. Wait for the development server to start. Once started, you can view the project in your web browser by navigating to the appropriate URL, typically http://localhost:3000.

Your app should now be running locally and using your AWS resources. If you encounter any issues, consult the Amplify documentation or reach out to the Amplify community for support.

For how to use the Kinetik Simulation Application, please refers to the User Manual for more information.

Recommendation Engine Backend

Backend Login

To make the backend of the recommendation engine runs smoothly, it is deployed on an EC2 with ubuntu as its system. You could find the EC2 in the EC2 dashboard with name Recomm-Ec2.

Login to the backend

You could use smtp/ftp tool like putty to login to the backend.

  • The username for the server is as default: ubuntu
  • There should be no password provided. Instead, provide a .ppk file (private key) for the SSH session.

Create backend by Yourself

If you would like to create your own backend by yourself with Amazon EC2 service, you could follow the common EC2 creation instructions. But you need to be cautious for following points:

  • Save the private key file in a place you would not forget. You could directly save .ppk file from EC2 creation. But if you save a .pem file, you could use tool like PUTTYgen to save a .ppk version of your private key.
  • Double-check the security policy of your EC2. You need to make sure the EC2 allows inbound traffic of tcp and http requests from your front-end ip. (Or you could set to 0.0.0.0 but this might cause potential security issue for production). Also, double-check the outbound traffic too.

Backend Structure

After login to the EC2, you could find the recommendation folder under /home/ubuntu/recommendation_backend. Inside the folder, there is one file called recommedationEngine.py consisting of the ML and flask, a requirements.txt, and a tests folder which has unit tests about backend.

Run the backend

Since we want the backend to run continuously, we need to use the process manager tool to keep it alive after we close the backend window. In the development EC2, we use pm2 to keep the backend alive. But if you would like more advanced process manager for greater volume production, consider using WGSI (which can be installed through pip install gunicorn)

The Flow of the Request

Though the front-end is hosted on Amplify and back-end is hosted on EC2, the flow of the request needs few more steps to complete. 

Front-End (Amplify) -> AWS HTTP Gateway -> AWS Lambda Function -> EC2 (Backend)
  • This ensures the outgoing data from front-end to gateway are HTTPS communication
  • So, the recommendation engine url that front-end calls is not the EC2 url, it is the gateway url.
  • The AWS Lambda Function calls the EC2 Flask app with HTTP communication.