Getting Started
Introduction
Enterprises today are inundated with data, yet extracting meaningful insights remains a challenge that can impede growth and strategic agility. Our recommendation engine is at the forefront of addressing this challenge, offering a sophisticated solution that transforms raw, scrapped data into actionable intelligence.
Overview
Our system is a convergence of advanced analytics and user-centric design. It functions by tapping into a comprehensive database of scrapped data, applying complex algorithms, and delivering precise answers to user inquiries. This engine is not just a tool but a strategic asset that can decode the intricacies of vast datasets to inform decision-making processes.
Purpose
The engine is crafted to empower businesses by:
- Generating Deep Insights: Leveraging multidimensional data correlations, it provides users with exhaustive answers that encompass all facets of their queries.
- Enhancing Answer Accuracy: By utilizing cutting-edge analytics, the engine ensures that the answers are not only relevant but also precise.
- Tailoring User Responses: It excels in discerning crucial data attributes, enabling customized responses to user-specific questions.
- Facilitating Real-time Analysis: The engine delivers immediate answers and allows for on-the-fly scenario analysis, adapting to the dynamic context of user questions.
- Demystifying Data Complexity: With its intuitive interface, the engine simplifies the complex data landscape, making sophisticated data-driven recommendations accessible to all tiers of users.
Running the Engine Locally
Cloning the Kinetik Simulation Project
- Open your GitHub repository in a web browser and locate the green “Code” button in the top right corner of the page.
- Click on the “Code” button to open a dropdown menu.
- From the dropdown menu, select the “HTTPS” option to obtain the repository URL in HTTPS format.
- Copy the repository URL to the clipboard.
- Open a terminal or command prompt on your local machine.
- Navigate to the directory where you want to clone the repository using the command
cd
followed by the directory path. - Type the following command to clone the repository to your local machine:
git clone <https://github.com/TaoHuang0/Kinetik.git>
- Press Enter to execute the command. Git will download the repository to your local machine.
- After the cloning process is complete, navigate to the cloned repository directory using the command
cd
. - Now, you can install the project dependencies using the appropriate package manager such as npm or yarn.
Installation
- Open a terminal or command prompt on your local machine.
- Navigate to the cloned repository directory using the command
cd
. - Install the project dependencies using
npm
andyarn
.npm install yarn install
- Wait for the installation process to complete. Once completed, you can run the project locally on your machine.
How to Run this Engine Locally Using AWS Access Keys:
Prerequisites:
Before you can run this Engine locally using AWS Access Keys, you will need to have the following:
An AWS account with appropriate permissions to access the lambda and API resources this app uses
AWS Access Keys for the account you will be using
Node.js and npm installed on your computer
The Amplify CLI installed your computer
Setting up Your AWS Access Keys
To set up your AWS Access Keys, follow these steps:
Log in to your AWS account.
Navigate to the IAM Dashboard
.
Click on Users
in the left-hand menu.
Click on your user name to open your user details.
Click on the Security credentials
tab.
Click on Create access key
to create a new access key.
Download the access key file or copy the access key ID and secret access key.
Running This Amplify App Locally
To run your Engine locally using AWS Access Keys, follow these steps:
- Clone your Amplify app’s repository to your local machine.
- Open a terminal window and navigate to the root directory of your app.
- Run
amplify init
to initialize the Amplify project.
- Run
amplify configure
to configure your AWS Access Keys. Enter the access key ID and secret access key that you obtained earlier, and choose the appropriate region for your app.
Run amplify push to deploy your app’s backend resources to AWS.
Start with Development Server
- Open a terminal or command prompt on your local machine.
- Navigate to the cloned repository directory using the command
cd
. - Start the development server using the appropriate command.
npm start
- Wait for the development server to start. Once started, you can view the project in your web browser by navigating to the appropriate URL, typically
http://localhost:3000
.
Your app should now be running locally and using your AWS resources. If you encounter any issues, consult the Amplify documentation or reach out to the Amplify community for support.
For how to use the Kinetik Simulation Application, please refers to the User Manual for more information.
Recommendation Engine Backend
Backend Login
To make the backend of the recommendation engine runs smoothly, it is deployed on an EC2 with ubuntu as its system. You could find the EC2 in the EC2 dashboard with name Recomm-Ec2
.
Login to the backend
You could use smtp/ftp tool like putty to login to the backend.
- The username for the server is as default:
ubuntu
- There should be no password provided. Instead, provide a
.ppk
file (private key) for the SSH session.
Create backend by Yourself
If you would like to create your own backend by yourself with Amazon EC2 service, you could follow the common EC2 creation instructions. But you need to be cautious for following points:
- Save the private key file in a place you would not forget. You could directly save
.ppk
file from EC2 creation. But if you save a.pem
file, you could use tool likePUTTYgen
to save a.ppk
version of your private key. - Double-check the security policy of your EC2. You need to make sure the EC2 allows inbound traffic of
tcp
andhttp
requests from your front-end ip. (Or you could set to0.0.0.0
but this might cause potential security issue for production). Also, double-check the outbound traffic too.
Backend Structure
After login to the EC2, you could find the recommendation folder under /home/ubuntu/recommendation_backend
. Inside the folder, there is one file called recommedationEngine.py
consisting of the ML and flask, a requirements.txt
, and a tests
folder which has unit tests about backend.
Run the backend
Since we want the backend to run continuously, we need to use the process manager tool to keep it alive after we close the backend window. In the development EC2, we use pm2
to keep the backend alive. But if you would like more advanced process manager for greater volume production, consider using WGSI
(which can be installed through pip install gunicorn
)
The Flow of the Request
Though the front-end is hosted on Amplify and back-end is hosted on EC2, the flow of the request needs few more steps to complete.
Front-End (Amplify) -> AWS HTTP Gateway -> AWS Lambda Function -> EC2 (Backend)
- This ensures the outgoing data from
front-end
togateway
are HTTPS communication - So, the recommendation engine url that front-end calls is not the
EC2
url, it is the gateway url. - The
AWS Lambda Function
calls the EC2 Flask app with HTTP communication.