Blog | Utkarsh's Blogs

Reinforcement Learning Series [01] - An introduction to RL

Mon, 27 Jan 2025 00:00:00 +0000

This will be series of articles based on the notes that I had made when I was learning RL from the Reinforcement Learning Course by David Silver from Google DeepMind. The articles are not just repetitions of the content delivered in the course, but my versions of the topics and how I understood them.

Since Reinforcement Learning is a very math intensive course, I have broken down the concepts and the mathematical equations to layman terms and explained them in my own words so as to make them as simple as possible.

In addition to the course by David Silver, I had also used the book titled, “An Introduction to Reinforcement Learning” by Sutton and Barto which is like a bible for any beginner who wants to get into this field. This book tries to explain the mathematical concepts with as many real like example as possible. Online version of the book can also be found here.

Introduction to Reinforcement Learning

Reinforcement Learning are a class of Machine Learning algorithms in which we have an Agent interacting with an Environment; which provides a numeric reward to the agent telling it how good or bad it was performing.

The goal of the agent in Reinforcement Learning is to learn to take actions in the environment which will maximize the reward. A real life analogous example of RL would be how you would train a puppy. You reward the puppy with a treat when it does what you expected it to do and punish it by not giving the treat when it does not do what you had asked for.

Following is a simple diagram representing the various entities in Reinforcement Learning:

Here, at time step t, the agent was initially in state Sₜ and thus got a reward Rₜ by the environment. Based on that reward:

The agent takes an action Aₜ .
Based on the action taken, it receives some reward Rₜ₊₁ from the environment in the next time step.
Then moves on to the next state Sₜ₊₁.

This cycle keeps continuing during the entire reinforcement learning process until a termination condition / state is reached.

Difference between Reinforcement Learning and other Machine Learning algorithms

A major difference in the case of Reinforcement Learning and other traditional Machine Learning algorithms, especially the Supervised Learning ones is that, there is no ‘Supervisor’ in this case. No one tells the algorithms about which actions to take. This is just a trial and error paradigm. The best thing to do is not told by anyone and the only way to know if it was a good or a bad action is by looking at the reward signal.
The feedback in Reinforcement Learning is not instantaneous, and is delayed. This means that, if the agent makes a decision now at a current time step, many steps later it will know if it was a good or a bad decision taken previously based on the cumulative reward at the end.
As we work we sequential decision making in Reinforcement Learning, time really matters. The data which the agent trains upon is highly correlated.

The actions taken by an agent at the current time step affects the subsequent data that it will receive in the future. An example to understand this would be, if an RL agent is controlling a robot which is moving in a particular direction, based on the action taken by the agent to move the robot in different directions, the data received will be related to those particular directions only.

The Reinforcement Learning Problem

In this section, we will be talking about the aspects of a Reinforcement Learning model such as Rewards functions, State functions, Environments etc.

Reward Functions

The Reward function (Rₜ) is a scalar feedback signal which indicates how well an agent is doing at any time step t. The objective of an RL agent is to maximize the cumulative reward. This means that the RL agent will take necessary actions so that the total reward received in the end is as high as possible.

The concept of Reinforcement Learning is based on the ‘Reward Hypothesis’, which states that : " All goals can be described by the maximization of expected cumulative reward “. In simple terms, every goal has some or the other intermediate reward and the achievement of that particular goal can be done by getting maximum amount of those intermediate rewards. An example for this can be, lets suppose someone’s goal is to loose certain amount of weight, the intermediate goal can be working out daily and loosing small amounts of weight each week and by maximising this intermediate rewards each week, the ultimate goal of reducing that certain amount of weight can be achieved over time.

History and State

As analogous to real life, History in RL is a sequence of previous observations, actions reward which happened during the training of the RL model. This is the informations which the RL agent has seen so far. Mathematically, it represents all the observable variables up the current time step t.

$$ \mathbf H_t = (\{\mathbf A_1, \mathbf O_1, \mathbf R_1 \}, ...., \{\mathbf A_t, \mathbf O_t, \mathbf R_t \}) $$

What happens next in the Reinforcement Learning is dependent on the history of the model. We aim to develop an algorithm which represents the mapping of the history to the future actions of the model. The agent will learn from the history and accordingly choose actions. Finally, according to the action taken by the agent, the environment will reward or penalize the agent.

Instead of using history, which is a long piece of information and might be not be completely useful, we use ‘State’, to determine what happens next. State is nothing but a function of the History.

$$ \mathbf S_t = f(H_t) $$

Environment State

The Environemnt State ( $\mathbf S_{t}^{e}$ ) is the private representation of the environment in which the agent is reacting. It represents the data which the environment uses to pick the next observation or reward.

The reaosn it is called ‘private representation’ is because, usually this data is not visible to the agent. Even if it might be visible, it may contain data which is irrelevant to the environment.

When we deal with multi-agent systems in the future, an individual agent can consider all other agents as part of the environment.

The environment state $\mathbf S_{t}^{e}$ is Markov by definition, what this means, we will discuss in upcoming sections.

Agent State

The Agent State $\mathbf S_{t}^{a}$ is the internal representation of the agent in a RL model. It is nothing but a set of numbers inside the agent, which it uses as information to pick its next action. This information is used by the reinforcement learning algortihms. Agent State can be any function of the history.

$$ \mathbf S_{t}^{a} = f(H_t) $$

Markov State / Information State

Information State (a.k.a Markov State) is a collection of all the useful information from the history which is used by the model to take future decisions.

Markov Property

Markov Property states that, any state $\mathbf S_{t}$ is Markov, if and only if,

$$ \matbh P[\mathbf S_{t+1} | \mathbf S_t ] = \mathbf P[\mathbf S_{t+1} | \mathbf S_1, ..., S_t] $$

This basically means that, the probability of the occurence of the next state $\mathbf S_{t+1}$ conditioned the given state $\mathbf S_{t}$ is same as the probability of the next state $\mathbf S_{t+1}$ if you show all the previous states to the system.

In simple meaning, to find the probability of the next state occuring, you can throw away information about all the previous states and just retain the information about the current state and you will see a similar future. There is a famous line which describes this property which says that “The future is independent of the past given the present”.

Lets take an real-life example to understand this, suppose you want to control a RC helicopter using RL, in such a case, the velocity, angular velocity, angular position and wind speed would all roughly be a markov state of the helicopter. The position or velocity of the helicopter 10 minutes ago doesn’t matter or affect the position of the helicopter in the next moment, just the information of the current states are enough.

Approaches in Sequential decision making

Problems or Objectives

There are two fundamental problems in sequenctial decision making:

Reinforcement Learning Problem: In a RL problem, the environment is initially unkown. It is not told how the environment works. The agent interacts with the environment to figure out the best plan of action based on hit and trial method. This is analogous to the real life use case of controlling a nuclear power plant in which you are asking a RL agent to control and handle the entire nuclear powerplant without giving information to the agent about how the power plant actually works.
Planning Problem: In this, the model of the environment is known to the agent in advance and you tell the agent all the rules of the game, dynamics of the environment etc. Instead of interacting with the environment, the agent performs internal computations on its models and figures out what to do next, improving its policy during this course of action.

The agent without ever interacting with the environment actually, simulates the different situations and possiblities of what can happen, and builds a tree of these different sitautiosn and possiblities. At last it just performs a tree search to reach that particular situation taking the necessary actions required for it.

Exploration vs Exploitation

Reinforcement Learning is a type of trial and error learning. The agent should discover a good policy by itself from its existance of the environment without loosing too much reward along the way. There are two basic pathways that an agent can take to achieve this goal:

Exploration means choosing the give up some known reward in the journey inorder to find out more information about the environment and use that information later on to maximize the rewards.

Exploration means exploiting the known information during the journey and maximizing the rewards along its way.

It is very important to keep a decent balance between these two. An agent should explore as well as exploit.

Working with W5500 on Arduino Microcontrollers

Sat, 07 Sep 2024 00:00:00 +0000

Local Area Network (LAN) is a a collection of devices physically connected to each other in a single location such as a home, office, building etc. Ethernet is a robust modern technology which allows us to connect the various devices over LAN and communicate to each other through an Ethernet Cable. In this article, we are going to see how to establish communication between an host which will be an Arduino microcontroller and a linux server acting as the client.

To establish this communication, we will be performing a simple loopback test using the Unified Data Protocol (UDP) wherein, whatever we send from the Arduino to the linux server will be displayed on the client end which is our server and then sent back to the micrcontroller which is the host creating a loop of data transfer.

For testing, any arduino based microcontroller can be used but we will be using an Arduino Nano. By default, arduino based microcontrollers do not support Ethernet communication directly, hence, we will be using Wiznet’s W5500 Ethernet to SPI module to interface ethernet with our microcontroller.

Components Required

Arduino Nano Microcontroller x 1
Wiznet W5500 Ethernet Module x 1
Ethernet Cable (Crossover cable preffered) x 1
Ubuntu Linux Computer (UDP Server) x 1
Jumper Cables

Hardware Setup

W5500	ARDUINO
5V	5V
GND	GND
SCK	D13
MISO	D12
MOSI	D11
CS	D10
RST	D9

Connecting the RST pin is optional, only if you want to Reset the W5500 module through code. It is preffered to reset the module before configuring the network parameters for communication.

Arduino Programming

Libraries Required:-

We are going to use the following external ethernet libraries for our program:

Let us start with writing the Arduino Code.

Step 1: Import the required libraries.

#include <Ethernet_Generic.h>
#include <EthernetClient.h>
#include <EthernetServer.h>
#include <SPI.h>

Step 2: Create a function which sends a confirmation to the Client when a message has been received.

void sendConfirmation(EthernetClient client) {
 // Confirmation message to be sent
 String confirmation = "Message Received\n";

 // Send the confirmation to the client
 client.print(confirmation);

 // Also print the confirmation to the Serial Monitor for debugging
 Serial.print("Sent to client: ");
 Serial.print(confirmation);
}

Step 3: Configure the network parameters for the Host (Arduino).

// Network settings
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; // MAC address which should be unique
IPAddress ip(192, 168, 1, 177); // Set the Arduino IP (Make sure no other device on the network uses this IP address)
EthernetServer server(80); // Set the server port

Step 4: Setup the parameters for the Serial Monitor, Start the Ethernet and the Server.

void setup() {
 // Start serial communication for debugging
 Serial.begin(9600);

 // Start Ethernet
 Ethernet.begin(mac, ip);

 // Start the server
 server.begin();

 Serial.print("Server is at ");
 Serial.println(Ethernet.localIP());
}

Step 5: Write the loop function which continously checks for any incoming character if the client is connected and stores all the charaters in a string.

void loop() {
 // Listen for incoming clients
 EthernetClient client = server.available();

 if (client) {
 Serial.println("Client connected");
 String message = ""; // Variable to hold the incoming message

 // While the client is connected
 while (client.connected()) {
 if (client.available()) {
 char c = client.read(); // Read the incoming data
 Serial.print(c); // Print the received data to Serial Monitor
 message += c; // Append the character to the message

 // Check if the message is complete (e.g., by checking for newline '\n')
 if (c == '\n') {
 // Send confirmation to the client
 sendConfirmation(client);

 // Clear the message variable for the next message
 message = "";
 }
 }
 }

 // Close the connection
 client.stop();
 Serial.println("Client disconnected");
 }
}

Complete Code:-

// Author: Utkarsh Anand
/*
Description:-
This code is for communicating between Arduino Nano microcontroller and a UDP server like Ubuntu Linux system through ethernet LAN connection using a Wiznet W5500 Ethernet to SPI module.

Code:- https://github.com/ChiratheRobotics/Safety_Embedded_board/tree/15_ethernet_integration/Mule_V3/Ethernet_Integration/Arduino_W5500/Arduino_W5500_UDP_Loopback_Test

*/



// Include all the required libraries

#include <Ethernet_Generic.h>
#include <EthernetClient.h>
#include <EthernetServer.h>
#include <SPI.h>

// Fuction Prototype
void sendConfirmation(EthernetClient client);

// Network settings
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; // MAC address which should be unique
IPAddress ip(192, 168, 1, 177); // Set the Arduino IP (Make sure no other device on the network uses this IP address)
EthernetServer server(80); // Set the server port

void setup() {
 // Start serial communication for debugging
 Serial.begin(9600);

 // Start Ethernet
 Ethernet.begin(mac, ip);

 // Start the server
 server.begin();

 Serial.print("Server is at ");
 Serial.println(Ethernet.localIP());
}

void loop() {
 // Listen for incoming clients
 EthernetClient client = server.available();

 if (client) {
 Serial.println("Client connected");

 String message = ""; // Variable to hold the incoming message

 // While the client is connected
 while (client.connected()) {
 if (client.available()) {
 char c = client.read(); // Read the incoming data
 Serial.print(c); // Print the received data to Serial Monitor
 message += c; // Append the character to the message

 // Check if the message is complete (e.g., by checking for newline '\n')
 if (c == '\n') {
 // Send confirmation to the client
 sendConfirmation(client);

 // Clear the message variable for the next message
 message = "";
 }
 }
 }

 // Close the connection
 client.stop();
 Serial.println("Client disconnected");
 }
}

void sendConfirmation(EthernetClient client) {
 // Confirmation message to be sent
 String confirmation = "Message Received\n";

 // Send the confirmation to the client
 client.print(confirmation);

 // Also print the confirmation to the Serial Monitor for debugging
 Serial.print("Sent to client: ");
 Serial.print(confirmation);
}

Ubuntu Setup

Step 1: Ethernet Connection

Ensure that ethernet cable is connected between the LAN port of W5500 and Ubuntu System.

Step 2: Network Configuration

Open terminal on Ubuntu System
Check your ethernet interface

$ ifconfig

In my system, the ethernet interface was called ‘enp89s0’.

Assign a static IP to your ethernet interface.

$ sudo ifconfig enp89s0 192.168.1.199 netmask 255.255.255.0

Note:- The IP addresses of Ubuntu System and Arduino should be on the same subnet.

Step 3: Test the connection

Ping the Arduino Nano to ensure the connection is working.

$ ping 192.168.1.177

You should receive replies from the Arduino, confirming the network setup is correct.

Step 4: Send and Receive Data

You can use ’telnet’ or ’netcat’ to communication with the Arduino server.
Download ’netcat’ using the following command

$ sudo apt-get install netcat

Connect to the arduino using the comamnd nc {IP Address of Arduino} {Port}.

$ nc 192.168.1.177 80

Type any test, and you should see the echoed response from the arduino on the serial monitor.

Leveraging Lazy Predict

Fri, 05 Apr 2024 00:00:00 +0000

The article was originally published by ACM Manipal on behalf of the author here.

In machine learning, selecting the right model for a given dataset can be a daunting task. With a plethora of algorithms to choose from and various hyperparameters to tune, the process often demands significant time and effort. On top of this, with the recent AI boom, the application of machine learning and deep learning algorithms has increased significantly especially in non-computer science applications. Researchers in diverse domains want to leverage this powerful tool into their work but lack a background in machine learning or high-level programming.

However, there’s a silver lining: Lazy Predict, a Python library that offers a shortcut to model selection by automating the evaluation of multiple algorithms on a dataset. In this article, we’ll explore Lazy Predict in detail, including its features, use cases, and example codes.

What is Lazy Predict ?

Lazy Predict is a Python library designed to streamline the process of evaluating and comparing multiple machine learning models simultaneously on a single dataset. Developed by Shankar Rao Pandala, it leverages the scikit-learn library to build, train and evaluate a variety of classifiers and regressors. With Lazy Predict, users can quickly get an overview of how different algorithms perform on their dataset without the need for extensive manual coding.

How does it work ?

Using Lazy Predict is incredibly simple, after importing the necessary modules, users can directly load their dataset and specify the target variable which needs to be predicted or classified. Lazy Predict then automatically splits the data into training and testing sets, initializes a collection of pre-defined classifier and regressor models, trains each model on the training data, and evaluates their performance on the test data. Finally, it generates a comprehensive report containing various performance metrics for each model, enabling users to compare their effectiveness effortlessly.

Code Example

For this article, we are using Lazy Predict for running classification models on the famous Optical Recognition of Handwritten Digits dataset by MNIST. The data set contains images of hand-written digits: 10 classes where each class refers to a digit.

To use the Lazy Predict library, it has to be installed using the python-pip package.


pip install lazypredict

Let’s start coding the application.

Import the required python packages


from lazypredict.Supervised import LazyClassifier
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

Load the dataset and separate the target and feature variables


data = load_digits()
X, y = data.data, data.target

Split the data into training and testing sets. Here, we are using a training to testing split ratio of 8:2


# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123)

Initialize the LazyClassifier. We are not using any custom metric, hence it has been set to None. Set verbose parameter to false to hide the detailed information while the model is being trained. Also, set the ignore_warnings flag to True to display any warnings which may come up.


# Initialize LazyClassifier
clf = LazyClassifier(verbose=0,ignore_warnings=True, custom_metric=None)

Start the model training and generate the model performance summary.


# Fit LazyClassifier
models_summary, predictions = clf.fit(X_train, X_test, y_train, y_test)

Display model performance summary


# Display model performance summary
print(models_summary)

Use Cases of Lazy Predict

Exploratory Data Analysis (EDA): Lazy Predict serves as a valuable tool during the initial stages of a machine learning project, providing insights into which algorithms may be suitable for further exploration.
Model Selection: By comparing the performance of various models, Lazy Predict helps practitioners identify the most promising candidates for fine-tuning and optimization.
Education and Learning: Lazy Predict can be used as an educational resource to introduce beginners to different machine learning algorithms and their performance characteristics.

Benefits of using Lazy Predict

Time-Saving: Lazy Predict automates the process of model selection, saving practitioners valuable time that would otherwise be spent on manual evaluation.
Efficiency: With Lazy Predict, users can quickly assess the performance of multiple models in a single step, facilitating rapid experimentation and iteration.
Accessibility: Lazy Predict’s user-friendly interface makes it accessible to users of all skill levels, from novices to seasoned machine learning practitioners.

Conclusion

Lazy Predict offers a convenient and efficient solution for evaluating machine learning models, making it an invaluable asset in the data scientist’s toolkit. By automating the process of model selection and performance evaluation, it enables practitioners to expedite the development of robust and accurate machine learning solutions. Whether you’re a beginner exploring the world of machine learning or a seasoned researcher seeking to streamline your workflow, Lazy Predict is a tool worth considering.

References:

Lazy Predict GitHub Repository: https://github.com/shankarpandala/lazypredict Pandala, Shankar Rao. “Lazy Predict.” GitHub, 2021, https://github.com/shankarpandala/lazypredict.

Navigating the Landscape of Research - Top tools and resources for researchers in 2024

Fri, 29 Mar 2024 00:00:00 +0000

The article was originally published by ACM Manipal on behalf of the author here.

In the ever-evolving world of research, staying organized, efficient, and up-to-date with the latest developments is paramount. Fortunately, the digital age has ushered in a plethora of tools and resources designed to streamline the research process. In this article, we’ll explore some of the essential resources and tools that have helped me through my undergraduate research journey.

1. Overleaf

LaTeX is the standard format for document writing in academia and Overleaf has revolutionized collaborative writing. This online LaTeX editor allows researchers to create, edit, and share LaTeX documents seamlessly. Its real-time collaboration features enable multiple authors to work on a document simultaneously, making it ideal for collaborative research projects. With Overleaf, formatting complexities are minimized, allowing researchers to focus on content creation without worrying about the intricacies of LaTeX coding.

2. Scispace

A literature review is an essential component of any research project. As essential as it is, understanding research papers can be difficult sometimes. Scispace played a very crucial role in making this simpler for me. It is a comprehensive research management platform that integrates various tools and services to facilitate every stage of the research process. From literature review and data analysis to collaboration and project management, Scispace offers a centralized hub for researchers to organize their work efficiently. Its AI-powered chatbot can read through entire papers and generate summaries, inferences, etc. easily Users can also ask different questions to the chatbot about the research presented in the papers.

3. Mendeley / Zotero

Mendeley is a leading reference management tool that helped me collect, organize, and cite sources effectively. It offers features for storing and annotating PDFs, generating bibliographies, and collaborating with peers. Mendeley’s social networking capabilities also allow researchers to discover new publications and connect with colleagues in their field.

Another popular tool for similar use cases is Zotero which offers great open source capabilities but I used and preferred Mendeley due to its cross-device synchronization capabilities handling papers on multiple devices.

4. Connected Papers / Litmaps

Connected Papers and Litmaps are visualization tools that helped me in exploring the interconnectedness of academic literature. By analyzing citation networks and identifying key relationships between papers, these platforms offer valuable insights into the landscape of scholarly research. Researchers can use Connected Papers and Litmaps to discover seminal works, identify emerging trends, and navigate complex research domains with ease.

5. Github

Although having a reputation for being the go-to tool in the software development industry, GitHub has increasingly become a valuable resource for researchers working on projects involving code or data analysis. Its version control capabilities facilitate collaboration and reproducibility by enabling my fellow co-researchers to track changes, manage revisions, and share code repositories.

6. Notion

When working on multiple research projects, keeping track of all the progress becomes critical and messy at times. That is where Notion helped me. It is a versatile productivity tool that combines note-taking, project management, and collaboration features in a single platform. Its flexible structure allows researchers to create customized databases, organize information intuitively, and collaborate with team members seamlessly. Whether used for brainstorming ideas, outlining research proposals, or managing lab workflows, Notion adapts to the unique needs of researchers across diverse disciplines.

7. Google Scholar

Google Scholar remains a go-to resource for researchers seeking access to scholarly literature across a wide range of disciplines. Its vast database of academic publications, patents, and citations enables researchers to discover relevant sources, track citations, and stay informed about the latest research trends. Powered by the Google search engine, its advanced search capabilities and personalized recommendations make it an indispensable tool for conducting comprehensive literature reviews and staying updated on cutting-edge research.

8. Grammarly

Clear and concise writing is essential for effective communication in academia. Grammarly is a writing assistant tool that helps researchers improve the clarity, correctness, and coherence of their written work. From grammatical errors and punctuation mistakes to style suggestions and tone adjustments, Grammarly provides real-time feedback and suggestions to enhance the quality of research manuscripts, grant proposals, and academic papers.

9. Papers with Code

As the importance of reproducibility and transparency in research continues to gain prominence, Papers with Code offers a valuable resource for researchers seeking to share and reproduce code implementations of academic papers. This platform provides code repositories linked to research papers, allowing researchers to access, replicate, and build upon existing work with ease. It also promotes open sourcing in academia, especially in the computer science field. Papers with code provide the ability to implement and upload on the website for research papers whose code is not published by the authors. It provides monetary incentives to people contributing to the code repositories.

In an era of rapid technological advancement and information overload, leveraging the right tools and resources is essential for navigating the complexities of the research landscape. Whether it’s managing references, analyzing data, or collaborating with peers, the tools mentioned above can empower researchers to work more efficiently, communicate effectively, and make meaningful contributions to their respective fields.