Here I delve into wonderful world of open-source.

<MainHeading postData={postData} />

Hello!! Long time no see. I recently did this project as part of the CS 527 (Topics in Software Engineering) course at the [University of Illinois Urbana-Champaign](https://illinois.edu/). It was some super exciting stuff and I wanted to write about it here.

For Fall 2022, the course was taught by professor [Darko Marinov](https://mir.cs.illinois.edu/marinov/). Super cool professor. Here's the course description:

> Fault-tolerant software, software architecture, software patterns, multi-media software, and knowledge-based approaches to software engineering. Case studies.

Each year, the content of the course slightly changes. This year, the emphasis was on testing. One of the 3 routes students have the opportunity to take is fixing flaky tests. What are flaky tests, you ask? Well, in short: they're tests that fail on subsequent runs with no changes to the code or the test itself. Imagine that: You changed nothing. You didn't touch a thing. But you run the test a second time and it fails. Frustrating. This is just one example. There are many reasons why a test might be considered flaky. [This paper](https://www.cs.cornell.edu/courses/cs5154/2021sp/resources/LuoETAL14FlakyTestsAnalysis.pdf) has more information about the type of tests.

Other reasons why a set of tests might fail is when you run them in a different order than indended. One type is a victim, an order-dependent test that passes when run in isolation but fails when run with some other tests. The other type is a brittle, an order-dependent test that fails when run in isolation but passes when run with some other test / tests.

Students who picked this route were tasked with fixing flaky tests. Now, finding them is another aspect of the problem. You'd have scripts that run tests multiple times, such as [pytest-flakefinder](https://pypi.org/project/pytest-flakefinder/). Each time it runs, it checks to see if all the tests pass. If flakiness is present, the script tells us. Thankfully, there was already a list of tests that were detected. All I had to do was choose tests, recreate the flakiness, fix the tests, open PRs, and get them merged. But, there was another problem. A lot of the tests on the list of detected tests that were provided to us were old. Archived code. You'd be awarded 5 points for getting a PR accepted. But only 1 for opening them. Why would someone accept a PR on a repostiroy that was not being maintained? They had absolutely no reason to.

Often times, recreating the flakiness was the biggest problem. And then once that's done, you'd have to fix the actual issue. By the end, I fixed a total of 26 tests across 7 open-source repositories. After this, you had to convice the maintainer of the repository that the change you're suggesting is a good one that they'd actually consider. I couldn't go and just find a source of flakiness and fix it. I had to find out exactly why it occurred, find out how to fix it, and find out if the fix makes sense in context to the rest of the code. Because more often than not, the test you're modifying has ramifications on other tests. So you fix the flakiness in one, and another one appears. But wait, you finally fixed the flakiness. All tests pass! But wait, why did it work? How do I convice the maintainer of the repository to accept my pull requests?

So many things to juggle. One challenge after the other. It taught me a lot about overcoming challenges and sticking with something. Because the time spent on some tests were long. Way too long. With no fruitful outcome. And it sucks. But you learn.

# Scripting!

The first PR I fixed was on the repository with the most amount of stars, and the one with the most recent commits. I had to write a script to get these repositories, because sifting through an excel sheet with 1000s of rows and checking each repository to see if it was maintained was tiring. Here's the script which scrapes an existing Excel sheet, notes down the number of stars and number of months since latest commit, and sorts in descending order of number of stars and ascending order of months since last commit. Both these pieces of information together in this specific order help to shortlist a repository to fix tests on. The higher it is on the list, the more chance your PR had of getting accepted.

```python
import os
import argparse
import datetime
import pandas as pd
from tqdm import tqdm
from github import Github

tqdm.pandas()

parser = argparse.ArgumentParser()
parser.add_argument('-t', '--github_access_token', help='GitHub access token to overcome API rate limitations')
parser.add_argument('-f', '--filepath', help='Filepath of .csv file containing repo data')
parser.add_argument('-c', '--colname', help='Column name in CSV file pertaining to repo URL')
args = parser.parse_args()

GITHUB_API_RATE_LIMIT = 5000
FILEPATH, COLNAME, GITHUB_ACCESS_TOKEN = args.filepath, args.colname, args.github_access_token

data = pd.read_csv(FILEPATH)
data = data[data['Status'].isna()]
REPO_URLS = data[COLNAME].unique()
NUM_REPOS = REPO_URLS.shape[0]

def check_number_repos():
    if NUM_REPOS > GITHUB_API_RATE_LIMIT:
        print(f'You can only make {GITHUB_API_RATE_LIMIT} requests per hour. Your file has {NUM_REPOS} unique repositories. Exiting.')
        exit(0)

def get_diff_month(d1, d2):
    return (d1.year - d2.year) * 12 + d1.month - d2.month

def get_repo_object(repo_url):
    try:
        repo_name = repo_url.split('github.com/')[1]
        return Github(GITHUB_ACCESS_TOKEN).get_repo(repo_name)
    except Exception as e:
        print(e)
        return None

def get_months_since_last_commit(repo):
    try:
        default_branch = repo.get_branch(repo.default_branch)
        latest_commit_date = default_branch.commit.commit.author.date
        months_since_commit = get_diff_month(datetime.datetime.now(), latest_commit_date)
        return months_since_commit
    except Exception as e:
        print(e)
        return None

def get_maintained_repos():    
    check_number_repos()
    print(f'Analyzing {NUM_REPOS} repositories...')
    df = pd.DataFrame()
    df['REPO_URL'] = REPO_URLS
    df['REPO_OBJECT'] = df['REPO_URL'].progress_apply(lambda url: get_repo_object(url))
    df['MONTHS_SINCE_LAST_COMMIT'] = df['REPO_OBJECT'].progress_apply(lambda repo_object: get_months_since_last_commit(repo_object))
    df['STARS'] = df['REPO_OBJECT'].progress_apply(lambda repo_object: repo_object.stargazers_count if repo_object is not None else None)
    df = df.sort_values(by=['MONTHS_SINCE_LAST_COMMIT', 'STARS'], ascending=[True, False]).drop(columns=['REPO_OBJECT', 'Unnamed: 0'], errors='ignore')
    df.to_csv(f'{os.getcwd()}/repo-info/repo-info.csv', index=False)

if __name__ == '__main__':
    get_maintained_repos()
```

# Fixing the test

The repo at the top of the list was [pingouin](https://github.com/raphaelvallat/pingouin), an open-source Python package used for statistical analysis. Alright. First step is to recreate the flakiness. Easy enough. I ran all the tests once with `pytest`. They all pass. I then use the pytest-flakefinder plugin, and run the tests again. Remember, this plugin runs the tests multiple times, in different orders, and often without resetting certain state information. When this happened, only the first test passed. The rest of the tests failed:

![When tests pass normally but they don't with the flakefinder plugin, we have recreated flakiness](/images/blog-posts/fixing-flaky-tests/running-pytest.png)

Great, now we have somewhere to start. The error message tells me exactly where the issue was occuring. So I go ahead and start debugging (with `print` statements, like a true programmer). Here's what was happening: after a DataFrame was initially read certain columns were not being added to the DataFrame in some tests, and without these columns, the test would fail. All I had to do to fix it was add 4 lines (2 to each test that was failing):

![Code changes](/images/blog-posts/fixing-flaky-tests/code-changes.png)

When re-running pytest-flakefinder, all the tests passed, which means the test was successfully fixed. Now, all that's left was to open a pull request.

# Opening the PR

Certain points needed to be addressed if I were to convince the developer to merge my PR.

1.  Give context. Tell the repo maintainer what's happening, and why it is happening.
2.  Convice. The developer might be running his suite of tests once, and be happy that they all pass. After all, who wouldn't be? Most developers don't think about flaky tests, atleast on pretty small open source libraries. So I had to be mindful and explain how to replicate it, and why it would be a problem in the future (hint: flaky tests are notoriously hard to debug if not caught early on)

Hitting all these points on the PR, this was what I had:

![Pull request](/images/blog-posts/fixing-flaky-tests/pull-request.png)

And after waiting a few days, the developer accepted my PR (yay!)

![Pull request accepted!](/images/blog-posts/fixing-flaky-tests/pull-request-accepted.png)

Okay. Great. One down. 5 points! I needed atleast a 100 to pass...

# What after?

The rest the came after weren't as easy as the first one. There were a multitde of problem that came up, which I very graciously mentioned in my report, which was a part of the final submission for the class. By far, I have never learnt more from a class than before. Sure, almost all of this involves self-teaching, but that's an important skill to develop, and the overall experience made me appreciate good quality code.

From the stars,

FR.

# The final report

## Problem

Fix flaky tests in Python with an emphasis on non-order-dependent categories

## Summary

### Results

*   Fixed flakiness for 26 tests across 7 repositories. Opened 8 pull requests, of which 2 got accepted.
*   Wrote Python script for aiding in selecting flaky tests to fix.

### Other Effort

*   3 PRs got rejected.
*   Tried fixing a lot more tests since progress4 but was largely unsuccessful due to a variety of reasons.

## Total Effort

(Estimated) 140 - 150 hours. (~50 hours from progress1 to progress3, rest from progress4 to end)

## Longer Description:

| PR Link                                                | Status   |
|--------------------------------------------------------|----------|
| https://github.com/raphaelvallat/pingouin/pull/303     | Accepted |
| https://github.com/didix21/mdutils/pull/76             | Accepted |
| https://github.com/pv8/noipy/pull/203                  | Opened   |
| https://github.com/hefnawi/json-storage-manager/pull/1 | Opened   |
| https://github.com/ganehag/pyMeterBus/pull/25          | Opened   |
| https://github.com/neurodsp-tools/neurodsp/pull/308    | Rejected |
| https://github.com/fooof-tools/fooof/pull/243          | Rejected |
| https://github.com/fooof-tools/fooof/pull/242          | Rejected |

*   Wrote a Python script to help finding repositories to choose for fixing flaky tests.
    *   Motivation: A lot of the tests (when chosen arbitrarily from `py-data.csv`) were on old repositories that were not maintained).
    *   Script looks at URL from `py-data.csv` or `pr-data.csv` and for each repository, notes down two pieces of information: number of stars, and number of months since last commit to default branch. Also, it only looks at repositories that have atleast one test which has not been fixed.
    *   [Script added to IDoFT repo](https://github.com/TestingResearchIllinois/idoft/tree/main/repo-info).
    *   Used this script to help find archived repositories (i.e. likely that archived repos had no commits for a long time), which I updated in `py.data.csv`.

*   The three pull requests which were rejected was very well documented, but maintainer of repository (same maintainer for all 3 PRs) felt that the situation under which the tests were flaky wouldn't occur in real life. This was the reason why the PR was closed. In my opinion it was not a fault with the PR itself, but the situation under which the fix would be useful.

*   Progress since progress4 was not very good. Only managed to open two PRs.

*   In general, fixing flaky tests in Python was challenging due to the following reasons (not in any order):
    *   Ran into issues during installing required libraries
    *   Flaky behaviour not found (either due to failure in re-creating or flaky test got fixed)
    *   Default tests were failing- no use fixing flakiness when default tests are failing
    *   Unable to reproduce flakiness. Used three different methods to test for flakiness:
        *   [`pytest-flakefinder`](https://github.com/dropbox/pytest-flakefinder) - Preferred method of finding flaky tests.
        *   [`pytest-random-order`](https://pypi.org/project/pytest-random-order/) - Ran tests in random order. Did not use much because developers might argue tests will never be run in any other order than they are written, which means my PR would have a high chance of getting rejected.
        *   [`ipflakies`](https://github.com/ailen-wrx/python-ipflakies) - Execution of this plugin took way too much time even with low number of iterations, was not suitable for quick test running. Thus, did not use plugin much.
    *   Unable to fix / find source of flakiness- Highest reason why so many tests were explored and not fixed. The packages were complex and in general was very difficult to find source of flakiness.
    *   Many tests are on repositories that are not maintained, so chance of PR getting accepted on an unmaintained project is very less
    *   Not many order-dependent tests to fix. These tests are (in my opinion) very easy to fix, but due to the fact that Python tests don't have such issues like in Java (as per my experience), fixing tests in general have been very difficult.
    *   Existence of flakiness does not warrant a fix. On a good portion of the tests I've seen so far, tests are meant to be run more than once- a lot of tests were very hard to debug and impossible to fix because they dealt with temporary files/directories, objects with persistent memory, etc. Fixing flakiness would require too many lines of code to be changed- developer would likely not agree with so much changes for an issue that isn't going to be a problem (an issue here referring to flakiness not being a problem because tests are always run in the same order / not run more than once.)


Fixing Flaky Tests

Here, I walk you through step-by-step on how I made this pathfinding visualizer.

Building a pathfinding visualizer using PyGame

In this post I explain how I learnt React and built my first ever proper website.

<MainHeading postData={postData} />

Hello. In this post I'll explain how I built my personal website using React. Prior to this, I had very little experience with fullstack development, my only two projects at the time being my [Sorting Visualizer](/projects/sorting-visualizer) and [Anonboard.](http://www.anonboard.site/) My previous blog was a static website made using [Jekyll](https://jekyllrb.com/) and hosted on Github Pages. What I didn't like about that website is that it was not very customizable. I had a lot of ideas in mind and Jekyll was limiting in many ways. It was super easy to get started with though, and I recommend Jekyll if you want to start your own blog without much effort.

And with that, I said goodbye to that website and I started building my own. One problem though- I didn't know how.

# The Stack

## ReactJS

[React](https://reactjs.org/) was on my to-learn list for a long time. I had analysis-paralysis of choosing which project to do and I finally said fuck it, and started learning. I only used two resources, one, the official [React website](https://reactjs.org/), specifically, the Tic Tac Toe [tutorial](https://reactjs.org/tutorial/tutorial.html), and two, the online course called [fullstackopen](https://fullstackopen.com/) offered by The University of Helsinki. I highly recommend both of these. Once finishing upto Part 2 on fullstackopen, I finished the Tic Tac Toe tutorial and felt I learnt enough to begin implementing.

## Next.js and MDX

I had 0 zero idea about how a blog is implemented, so I did what every proper person in this field does, and I used Google. I came across [this post](https://www.joshwcomeau.com/blog/how-i-built-my-blog/) by [Josh Comeau](https://www.joshwcomeau.com/), and the tech stack he used for his own blog seemed perfect for a few reasons:

*   Next.js has [static site generation](https://nextjs.org/docs/basic-features/data-fetching/get-static-props) capabilities, which means that for content that doesn't change (i.e. a blog post), the page could be served up blazingly fast.
*   Next.js has [server side rendering](https://nextjs.org/docs/basic-features/pages#server-side-rendering) capabilities as well. This was particularly important for me, since I wanted to host my projects on the same website, and they could not be implemented without server side rendering. (If you're curious, two projects I've currently hosted are built using [p5.js](https://p5js.org/), which is a JavaScript library for creating visualizations, and requires the `window()` object to be available, and the `window()` object wouldn't be available if the page was statically generated).
*   Next.js has a super easy way to implement routes, much better than [Express](https://expressjs.com/) in my opinion.
*   Next.js is a React framework, and React is what I wanted to learn.
*   With [MDX](https://mdxjs.com/), I can make new blog posts super quick, and I could also migrate all my posts from my old website without too many modifications (the previous website was built using Jekyll, which hosts blog posts written in [Markdown](https://en.wikipedia.org/wiki/Markdown)).
*   Also, with MDX, I'm not limited to the components rendered from Markdown, I can make whatever custom React component I want and stick it in the middle of my blog page. For instance, here's my custom `<RainbowTitleText />` component:

<RainbowTitleText content={"It's Wednesday my dudes"} size="4xl" />

99% of the text on this page is from a Markdown file. Pretty sick huh?

## ChakraUI

My only experience with a front-end framework before this was [Bootstrap](https://getbootstrap.com/). I heard so much about [TailwindCSS](https://tailwindcss.com/), but as a beginner, I felt it was a bit of a hassle to get started with. My next alternatives were between ChakraUI and [Mantine](https://mantine.dev/). While I could not decide between the two, I decided that I'm spending too much brain energy on a frivolous decision, and what mattered more was actually getting started. So I went with ChakraUI. And I have no regrets. One very cool feature that I absolutely love about ChakraUI is how easy it is to implement site-wide dark/light mode. I can toggle the mode by clicking on the icon in the top right corner of the page. Actually, to show how cool MDX is, lemme plop up the button right here. Go on, click it.

<Center>
  <ColorModeToggleButton />
</Center>

Boom.

This button is the same component as the one on the top right of the page. All I had to do was import it in. This is one very important reason why many people use React- it's super easy to re-use components. All the code I had to write for this button was the following:

```javascript
<Center>
  <ColorModeToggleButton />
</Center>
```

Anyway, let's get started.

# 1 - Landing Page

I'm a big fan of cool-looking landing pages, and I'm super proud of how [mine](/) turned out. But first things first though, I will give credit where credit is due. The animated ocean waves and clouds aren't mine, I found them on [CodePen](https://codepen.io/trending). While I could've done all this on my own, I absolutely abhor CSS, and I'd rather not learn it just for this.

*   Ocean Waves - [CodePen by Ted McDonald.](https://codepen.io/tedmcdo/pen/PqxKXg)
*   Clouds - [CodePen by Mark Bowley](https://codepen.io/Mark_Bowley/pen/LYZEBq)

The stars are done using an NPM package called [react-tsparticles](https://www.npmjs.com/package/react-tsparticles). (Toggle dark mode on the [root page](/) to see the stars). The sun and moon are just a simple amalgamation of some React and some CSS.

```javascript
import { useColorMode } from '@chakra-ui/react';

const sphereStyles = {
  height: '100px',
  width: '100px',
  margin: '5% 0',
  borderRadius: '50%',
  zIndex: -2,
  position: 'absolute',
};

let sunStyles = {
  ...sphereStyles,
  background: 'orange',
  boxShadow: '0 0 10px orange, 0 0 60px orange, 0 0 200px yellow, inset 0 0 80px yellow',
};

let moonStyles = {
  ...sphereStyles,
  background: '#F7EAC6',
  boxShadow: '0 0 70px 70px rgba(255, 248, 216, 0.2)',
};

export default function SunAndMoon() {
  return <div className={sphereStyles} style={useColorMode().colorMode === 'dark' ? moonStyles : sunStyles}></div>;
}
```

<Center py={20}>
  <SunAndMoon />
</Center>

# 2 - Blog Page

One thing I didn't want in this page was for the blog posts to be displayed like how they'd be displayed in a traditional blog- title followed by a short summary followed by a date. Meh. Too boring. I wanted to spice it up. I'm a big fan of [Masonry layouts](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Grid_Layout/Masonry_Layout), and I wanted to somehow implement that here. So I decided to have each post rendered as a component that looks like a card, and I think it turned out pretty good. If you're on desktop, take a look for yourself at [/blog](/blog). (Or view the desktop version of the site if you're on mobile.) The masonry part of it is handled by the [Plock](https://github.com/itsrennyman/react-plock) library.

Here's how a Card component looks like. The arbitary data inside the card below is passed in through the `props` object.

<Center>
  <Card
    pageType="blog-post"
    data={{
     thumbnail: '/images/blog-posts/personal-website-walkthrough/card-component-image.png',
     title: 'An Example Card Component',
     summary: 'This is a card component that I made for my blog page.',
     category: 'Category',
     tags: ['Tag 1', 'Tag 2', 'Tag 3'],
     date: postData.date,
    }}
  />
</Center>

## Filtering & Sorting Posts

This part is surprisingly pretty simple. As of the date of writing this post, there are two ways of filtering posts: 1) Search query, 2) Filtering by tag/category. All you have to do to filter posts with multiple criteria is to chain on multiple `filter()` commands. I added a `useState()` hook to both the search bar and the sort dropdown, and to filter the posts with these multiple criteria, all I have to do is this:

```javascript
allPostsAllData = allPostsAllData
  .filter(postData => !categoryFilter || postData.category.toLowerCase() === categoryFilter.toLowerCase())
  .filter(postData => !tagFilters.length || tagFilters.every(v => postData.tags.map(tag => tag).includes(v)))
  .filter(
    postData =>
      !searchQuery ||
      postData.tags.map(tag => tag.toLowerCase()).filter(tag => tag.includes(searchQuery)).length ||
      postData.title.toLowerCase().includes(searchQuery) ||
      postData.summary.toLowerCase().includes(searchQuery) ||
      postData.category.toLowerCase().includes(searchQuery)
  )
  .sort(({ date: a }, { date: b }) => {
    const date_a = parse(a, 'yyyy-MM-dd', new Date());
    const date_b = parse(b, 'yyyy-MM-dd', new Date());
    if (date_a > date_b) return sortOption === 'date-asc' ? 1 : -1;
    else if (date_a < date_b) return sortOption === 'date-asc' ? -1 : 1;
    return 0;
  });
```

If you're wondering why there's so many `toLowerCase()` calls, it's because I wanted to preserve the original casing of the tags and categories (which may consist of multiple words). To make sure that all posts show up when no search query or tag or category is specified, look at the first condition in each `filter()` function- it checks to see whether the filter exists in the first place. If it doesn't, no data is filtered, i.e. all data is returned.

# 3 - Blog Post Page

This is the part where MDX comes into play. First of all, what is MDX?

> MDX allows you to use JSX in your markdown content. You can import components, such as interactive charts or alerts, and embed them within your content. This makes writing long-form content with components a blast. [Source](https://mdxjs.com/)

What MDX allows us to do is modify the HTML that is returned. This is super powerful, because it allows me to customize to my heart's content. The NPM packages that I used is called [next-mdx-remote.](https://github.com/hashicorp/next-mdx-remote)

I use Next.js's [Dynamic Routes](https://nextjs.org/docs/routing/dynamic-routes) in order to have a custom slug for each blog post. The following is the content within the `[slug].jsx` file:

```javascript
import { serialize } from 'next-mdx-remote/serialize';
import { MDXRemote } from 'next-mdx-remote';
import rehypeSlug from 'rehype-slug';
import remarkGfm from 'remark-gfm';
import { getPostData, getAllPostSlugs } from '../../utils/posts-markdown-helper';

import Layout from '../../components/Layout';
import MDXComponents from '../../components/Blog/MDX/MDXComponents';

const options = {
  mdxOptions: {
    rehypePlugins: [
      rehypeSlug, // add IDs to any h1-h6 tag that doesn't have one, using a slug made from its text
      remarkGfm,
    ],
  },
};

export async function getStaticProps({ params }) {
  const postData = await getPostData(params.slug);
  const mdxSource = await serialize(postData.contentHtml, options);
  return { props: { mdxSource, postData } };
}

export async function getStaticPaths() {
  const paths = getAllPostSlugs();
  return {
    paths,
    fallback: false,
  };
}

export default function Post({ mdxSource, postData }) {
  return (
    <Layout padding={10} currentRoute='post' siteTitle={postData.title}>
      <MDXRemote {...mdxSource} components={MDXComponents} scope={{ postData: postData }} />
    </Layout>
  );
}
```

The `MDXComponents` component is where I specify the styles for all the HTML elements that Markdown returns. The code below contains a mix of custom components and ChakraUI's components and has extra props removed for easy reading. Referring to this should be enough to get you started on your own implementation. Notice that each key inside the `MDXComponents` object is returning a React functional component- you can use this to return whatever you like.

```javascript
const MDXComponents = {
  a: props => <AnchorLink {...props} />,
  p: props => <Text {...props} />,
  h1: props => <SubHeading as='h1' size='xl' {...props} />,
  h2: props => <SubHeading as='h2' size='lg' {...props} />,
  h3: props => <SubHeading as='h3' size='md' {...props} />,
  h4: props => <SubHeading as='h4' size='sm' {...props} />,
  h5: props => <SubHeading as='h5' size='xs' {...props} />,
  ol: props => <OrderedList {...props} />,
  ul: props => <UnorderedList {...props} />,
  li: props => <ListItem {...props} />,
  pre: props => <CodeBlock {...props.children.props} />,
  img: props => <CustomImage {...props} />,
  code: props => <Code {...props} />,
  table: props => <CustomTable {...props} />,
  thead: props => <CustomTableHead {...props} />,
  tbody: props => <CustomTableBody {...props} />,
  th: props => <CustomTableTh {...props} />,
  tr: props => <CustomTableTr {...props} />,
  td: props => <CustomTableTd {...props} />,
  blockquote: props => <BlockQuote {...props} />,
};

export default MDXComponents;
```

# Final Thoughts

Aaaaand, that's it. Pretty short yes, I did not want to write too much about it. The basics are there. You should be able to replicate the functionality of this website if you wanted to with the content in this post. Overall, this took me about 1 month to finish, with an average of 1-3 hours worked on it each day. Could've been better I suppose, but I wasn't giving it my full attention, and since I'm new to Next.js, a lot of time was spent on some annoying [hydration](https://github.com/vercel/next.js/discussions/22276) issues. In Next.js, you cannot nest a `<p>` tag inside another `<p>` tag. This is a limitation of the framework. Blasphemous, I know. Couple this with the fact that ChakraUI renders a `<p>` tag for a lot of its components, I had a super-fun (read: awful) time debugging everything.

Was it worth it? Hell yeah. Next.js is blazing fast. And I can fire up a new website so much more quicker now. Truly, the fastest way to learn something is to do your own project. I'm glad I stuck with this, because another version of myself would be stuck ideating on what the perfect next project would be and never would have started. Just take a leap of faith I guess.

From the stars,\
FR.


How I built my first website with React

In this post I explain how I used computer vision in order to batch-digitize old photos.

<MainHeading postData={postData} />

> Side note: Almost all of the code in this post is taken from this [StackOverflow answer](https://stackoverflow.com/a/56617224/6593862). I only learnt the basics of computer vision after I got the code to run. So while I can't say the code in this post is done entirely by myself, I can say that I've learnt a bunch and I'm excited to delve deeper into this field, mainly do a revamp of my undergrad university's final year project: Classifying Book Genre using Machine Learning Techniques. Something for the future :)

My dad had the cognizance to capture a lot of photos from the good old days (as I'd like to call it). In our house, there exists a shelf filled with 30-40 old photo albums ranging from the 1980s all the way to 2010. They were sitting there collecting dust, so I wanted to forever immortalize them all at once, so I decided to use the power of computer vision.

Before this idea came to me, I wanted to use [Google PhotoScan](https://www.google.com/photos/scan/), but it had a bunch of problems:

*   Scanning just one photo required you to move your phone to 4 different corners of the photo.
*   There needed to be sufficient lighting.
*   With the phone's flash on, the supposed "glare-free" effect was nowhere to be seen.
*   On average, a single photo required 10 seconds to be scanned.
*   The quality of the scanned photos were not good.

So after a bit of thinking, I decided to use [OpenCV](https://opencv.org/). I have a printer-scanner lying around in my house, and with that, I could scan 2-4 photos at once. For instance, here's an example scan which I took:

![An example photo scan that I took](/images/blog-posts/digitizing-old-photos-using-computer-vision/example-scan.png)

# The Script

Here's a step by step on everything I did to batch-digitize the photos:

## Grayscaling

First step is to gray-scale the images. Why? With respect to code execution alone, two reasons that can be summarized from [this Quora answer](https://qr.ae/pvPj1p) are:

*   Colour doesn't help us identify important edges or other features.
*   It is faster since we don't have to deal with the third channel used for colour.

Also, we will be applying thresholding later on, which requires that the image be grayscale.

After grayscaling, we have the following:

```python
image = cv2.imread(f'sample.jpg')
original = image.copy()
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
```

![Grayscaled example image](/images/blog-posts/digitizing-old-photos-using-computer-vision/grayscaled-example-image.png)

## Gaussian Blurring

> Named after mathematician Carl Friedrich Gauss (rhymes with “grouse”), Gaussian (“gow-see-an”) blur is the application of a mathematical function to an image in order to blur it. “It’s like laying a translucent material like vellum on top of the image,” says photographer Kenton Waltz. “It softens everything out.” A type of low-pass filter, Gaussian blur smoothes uneven pixel values in an image by cutting out the extreme outliers. [Source.](https://www.adobe.com/creativecloud/photography/discover/gaussian-blur.html#:~:text=What%20is%20Gaussian%20blurring%3F,%2C%E2%80%9D%20says%20photographer%20Kenton%20Waltz.)

I'm using the Gaussian Blur here to reduce high frequency image noise and to enhance image structure which increases the rate at which image features will be detected. Blurring moves you up in the [scale space](https://en.wikipedia.org/wiki/Scale_space).

The first parameter is the source image, the second parameter is the kernel size (the higher the kernel size, the more blurry the image will be), and the third parameter is the standard deviation of the Gaussian function. When this is 0, it is calculated based on the kernel size.

```python
blurred = cv2.GaussianBlur(gray, (3, 3), 0)
```

![Blurred example image](/images/blog-posts/digitizing-old-photos-using-computer-vision/blurred-example-image.png)

## Threshold

Thresholding is simply a way to convert a grayscale image into a binary image. The thresholding process is usually done using a [threshold value](https://en.wikipedia.org/wiki/Thresholding_$image_processing$). The threshold value is the value at which the image is split into two different regions. If a certain pixel exceeds the threshold value, it is assigned to the foreground, otherwise it is assigned to the background.

The first parameter is the source image, the second parameter is the threshold value, the third parameter is the maximum value, and the fourth parameter is the type of thresholding.

```python
thresh = cv2.threshold(blurred, 230, 255, cv2.THRESH_BINARY_INV)[1]
```

![Thresholded example image](/images/blog-posts/digitizing-old-photos-using-computer-vision/thresh-example-image.png)

## Finding Contours

> Contours can be explained simply as a curve joining all the continuous points (along the boundary), having same color or intensity. The contours are a useful tool for shape analysis and object detection and recognition. In OpenCV, finding contours is like finding a white object from a black background. So remember, object to be found should be white and background should be black.
> [Source](https://docs.opencv.org/4.x/d4/d73/tutorial_py_contours_begin.html).

```python
cnts = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
```

With this, we have the bounds of our image.

## Finding ROIs (Regions of Interest)

To crop only the photo part of the scan, we need to find out the ROIs. We can do this with the help of the `boundingRect()` function. The `cv2.boundingRect()` function of OpenCV is used to draw an approximate rectangle around the binary image. This function is used mainly to highlight the region of interest after obtaining contours from an image.

We can loop through our contours to find the bounding rectangle of each contour. For a set of contours, there will be multiple ROIs, and to get only the image from the scan, we can filter out the resulting contours that have a very low file size. This was kind of a hacky fix since I'm not a computer vision aficionado by any means, but what I did was I saved each image, and if the image was below a certain filesize, I would delete it. This meant that whatever image I saved, was an ROI pertaining to the actual image itself.

```python
image_number = 0
for c in cnts:
    x, y, w, h = cv2.boundingRect(c)
    cv2.rectangle(image, (x, y), (x + w, y + h), (36, 255, 12), 2)
    ROI = original[y:y+h, x:x+w]
    cropped_img_file_name = f'albums/{album_name}/cropped/ROI_{image_number}.jpg'
    cv2.imwrite(cropped_img_file_name, ROI)
    file_size_mb = os.path.getsize(cropped_img_file_name) / 1024 / 1024
    if file_size_mb < 0.05:
        os.remove(cropped_img_file_name)
    else:
      image_number + =1
```

I got the resulting images:

![ROI Image 1](/images/blog-posts/digitizing-old-photos-using-computer-vision/roi_image_1.png)
![ROI Image 2](/images/blog-posts/digitizing-old-photos-using-computer-vision/roi_image_2.png)

# Code

Here's my code suited to the following usecase- each batch digitized album exists as a bunch of images in a subdirectory called `original`, under a directory called `album_<number>`, with a sibling directory called `cropped`. The images under the directories not listed under `albums_to_exclude` are scanned through, and the resulting cropped images are stored to the `cropped` directory.

```python
import cv2
import os

for album_name in os.listdir('albums'):
    if album_name not in albums_to_exclude:
        print(f'Processing {album_name}...')
        for img in os.listdir(f'albums/{album_name}/original'):
            if img != '.DS_Store':
                image = cv2.imread(f'albums/{album_name}/original/{img}')
                original = image.copy()
                gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
                blurred = cv2.GaussianBlur(gray, (3, 3), 0)
                thresh = cv2.threshold(blurred, 230, 255, cv2.THRESH_BINARY_INV)[1]

                # Find contours
                cnts = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
                cnts = cnts[0] if len(cnts) == 2 else cnts[1]

                # Iterate thorugh contours and filter for ROI
                for c in cnts:
                    x, y, w, h = cv2.boundingRect(c)
                    cv2.rectangle(image, (x, y), (x + w, y + h), (36, 255, 12), 2)
                    ROI = original[y:y+h, x:x+w]
                    cropped_img_file_name = f'albums/{album_name}/cropped/ROI_{image_number}.jpg'
                    cv2.imwrite(cropped_img_file_name, ROI)
                    file_size_mb = os.path.getsize(cropped_img_file_name) / 1024 / 1024
                    if file_size_mb < 0.05:
                        os.remove(cropped_img_file_name)
                        image_number -= 1
                    image_number += 1
print('Done.')
```

# We're Done!

And voilà! From here, we have a collection of images that are all the ROIs of the scan. We can single-handedly go through hundreds of images at a single go. Of course, the effort of manually scanning the photos is there, but that is a non-negotiable step, and a much better alternative to Google PhotoScan, considering that we can scan multiple images at the same time.

One thing this program does not have is auto alignment of images. During the scan, if the image is not aligned, it will not be rotated to align with the scan. This is something that is pretty easy to do with OpenCV, but I did not find this to be a huge issue, since aligning the photos by hand is not terribly difficult.

Aaaand, that's all I have to say about this. My first blog post in 2 years. More are coming soon. I hope you enjoyed it.

From the stars,\
FR.


Digitizing Old Photos using Computer Vision

A post on how I leveraged machine learning in order to predict the revenue of a movie, and also determine which features are the most important contributors.

Predicting Movie Revenue using Machine Learning

In this post, I explain how I set up a very basic automatic file download system for an on-premise data warehouse using AWS S3 and a bit of bash scripting.

<MainHeading postData={postData} />

The company I'm currently interning at was planning to set up an on-premise datawarehouse. Usually, this it too expensive so many companies shift to Amazon Redshift for their data warehousing needs. But we had the hardware set up and ready. Usually, there would be people accessing the warehouse for different needs.One such group of people worked with .csv files that arrived on a daily basis. They wasted extra time on ETLing everything so I was tasked with implementing a possible time saving alternative. I had to set up a pseudo data lake in which these files would be stored. For those who don't know what a [data lake](https://en.wikipedia.org/wiki/Data_lake) is:

> A data lake is a system or repository of data stored in its natural/raw format, usually object blobs or files. A data lake is usually a single store of all enterprise data including raw copies of source system data and transformed data used for tasks such as reporting, visualization, advanced analytics and machine learning. A data lake can include structured data from relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, documents, PDFs) and binary data (images, audio, video).

For my purposes, the data inside this lake would limited to .csv files. Files would come in on a daily basis and be stored in the lake. The on-premise data warehouse needs to take only the new files and upload them to its warehouse, on a timely basis. For the proof of concept, I was told to implement the warehouse using [Clickhouse](https://clickhouse.tech/), which is a columnar-oriented database system. For those who don't know, a columnar format like ORC or Parquet is much, much more efficient than row based formats like .csv. It's a godsend if you're working on the AWS ecosystem, because you get to save on both time and money. How you ask? In a simple sense, AWS charges you for both storage and query execution time. Since formats like Parquet take up much less space and are more efficient, you end up saving a lot of money in the long run. This comparison shows how amazing columnar-oriented file formats are:

![CSV vs. Parquet Comparison](/images/blog-posts/bash-scripting-aws/csv_vs_parquet.png "A massive improvement!")

As cool as columnar-oriented files are, I did not use them for my proof of concept. Simply because I was just learning the AWS ropes and had no need for them except knowing that they existed. However this could be easily achieved using [AWS Glue](https://aws.amazon.com/glue/), which an ETL service. You could easily set up and [AWS Lambda](https://aws.amazon.com/lambda/) trigger which would be invoked each time a new file is uploaded. This Lambda function would then convert the .csv files to Parquet files and store them in a separate S3 bucket.

> ETL is short for extract, transform, load, three database functions that are combined into one tool to pull data out of one database and place it into another database. Extract is the process of reading data from a database.

Woah! This blog is about bash scripting. Why am I talking about all of this? Because Clickhouse isn't available for Windows. And what is bash script anyways? In very simple terms, it's just a file containing a bunch of bash commands that you can run sequentially. I had access to an OpenStack Linux instance running CentOS 7. Pretty cool. All I had to do was SSH in and boom! Linux. It was kinda unnerving to interact with an OS using the terminal alone, but pretty soon I came to *love* it. Not more than GUIs mind you, but it has a special charm of its own.

*Anyway.*

This is how I envisioned my script would work:

*   Check if new files have been added on S3 using AWS CLI.
*   If no, download all files with .csv extension using `aws sync`.
*   If yes, get list of new files by doing comparison between local list of downloaded files and list of all files on S3.
*   Download new files using `aws cp`.

Onwards!

# The Script

First step is to check if there are files to download. I do this by listing all files in my S3 bucket and storing the output to a text file.

```bash
aws s3 ls --recursive s3://bucket-faaez/data > files_in_s3.txt
```

Now, the files are listed with their full file paths, i.e with the name of the bucket and the folder they are in. We don't want this, we only want the names of the .csv files. So I use [sed](https://en.wikipedia.org/wiki/Sed) which is stream editor. It allows us to easily manipulate files using regexes. All I wanted was the name of the files, so I used a regex which would remove all text before the '/'. This unfortunately caused some empty lines to appear, which again can e removed using stream editor.

```bash
# the -i is to edit the files in place
# otherwise, the output would show on the terminal
sed -i 's:.*/::' files_in_s3.txt
sed -i '/^$/d' files_in_s3.txt # removes empty lines
```

![Code for removing slashes](/images/blog-posts/bash-scripting-aws/sed_remove_slash.png)

Now's the interesting part. Remember I said that the warehouse only needs to downloaded the new files? Well here's how we do it. I keep a local text files containing the names of all the files that have been downloaded so far. Initially, this file would be empty. Then using the list of all files from S3, we perform a set difference between the files to get the files that we need to download. For those of you who are hazy on set theory, A-B basically says to take elements existing in A but not in B. So basically, all the new files.

We can do this using the `comm` command, which stands for common. It requires that the files be sorted, so we have to sort them beforehand. The -1 argument removes lines unique to the first file (passed as the first argument) and -3 removes lines that appear in both files. So what we're left with is the files left in the second file. The output of this is the saved to another text files, using `>` which is the redirection operator. Basically redirects your output to another place. On another hand, so many text files!

```bash
comm -1 -3 <(sort files_in_local.txt) <(sort files_in_s3.txt) > files_to_download.txt
```

![The comm command in action](/images/blog-posts/bash-scripting-aws/comm.png)

Next, we need to check if our local database is already up to date. This is easy, as we have to check if the files\_to\_download.txt is empty or not. If it is, we exit the script. Else, we move on. I iterate through each name in the text file and pass them to the `aws cp` command, with the `--quiet` option, so that my script's output looks clean.

```bash
if [[ ! -s files_to_download.txt ]]; then
        echo "Local database is up to date!"
        echo "Exiting..."; echo -e "\n"; exit 1
else
        echo "New files found. Downloading..."
fi

while IFS="" read -r p || [ -n "$p" ]; do
        echo "Downloading $p"
        aws s3 cp s3://temp-bucket-faaez/cme-data/$p /home/centos/aws-s3-data --quiet
        echo "$p" >> files_in_local.txt
done < files_to_download.tx
]}
```

The next step is to upload these files to our Clickhouse database. The tables have already been created beforehand, with the necessary types. Unfortunately there is no way to do this dynamically, as Clickhouse is relational, as many other databases out there. Once we create our tables before hand, we upload them using `clickhouse-client`. Additionally, we don't have further need for those downloaded .csv files, so we remove them too. And in the end, we remove all text files except files\_in\_local.txt.

The `s | grep XYZ > file_checker.txt` line is necessary because there were two different .csv files coming in, each with their own different names appended to the end. Let's say those two different files had "ABC" and "XYZ" and the end of their names. Since they're both different, they needed to be uploaded into different tables. We can filter them out using [grep](https://en.wikipedia.org/wiki/Grep), which is one of my favourite commands. It stands for **\*g**lobally search a **re**gular expression and **p**rint.\* Basically, search for files which match a certain search criteria which you specify using regex.

> *Linux Life Hack:* If you can't remember the a command you typed before, just do `history | grep <key>` where key is any word in that command. You'll be presented with your command history and can easily find out the exact commands you executed before. You're welcome.

```bash
ls | grep XYZ > file_checker.txt

if [[ -s file_checker.txt ]]; then
        echo "Inserting XYZ files into database..."
        for filename in *XYZ*; do
                clickhouse-client --query="INSERT INTO xyz FORMAT CSVWithNames" < $filename
        done
else
        echo "No new XYZ files found."
fi
```

# And We're Done!

Except we're not. Once I built my solution, I went and asked if it's alright. They said no, the main reason being that I'm storing the names of these files in a plaintext format. They stated that flat files are prone to corruption, and it'll be nasty if the file size grows too long. My solution does not hold up against scale. I also found some other reasons why it wouldn't be good:

*   as the text file grows, searching it becomes slow.
*   using `comm` requires us to sort both files, which would eventually cause a bottleneck.
*   `aws ls` can only list a maximum of 1000 files, so my solution wouldn't work if there were more than a 1000 files.

In hindsight, I should've thought about this. Listing *every single file* on the S3 bucket each time it had to update the database seemed wasteful. Moreoever, the bucket would only grow about 2 files per day. At a certain point down the line, it would hinder efficiency. This solution, as I stated before, would not be scalable if there were other types of files coming in.

Nevertheless, this was a good introduction to the world of shell scripting. Can I do this again from scratch? Heck no. Thank you Google, and thank you all for reading this blog post. See you later. And here's the entire script in case you'd want to see it. Why would you want to see it??

From the stars,\
Faaez.


Bash Scripting for Automated AWS S3 Downloads

A post on how I analyzed various neighbourhoods in Chennai using the Foursquare API to find which neighbourhood you should live in if you love food.

<MainHeading postData={postData} />

As part of a course I'm doing on Coursera, I am supposed to write a blog post for my final project. What better platform to do that other than your own website?

The certification course is called the IBM Data Science Professional Certification. While it is a great starter course, I found it to be to great in width and very low in depth. I would recommend it to most who have no knowledge of data science.

Anyway, here's the content.

# The Project

The final capstone project is called 'Battle of the Neighborhoods'. Here's what the instructions say:

> Now that you have been equipped with the skills and the tools to use location data to explore a geographical location, over the course of two weeks, you will have the opportunity to be as creative as you want and come up with an idea to leverage the Foursquare location data to explore or compare neighborhoods or cities of your choice or to come up with a problem that you can use the Foursquare location data to solve.

I am supposed to use the Foursquare API (a location based service), in order to get location data and use it to compare different neighborhoods and use it to solve a problem. I came up with a very trivial problem since I was short on time (since Coursera works on a monthly subsription basis and I didn't want to pay any more). However in the future I will come up with an actual problem that needs to be solved.

# The (Fake) Preamble

Mr. Nolan is going to be moving in to the city of Chennai, located at the south edge of the Indian subcontinent. He needs to find suitable housing. However, he frets. He doesn't worry about the locality one bit; except the fulfillment of one condition: there needs to be a lot of restaurants/food stalls nearby. You see, Mr. Nolan is a foodie. Not a day goes by where he does not make or savour a new dish. He has come to me for help, asking me to analyze the different areas in the city of Chennai and find which neighborhoods would be the best for a foodie like him to move in to. Me, being the perfect philanthropist as I am, have decided to help Mr. Nolan using a bit of data and a bit of science.

# Introduction

The aim here is to find neighborhoods with a high frequency of restaurants/food stalls/cafés. Firstly, the number of neighborhoods and their respective coordinates need to be retrieved, so that Foursquare can find nearby venues. The preferred method is using the `geopy` package but for some reason, it did not work here. So I used `urllib` and `bs4` to get the coordinates. Using this data, Foursquare should be used to search for nearby venues and get their categories.

These venues are then clustered using k-means. The cluster in which eateries are of the highest frequency will be the set of neighborhoods we are looking for. All of these neighborhoods would be suitable for Mr.Nolan to move in. This problem can also be easily extended to fit other requests, such as finding the neighborhoods with low real estate prices, neighborhoods with a wide variety of grocery shops, neighborhoods closest to public transportation systems, etcetera.

The target audience here is people who are moving to a new city and require some knowledge about the neighborhoods beforehand so that they can decide the place they want to live in.

# Data

## Wikipedia Scraper

Since available data for Chennai city was sparse online, I've manually scraped the list of neighborhoods from [this](https://en.wikipedia.org/wiki/Areas_of_Chennai) Wikipedia page using `bs4`, and then grabbed all the hyperlinks. Using `urllib`, these links are visited individually and the coordinates and pincodes are scraped and put into a `pandas` dataframe.

```python
url = 'https://en.wikipedia.org/wiki/Areas_of_Chennai'
page_unparsed = urllib.request.urlopen(url)
soup = BeautifulSoup(page_unparsed, 'html.parser')
wiki_rows = [] # each row in the wikipedia table
urls = []
names = []
wiki_table = soup.find_all("table", {"class": "wikitable"})
for row in wiki_table:
  wiki_rows.append(row.find_all('a', href=True))
# gets names and links of each neighborhood so that further scraping can be done
for i in range(len(wiki_rows[0])):
  urls.append('https://en.wikipedia.org' + wiki_rows[0][i]['href'])
  names.append(wiki_rows[0][i].text)
# getting data from each neighborhood
latitudes = []
longitudes = []
pincodes = []
for url in tqdm_notebook(urls, total = len(urls), unit = 'url'):
  try: # because some links are broken
    page_unparsed = urllib.request.urlopen(url)
    soup = BeautifulSoup(page_unparsed, 'html.parser')
  except:
    continue
  coords = soup.find("span", {"class" : "geo-dec"})
  pincode = soup.find("div", {"class" : "postal-code"})
  if coords == None:  # because some pages do not have coordinates listed
    latitudes.append(np.nan)
    longitudes.append(np.nan)
  else:
    coords = coords.text.split()
    latitudes.append(float(coords[0].replace('N', '').replace('°', '')))
    longitudes.append(float(coords[1].replace('E', '').replace('°', '')))
```

```python
neighborhoods = pd.DataFrame(list(zip(names, latitudes, longitudes)), columns =['Name', 'Latitude', 'Longitude']) 
neighborhoods = neighborhoods[neighborhoods['Latitude'].notnull()]
neighborhoods = neighborhoods[neighborhoods['Longitude'].notnull()]
neighborhoods.head()
```

This is what my dataframe looks like:

|    | Name       |   Latitude |   Longitude |
|---:|:-----------|-----------:|------------:|
|  0 | Adambakkam |    12.99   |     80.2    |
|  1 | Adyar      |    13.0063 |     80.2574 |
|  2 | Alandur    |    13.003  |     80.204  |
|  3 | Alapakkam  |    13.049  |     80.1673 |
|  4 | Alwarpet   |    13.0339 |     80.2486 |

## Foursquare

The next step is to get all the venues in each neighborhood within a specified radius, in this case, 500 metres. To use Foursquare, we need to create an account as they offer a limited number of API calls per day for a free user (more if you give your credit card details). After signing up, we will get a `CLIENT_ID` and a `CLIENT_SECRET` which are then appended to a URL along with other parameters, which is then used to send a GET request to that URL. The resulting JSON file is then parsed and stored for further use.

```python
url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
  CLIENT_ID[::-1], 
  CLIENT_SECRET[::-1], 
  VERSION, 
  lat, 
  lng, 
  radius, 
  LIMIT)
  # make the GET request
  results = requests.get(url).json()["response"]['groups'][0]['items']
```

After doing this for each neighborhood, the resulting venues are again put into a `pandas` dataframe.

|    | Neighborhood   |   Neighborhood Latitude |   Neighborhood Longitude | Venue                   | Venue ID                 |   Venue Latitude |   Venue Longitude | Venue Category          |
|---:|:---------------|------------------------:|-------------------------:|:------------------------|:-------------------------|-----------------:|------------------:|:------------------------|
|  0 | Adambakkam     |                 12.99   |                  80.2    | Pizza Republic          | 4bf58dd8d48988d1ca941735 |          12.991  |           80.1986 | Pizza Place             |
|  1 | Adambakkam     |                 12.99   |                  80.2    | Loiee                   | 4bf58dd8d48988d16a941735 |          12.9922 |           80.199  | Bakery                  |
|  2 | Adambakkam     |                 12.99   |                  80.2    | Thalapakattu Hotel      | 4bf58dd8d48988d142941735 |          12.992  |           80.1989 | Asian Restaurant        |
|  3 | Adambakkam     |                 12.99   |                  80.2    | The Great Kabab Factory | 5283c7b4e4b094cb91ec88d7 |          12.9938 |           80.2017 | Kebab Restaurant        |
|  4 | Adyar          |                 13.0063 |                  80.2574 | Bombay Brassiere        | 54135bf5e4b08f3d2429dfdd |          13.007  |           80.2564 | North Indian Restaurant |

# Methodology

After this, the venue categories are one-hot encoded and then the 10 most frequent venues (only five shown here) in each neighborhood are found using code that I totally did not copy-paste from the tutorial notebooks. The result is this:

|    | Neighborhood   | 1st Most Common Venue   | 2nd Most Common Venue   | 3rd Most Common Venue         | 4th Most Common Venue   | 5th Most Common Venue   |
|---:|:---------------|:------------------------|:------------------------|:------------------------------|:------------------------|:------------------------|
|  0 | Adambakkam     | Pizza Place             | Bakery                  | Kebab Restaurant              | Asian Restaurant        | Women's Store           |
|  1 | Adyar          | Indian Restaurant       | North Indian Restaurant | Vegetarian / Vegan Restaurant | Electronics Store       | Juice Bar               |\
|  2 | Alandur        | Hotel                   | Fish Market             | South Indian Restaurant       | Movie Theater           | Donut Shop              |
|  3 | Alapakkam      | Indian Restaurant       | Fast Food Restaurant    | Women's Store                 | Donut Shop              | Flea Market             |
|  4 | Alwarpet       | Indian Restaurant       | Lounge                  | Hotel                         | Japanese Restaurant     | Restaurant              |

After this, k-means clustering is used to group these neighborhoods. I forgot to choose the best k for this algorithm. I didn't notice until after I published this. Oh well. After they are clustered, we can use Folium (which is a map visualization library for Python), to see all the neighborhoods and the cluster they belong to on a nice map:

```python
chennai_merged = chennai_merged[chennai_merged['Cluster Labels'].notnull()]
# create map
map_clusters = folium.Map(location=[13.067439, 80.237617], zoom_start=11)
colors = ["#ff0000", "#3d84ad", "#000000", "#ffff00"]
# add markers to the map
markers_colors = []
for lat, lon, poi, cluster in zip(chennai_merged['Latitude'], chennai_merged['Longitude'], chennai_merged['Name'], chennai_merged['Cluster Labels']):
    label = folium.Popup(str(poi) + ' Cluster ' + str(cluster), parse_html=True)
    folium.CircleMarker(
        [lat, lon],
        radius=5,
        popup=label,
        color=colors[int(cluster)],
        fill=True,
        fill_color=colors[int(cluster)],
        fill_opacity=0.7).add_to(map_clusters)
map_clusters
```

![Folium map showing different neighbourhoods on a map of Chennai](/images/blog-posts/neighbourhood-clustering/folium.png)

Not much can be inferred from this map. Due to the fact that the clusters are grouped together not due to their Euclidean distances on the map, but due to properties of the venues themselves. Inspecting the dataframe which has the cluster labels along with the most frequent venue in each neighborhood, we can see that:

*   Cluster 0 (Red) have Indian Restaurants as their most frequent venue.
    ![Screenshot of notebook showing most frequent venues](/images/blog-posts/neighbourhood-clustering/clus0.png)

![Barplot showing most frequent venues](/images/blog-posts/neighbourhood-clustering/clus0_bar.png)

*   Cluster 1 (Blue)'s frequent venues are mostly related to food. Evident by counting the occurences of each.
    ![Screenshot of notebook showing most frequent venues in Cluster 1](/images/blog-posts/neighbourhood-clustering/clus1.png)

*   Cluster 2 (Black) has pharmacies as the most frequent venues.
    ![Screenshot of notebook showing most frequent venues in Cluster 2](/images/blog-posts/neighbourhood-clustering/clus2.png)

*   Cluster 3 (Yellow) has multiple forms of transport as the most frequent venues.
    ![Screenshot of notebook showing most frequent venues in Cluster 3](/images/blog-posts/neighbourhood-clustering/clus3.png)

Use these results, we can solve our trivial problem. Nolan should move to neighborhoods in either Cluster 0 or Cluster 1, as they have a high concentration of restaurants and other food-related venues.

# Results

While it looks like we have solved our problem, there is one flaw. Clusters 1 and 2 (Red and Blue) are grouped sporadically, in such a way that Nolan would not have a problem finding food in the majority of the city. Of course, Foursquare data for the city of Chennai is considerably sparse compared to other well developed cities. It also probably does not take into factor the ton of small food shops scattered throughout the city. Using Foursquare, the individual ratings for each venue could also be retrieved, but it did not seem to have rating data for Chennai. This would've helped pick out individual restaurant suggestions and give Nolan a neighborhood with highly rated restaurants.

# Conclusion

Mr. Nolan would not have trouble finding food in the city of Chennai. For lesser travel times, he can choose any of the neighborhoods in Cluster 0, however he'll find that most of them are Indian restaurants. Independent of travel distance, the cluster choice does not matter much as there are more restaurants than any other venue. There is simply not enough data to do an in-depth analysis. However, individually marking the venues which are food-related is also a possibility- something to do for the future.

The outcome of the project was impacted by the limited effectiveness of the Foursquare API for a city like Chennai. Using some other more developed city would've probably yielded better results. The way the neighborhoods were suggested were majorly due to Indian restaurants alone.

Further analysis could've been done by using the rating of each venue however yet again ratings were not available for venues in Chennai. If ratings were available, individual restaurants in each locality could've been suggested.

# Final Thoughts

I am in no way entirely happy with the content I posted. It's just too simple. There is so much more that could be done but for now this shall suffice. I will probably get back to this in the future. *Probably.*

From the stars,\
FR.


Using the Foursquare API to analyze Chennai's neighborhoods

This post is a rant about how the current Indian education system is doing more harm than good.

<MainHeading postData={postData} />

This post will give you a small insight to why the Indian education system is utter garbage, followed by my experiences with this godforsaken assemblage of marks and money. As it is right now, it's an archaic factory pumping out millions of "engineers" who have memorization powers unlike never seen before. 

Why am I ranting? Because I like to. I've ranted time and again to friends of mine, knowing perfectly well that I'm better off trying to conjure my very own Patronus (which is a hedgehog by the way). I know that I cannot possibly change the rusty minds of our 'humble teachers', so I've stopped trying entirely. Now I'm ranting online. Great. 

I'm studying Computer Science at a Tier 3 (the lowest tier) university at the far south edge of India. Couldn't do anything about this due to circumstances I do not wish to explain here. Anyway, I chose CS because I love it. I love how you can tell precisely tell what you want your computer to do with mere lines of code. I love the sound the keyboard makes when you type.

This isn't the case for the majority of the students in my classroom though. I guarantee you, if you walk up to a student and ask him why he chose this major, he'd reply,

> *'Duh. Cuz my parents said so.'*

Here's the first problem. The parents. Yes, I get it. You want a good life for your kid. You want him to get "settled". You don't want him to face what you faced. But what do you mean by "settled"?  Does it mean you control what your child does and force him to repeat the vicious cycle of study, marry, have kids and retire? This way of living has become commonplace here in India. Does it mean you don't want him to live his dream life?

I once asked a classmate, why he chose CS, when he obviously didn't belong here. With a fierce glint in his eyes, he replied, 

> *Dude, I love cricket. If I had kept practicing I would've been in \[some division] by now. But what do I do, my parents won't allow that.*

One reason why the system isn't working is that the parents are forcing their decrepit ideologies into the minds of their children. I'm eagerly anticipating the next generation of parents who will hopefully allow their kids to study what they want. I'm grateful to this day to my parents for giving me the freedom to choose what I want.

# Actual proof that the system isn't working.

As it is right now, the system is more about getting enough marks rather than actually understanding a subject. Students fret over these superficial marks more so than the knowledge they'd be able to gain. It is universally known that *knowledge = power*. Marks don't mean shit.

I got hold of a database containing academic details of fellow students. With this, I could see how everyone's GPA progressed over the semesters. As shown below, the graph below exhibits a steady increase. This is not normal. It can one of two things: either the students are actually learning (which I assure you, isn't the case), or the students figured out a way to play the system.

![Average class GPA progression over semesters](/images/blog-posts/education-system-rant/gpa-progress.png)

# Why the steady incline?

*Logically*, this increase in GPA progression over the semesters can be explained by the following reasons:

*   subjects getting easier
*   teachers slacking off
*   teachers awarding free marks
*   students 'playing the system'

**Subjects getting easier:**
While the first two semesters barely had any CS subjects, progression of subjects slowly went from basic programming paradigms to complex concepts like page replacement algorithms in operating systems to lexical analyzers in compiler design. In no way did the subjects get easier. This is expected, as a well structured study program would ramp up in difficulty as you move on to more complex subjects. I pity the one who created the syllabus. If he'd see how far the college has fallen in implementing it, he'd surely shed a tear or two. Surely. The subjects never got easier, it's just that pedagogy went to shit.

**Teachers slacking off:**
I've only seen one teacher who was actually good in teaching and was passionate about it. The rest taught the subjects half-heartedly and corrected papers just for the sake of it. The fact that the majority of the people don't think that this is a big problem still baffles me to this day. I once had a teacher correct my exam paper in a mere 10 seconds. Yes, 10 seconds. And he did it right in front of me. I even got awarded marks for a wrong answer I wrote. So there's that. The way they blatantly display their unconcerned smirks in front of hundreds of students is honestly sickening.

**Teachers awarding free marks:**
Teachers award extra marks just so the student can pass a paper. What does this lead to? A batch of mediocre students just managing to coast along by short-term rote memorization. Great.

**Students playing the system:**
I think it's safe to say that the majority of education systems in the world are 'playable', the one in India even more so. You figure out how it works, and you trick the system into thinking you actually know what you're doing, but in reality you're just pushing the right buttons.

I also played the system. When you have unmotivated teachers, a lack of resources, lack of mentorship and a mediocre college, the motivation to actually study is non-existent. I 'studied' on the last day for each of my exams and managed to get a 9.1 GPA. Don't ask me why I never actually studied the subjects. I had no reason to.

# A story of broken dreams

While joining, I had high hopes. I wanted to make the best use of the resources given to me. But reality hit. And it hit hard. The university wasn't going to help me. It had no resources to give.

I had come from a syllabus which required you to truly understand the subjects you're studying and was thrown into a system where rote memorization and simple pencil diagrams were commonplace and a surefire way to fetch you marks. The assignments I wrote by myself would get the same marks as the 40 others who plagiarized it from a single guy who was gracious enough to share it on the WhatsApp group. Sometimes this gracious guy was me.

Practical subjects are only given a credit value of 1 while theory subjects were given a value of 3. You'd have people who knew nothing about algorithms get an A grade on the theory paper while the same person couldn't get a simple bubble sort algorithm to compile without errors. People who did not know how to run MySQL from the Linux terminal ended up passing exams which required you connect databases to a PHP website. Executing your code succesfully amounts to a mere 15% of your entire grade. Writing the code by hand on paper gains you 50% of the marks. And it isn't even checked to see if it's correct. All the invigilators want is a piece of paper filled with syntax errors.
*Why did I even bother putting that extra semicolon?*

I am one of the few in my class who would compile everything without errors. The rest couldn't even manage to compile a program succesfully, yet 90% of them would pass. How is this is any way fair to the people who are actually interested and put an effort to study? It's like the people who do a task correctly and people who don't attempt it both get the same rewards. Put yourself in this situation and you can realize how frustrating it gets after a while.

Pretty soon I realized what an absolute mess this system is, and I was better off forging my own path.

Anyway. I'm ending this short. I want to write more but thinking about the time and money wasted just makes everything feel so drab. Here's a nice tweet to end it on a positive note:

![A relevant tweet from the Twitter user @MatPatGT](/images/blog-posts/education-system-rant/matpat-tweet.png "So true.")

# How to fix this?

My attention-span has been depleted, therefore I'll just paste an excerpt from my Quora answer here:

> All these problems have an easy solution. If the students were a tad bit involved and interested in what they were studying, progress could be boundless. But alas, they aren’t. Only 5% of a given class truly is.
> The education system here is a joke. My friend sleeps in class, yet studies on the last day before exams and manages to get a 10 point GPA. This mediocrity that Indian students possess stems from deeper problems, one being the archaic mindset of parents, due to which they force their children to study a “prestigious” degree which they have zero interest in, just so they could be in a stable job, earn just enough money for their eventual 2 kids to repeat the same process, and retire at the age of 60 filled with regret. I don’t see this problem going away anytime soon.
> Only time will tell.

From the stars,\
FR.


The gargantuan inefficiency of the Indian education system

This post is about the exploratory data analysis that I conducted on movies from IMDb and used it to achieve some interesting insights on the movie industry.

<MainHeading postData={postData} />

Sup. I'm back. And I've got something cool to show ya.

I've been working on a small data science project for the past few days, and I've just completed it.

After trying out various domains in the computer science field, I've finally come a decision that I'm most interested in domains such as data science, artificial intelligence, ML/DL. Although my skills in the latter 2 are almost non-existent, I decided to take what I know in data science and apply it in a small project to see how it goes.

What I built is a webscraper using BeautifulSoup4, which scrapes a list of Hollywood movies released from 2000-2018 and gets various information such as IMDb and Metacritic ratings, genres, gross profit etc. This is the first project that I've been truly interested in for a long time and I am glad I was able to complete it and gain some insights.

I think that's what I like about data science. Gaining insights on data which would otherwise be invisible. I was mainly interested in seeing how the IMDb scores and metascores varied, but the deeper I went in I found out I was also interested in things like which genre produced the most profit, and see whether normal distributions actually apply to real-world data. (This concept still fascinates me)

# The Web Scraper

The hardest part. I went in with a barebones knowledge of bs4, and it proved to be a hard time extracting the data from the website. Even after extracting them, a *lot* of data cleaning had to be done before I could perform data analysis. This was the most difficult (*and fun*) part of the entire project, and I'm ashamed to say that looking back now, it doesn't seem that hard. Trivial, in fact.  Maybe I just have a rapid knowledge consumption rate unlike all you plebs.

![GIF of James Francom looking back and winking](https://media.giphy.com/media/wrBURfbZmqqXu/giphy.gif)

*Just kidding.*

I figured out that maybe I wasn't scraping data efficiently, as I had to do a lot of string manipulation to get the data I wanted. I mean, look at this:

```python
meta_ratings.append(float(rating.text.split(" ")[0].split("\n")[1])/10) # don't even ask.
```

Maybe I've overcomplicated it due to lack of bs4 knowledge, but it works. And that's all the matters. For now.

```python
movie_names = []
movie_years = []
imdb_ratings = []
meta_ratings = []
movie_earnings = [] 
movie_genres = []
runtimes = []
pages = 10  # 10 pages per year.
start_year = 2000
end_year = 2018
for i in tqdm_notebook(range(start_year, end_year+1), total = end_year-start_year, unit = 'year'):
  k = 1
  for j in range(1, pages + 1):
    imdb_url = 'https://www.imdb.com/search/title?release_date=' + str(i) + '-01-01,' + str(i) + '-12-31&sort=num_votes,desc&start=' + str(k) + '&ref_=adv_nxt'
    page_unparsed = urllib.request.urlopen(imdb_url)
    page_parsed = BeautifulSoup(page_unparsed, 'html.parser')
    k += 50
    movie_divs = page_parsed.find_all('div', class_ = 'lister-item mode-advanced')
    for movie in movie_divs:
      skip_movie = False 
      meta_rating_unparsed = movie.find('div', class_ = 'inline-block ratings-metascore')
      gross_unparsed = movie.find_all('span', attrs = {'name' : 'nv'})
      if meta_rating_unparsed is None or gross_unparsed is None:
        continue
      temp = movie.find_all('span', attrs = {'name' : 'nv'})
      if len(temp) is not 2:
        continue
      else:
        movie_earnings.append(float(temp[1].string.strip('$').strip('M')))
      meta_ratings.append(int(meta_rating_unparsed.text.replace(" ", "").split("\n")[1]))
      movie_genres.append(movie.find('span', attrs = {'class' : 'genre'}).text.strip().split(",")[0])
      imdb_ratings.append(float(movie.find('div', class_ = 'inline-block ratings-imdb-rating').text))
      movie_names.append(movie.find('h3', class_ = 'lister-item-header').find('a').text)
      year = movie.find('span', class_ = 'lister-item-year text-muted unbold').text.split(" ")
      if len(year) == 1:
        movie_years.append(year[0][1:5])
      else:
        movie_years.append(year[1][1:5])
      runtimes.append(int(movie.find('span', class_ = 'runtime').text.strip('min')))
```

# Insights & Data Visualization

*There is an inherent flaw in my dataset. Each movie had multiple genres, but I took only the first genre listed. I do not know how to incorporate multiple genres into this project. And for that reason, all findings related to genre might be wrong. A sad thing to say the least, but I carried on with it as I was too invested.*

## Rating Metric

I used Seaborn, because it's Seaborn. The first thing I wanted to see was if IMDb ratings and metacritic ratings have any correlation with each other. The scoring metric (taken from their respective websites) for each are as follows:

### IMDb

> We take all the individual ratings cast by IMDb registered users and use  them to calculate a single rating. We don't use the arithmetic mean (i.e. the sum of all votes divided by the number of votes), instead the rating displayed on a title's page is a weighted average.

Interesting. The reason for them using a weighted average is to enforce the concept of [Wisdom of the crowd](https://en.wikipedia.org/wiki/Wisdom_of_the_crowd).

### Metacritic

> While we solicit votes from our site visitors on movies, games, and music, and television shows we do not include those votes in the METASCORE. The METASCORE is a weighted average of the published critic reviews contained in the chart on that page, and thus does not include any votes or comments from our users.

Interesting! While IMDb is wholely dependent on the individual users, Metacritic uses scores from public critic reviews. Looking at the scoring metric for both of these, I can guess that correlation between them will be positive, as the people rating them are in fact, humans.

![Scatterplot of IMDb vs. Metascore values](/images/blog-posts/data-analysis-imdb/1.png)

As evident by the KDE plot below, IMDb and Meta ratings have the same opinions on movies with a ratings of 6 - 6.5 (or 60 - 65).
With both these graphs, we can make our first inference:

![KDE plot of IMDb vs. Metascore values](/images/blog-posts/data-analysis-imdb/2.png)

**Inference 1:** People and critics tend to have the same view on movies. Both have the same opinions one movies with a rating of 6 (or 60).

This distribution plot for each metric is shown below. I'm guessing this is because meta ratings are done by critics, who take a lot into account and is kinda their job to do this sort of stuff. So their ratings are gonna vary a bit. That's why the distribution mirrors a normal distribution. Meanwhile, IMDb ratings are done by us, normal folk. If we like a movie, we give it a high rating, and we tend to do so a lot.

![Distribution plot of IMDb and Metascore values](/images/blog-posts/data-analysis-imdb/3.png)

**Inference 2:** Meta ratings follow a normal distribution while IMDb ratings tend to favour the 5 - 8 range.

Violin plots for rating distributions for each genre didn't give much info, other than the fact that biography movies tend to get high ratings. For 7 genres, the number of movies were 10 or below, so I'm not going to take them into account. These 7 genres cause their respective violin plots to look a little weird.

![Violin plots of Horror, Biography, and Animation genres](/images/blog-posts/data-analysis-imdb/4.png)

**Inference 3:** Biography movies tend to have high ratings.

Boxplots show us an interesting thing. This dataset has a *lot* of outliers. I don't know if it is due to the fact that the dataset is massive, or the inherent unpredictability of movies.

![boxplot-genre](/images/blog-posts/data-analysis-imdb/5.png)

**Inference 4:** In the world of movies, there tend to be a lot of outliers.

Here are the lowest and highest rated movies of each rating metric:

**IMDB**

*   Highest rated: The Dark Knight (2008). Score: 9
*   Lowest rated: Saving Christmas (2014). Score: 1.5

**Meta**

*   Highest rated: Boyhood(2014). Score: 100
*   Lowest rated: Death of a Nation(2018). Score: 1

## Gross

Each movie had anywhere from 1 to 3 genres. To simplify the process, I figured the first genre for each movie would be most accurate. Looking at the dataset, only the first 9 genres are the most abundant.

![Count of genres](/images/blog-posts/data-analysis-imdb/6.png)

First thing I wanted to see was if there was any correlation between the IMDb score and box office. There wasn't. Same can be inferred for meta ratings, as they are positively correlated.

![Scatterplot of IMDb v. Box Office](/images/blog-posts/data-analysis-imdb/7.png)

**Inference 5:** No correlation between IMDb or meta scores and movie box office.

Next, I wanted to check out the mean gross for each genre. Barplot would be perfect for this.

![Barplot of average gross per genre](/images/blog-posts/data-analysis-imdb/8.png)

Hmmm. Notice the big black line for the Family genre? It indicates that there is a high chance of it being erroneous. It's called an [error bar.](https://en.wikipedia.org/wiki/Error_bar) The reason for this is the fact that there are only 2 movies in the Family genre, and one of them was Beauty and the Beast, which earned a whopping $504 million box office. That's a lot.

![Dataframe screenshot showing two of the only family movies in the dataset](/images/blog-posts/data-analysis-imdb/9.png)

I wanted to see how much profit Family movies made per minute. It turned out to **2.2 million per minute of screentime!**. Family movies are the most profitable, followed by Animation and Action.

![Top 5 genres sorted in descending order of profit per minute](/images/blog-posts/data-analysis-imdb/10.png)

**Inference 6:** Family movies had the best profit per movie value.

**Inference 7:** The most profitable movie genres are Family, Animation, and Action.

The sum of gross for each movie show that Action movies made the most money.

![Barplot of sum of grosses per genre](/images/blog-posts/data-analysis-imdb/11.png)

**Inference 8:** Action movies had the most profits (67k million), followed by comedy movies (34k million) and then animation movies (25k million).

## Genre

The barplot of the count of each movie shows that Comedy movies were the most frequent.

![Barplot of number of movies in each genre](/images/blog-posts/data-analysis-imdb/12.png)

**Inference 9:** Comedy movies were the most frequently released, followed by drama and action movies.

The average rating per genre doesn't seem to change much for IMDb. This follows up with my finding that IMDb ratings tend to favor the 6-8 range, and this is evident by the barplot below.

![Barplot of average IMDb rating per genre](/images/blog-posts/data-analysis-imdb/13.png)

**Inference 10:** For IMDb ratings, on average, horror movies got the lowest ratings while war, documentary, and musical movies seem to get the highest.

**Inference 11:** IMDb movies tend to approximately get the same ratings regardless of genre.

Meanwhile, for meta ratings, the average rating per genre tend to be more varied than the IMDb ratings.

![Barplot of average meta rating per genre](/images/blog-posts/data-analysis-imdb/14.png)

**Inference 12:** For meta ratings, documentary movies get the highest average rating, while Sci-Fi and Horror get the lowest.

**Inference 13:** Mean ratings for each genre for metacritic tend to vary for each genre, unlike IMDb ratings.

From Inferences 10 and 12, we can derive another inference:

**Inference 14:** Horror movies are the most disliked, Documentary movies are the most liked.

## Runtime

![Notebook screenshot of runtimes of two genres](/images/blog-posts/data-analysis-imdb/15.png)

**Inference 15:** Biography movies had the longest average duration. (117 mins). Musical movies had the lowest. (90 minutes)

![Notebook screenshot of two movies with the shortest and longest runtimes](/images/blog-posts/data-analysis-imdb/16.png)

**Inference 16:** Longest movie was 'The Best of Youth', which ran for 366 minutes (6 hours). Shortest movie was 'Ghost of the Abyss', which ran for 61 minutes. (1 hour)

The scatter plot for runtime against gross shows that they are slightly correlated.

![Scatterplot of gross against runtime](/images/blog-posts/data-analysis-imdb/17.png)

**Inference 17:** Runtime and box office are slightly correlated. Meaning, if the duration of a movie is more, it earns more.

## Year

Boxplot of mean box office per year shows that the gross has increased as the years progressed. Maybe this is because more people are starting to watch movies or spend more on movies than before. Also notice the error bars. They seem to be pretty high here.

![Barplot of average gross per year](/images/blog-posts/data-analysis-imdb/18.png)

**Inference 18:** Average earning for movies increased as time went on.

The bar plot of count of movies per year show that as years went on, the number of movies released per year decreases. I cannot fathom why. Maybe people got tired of updating the IMDb list. Probably. I hope.

**Inference 19:** Number of movies released per year decreased.

# Summary

The following list encompasses all my findings as brief as possible.

1.  IMDb ratings tend to favour the 5 - 8 range
2.  Critic ratings follow a typical normal distribution.
3.  IMDb ratings and critic ratings are not correlated, therefore they do not judge on a similar basis.
4.  Family movies are the most profitable on average.
5.  Action movies had the highest profits (by summation, not average).
6.  A large amount of movies released belongs to the following genres: Comedy, Action and Drama.
7.  People tend to give generalized ratings regardless of genre.
8.  Critics tend to base their ratings to genre.
9.  Documentary movies are the most liked, while Horror movies are the most disliked.
10. Biography movies ran the longest while Musical movies were the shortest.
11. Movie runtimes ranged from 61 mins to 366 mins.
12. Runtime and box office are positively correlated.
13. As the years progressed, movies earned more.
14. As the years progressed, overall number of released movies declined.

# I'm done.

Phew. Another project completed. I had such high energy when doing the project itself but writing this blog post was a drag. I don't like to hold onto things for long. I'm glad I've finally done with it. It was a learning experience for sure.

The thing that's bugging me is the inherent flaw in my dataset, which I've mentioned above. This may cause all the findings related to genres to be false. That's a bummer. I don't know a workaround for this. Well, it's finally done.

![GIF of Jon Snow saying "My Watch has ended"](https://media.giphy.com/media/3oEjHUf7j0aFDce0dG/giphy.gif)


Exploratory Data Analysis on IMDb Movies

This post entails my experiences with my first ever Kaggle competition, which was fun, and at the same time, nerve-wracking.

<MainHeading postData={postData} />

Sup ya'll. Been a long time since my last post. About a month I guess. Why? I don't know. Maybe it's because I'm lazy or maybe it's because I tend to procrastinate a lot or maybe it's an ugly combination of both. Anyway...

I'm doing a 5 month course on deep learning called [PadhAI](https://padhai.onefourthlabs.in). The end goal is to translate text in a picture to another language (Hindi to Tamil and vice versa). Seems pretty simple but it contains a LOT of steps. Kinda daunting at first but once you start understanding the concepts separately, you get a better idea of the big picture. In the first phase of the course, we were supposed to build a like/dislike predictor for mobile phones. The dataset given contained various feature like screen size, battery capacity, RAM... and much more. There were two contests, one using MP Neuron and Perceptron.

After going over the initial hurdles of submitting the sample solution, I was able to successfully implement the MP Neuron model with an initial test categorization accuracy of about 77%. But this was just the basic model. I had not done any modifications to my model to make it more accurate.

So for the next two days, I tried the following:

*   binarization of columns using the mean of each column
*   binarization of columns using the mean of phones with a positive rating
*   feature selection
*   a lot more bullshit that I randomnly implemented without thinking and hoping it would boost accuracy. Spoiler alert: It didn't.

But to no avail. Frustrated, I tried other wacky solutions, and one seemed to work. I binarized each column based on the mean of phones with a negative rating. Seems counterintuitive but it worked. For the test data anyway. It boosted the accuracy from 77% to 82%. (this might not seem like a lot but in fact it is, in the world of machine learning). I was ecstatic. My public ranking went from somewhere around 900s, to the #110. I became the top 10% of that contest. Excited, I posted a WhatsApp status thinking I was Einstein or some shit.

![Screenshot showing me advancing 585 places on the Kaggle leaderboard](/images/blog-posts/kaggle/kaggle-yay.jpg "Woohoo!")

Later on, I figured out that binarizing by the mean of disliked phones is not the right thing to do, and it just managed to work for the sample test data by luck and will not help when it comes to the training data. So, being the scaredycat I am, changed my code to binarize the columns using the mean of each column of the liked phones. Accuracy dropped back to 77%, but I was willing to take my chances. For those who don't know, the leaderboard is split into public leaderboard and private leaderboard. Public leaderboards uses 30% of the test data but the private leaderboard uses 70% of the test data.)

To my utter dismay, it did not work. I woke up the next day, expected a top 10% score on the leaderboard but no. My rank had shot down *985 places*. I felt so disheartened I didn't look at the course for an entire week, lol.

![Screenshot showing my rank shooting down 985 places](/images/blog-posts/kaggle/kaggle-sad.png "I want to jump off a cliff.")

Thinking back, there might be a few reasons why my rank decreased:

*   Wrong implementation of mean binarization
*   Failure to do feature selection. Some columns could not have had any effect on the rating of the phones, and dropping them would've increased the accuracy.
*   Selective binarization on each column. I still don't know how this works.
*   Kaggle sucks. Everytime you submit your model, you cross your fingers and hope that you get a good accuracy. Looking at the leaderboards, it looks like the scoring is based on luck for the most part. I've seen people who have shot up 900 places and people have shot down 900 places. Unfortunately I was among the latter.

I mean, look at this guy:

![Screenshot of a guy jumping up 556 places to take rank #1](/images/blog-posts/kaggle/kaggle-omg.png "WHY. JUST WHY?")

![GIF of guy saying wtf](https://media.giphy.com/media/aZ3LDBs1ExsE8/giphy.gif)

Here's a quora comment that captures my experience:

> There is an art and a science in doing so and experience Kagglers are able to make the most out of it without overfitting. If not done well, that usually lends itself to worse scores on the private LB, sometimes disasters. In general the key is to build a model that generalizes well. Local cross validation is essential for that. But the public LB provides also some signal. Kaggle Masters can use cross validation and public LB to optimize private LB scores.

*I think I get the gist of how Kaggle works. I don't like it, but I kinda like it.*

![GIF of one confused Obama](https://media.giphy.com/media/pPhyAv5t9V8djyRFJH/giphy.gif)

By the time I wrote this, I've finished 2 other contests. I will write another blog post when something exciting happens.

*Till then, it's sayonara.*

From the stars,\
FR.


My first experience with Kaggle competitions

A short frustrating story on how I preemptively jumpted into the intricate world of logo design.

<MainHeading postData={postData} />

As the title gracefully suggests, I just made my first ever logo. Looks pretty cool right? If you don't get what it's supposed to mean, (I don't blame you), within the cube exists the abbreviation of my full name.

![My new logo](/images/blog-posts/my-new-logo/logo.png)

Looks seemingly perfect, right? (in terms of symmetry I mean). Wrong. I was too excited that I literally just watched [this](https://www.youtube.com/watch?v=q5Rfu5OJ--M) 2 minute video and jumped right into it. The symmetry is way off, but to the common eye it looks good enough. So I stopped here. As for the color scheme, I picked them up randomly from the colour palette, and they looked good enough so I let them be. There's also some minor protrusions at three corners, but...

![GIF of woman saying "Ain't nobody got time fo' dat"](https://media.giphy.com/media/10PcMWwtZSYk2k/giphy.gif)

I've wanted to try logo design for a long time, and I finally picked up the tiny scraps of motivation from the ground lost eons ago, and managed to conjure up this magnificent, ethereal, abstract and an absolute beast of a logo that even Bob Ross would be proud of. (Okay I'll stop). There is so much I can improve upon this logo, but for now this will do.

I still don't know if I might try logo design again. Sure, it was kinda frustrating, but maybe it was because I didn't know my way around the different tools which led me to randomly click each tool until I went "Aha! That's the one!". On the other hand, I loved the creativity aspect of it: making up designs in my head and having them come to life. Maybe I should try again, but this time, I actually know how to use Illustrator by watching a tutorial or something.

If you want me to design a logo, hit me up. I'll do it for free. (just the first one tho) (and i'm a lazy guy, don't expect a fixed deadline or some shit lol)

See ya later,\
FR.


My First Ever Logo

An interesting story on how I built a script to automatically retrieve exam results from a university exam portal using Selenium and Python.

<MainHeading postData={postData} />

My second blog post! (woohoo!)
I'm going to explain how my results scraper works. A very simple script that I built in 2 days. Why an entire post for this you say? Because this script holds a special place in my heart since it's the first ever project I ever attempted- and successfully completed.

The script works using [Selenium](https://www.seleniumhq.org/), a portable framework for testing web applications which can be used to interact with HTML elements. If you look at my results page, it looks like this:

![Landing page of BSACIST's old results portal](/images/blog-posts/exam-results-scraper/landing-page.png)

Pretty basic. An input element, a dropdown list, and an enter button. For the sake of simplicity (and just because I was lazy), I decided to not have any interaction with the dropdown list. Which means I could get the results of everyone who wrote just the most recent semester examination. Also, further down the line, I lost the need to implement this because.... *sigh*... I'll explain at the end.

So I get the IDs of the input box and button, which I used to paste a number into the the input box, and then I press enter.

So then we arrive at the following:

![Results screen of BSACIST's old results portal](/images/blog-posts/exam-results-scraper/results-screen.png)

For whatever reason, some roll numbers don't exist, due to which my script initially failed. It took me half a day to come up with the solution, mainly because this was my first time working with Selenium and I had *no clue* about the intricacies of web scraping. Moreover, I possess the attention span of a goddamn goldfish. Moving on...

The data resides in HTML tables, so I get the info by scanning the page for the td tag. Since I get all data (I mean *all*) from the tables, I filter it out by enforcing two more conditions: `if len(i.text) < 2 and i.text.isalpha(): `

So I take these grades, append them to the list, and input it to my GPA calculating function. The outcome is then again, a list, which co-exists along with 2 other lists, containing the student's name and roll number. *List-ception!*

```python
def calc_gpa():
    creditPerSubject = [3, 3, 4, 3, 1, 1, 1, 1, 4, 3]  # the credit pertaining to each subject, in order.
    totalcredits = 0
    i = 0
    for credit in credits:
        totalcredits += (credit * creditPerSubject[i])
        i += 1
    return round((totalcredits/24), 2)
```

The script does this for each number, and it appends them one by one to a .csv file. Near the end of the script, I was getting frustrated, because I have this tendency to lose motivation quickly. So I just used our good ol' companion StackOverflow to get the code to write these lists to the .csv file. Finally, the outcome is this:

![Screenshot showing an Excel sheet filled with students' GPAs](/images/blog-posts/exam-results-scraper/results-csv.png)

So. The gist of it is as follows:

*   run a for loop through the roll number range
*   input roll number into box, click the button
*   get only the grades from the page
*   do some magic and get the gpas
*   write 'em to a .csv file
*   boom

All in all, it takes a pretty long time to do everything, considering there's about 200 students. I had an itch at the back of mind, thinking to myself, "maybe I can make it faster if I use multithreading or some shit". Well it turns out my hunch was right, some dude pointed out in the comments of my Quora answer:

![A cropped comment from my Quora answer](/images/blog-posts/exam-results-scraper/quora-comment.png)

Maybe another day.

# Uneventful Occurences

Now, in an unfathomable twist of events, my esteemed university decided to "upgrade" to a commercially outsourced learning platform, which means results aren't publicly available anymore. Students have to login to view their individual results privately. (The platform sucks by the way, it doesn't even show the GPA, forcing us to calculate it manually, it also scales awfully on mobile, 4 popups scream at my face everytime I login... I could go on but you get the point)

I'd like to thank my college for making stupid ass decisions, and also rendering my script useless within mere months of writing it.

# Data Visualization using Seaborn

Currently, I'm learning about data visualization using Seaborn and I thought it'd be a good idea to implement some plots using the dataset that I just scraped.

I thought I could add the dropdown list to my script to get more info for my dataset, but sadly my university has taken down the website (as of Jan 22, 2019), so this will have to do.

Nothing complicated, just wanted to try something out, so I drew a distribution plot of GPAs using Seaborn and I got the following output:

![Distribution plot of GPA](/images/blog-posts/exam-results-scraper/dist-plot-gpa.png)

The mean GPA was 6.54. (higher than I expected, lol) I also wanted to see if there was any correlation between the roll number and the GPA using a scatter plot, and as you and I might have guessed, there wasn't.

![Scatter plot of GPA vs. Roll Number](/images/blog-posts/exam-results-scraper/scatter-plot-gpa.png)

Till we meet again,\
FR.
