Automating not Just the Boring Stuff
As I go forward in my development as a programer and data scientist I always wish the days were longer, there is always more to learn, do, and accomplish. I am reminded of a quote that says that art is never finished merely abandoned, and while I would never dream of saying that I feel anything as deeply as any artist, this is something that I can relate to with my projects.
So naturally with those kinds of feelings to serve as the foundation for my motivation I wanted to dive into automation. To see what I could learn about making computer scripts and pipelines to do checks and clean ups in a predictable and reliable fashion.
This grew to be more tedious and difficult than I thought because each data set that I had initially tried to use it with presented with its own unique quirks that I could not find a way to resolve. So, I would either have to drop more data then I was willing to, or have the data to barely be touched.
With this revelation I decided to have a deeper look into the process of automation with python. The automation process that I have had the most success with has been web-scraping. This is the process of getting data readily available on websites to be stored and used later, but I wanted to figure out how I could modify this into something more useful such as clearing out emails and setting appointments as well as personal projects that I find interesting.
Something that I saw was a common thread among most joined scripts for automation was the conversion of file types. I didn’t always understand the reason for this so I think it has to do with the intricacies of each file type.
I also have little experience with image processing so I hope to also gain experience with that. The first step for me was to create better and constant pipelines to so simple processes, because even though this was something that I did not find impressive or particularly useful I did want to familiarize myself with the process.
In regards to web-scrapping I do have some experience with using bots, they are able to have varying behavior and able to by pass different types of checks but ultimately I was not able to use them to their full potential and did things on my own.
I would like to design or better understand such bots as I think they would make my own endeavors in continuing to web-scrape that much easier. As with all web-scraping though I would have to start at a small scale and increment the process so as to not get banned from these websites, because for as much as we like to think that we’re working in the understanding and progressive future people still would rather interact with people then with bots.
The other project that I believe to be in my reach is a bot to help me respond to emails. The job hunting process has devolved into copying and pasting the same email several dozen times in order to try and get any sort of response from people. So something that is able to parse through the content of an email and change certain things as well as being able to send it would be extremely useful to me right now.
I’ll keep working on these projects and write about my progress and development.
