See me in action on my video explaining what I did, here:
Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts
Saturday, May 07, 2016
Video Presentation of my Python Project
I had to present my Python project to the other students in the MOOC, and I did this using a very nifty software called Screencast-o-matic which lets you film yourself using the webcam and also records your screen and mouse action at the same time. For free, too, in the basic version which was perfect for what I had to do.
See me in action on my video explaining what I did, here:
See me in action on my video explaining what I did, here:
Thursday, May 05, 2016
My Python Capstone Project
I've done it! I've made it all the way through the Python for Everybody MOOC by the University of Michigan with the très sympa Dr Chuck (Dr Charles Severance) at the helm who took us through the highways and byways of Python and supported us with substantially complicated scaffolding. Much needed, in my case.
The last module in the 5-part course was the Capstone where we had the opportunity to do an optional project. Always game to get the most out of things, I decided to take the bull by the horns and sign up.
For the project, we had to find a data set, 'scrape' it to find some specific information, put that into a database, and finally, visualise the results. There are data sets about many different fascinating subjects such as:
and so on. For my project however, I chose the Transport for London data set available in their Application Programming Interface (API). It provides access to real time data on the most highly requested information across all modes of transport. It also provides data on accidents across London. I wanted to find out about bicycle accidents (just because), and discover where most accidents happen. I thought it would probably be the City of London which is densely populated during the day and has high cycling activity (couriers etc.).
The first thing I had to do, I discovered, was apply for an API key giving me permission to scrape the data. Then I had to write the code, which I based on code we had seen during the course (click on the images to see them bigger and better).
This code creates the database, connects to the TfL API, asks for the year to download and inserts the longitude, latitude, severity and victim (cycle, car, motorcycle, etc.) into the database for that year. Then it saves the data and closes the connection.
It took me a few days because I wasn't sure about exactly what I needed to do - did I need to create a dictionary, or two, or none...? That's half the problem actually, for me - identifying the structure of the code you need to write for the job you want to do.
My code is really simple too. It just asks for one year, not multiple years. It assumes there are no errors in the year entered (e.g. 2016 which is not available yet). I could make it more robust, but to start with, I just wanted to make it work!
This is what the data looks like in the API:
This is what it looks like 'pretty printed':
You can see more clearly the information I wanted to download in the 'pretty printed' format.
This is what the database my code created looks like, it has 23116 rows of data:
I was astounded the first time the code worked and saw the database loaded with data. Someone I spoke to recently called the feeling a 'nerdy moment'. Never thought I'd ever have one of those, I must say!
Having got the data, I then had to write some code to select the cyclist accident set, choosing 'severe' accidents rather than fatal ones (too sad), iterating through the data, and writing the longitude and latitude locations only to a javascript file.
I was thankful to have some scaffolding to help me write that too!
Once I had the geolocation data, I then had to visualise it. I had already used some visualisation code earlier in the MOOC, so just had adapt it to visualise my data. It actually took me three days because I ran into a problem and had no idea what to do. The code was written in html, which I know nothing about. I hunted around for a solution on the internet, including that fabulous resource Stackoverflow, but couldn't find an answer.
I was stumped. Then I moaned to my DB about my problem, and he said that I should check the latitude and longitude coordinates because they might not be in the right format. And he was right! They were back-to-front in my code! Once I'd fixed that, up popped the little red labels as they should (with another nerdy moment).
These maps show the severe accidents for 2008. There were 429, and you can see from the second map that the highest concentration was indeed the City of London.
Job done!
It's been a really satisfying few months, going from being a complete Python beginner/never having touched coding before, ever, and having been crap at maths, to producing an amazing, functioning final result that I had to understand to make work (more-or-less, let's just ignore the html...). Dr Chuck was an entertaining teacher who could engage with us across a screen (no mean feat), and who even set up live 'office hours' during the Capstone so we could interact with him directly. He was aided and abetted by a team of kindly mentors who were available to help us out and give advice in the forums.
I am very happy with the results, and aim to go on and tackle C# next!
The last module in the 5-part course was the Capstone where we had the opportunity to do an optional project. Always game to get the most out of things, I decided to take the bull by the horns and sign up.
For the project, we had to find a data set, 'scrape' it to find some specific information, put that into a database, and finally, visualise the results. There are data sets about many different fascinating subjects such as:
- the last words of inmates in Texas before execution since 1984
- the "Million Base" of 2.2 million chess matches
- a Twitter data set
- World Health Organisation data set
- Family food data set
- Million Song data set
and so on. For my project however, I chose the Transport for London data set available in their Application Programming Interface (API). It provides access to real time data on the most highly requested information across all modes of transport. It also provides data on accidents across London. I wanted to find out about bicycle accidents (just because), and discover where most accidents happen. I thought it would probably be the City of London which is densely populated during the day and has high cycling activity (couriers etc.).
The first thing I had to do, I discovered, was apply for an API key giving me permission to scrape the data. Then I had to write the code, which I based on code we had seen during the course (click on the images to see them bigger and better).
This code creates the database, connects to the TfL API, asks for the year to download and inserts the longitude, latitude, severity and victim (cycle, car, motorcycle, etc.) into the database for that year. Then it saves the data and closes the connection.
It took me a few days because I wasn't sure about exactly what I needed to do - did I need to create a dictionary, or two, or none...? That's half the problem actually, for me - identifying the structure of the code you need to write for the job you want to do.
My code is really simple too. It just asks for one year, not multiple years. It assumes there are no errors in the year entered (e.g. 2016 which is not available yet). I could make it more robust, but to start with, I just wanted to make it work!
This is what the data looks like in the API:
 |
| TfL API raw data |
This is what it looks like 'pretty printed':
 |
| TfL API data in readable format |
This is what the database my code created looks like, it has 23116 rows of data:
 |
| Database |
Having got the data, I then had to write some code to select the cyclist accident set, choosing 'severe' accidents rather than fatal ones (too sad), iterating through the data, and writing the longitude and latitude locations only to a javascript file.
 |
| Code to select geolocation data |
 |
| The geolocations of severe cycle accidents in London |
Once I had the geolocation data, I then had to visualise it. I had already used some visualisation code earlier in the MOOC, so just had adapt it to visualise my data. It actually took me three days because I ran into a problem and had no idea what to do. The code was written in html, which I know nothing about. I hunted around for a solution on the internet, including that fabulous resource Stackoverflow, but couldn't find an answer.
I was stumped. Then I moaned to my DB about my problem, and he said that I should check the latitude and longitude coordinates because they might not be in the right format. And he was right! They were back-to-front in my code! Once I'd fixed that, up popped the little red labels as they should (with another nerdy moment).
 |
| Great London severe accident sites |
 |
| Central London severe accident sites |
It's been a really satisfying few months, going from being a complete Python beginner/never having touched coding before, ever, and having been crap at maths, to producing an amazing, functioning final result that I had to understand to make work (more-or-less, let's just ignore the html...). Dr Chuck was an entertaining teacher who could engage with us across a screen (no mean feat), and who even set up live 'office hours' during the Capstone so we could interact with him directly. He was aided and abetted by a team of kindly mentors who were available to help us out and give advice in the forums.
I am very happy with the results, and aim to go on and tackle C# next!
Sunday, January 17, 2016
Pumpkins etc.
Until this week, I hadn't been ill for over a year, which I put down to the wonders of propolis and being tanked up with vitamin D. This year's batch of rhinoviruses are a tough bunch, it seems, and got through the armour plating of my defences. I've been off work, but it could have been worse - I could have gone temporarily deaf in one ear, like my DB.
By far the most comforting nourishment while feeling awful is soup. For Christmas, I was given a 10kg pumpkin, a massive thing. It sat on the floor in the hall until I realised a couple of weeks ago that it was starting to go off, whereupon I had to do something with it immediately, or consign it to the compost bin.
This happened on a Sunday afternoon, the rest of which was thus spent peeling, de-seeding and cutting up. I put two oven-tray fulls of pumpkin chunks in the oven to roast and freeze, put another bag-full in the fridge, and made pumpkin soup out of the rest. I'm a big fan of potimarron soup, and butternut squash soup, but I find pumpkin soup a bit bland. My first attempt just had pumpkin, onion and potato and chicken stock. That's my classic recipe for potimarron and butternut, or a mixture of both. It's not enough for pumpkin unless you have a super rich stock, which I didn't. The resulting soup was disappointing to say the least.
For my second try, I took the bag of chunks out of the fridge this week (just before they went off), and added carrot, turnip and a parsnip to the pot. They made all the difference, and I ended up with a super-tasty, hearty soup which has kept me going at lunch times this week.
I still have two bags of frozen roasted pumpkin in the freezer, pumpkin that is responsible for the demise of my oven. At least, while cooking it, and indeed, once the pumpkin was cooked, my oven started making odd noises and at some point gave up the ghost. It was thirteen years old and much used so I can't complain. It didn't die during cooking Christmas lunch, or an important dinner party (not that I have any of those), or even half-way through a roast chicken. No, it was faithful to the last, and waited until the pumpkin was nicely roasted before expiring. Thank you dear oven.
I am replacing it with a French oven, a Sauter (let's jump!) which is the equivalent of my Sholtès and the winner of a Que Choisir ('Which') comparison of ovens. I could have replaced it with another Sholtès, but I read a review from someone who had done just that, and while they were happy with it, they remarked that the components have been made with shoddier quality materials than the old one. A sign of the times, obviously, so I decided to support French manufacturing and save 150Eur or so.
What else? Well, I've been making good use of Hotspot Shield by watching BBC iPlayer - 'War and Peace', 'Dickensian', 'Death in Paradise', 'Silent Witness', and also 'Endeavour' on ITV player. Sitting at the dining table (which is not used for dining) in my cosy dressing gown, in front of the computer with the earphones on, a cup of tea and a slice of Christmas cake to hand, and watching tele is a great way to be ill.
I've also completed the second course of Python and signed up for the third which is starting this week. As I fear I might not be up to the task of all this learning, I am doing, alongside, the CodeAcademy course on Python to give me another point of view, and more exercises. I have to keep at it as there's nothing easier to forget. As with all languages, the most difficult part is thinking in the right way. I can understand what's going on when I see it, but actually writing it down from scratch is much trickier. Exactly like people who say they can understand but not speak a language.
I will get there in the end. I will.
By far the most comforting nourishment while feeling awful is soup. For Christmas, I was given a 10kg pumpkin, a massive thing. It sat on the floor in the hall until I realised a couple of weeks ago that it was starting to go off, whereupon I had to do something with it immediately, or consign it to the compost bin.
 |
| My 10kg pumpkin looked like this (from here) |
For my second try, I took the bag of chunks out of the fridge this week (just before they went off), and added carrot, turnip and a parsnip to the pot. They made all the difference, and I ended up with a super-tasty, hearty soup which has kept me going at lunch times this week.
I still have two bags of frozen roasted pumpkin in the freezer, pumpkin that is responsible for the demise of my oven. At least, while cooking it, and indeed, once the pumpkin was cooked, my oven started making odd noises and at some point gave up the ghost. It was thirteen years old and much used so I can't complain. It didn't die during cooking Christmas lunch, or an important dinner party (not that I have any of those), or even half-way through a roast chicken. No, it was faithful to the last, and waited until the pumpkin was nicely roasted before expiring. Thank you dear oven.
I am replacing it with a French oven, a Sauter (let's jump!) which is the equivalent of my Sholtès and the winner of a Que Choisir ('Which') comparison of ovens. I could have replaced it with another Sholtès, but I read a review from someone who had done just that, and while they were happy with it, they remarked that the components have been made with shoddier quality materials than the old one. A sign of the times, obviously, so I decided to support French manufacturing and save 150Eur or so.
What else? Well, I've been making good use of Hotspot Shield by watching BBC iPlayer - 'War and Peace', 'Dickensian', 'Death in Paradise', 'Silent Witness', and also 'Endeavour' on ITV player. Sitting at the dining table (which is not used for dining) in my cosy dressing gown, in front of the computer with the earphones on, a cup of tea and a slice of Christmas cake to hand, and watching tele is a great way to be ill.
I've also completed the second course of Python and signed up for the third which is starting this week. As I fear I might not be up to the task of all this learning, I am doing, alongside, the CodeAcademy course on Python to give me another point of view, and more exercises. I have to keep at it as there's nothing easier to forget. As with all languages, the most difficult part is thinking in the right way. I can understand what's going on when I see it, but actually writing it down from scratch is much trickier. Exactly like people who say they can understand but not speak a language.
I will get there in the end. I will.
Thursday, December 17, 2015
Catch-up
At the end of my last post, I mentioned that I was studying Python, a computer language. I've never done anything like it before (although now I come to think of it, learning Arabic was quite as tortuous), so it's tough pounding it into my brain (which is obviously full of incredibly useless useful other stuff...).
Still, headway is being made; I finished the first course, Python for Everybody (even me) - and I'm now on the second of the five-part course - Data. Helping me learn is the Anki app, in their own words:
Why Python? Why learn to code? Well, you never know what the future holds in store, and my current job could disappear, and as a developer, you can work anywhere in the world. Particularly in France where, when you hit 50, you're considered dead meat, unless you can code. Java programmers are never out of work apparently. I plan to go onto Java once I've got the hang of the more straightforward language Python.
Also, I got a promotional email from Coursera in which a woman who had been in admin retrained as a programmer starting with the Python for Everybody course. If she can do it...
Let's just say that it's a challenge, and definitely keeping me on my toes!
I've been on my toes in more ways than one. Remember I mentioned that I was walking regularly during the week using the WalkActive technique, or as near as one can without going through the training programme? Well, it's taken inches off my waist and I can now get into trousers I was on the point of throwing out. My DB says it's very noticeable, so well done Joanna Hall (the founder).
I could definitely do with walking off work's Christmas lunch! I'm stuffed!! We had it at work and everyone brought something so there were lots of good things to eat including: boeuf bourguignon, lobster bisque tart, quiche, salads, charcuterie, luxury butter, fabulous cheeses, chocolate moelleux cake, chestnut log, buche, clementines, lichis and so on. Ouf.
We are all at work in the family this week. My youngest is doing his work experience at the local opticians' who are parents of a best buddy and very welcoming so he's having a lovely time. My eldest dropped law as being way too boring and with people he didn't get on with, and is now doing a sandwich course in marketing, and working at a local dynamic start-up with people he does appreciate and having a fab time. He's putting into practice all the theory and ideas he learns at college and at his age, having a job is a miracle! France is a rubbish country for jobs for the under 25s and over 50s and the unemployment statistics bear witness to my damning judgement. He got the job by networking - the only way these days, it seems - the CEO is the mother of a good buddy who he's known for years. She told him that he might not have the technical skills yet, but he has the human qualities she wants for her team. Was I proud?
So as the holiday season kicks off, I hope you all have a very merry Christmas, or at least the Christmas which suits you.
And if you need to send some Christmas ecards that are not cheesy, whimsical or nauseating, (and free), I can recommend jimpix.co.uk for cards such as this one:
Still, headway is being made; I finished the first course, Python for Everybody (even me) - and I'm now on the second of the five-part course - Data. Helping me learn is the Anki app, in their own words:
It's also open source and free. And it works.Anki is a program which makes remembering things easy. Because it's a lot more efficient than traditional study methods, you can either greatly decrease your time spent studying, or greatly increase the amount you learn.Anyone who needs to remember things in their daily life can benefit from Anki. Since it is content-agnostic and supports images, audio, videos and scientific markup (via LaTeX), the possibilities are endless.
For example:
- Learning a language
- Studying for medical and law exams
- Memorizing people's names and faces
- Brushing up on geography
- Mastering long poems
- Even practising guitar chords!
Why Python? Why learn to code? Well, you never know what the future holds in store, and my current job could disappear, and as a developer, you can work anywhere in the world. Particularly in France where, when you hit 50, you're considered dead meat, unless you can code. Java programmers are never out of work apparently. I plan to go onto Java once I've got the hang of the more straightforward language Python.
Also, I got a promotional email from Coursera in which a woman who had been in admin retrained as a programmer starting with the Python for Everybody course. If she can do it...
Let's just say that it's a challenge, and definitely keeping me on my toes!
I've been on my toes in more ways than one. Remember I mentioned that I was walking regularly during the week using the WalkActive technique, or as near as one can without going through the training programme? Well, it's taken inches off my waist and I can now get into trousers I was on the point of throwing out. My DB says it's very noticeable, so well done Joanna Hall (the founder).
I could definitely do with walking off work's Christmas lunch! I'm stuffed!! We had it at work and everyone brought something so there were lots of good things to eat including: boeuf bourguignon, lobster bisque tart, quiche, salads, charcuterie, luxury butter, fabulous cheeses, chocolate moelleux cake, chestnut log, buche, clementines, lichis and so on. Ouf.
We are all at work in the family this week. My youngest is doing his work experience at the local opticians' who are parents of a best buddy and very welcoming so he's having a lovely time. My eldest dropped law as being way too boring and with people he didn't get on with, and is now doing a sandwich course in marketing, and working at a local dynamic start-up with people he does appreciate and having a fab time. He's putting into practice all the theory and ideas he learns at college and at his age, having a job is a miracle! France is a rubbish country for jobs for the under 25s and over 50s and the unemployment statistics bear witness to my damning judgement. He got the job by networking - the only way these days, it seems - the CEO is the mother of a good buddy who he's known for years. She told him that he might not have the technical skills yet, but he has the human qualities she wants for her team. Was I proud?
So as the holiday season kicks off, I hope you all have a very merry Christmas, or at least the Christmas which suits you.
And if you need to send some Christmas ecards that are not cheesy, whimsical or nauseating, (and free), I can recommend jimpix.co.uk for cards such as this one:
 |
| jimpix.co.uk *Merry Christmas* |
Subscribe to:
Posts (Atom)