Reading "Forbidden" Web Pages with C#

In my session at Techdays last week I was showing how to make an asynchronous method that reads a web page. I was bemoaning the way that my method worked for www.bbc.co.uk but not for the wonderful site that is www.robmiles.com. When I try to read "my" page the request fails with a "403 Forbidden error".

Well, many thanks to Erik van Telgen, René Vermijs and @HenroOnline who all came back with the answer. Turns out that, unlike the liberal BBC, my blog host insists that that any read requests come from browsers, not programs. Fortunately it is easy to modify the web request to appear to be from a browser, and so get the Html back.

public async Task<string> GetPageAsStringAsync(string url)
{
    HttpClient x = new HttpClient();
    x.DefaultRequestHeaders.Add("user-agent", 
        "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)");
    HttpResponseMessage response = await x.GetAsync(url);
    string content = await response.Content.ReadAsStringAsync();
    return content;
}

This version of "GetPageAsStringAsync" returns the contents of a page and impersonates a browser when it does it.

string site = await GetPageAsString(@"http://www.robmiles.com");

It's very easy to use, as you can see above. And this works perfectly. Many thanks for your help folks.

I really hope that I get invited back to Techdays next year, I'm sure by then I'll have another bunch of technical problems I can get help with...

Eye Test

Coming soon to a face near me...

So, I went for an eye test yesterday. Cue all the eye test jokes.

"Tell me sir, have your eyes ever been checked?"
"No, they've always been blue."

"Can you see that eye-test on the wall?"
"What wall?"

etc etc

Anyhoo, I was pleased to find out that my eyes haven't changed that much since last time. I've had my present specs for quite a few years. A year and a half ago a bit fell off and I made a "temporary" repair using heat shrink tubing that has been a bit less temporary than I intended.

This time I've just gone for a single pair of glasses. No messing around with "two for one" deals. Experiments have shown that I can't wear two pairs of glasses at the same time, and that I'll probably lose the second pair anyway. They should arrive in a couple of weeks. Rather excited.

Plane Rainbow

As we were in the shuttle bus out to the plane to fly back last night there was the most amazing rainbow over the airport. When we got off to head into the plane I threw all my bags down on the concrete, grabbed the camera and took a few shots.  I'm quite pleased with the composition, although it really had nothing at all to do with me, and everything to do with dumb luck.

TechDays Sessions Fun

First Great Audience for the Universal Application Session

I don't know what it is about audiences from the Netherlands. But they are around the best in the world. Up for anything, attentive and just generally fantastic.

Await and Async audience

Both sessions seemed to go really well (at least most of the jokes got appropriate laughs or groans). I've passed the slides and demo programs over to the conference folks and they should be available soon. I'll post them here on the blog (with a bit of extra howto stuff) next week as well.

Thanks to the Techdays crew for inviting me. I was in the Hague for just about 24 hours, but they were lovely...

Flying to TechDays

Flew out to Techdays today. The weather was nice and now I'm allowed to use my phone all the way through the flight I did just that. This was the view from the plane as we flew over the UK coastline.

My sessions are tomorrow, but tonight we had a dinner in the Speaker's Lounge which was great. I was able to catch up with lots of folks I've not seen for a while. Good to see you all again.

The bags the are giving the delegates this year are really nice, and the speakers get one each too. And the speakers lounge has some very groovy furniture.

Mad Max: Fury Road

Over last weekend we went to see Mad Max: Fury Road.  Wow.

It is as if the director had watched every Michael Bay Transformers movie and then decided that he could go to around 11 or 12 on that scale. Or more. The action is relentless, violent and in some parts a bit queasy making (at least for wimpy old me).

Tom Hardy does a great job as Max. Not that he has to say much. I think they gave him his script written on the back of a cigarette packet. Charlize Theron is great in the film as well. She has lines and everything. All the souls in the movie are tortured, torturing or both.

If you like your action fast moving and with lots of spectacle then you'll love it. If you prefer films where the most action packed thing that happens is that Lady Faversham comes to visit and they can't find the best tablecloth, then you might want to give it a miss.

TechDays Writing

Spent a big chunk of today writing content for TechDays later this week. I'm giving two sessions. Rather looking forward to it.

The over eager dog that is Lightroom

Some software drives me nuts. The kind I'm thinking about is the stuff that is useful enough to be indispensable, but actually really annoying to use sometimes.

Such as Adobe Lightroom. Lightroom is a fantastically useful program for managing your photos. If you are a serious photographer, you use Lightroom. It's that simple. I only started using it a few years ago, but it has transformed the way that I work with images. And just about all for the better.

Unfortunately though, it has a few problems. Sometimes it is a like a big, helpful, over-eager dog that is always trying to do the best for you. And most of the time it succeeds. But sometimes it does insane things.

Take, for example, importing pictures.  This should be easy. Find the folder where the pictures are, and then copy in all the ones that aren't in the catalogue.

Except that whenever you click on (or perhaps even wave the mouse near) a folder at the start of an import Lightroom insists on running off and seeking out all the images in that location. In the entire directory hierarchy. Even though there is absolutely no need to do this.

And it always thinks I want to import from the SD card that I keep plugged into the Surface. So when I start importing it proudly displays a whole bunch of clipart I'm never going to want to look at.

What I'd love is for someone to come back to me and tell me that there are magical options that can be used to turn off this rather daft behaviour. Any ideas folks?

Batmobile

They had this Car Expo in Bristol today. Including a version of I think one of the best Batmobiles.

No Photographs

I carted a camera all around Bristol today without taking a single photograph. Never mind, here's a picture of Hull centre instead...

No more SatNav

We drove down to Bristol today. We've now got a policy of not using the SatNav on journeys like this. 

Experience has shown that all it does is make you unhappy by warning you of traffic disruptions that you can't do anything about. I'd much rather sail along in blissful ignorance of the enormous tailback waiting just around the corner. And the alternative route is quite often a lot less pleasant than sitting in a stationary car for a while.

Flickr Magic View is quite, er, magical

I've been hosting my pictures on Flickr for a long time. When if first came out Flickr was genuinely revolutionary, with unlimited online storage for subscriber's pictures.

Unlimited storage is pretty much the norm these days, and Flickr has been a bit left behind by other, fancier sites. But of late they've done some rather nice things. If you've got a Flickr account (paid or free) you might want to take a look at the new Magic View option in the Camera Roll part of your site. It does a rather good job of working out the subject of your shots and then itemising them.

I'd no idea I'd taken so many pictures of birds.....

Spooks: The Greater Good

Spooks: The Greater Good is a happy tale of fairies and unicorns, living together in "Happy Paradise Land". Everyone is nice to everyone else and there are always lots of cakes for tea.

On the other hand, this might be a lie. Why should you trust me? Why should you trust anyone? After seeing this film I won't be doing much trusting. Everybody is pretty horrible one way or another, but then again the TV show this is based on wasn't big on Rainbows and Cupcakes either.

If you like twisty stories with a bit of action thrown in, or you enjoyed the TV show, then it is well worth a trip to the cinema.

The Ladykillers at Hull Truck

At the weekend we went to see The Ladykillers at Hull Truck. It's a play based on a film. Which was based on a film. Or something. The script is written by Graham Linehan of Father Ted fame.

Is that enough links for you?

Anyhoo, the play was great. We seem to be re-discovering live theatre at the moment, we've been twice in almost as many weeks. The Hull Truck is a great venue and we've discovered that if you sit right at the back, on the high seats, you get loads of legroom of the straight down variety and a commanding view of the stage. The theatre space is very intimate and nobody is very far from the performers. We're scanning the programme for the next thing to go and see.

The Ladykillers, a tale of dishonour amongst thieves and a most unlikely heroine, is a hoot with great performances from all. It's in Hull for the next week or so. Go see it. You'll have fun.

Free T Shirts at Windows 10 Three Thing Game

We've just about finalised the itinerary for the Windows 10 Game Jam/Three Thing Game we are holding next month. If you're a developer in the Humberside region you really should come along. We've got sessions about MonoGame, Unity and Marmalade, a room full of Windows 10 machines for you to play with. And Free T Shirts:

Find out the schedule, and sign up, here.

Thwaite Gardens Open Day 2015

Every year the Friends of Thwaite Gardens open up the gardens for folks to look round. And if I can, I go round and point my camera up a tree or two....

Break into Code with TouchDevelop and the Imagine Cup

One other neat thing that I saw at the summit last week was Minecraft running under the control of a TouchDevelop program. TouchDevelop is a great way to get playing with writing code.

And there's a neat little competition running as part of the Imagine Cup that you might like to have a go at if you are a bit younger than me. Essentially you get to re-invent/re-imagine the classic Breakout game. And there are big money prizes. Find out more here:

https://www.imaginecup.com/breakintocode

Bryan Ferry and Judith Owen Live at Bridlington

Judith Owen

Tonight we headed off to Bridlington Spa to see Bryan Ferry and his support artist. Who turned out to be Judith Owen.  I'd not heard of her before. Great voice, great piano playing and an amazing band. of musicians providing support.

An Amazing Backing band

If you are as old as me (and good luck with that) you'll remember such people as Waddy Wachtel ,Leland Sklar and Russ Kunkel. Pick a hit album from the seventies, eighties, nineties, noughties or now and it's odds-on that at least one of them played on it. To pick just one random example, Waddy Wachtel played on "Lonely Boy" by Andrew Gold, one of the best records ever. Ever.

Just as they finished their set I looked up Judith Owen on Xbox Music and found out another interesting fact. She is married to Harry Shearer, the voice of Mr. Burns in the Simpsons. During the interval I wandered off to find them selling and autographing CDs. I got the whole band and did the whole star struck bit, shaking hands twice and whatnot.

Then I noticed a chap who was helping to take pictures of folks posing with the band. Who looked a lot like Mr. Shearer. And was. I asked him to sign the CD too, although he protested that he had nothing to do with the music. I reassured him that I've been taking credit for things I haven't done for years, and that persuaded him to put pen to paper. I then shook hands with "Mr Burns", complemented him on his wife's singing and her choice in backing musicians, and headed back to my seat. I thought the evening had already peaked.....

But I was wrong. Bryan Ferry came on and did an absolutely storming set. Backed by a fantastic team of musicians he ripped through a choice selection of his back catalogue, including a few Roxy Music numbers and tracks from his new album before bringing the crowd to its feet for the last few songs. Including a blazing version of "Do the Strand".

General consensus at the end of the evening was that "Bryan has still got it going on.".  HIs tour is continuing round the country. If you get the chance to go, just go. Thank me later.

Docker at C4DI

After all of the shenanigans of the last few days I was probably not in the best mental shape to go along to the Developer Meetup at C4DI tonight. But I went along to meet the folks and hear about Docker. Adam Carlile works at Board Intelligence and uses Docker to build highly secure, discrete systems that generate confidential management reports.

Docker is especially useful to Adam because it provides a way of packaging up a particular configuration of machine into a "lump" of data which can then be transported and executed in a secure way on another device. It is essentially a highly portable virtual machine configuration, but with the advantage that the Docker container is very lightweight and can be layered in a way that makes it very easy to create and amend configurations.

If you like playing with operating systems, or have a need to distribute work in a secure and managed way, then Docker is well worth a look.

Devices and Networking Summit Day 2

And so to day two of the summit which features more impressive tech followed by a nerve jangling train ride and late night return to Hull. Now read on....

Keeping Control of Security and Privacy in a World of Devices

In this session we had some talks on different aspects of security. First up was a discussion of techniques to address "man in the middle" attacks, where a bad person interposes themselves between you and the person you are talking to. This technique of intercepting and relaying messages is particularly dangerous in a world where people are happy to use the nearest open WiFi connection with no thought of security. Then we moved on to a fascinating discussion of how we can remove the faces of "innocent bystanders" from photos that we take when we are out and about. This is a surprisingly tricky thing to do and led to a great discourse about how privacy can be managed in the wild.

The Next Big Hurdle

The next session I went to took a high level view of the development process, with the goal of working out how to make properly useful devices, both now and in the future. This is something that we are not terribly good at today, but with increasing levels of complexity and a greater range of uses than ever before, it is important that we make sure that the devices we surround ourselves with aren't more frustrating for us than they are worth.

The talk was placed in the context of producing something really useful, a way of allowing blind people to navigate around cities on their own. The project uses bone-conduction headphones which relay navigation advice generated by a smartphone. The most impressive thing about this for me was the intent to build a system that works even when the user is doing something else. In other words the user would be able to carry on a conversation even while finding their way around.

Home Networking

I wasn't quite sure what to expect from this session, but what we got was some very good discussion of the problems faced when you connect households to the internet. Things have moved on from the early internet where a home had one computer and one wire to the router. Nowadays there will be many devices connected via WiFi and a huge problem diagnosing what has broken when things fail. I saw the results of an interesting study looking for reasons why home networks fail (a hint: it's usually the WiFi connection) and some very interesting ideas about moving the home router into the cloud.

Lunchtime Demonstrations

During the lunch break they had demonstrations of some really neat stuff. I wandered around with my camera and took some pictures.

This is a way of making 3D rapid prototype printing even more rapid. Rather than printing out a filled in model the printer produces a 3D outline. There are a number of technical challenges to address here. The printer needs a larger hole in the nozzle to print thicker strands, the output has to be carefully cooled as the models are built and the slicing software has to be re-written to drive the printer in a completely different way. Very clever.

These are Picco devices. Tiny miniature screens that you can use to send playful, or useful messages to folks you know.

Inside each tiny device is an .NET Microframework powered processor, a WiFi interface, SD card and display unit. The devices and their 3D printed cases were created to explore product development as much as invent new kinds of interaction. Very clever and great fun to play with.

This picture shows some prototype "Disruptive Display" devices. These provide a completely new kind of display medium that "remembers" what light you shine onto it. By using light sensitive transistors in the construction of display drivers you can make a display which can be triggered to retain the level of light that is falling on it at a particular moment. The way I saw this, it meant you could make a wall display that you could "program" using a video projector whenever you felt like changing the scenery. The display doesn't really "remember" anything, each individual pixel retains their setting. You could also use this to "paint" with a torch, and then flick a switch and create a new picture.

There were also folks showing off circuits you can draw, some amazing touch devices on flexible materials, devices that can create radio signals entirely in software, posters you can vote on and a networking system for servers that can reconfigure its topology dynamically  to match the use case. You can find details of all the projects here.

Micro Datacentres and Cloudlets for Mobile Computing

The final presentation of the day was from Victor Bahl, who talked about the way that the cloud is becoming part of the way our machines work. Heavy duty tasks such as face recognition reap a huge benefit from cloud computing. The results are obtained more quickly and place less of a load on the batteries in the mobile device. But of course to use the cloud your device must send the data to be analysed and then get the result back, which depends on fast data transfer.

In many applications it is particularly important to get responses back quickly. Introducing a few milliseconds delay into the loop can make the difference between life and death for a video game player.

Victor talked about the way performance can be improved by streamlining the protocols used to send requests and also how large monolithic data servers were not the complete answer to the problem. Instead he described how smaller numbers of servers, "cloudlets" are now being deployed. These provide high performance local resources and reduce network traffic.

I wasn't able to stay to the end of this session because we had to grab a cab to the station. I had a tight schedule for the journey back. When I organised the tickets I thought that an interval of twenty five minutes would be plenty of time to get from the Eurotunnel train to the Hull one. Of course I was making the assumption that everything would run on time, which turned out to be mostly right. But it did mean for an exciting sprint across a tiny part of London.

Thanks so much to Microsoft Research for Inviting me. It was a great event and it was lovely to catch up with some folks I've not seen in a while.