VarCity: 3D and Semantic Urban Modelling from Images

In this video we see the results of a 5 year VarCity research project at the Computer Vision Lab, ETH Zurich. The aim of the project was to automatically generate 3D city models from photos such as those openly available online via social media.

The VarCity system uses computer algorithms to analyse and stitch together overlapping photographs. Point clouds are then created on the basis of overlapping points and then used to generate a geometric mesh or surface model. Other algorithms are used to identify and tag different types of urban objects like streets, buildings, roofs, windows and doors. These semantic labels can then be used to query the model to automatically determine meaningful information about buildings and streets as the video describes. In this way the VarCity project demonstrates one way in which comprehensive 3D city models could effectively be crowd sourced over time.

It is also interesting that VarCity is using computer vision to connect real-time video feeds or content from social media to actual locations. This is used to determine local vehicle and pedestrian traffic. As the video suggests, there may be limitations to this method for determining urban dynamics across the city as it is dependent of accessibility of a suitably large number of camera feeds. This also has implications for privacy and surveillance. The VarCity team address this by showing representative simulated views that replace actual scenes. As such the 3D modelling of urban regions can no longer be viewed as a neutral and purely technical enterprise.

The wider project covers four main areas of research:

  • Automatic city-scale 3D reconstruction
  • Automatic semantic understanding of the 3D city
  • Automatic analysis of dynamics within the city
  • Automatic multimedia production

A fuller breakdown of the VarCity project can be viewed in the video below.

The work on automatic 3D reconstruction is particularly interesting. A major difficulty with the creation of 3D city models has been the amount of manual effort they require to create and update through traditional 3D modelling workflows. One solution has been to procedurally generate such models using software such as ESRI’s CityEngine. With CityEngine preset rules are used randomly determine the values for parameters like the height of buildings, the pitch of the roofs, the types of walls and doors. This is a great technique for generating fictional cities for movies and video games. However, this has never been fully successful for the modelling of actually existing urban environments. This is because the outputs of procedurally generated models are only as good as the inputs, both the complexity of the rules used for generating the geometry, but also the representational accuracy of things like the models for street furniture and textures for buildings if they are to be applied.

Procedural generation also involves an element of randomness requiring the application of constraints such as the age of buildings in specific areas which determines which types of street furniture and textures should be applied. Newer districts may be more likely to feature concrete and glass whereas much older districts will likely consist of buildings made of brick. The more homogeneous an area is in terms of age and design the more easy it is to procedurally generate, especially if it is laid out in a grid. Even so there is always the need for manual adjustment which takes considerable effort and may involve ground truthing. Using such methods for particularly heterogeneous cities like London are problematic, especially if regular updates are required to capture changes as they occur.

For my own part I’m currently looking at the processing of point cloud information so it will be fascinating to read the VarCity team’s research papers, available here.

Pointerra: Points in the Cloud

Pointerra New York

Pointerra are an Australian geospatial start-up offering point cloud and LiDAR data as a service. Their platform which is deployed on Amazon Web Services enables online visualisation of massive point clouds in 3D via a standard browser.

The U.S. Geological Survey point cloud of New York visualised above has a massive 3.1 billion points. These can be navigated in 3D, viewed with or without a base map, and visualised by intensity, classification or height, as depicted here. Quality settings can be adjusted to speed up render times. Even on the highest setting the point cloud updates in a matter of seconds on our rigP.

Pointerra St Pauls

This second Pointerra example of St Paul’s Cathedral in London is visualised with RGB values. Being able to view point clouds on the web is great. With their plans to be “the Getty Images of 3D data”, as reported by The Australian, it will be interesting to see how the platform develops and what features get added over time. The platform isn’t yet live but you can try it here today.

Drones for Participatory Planning with Flora Roumpani

In this recent video from AMD’s RADEON Creator series fellow CASA PhD candidate Flora Roumpani discusses her involvement in the UCL Development Planning Unit’s ReMap Lima project.

The project sought to map the favelas on the outskirts of Peru’s capital Lima in order to help the communities living there to better understand the planning challenges they face and more effectively participate in the informal local planning processes that tend rely on short term and hoc responses that risk creating new problems for every one solved.

The project involved flying drone’s over Lima and turning the captured data and imagery into digital maps and 3D models that could be used for further analysis and communication. By creating fly-through visualisations and 3D printed models that could be shared with the favela communities Flora helped them to better understand and respond to the problems they were facing.

Read more about Flora’s involvement here or checkout the ReMap Lima project blog here.

Exploring London’s Underworld in VR

In collaboration with academic and urban explorer Bradley Garrett the Guardian have released this new interactive tour virtual reality of London’s sewers. The tour has been specially created for Google Daydream. However, there is also an interactive demo for your desktop web browser. For more information check out the Guardian VR page.

CleanSpace: Mapping Air Pollution in London

phones-two

Today I received a personal air quality sensor, the CleanSpace sensor tag. The device is a carbon monoxide (CO) sensor which is designed to be carried by the user and paired with the CleanSpace Android or iOS app via blueetooth. While the sensor takes readings the app provides real time feedback to the user on local air quality. It also pushes the anonymised sensor readings to a cloud server which aggregates them to create a map of air quality in London.

As well as providing data for analytics the app is intended to encourage behaviour change. It does this by rewarding users with ‘CleanMiles’ for every journey made on foot or by bike. The clean miles can then be exchanged for rewards with CleanSpace partner companies and retailers.

Another interesting aspect of the project is that the sensor tag is powered using Drayson Technologies’ Freevolt. This enables the device to harvest radio frequency (RF) energy from wireless and broadcast networks including 3G, 4G and WiFi. In theory this means that the device can operate continually without needing to have its batteries recharged because it can draw energy directly from its environment. In this way the CleanSpace tag provides a perfect test bed for Drayson’s method of powering Low Energy IoT devices.

The project kicked off with a campaign on Crowdfunder last autumn which raised £103,136 in 28 days. The campaign was initiated shortly after the announcement of results from a study at Kings College which found that nearly 9,500 deaths per year could be attributed to air pollution. Two pollutants in particular were found to be responsible: fine PM2.5 particles in the air from vehicle exhaust along with toxic Nitrogen Dioxide (NO2) gas released through the combustion of diesel fuel on city streets. While the CleanSpace tag does not measure PM2.5 or NO directly it is believed that recorded levels of CO can provide a suitable surrogate for other forms of air pollution given their shared source in car fuel emissions.

While the UK government are under pressure to clean up air pollution from the top-down, Lord Drayson who leads the CleanSpace project argues that there is also need for a complementary response from the bottom up:

“I think the effect of air pollution is still relatively underappreciated and there is work to do in raising awareness of the impact it has.”

“Yes, the government has a role to play, but this isn’t solely a government issue to tackle. The best way to achieve change, and for legislation and regulation to work, is for it to grow from and reflect the beliefs and behaviours of the general public as a whole.”

I’m looking forward to seeing what the device reveals about my own exposure to air pollution on my daily commute. It’ll also be interesting to see how my contribution fits in with the broader map being built up by the CleanSpace user community. After collecting some data I’m keen to compare the apps output with the data collected by the London Air Quality Network based at King’s College.

I’m a card carrying walker. At the same time I’m struck by the paradox that every CleanMile walked or cycled is essentially a dirty mile for the user. I can see the device and app appealing massively to those who already walk and cycle, and want to contribute to raising awareness on the issue of air pollution. However, with the sensor retailing at £49.99 the CleanMile rewards will have to be sufficiently compelling to encourage a wider base of new users participate, especially if the project is expected to have a genuine impact on the way they commute. Of course, it has to start somewhere! It’s an exciting challenge so I’m looking forward to seeing how it goes.

Microsoft HoloLens: Hands On!

It’s taken a while but I finally had my first hands on look at Microsoft HoloLens last night. The demonstration was given as part of the London Unity Usergroup (LUUG) meetup a talk by Jerome Maurey-Delaunay of Neutral Digital about their initial experiences of building demos for the device with Unity. Neutral are a design and software consultancy who have a portfolio of projects including work with cultural institutions such as the Tate and V&A, engineering and aviation firms like Airbus, and architectural firms such as Zaha Hadid architects who they are currently assisting to develop Virtual Reality visualisation workflows.

During the break following the presentation I had may first chance to try the device out for myself.  One of the great features of HoloLens is that it incorporates video capture straight out of the box. Although clips weren’t taken on the night these videos from the Neutral Digital twitter stream provide a good indication of my experience when I tested it:

After using VR headsets like the Oculus Rift and HTC Vive the first thing you notice about the HoloLens is how unencumbered you feel. Where VR headsets enclose the user’s face to block out ambient light and heighten immersion in a virtual environment, the HoloLens is open affording the user unhindered awareness of their surrounding [augmented] environment over which the virtual objects or ‘holograms’ are projected. The second thing you notice is that the HoloLens runs without a tether. Once applications have been transferred to the device it can be unplugged leaving the user free to move about without worrying about tripping up or garroting themselves.

Being able to see my surroundings also meant that I could easily talk face to face with Jerome and see the gestures he wanted me to perform in order operate the device and manipulate the virtual objects it projected. Tapping forefinger and thumb visualised the otherwise invisible virtual mesh that the HoloLens draws as a reference to anchor holograms to the users environment. A projected aircraft could then be walked around and visualised from any angle. Alternatively holding forefinger and thumb while moving my hand would rotate the object in that direction instead.

Don’t be fooled by the simplicity of these demos. The ability of HoloLens to project animated and interactive Holograms that feel anchored to the user’s environment is impressive. I found the headset comfortable and appreciated being able to see my surroundings and interact easily with the people around me. At the same time I wouldn’t say that I felt immersed in the experience in the sense discussed with reference to virtual reality. The ability to interact through natural gestures helped involve my attention in the virtual objects I was seeing, but the actual field of view available for projection is not as wide as the video captures from the device might suggest.

As it stands I wouldn’t mistake Microsoft’s holograms for ‘real’ objects, but then I’m not convinced that this is what we should be aiming for with AR. While one of the prime virtues of virtual reality technologies like Oculus and Vive is their ability to provide a sense of ‘being there’, I see the strength of augmented reality technologies elsewhere in their potential for visualising complex information at the point of engagement, decision or action.

Kind thanks to Neutral Digital for sharing their videos via Twitter. Thanks also to the London Unity Usergroup meetup for arranging the talks and demo.

One Man Game Jam: HTC Vive Basketball

HTC Vive BasketballLast week CASA finally received the HTC Vive. Everyone in the office had great fun exploring Valve’s demo experience The Lab. During the week the Longbow emerged as a particular favourite and caused several of us to discuss which sports might work in VR as viable training simulations. Wanting to get to grips with the HTC Vive hand controllers I decided to take up the challenge by creating a basketball simulation for the Vive in Unity.

I started by downloading a SketchUp model of a basketball court from the 3D warehouse. The model had no walls and a lot of reversed faces so I quickly fixed it up in SketchUp with the aid of the S4U Material plugin, ThomThom’s Material Tools and ThomThom’s excellent CleanUp³ plugin. I also obtained a royalty free basketball model from TurboSquid.

Basketball Court SketchUp

As the Unity importer for SketchUp had failed last time I used it I exported the model from SketchUp in Collada format, and converted it to FBX out of habit using the Autodesk FBX converter. After importing the models into Unity I downloaded the SteamVR plugin and added the CameraRig prefab to my scene to handle the basic Vive interaction.

Basketball Court Unity

Trigger colliders were placed in the basketball hoops with a C# script attached to count the score. The Steam scripts for TestThrow and Teleporter were then added to the hand controllers and modified to enable the player to navigate the entire basketball court and to spawn and throw the ball. The ball physics were handled with a simple Unity physics material which was surprisingly effective.

Using the Vive hand controller works well with two qualifications: Firstly it isn’t possible to apply back spin to the ball; secondly there is a high risk of throwing the hand controller out of the window. Risk of breakage in injury aside the final game is really challenging but great fun. I thought I’d actually got the drop on Basketball games in VR but it looks like HelloVR are adding a basketball experience to their social VR platform Metaworld. Could be fun!