Many of the components are at an advanced stage of preparation however there is still significant work to be done to integrate them. There is a particular risk that the ooniprobe API won't be built in time for us to use in the first iteration of our system. We can work around this by using some scripts in the short term however work on this has not yet started. Another risk is that the anroid probe is not currently in active development yet it needs significant updates to work with the new API. Test broadband lines will become available in a couple of weeks time and this will let us compile some initial reports from general URL lists and manually-crafted lists of user-submitted URLs.

Funding and sponsorship

Andrews & Arnold

Andrews & Arnold (AAISP) are a UK ISP providing top quality leading edge telecommunications solutions for technical and professional customers. They are corporate members of the Open Rights Group. They only provide unfiltered internet connections. More information is available on their website.

AAISP are sponsoring this project with hardware, network infrastrucutre and the technical expertise of their staff. They have provided us with access to half a dozen home-broadband connections across all major UK ISPs so that we can test whether URLs can be reached on each network. They have also provided physical space in their offices for the termination of these connections, the networking infrastructure necessary to monitor and interconnect the lines, and server hardware on which our probes will run. This infrastructure investment has been commissioned, and is managed, by AAISP staff.

Bytemark hosting

Bytemark started in 2001 as a consulting firm based in York, UK. Its directors are experienced software and network engineers Matthew Bloch and Peter Taphouse. They combined their experience to build the UK’s 'nerd hosting outfit of choice'. They are long-term supporters of the Open Rights Group. More information about the company can be found on its website.

Bytemark have provided the project with two Virtual Private Servers (VPSs) on which we are currently hosting our middleware API endpoints, databases, staging server for the website, the team's mumble server and a few other development tools.

Donations and membership

In January 2014 a generous donor offered ORG £3000 in matched funding to hire a project manager to support ths project. We held a funding drive and raised a total of £3354 in donations and new membership subscriptions over a three-week period, more than matching our target, and releasing the full £3000 donation from our benefactor. Richard King (Rk) was hired to work one day per week as project manager in February 2014. By virtue of the successful funding drive his contract was extended to two days per week in April 2014.

Advocacy and publicity

In March Richard King spoke about the project at ORG local groups in Edinburgh, Sheffield and Manchester. ORG has also sent a couple of emails out to its general supporters mailing-list asking for donations and telling people how to get involved.

Richard has approached the podcast Linux Outlaws to ask for a slot to discuss the project. Arrangements for this are being finalised.

Richard has suggested running a series of interviews with project contributors to showcase their work, highlight their individual reasons for joining the project, and publicise how others can get involved. Work on this has not yet started.

We have yet to engage with politicians, companies or the media on this project specifically, however ORG is running a wider campaign against web blocking and has commented in the press as well as publishing blog posts. Given the journalistic interest in O2's URL checking tool in December last year (before it was pulled) we anticipate our own tool being a rich source of stories in the technical press. Richard will work with ORG's communications director to maximise the potential publicity for the project.

Documentation

The project on-ramp starts on the blocked.org.uk help page, which explains what we're trying to achieve, where we are currently and how people can get involved.

Piecemeal project and technical documentation is available on this wiki (see the Censorship Monitoring Project category).

User documentation has not yet been started.

There are no language translation plans yet.

Infrastructure

In March we received the following update from Alex Bloor, Business Development Manager at Andrews & Arnold, on the fixed-line ISP infrastructure part of the project:

For a little while we have had a new HP Microserver ready for this project. We've also installed a telecoms cabinet on a wall in a spare office ready to take the routers sent by the various broadband providers. I am pleased to say we have now placed orders with five ISPs, those being:

Sky

Talktalk

Plusnet

BT

VirginMedia

From each ISP we've now ordered a phoneline and broadband package. We also intend to install an A&A Home::1 ADSL line and phone line as a sixth connection for comparison purposes.
All ISPs were quoting the same week for installs; week commencing 7th April; presumably the slightly long leadtime is due to the PSTN line requirement. The upshot is that we have staggered these installs across the week - Monday 7th = Sky, Tuesday 8th = TalkTalk, Wednesday 9th = Plusnet, Thursday 10th = BT, Friday 11th = VirginMedia. I am anticipating the slight possibility of significant deja-vu on the part of a BT Openreach engineer!
There's a little more preparation we can do before they arrive, and then quite a bit more once they have done the installs and we have service on all of them.

We have not yet started discussing how or where to build the mobile-line ISP infrastructure part of the project. Richard is negotiating with a possible sponsor for this but this is at an early stage.

The Bytemark virtual private servers are commissioned and working.

Software development

Probes

Android Probe (Censor Census / Bowdlerize)

Bowdlerize is the code name for an Android app-based probe, also known as Censor Census. URLs are sent to Android devices with the app installed. Upon receipt of a probe payload the app checks the URL via a HTTP HEAD request to test if it is censored or not. The results of these tests are then reported back to our database.

Development of this probe has been stalled for the last two months and its author has not been communicating with the project. We need to attract other contributors to help by continuing their work.

The android probe uses the old v1.1 API so it needs upgrading to the latest 1.2 version.

The format in which this probe sumbmits its reports is currently undocumented. We need to discuss adapting its reports to conform to the OONI specification to make them comparable with those generated by ooniprobes.

OONI Probe (Lepidopter)

The Open Observatory of Network Interference (OONI) is part of the TOR project. ooniprobe is tool for performing internet censorship measurements. It has a corresponding back-end called oonib.

Lepidopter is the codename for a Debian wheezy image for the Raspberry Pi generated using a set of Spindle scripts. The aim is to produce reproducible, unbooted, clean setups that require no manual intervention. The scripts will eventually be able to build a stand-alone, bootable ooniprobe instance, which can join our network of probes and contribute test results without local intervention. Just flash the image onto an SD card, plug it into the pi, switch it on and that's it!

Development has been stalled for the last few months waiting for the middleware API to mature. Some testing has been done on the image as-is, launching ooniprobe manually over SSH and feeding it URL lists manually, however we have not yet had access to any networks where web-filtering is active so have not detected any censorship. When the AAISP broadband connections come online we will be able to run some initial tests using the Alexis list of the top 1^6 websites and then generate some initial reports for the project.

The scripts we have currently can:

setup_spindle_environment: Sets up a chroot using wheezy and installs the pre-requisites needed for spindle
wheezy-stage0: Create and partition an SD card image, perform the initial debootstrap on the host and copy the files to the SD image.
wheezy-stage1: Complete second stage of debootstrap under QEMU after first setting up a squashfs filesystem derived from Aboriginal Linux. Setup dropbear.
wheezy-stage2-lepidopter: Add in Raspberry Pi 'firmware' and do misc config (e.g. fstab, network interfaces, hostname). The resulting image is bootable.
wheezy-stage3-lepidopter: Install and configure a few useful packages (such as ifplugd, sudo).
wheezy-stage4-lepidopter: Install ooni-probe and needed dependencies.

The image still needs:

A way to register itself with the middleware, including passing the probe's unique .onion address to the middleware so that it can receive commands, and to receive an API key in return.
A way to receive URLs to test.

This second point needs some work. At the moment ooniprobe does not have a functioning API. Instead it works on URLs passed to it from the command line or via a text file. Neither of those methods is ideal for our system because we want to be able to send probes new URLs whenever they are submitted to us by blocked.org.uk users. In the short term we can build some scripts that will download updated text-files of URLs and then launch ooniprobe to run the tests and report the results. In the medium-term we want to implement the ooniprobe API spec upstream, adapt our own API in line with this spec, and then use this as a dynamic mechanism of controlling probes.

Middleware

Version 1.1 was the initial implementation of the middleware API and is hosted at api.bowdlerize.co.uk. This version is considered deprecated.

Version 1.2 of the API is in active development. The current github code for the 1.2 API is now live on the dev-censor-1 ORG server and the database has also been updated. There are still a few rough edges to fix. The API is reachable on [1] (although the SSL cert is for api.bowdlerize.co.uk, so it might be better to use that address for now if your client software is strict about certificate validation). A java library for version 1.2 is also in active development.

Currently, only the Android probe is using the middleware as a source of URLs to test, and this uses the legacy version 1.1. The origins of the actual list of URLs are obscure and undocumented. In the short term we need to make sure the list is being populated by submissions from the blocked.org.uk front-end.

The OONIProbe does not yet have an API feature so we will need to build one for it.

The draft API specification for ooniprobe, which is an upstream project of ours, works "the other way round" to how our Android probe communicates with our current API: ooniprobe expects to act as the API server, with the oonib back-end client pushing URLs to it together with "start test" and "stop test" commands, whereas our back-end currently acts as a server with the Android probes sending periodic pull requests for new URLs to test. We will need to rationalise these approaches. Since ooniprobe is an upstream project we should really use their model for the API. This may mean doing some re-work on our own API and the Android client.

We need to decide whether to maintain our own, separate database of results, and then transform these wholesale into submissions to the oonib database; or produce reports directly in an oonib-compatible format and use oonib exclusively for results.

Website

The new website front-end design is in active development. Template proofs have been published on github and some comments have been received.

A staging server is available, hosted on one of the bytemark servers, and an instance of the MODX CMS has been installed there. The web designers are currently implementing the template proofs in MODX.

We still need to migrate page content from the old site or write new content. We also need JavaScript development in two areas: communication with the API (to queue submitted URLs for testing and to retrieve and display results); and implementing the form hiding/revealing features (so that filling in the first bit of the form reveals the next part).

Testing and QA

No organised bug reporting, triage or resolution process has been implemented yet. Contributors are encouraged to raise issues on github for bugs, feature requests and other suggestions.

Interpretation of results

We have not yet started considering how we might interpret results.

Censorship Monitoring Project Status

Contents