Censorship Monitoring Project Architecture

Overview

(moved from "functional spec" page by Richard (talk) 09:34, 5 January 2014 (GMT))

i) A front end that can get user input and feedback, report on what's blocked and (perhaps) communicate with network operators (this could be based on the MySociety framework) - All RIPE LIRs should have had added an abuse-c attribute by now which could be used for reporting.

ii) Rasberry Pis & Android devices to act as probes

iii) Some middleware to collect 'tasks' from the front end system, allow them to be picked up by the probes and to collect the data back from the probes. (OONIB etc)

iv) A database for storing users, probes and URLs to be tested

Hardware

Probes

Based on RPi


  • Memory card class (speed), size, price
    • Size depends on the amount of data we want to keep before submission to "backend".

Android Phone + App

Web / database infrastructure

  • Web server
    • SSL + IP for blocked.org - used by Users / ISPs and Administrators
    • SSL + IP for api.blocked.org (suggested) - used by probes and power users
      • Separate hostnames makes it easier to split the roles described in the functional spec to different servers / load balancer pools.
  • URL Crawlers
    • For processing received URLs from a clean (and non TOR) connection
    • Technically a type of probe but can be part of the process of receiving a URL to test
  • Database
    • Middleware / API
    • Blocked.org
      • Schema Design

Software

Probes

RPi

  • RPi boot image
    • Providing an image ready to roll RPi OS and ooni.
  • OS updates (security updates)
    • Since RPi is going to be 24/7 online it's quite important to assure that at least the security updates being performed.
    • However it could cause problem with ooni version specific packages.
  • Network configuration
    • Automatically recognize the mobile network being used and configure accordingly.
    • Has general internet access (via ethernet or wifi).

ooni probe

  • Installation: packages and/or install script
    • People should be able to install it themselves (automatic or manual).
    • Building by users adds transparency to the whole project.
  • Upgrade: would be nice to ensure that upgrade (auto, manual or both) is possible.
  • Rulesets, tests, decks
    • Can routinely connect to a queue service on the internet, pull done URLs to check, and send back structured data in response.
    • Support deployment of "Ooni Probe" tests.
    • Has some understanding of the various styles of blocking responses, and can extract metadata regarding the block.

Android Device

See Android_app for a breakdown / specification of an operational URL probe app.

ooni backend / middleware

OONI Backend

  • API: ooni-backend provides an API that allows applications to interact with ooni-probe

Middleware

  • See Database for DB schema
  • See API for middleware API (Probe / user registration, URL dispatch etc) discussion / specification

User Registration Process

  • A user signs up with an email address and a password
    • Receives a private key (used for signing requests to the API)
  • Account is in pending stage
  • ORG confirms account
  • User receives email address notifying they can activate probes

User Signup Process.jpg

Probe Registration

  • A user hits /prepare/probe to receive a one key used to seed the probe UUID
    • Each call generates a new seed
  • The probe generates a UUID
    • A combination of the seed and something unique to the device (e.g. MAC address)
  • The user POSTs relevant info to /register/probe signed with the users private key
    • Receives back a private key which is for use by the probe for probe related activities e.g. updating GCM info, requesting tests or submitting results
  • The probe information is recorded against the user

Probe Registration.jpg

URL Submission

  • User on Blocked.org / a probe with a UI submits a URL
    • DB checked for duplication
      • If duplicated possibly reschedule on assumption user is inquisitive and URL may have been blocked/unblocked since last checks
      • Or Reject
    • URL inserted with status of "pending"
    • Return submission status to user
  • Minutely cron kicks off to find all URLs that are pending
    • Check URL is resolveable
    • HEAD / GET URL
      • Confirm headers / content length etc match
    • Update row with status of ready and headers / return code etc

Main site - blocked.org.uk

See draft functional specification / collection of user stories.

Notes on where to go with blocked.org.uk from the 18.01.14 meeting.

Research

  • Collate results in repository
 o Help other ORG activists to build their own probes and tools for testing, submitting URLs to us.
  • Threats
 o Threats: Hypothetical security risks for Ooni's associated Roles.

Other ideas / discussions

Collaboration with the RIPE NCC Atlas Program

With your help, the RIPE NCC is building the largest Internet measurement network ever made. RIPE Atlas employs a global network of probes that measure Internet connectivity and reachability, providing an unprecedented understanding of the state of the Internet in real time.

What is RIPE Atlas

It is the next generation active Internet measurement system from the RIPE NCC. It is currently in the prototype stage. It is intended to scale up to many thousands of measurement probes distributed around the globe. You can read much more about all aspects of RIPE Atlas on RIPE Labs.

How can it help?

If ORG became a sponsor ( https://atlas.ripe.net/get-involved/become-a-sponsor/ ) then it would be able to schedule custom jobs (possibly just Layer 3 but hopefully full stack tests) from hundreds if not thousands of unfiltered datacenter grade probes.

ORG currently hosts a probe (14089) from its office in Borough.

Further Reading

https://atlas.ripe.net/

Mobile application

Fixmyinternet

Can we use the Mysociety system used for Fixmystret as a jump start?

Web browser extension

  • Chrome plugin
    • Act as a probe via Chrome Apps
    • Extension to flag when a URL hasn't loaded due to filtering / blocking

Abbreviations

  • RPi: Raspberry Pi
  • OS: Operating System
Retrieved from "https://wiki.openrightsgroup.org/w/index.php?title=Censorship_Monitoring_Project_Architecture&oldid=42607"