ORG-tech-vols IRC meeting log 2014 05 02

(20:25:04) Guest66498: hi all, hope you dont mind me listening quietly
(20:25:57) graphiclunarkid: Guest66498: Not at all - welcome! In fact we don't mind you participating noisily either ;-)
(20:26:08) Guest66498: :-)
(20:26:25) graphiclunarkid: I guess we'll be kicking off the meeting in about five minutes.
(20:27:30) Guest66498 is now known as stripe
(20:29:46) dantheta [~daniel@dsl-217-155-42-217.zen.co.uk] entered the room.
(20:30:41) graphiclunarkid: Hey dantheta
(20:30:45) dantheta: Hiya!
(20:30:49) graphiclunarkid: How's it going?
(20:30:51) stripe: hi
(20:31:01) dantheta: Yep, pretty good. It's been a good week!
(20:31:06) graphiclunarkid: :)
(20:31:43) graphiclunarkid: We might as well kick off - do you wanna start with your news, dantheta?
(20:32:14) graphiclunarkid: (Quick nod to _guy, hellais, plett, vasilis, xarlekino in case they want to join in)
(20:32:23) dantheta: OK - I've been taking a look around the A&A boxes, doing a little bit of research and config file creation.
(20:32:24) _guy left the room (quit: Read error: Connection reset by peer).
(20:32:38) graphiclunarkid: Oops - guess not! :D
(20:32:40) _guy [~guy@188.246.204.91] entered the room.
(20:32:44) dantheta: I've also been playing with message queues for pushing/receiving results from probes
(20:33:06) dantheta: One of the things that I noticed while I was running the alexa 10k on Vodafone was the setup and teardown of the https connection to the ORG api was quite slow
(20:33:21) dantheta: Running an SSL message queue persistent connection would make these a lot, lot faster
(20:33:32) dantheta: We've also got the 10k vodafone results in the database
(20:34:13) graphiclunarkid: dantheta: I should have some time this weekend to help with general hacking duties if you need some testing doing or anything.
(20:34:34) vasilis: Hi everyone
(20:34:38) graphiclunarkid: Hey vasilis
(20:34:50) dantheta: That's cool - I realized that although we've got API interrogation endpoints which report things on an individual URL, we don't have an "export all" endpoint for fetching data in bulk
(20:35:19) dantheta: I figure that's more useful that trying to design an endpoint for every query type that somebody might be interested in
(20:35:47) dantheta: Although the more useful general-case queries should definitely have API support to support mashups etc
(20:35:54) graphiclunarkid: dantheta: Seems reasonable. We can always add selectors into the endpoint later if people want to refine their queries.
(20:36:14) dantheta: Yep, that's cool.
(20:36:41) graphiclunarkid: I was thinking about building a test harness around the API - and also maybe starting work on a browser plugin...
(20:37:01) graphiclunarkid: (In my "spare time" - lol!)
(20:37:11) dantheta: I have had a mini-client that I've been using for testing while the 1.2 changes were being put in, but it's quite interactive
(20:37:46) graphiclunarkid: I also want to see if we can resurrect the Android client and make it talk to the 1.2 API.
(20:38:30) dantheta: I don't think there's anything that would prevent it - semantically, the 1.2 API is sufficiently similar to its predecessor
(20:38:44) dantheta: My java's really not up to that though ...
(20:38:46) graphiclunarkid: I notice korikisulda hasn't been active on the project for a while - was thinking of dropping them a line and seeing if they think their java library would suit...
(20:39:10) stripe: Ive got a linux box on talktalk if any scripts need to be run
(20:39:10) graphiclunarkid: It would be really nice to get some use out of it since they put so much effort into writing it.
(20:39:17) dantheta: Yes, that is true.
(20:39:38) graphiclunarkid: Thanks Stripe! Actually there is one thing you could look at on TalkTalk if you want to help out...
(20:39:55) stripe: no problem just ask
(20:39:57) graphiclunarkid: stripe: we need to document how they behave when you request a blocked site.
(20:40:07) graphiclunarkid: stripe: Hang on and I'll dig out the github ticket...
(20:40:42) dantheta: (sorry, I might have closed that one last night... !)
(20:40:46) stripe: will do, normally a court order notice on the opal server in manchester
(20:41:19) graphiclunarkid: Hmm - looks like dantheta was way ahead of me! It was this one https://github.com/openrightsgroup/Blocking-Middleware/issues/12
(20:41:41) dantheta: We've also got some config for detecting HomeSafe blocking
(20:41:48) dantheta: It's awfully twitchy
(20:41:56) dantheta: It seems to block bad language
(20:42:13) graphiclunarkid: stripe: This would be for TalkTalk's default-on "parental controls" and not the court-ordered copyright-infringement-related blocking.
(20:42:57) stripe: ok have it bookmarked will read up on it and try no problem
(20:43:32) dantheta: I've just got one more bit, if that's OK -
(20:44:11) dantheta: I've got a proof of concept of realtime blocking detection on multiple ISPs using a messagequeue hanging off the back of the 1.2 API.
(20:44:34) graphiclunarkid: stripe: Thanks - even though dantheta has closed the ticket it would still be useful to see if what you find on your line matches what happened when he tried.
(20:44:39) dantheta: I'd like to install RabbitMQ on the ORG dev-censor-1 server, if that's OK
(20:45:16) graphiclunarkid: dantheta: Sure - go ahead. It's a dev server so do whatever you think is necessary.
(20:45:23) graphiclunarkid: dantheta: Also, what is RabbitMQ?! ;;-)
(20:45:39) ***graphiclunarkid googles
(20:46:49) graphiclunarkid: Oh right, a thing that does message queueing. Probably could have guessed that from the context and the name!
(20:47:16) stripe: graphiclunarkid no problem at all, results to github or mailing list?
(20:47:33) graphiclunarkid: Stripe: either is fine.
(20:47:45) stripe: ;)
(20:47:46) graphiclunarkid: stripe: Thanks!
(20:49:12) dantheta: I've also been looking at ooni data a bit!
(20:49:23) graphiclunarkid: dantheta: Cool - what did you find out?
(20:49:52) dantheta: I think I might have found a bug in the twisted network library ...
(20:50:03) graphiclunarkid: Heh!
(20:50:17) dantheta: I only found that last night though, so a little bit more research needed.
(20:51:02) dantheta: Oh, one more thing - I added a basic client example script to the API repository that demonstrates URL submission.
(20:51:18) vasilis: some news from me as well, I'm almost ready with the deployment script (fabfile) of ooniprobe.
(20:51:51) dantheta: Is that for the A&A servers, or rpi?
(20:52:20) vasilis: There is an interesting way of categorizing URL against Opendns()
(20:52:48) vasilis: http://hellais.github.com/syria-censorship
(20:53:09) vasilis: dantheta: For A&A servers.
(20:54:22) vasilis: I tested and added many Ubuntu flavours to the ooniprobe install script.
(20:54:47) vasilis: Re-organized a bit the Documentation.
(20:55:15) vasilis: Crash tested ooni-backend..
(20:56:34) graphiclunarkid: vasilis: I'm not sure how fabric works - does each application have its own fabfile, so that each can be run independently, or is it more like a single file per host that installs everything needed?
(20:58:33) vasilis: The RPi img script is ready, I'm going to add the raspi-config used to change hostname, password, expand the fs (and use the whole available SD card space) and add some ooni/ORG related feautures.
(20:58:41) dantheta: I'm sorry to seem contrary at this point - I'm not 100% convinced by fabric
(20:58:59) dantheta: Its features seem much lower-level than puppet or ansible
(20:59:31) graphiclunarkid: dantheta: More like an application config tool than a server config tool?
(20:59:34) vasilis: graphiclunarkid: I'm not a fabric expert, you can use one fabfile or multiple...
(20:59:43) dantheta: More like a way to run a set of commands on lots of hosts
(20:59:52) vasilis: dantheta: I just found it quite simple.
(21:00:31) vasilis: We can definetely use anything else, fabric doesn't require any software on host.
(21:00:38) vasilis: *running
(21:01:12) vasilis: I just took the fabfile from ooniprobe and started from there...
(21:01:21) dantheta: I'd prefer something that integrates with the target's package manager, and knows how to inspect the current state of the machine in order to make the minimum changes to reset its state
(21:03:28) dantheta: I can demo the ansible/puppet equivalent, if that would be useful?
(21:04:01) vasilis: dantheta: Nice!
(21:04:20) vasilis: dantheta: Let's do this off meeting?
(21:04:33) dantheta: Sure - I hope that would be OK. Sorry, I never like to sound negative!
(21:06:24) graphiclunarkid: dantheta: I'm keen to help out with the web integration stuff in Alexxx's absence - though I think he might be returning next week. Is there anything on that front that you're waiting for?
(21:07:12) graphiclunarkid: Vasilis: that website categorisation stuff looks like a neat trick - I couldn't see the source code though. Has hellais written anything up about it or did I just miss the link?
(21:07:38) vasilis: dantheta: Not at all :)
(21:07:57) graphiclunarkid: vasilis, hellais: I'm sure Chris from the Chokepoint project would be interested (he's been writing wikipedia pages about categorisation and posting them to the tech-vols list!)
(21:08:54) vasilis: graphiclunarkid: Not really I found it "randonmly", while I was looking for a way to categorize the URL lists.
(21:09:20) dantheta: graphiclunarkid: All good on the web front - I've got the ModX database to look at. The realtime checking demo that I was trying out requires direct submission to the API, but that is only at the demo stage.
(21:09:47) vasilis: It seems that lots of companies are using Opendns to categorize URLs.
(21:10:05) vasilis: I think they even offer a commercial service.
(21:10:25) graphiclunarkid: vasilis: I think there's some really interesting stuff to be explored about comparing how different filtering systems categories sites - after version 2.0 of course! I wonder how OpenDNS arrive at the categories...
(21:10:50) graphiclunarkid: dantheta: That's cool. I might try to close a few of the more mundane web-dev jobs over the weekend (e.g. updating headers and footers).
(21:15:19) dantheta: OK - give me a shout if there's anything I can help with.
(21:16:05) dantheta: I might also have sorted out a reasonably secure way for javascript on the blocked.org.uk page to submit to the API, but I'll need to experiment a bit.
(21:16:23) graphiclunarkid: dantheta: Neat!
(21:17:00) dantheta: It does require a helper PHP script on the blocked.org.uk page, but it does mean that the website's API credentials don't wind up publicly viewable
(21:17:07) dantheta: s/page/server/
(21:17:15) stripe: nice
(21:17:29) vasilis: graphiclunarkid: any news about the offered RPi devices?
(21:18:50) graphiclunarkid: vasilis: Regrettably that's still on my to-do list. I was hoping to get to it today but didn't get round to it.
(21:19:13) graphiclunarkid: vasilis: I'm kinda still waiting to hear back on the thread I started about MVOs versus MVNOs so I know what the scope of the proposal should be though.
(21:19:44) graphiclunarkid: If we only need to monitor four networks to cover all mobile filtering in the UK then it'll be easy street.
(21:20:21) dantheta: As far as I can tell, EE and Vodafone overlap considerably. If I can get my T-Mobile (EE) connection to stay stable long enough, I may even be able to prove it!
(21:20:22) graphiclunarkid: If we also need to monitor some MVNOs (e.g. virgin mobile, giffgaff) then we need to figure out coverage of user-base versus how much money we want to spend.
(21:20:48) graphiclunarkid: dantheta: Yeah - it's possible more than one MVO is using the same filtering provider.
(21:21:11) graphiclunarkid: There are 80+ virtual mobile network operators in the UK and we can't possibly monitor them all.
(21:21:34) graphiclunarkid: However if they all either offer no filtering (e.g. A&A) or their upstream MVO's filtering then we only need to monitor the MVOs.
(21:21:41) graphiclunarkid: That's just O2, EE, Vodafone and 3.
(21:23:35) stripe: sounds like a good starting point, then if anyone finds anything different the search could be extended as needed?
(21:24:36) dantheta: I'll see if I can dig up a virgin mobile SIM to see if their blocking is comparable to their upstream provider
(21:24:43) vasilis: graphiclunarkid: Well it could be that some users or organization would like to run probes in order to provide results from their network...
(21:25:47) graphiclunarkid: dantheta: That might be useful, cheers.
(21:26:16) graphiclunarkid: vasilis: Indeed.
(21:26:23) graphiclunarkid: stripe: Yes - I think we'll start off with at least that.
(21:27:06) graphiclunarkid: Folks I need to duck out for a bit to take a phone call but I'll be hanging out here for a while longer so feel free to carry one.
(21:27:08) graphiclunarkid: *on
(21:28:03) vasilis: Comparing same URL lists on different networks would be a really interesting idea as well ;)
(21:29:12) dantheta: vasilis: I've found the twisted error that's upsetting ooniprobe, I think
(21:29:53) dantheta: In twisted-web/http.py line 421
(21:30:12) stripe: vasilis and picking some of the more obscure blocks to reduce the searching on other networks to get an idea if they are using the same blocking mechanism
(21:30:36) vasilis: stripe: Yep!
(21:31:24) dantheta: vasilis: It tries to split the http header line by a ':', but there isn't a ':' in the line so it throws a ValueError
(21:37:06) dantheta: So, anyone have any thoughts on github vs. self-hosted facilities posted to the mailing list earlier?
(21:38:16) vasilis: I suggest to move to our own self-hosted services!
(21:40:04) dantheta: That could be cool, but would it make it harder to attract devs to the projects? Would it make us more obscure?
(21:41:36) graphiclunarkid: dantheta: My suggestion is that if we're going to host our own we should roll it out as a second technical project separate to the CMP.
(21:42:06) dantheta: I'm open to either option, I think. Self-hosting git is very easy. Trac provides issues and wiki. I guess the real decider for me is how much GitHub's network effect contributes
(21:42:27) dantheta: And that's something I don't think I can readily quantify
(21:43:44) dantheta: Anyway, just a thought.
(21:43:55) graphiclunarkid: dantheta: I would certainly like us to consider how we can make the "on-ramp" as shallow as possible for new contributors.
(21:44:10) stripe: could they both be run and mirrored, pleasing everybody?
(21:44:16) graphiclunarkid: But I don't think github is the only way of getting network effects
(21:44:35) graphiclunarkid: And anyway we can always keep github as a mirror of our self-hosted stuff - we'd still be independent of them.
(21:44:55) graphiclunarkid: Yeah, what stripe said!
(21:45:03) dantheta: graphiclunarkid: Always good to have backup options!
(21:45:27) stripe: dantheta well said
(21:46:08) graphiclunarkid: The main thing we need to kick this off if we're going to is people who want to start and work on the project ;-)
(21:47:54) dantheta: Infrastructure projects tend to be quite popular
(21:48:48) dantheta: Apologies, I'm going to have to toodle off shortly. Anybody need anything from me over the bank holiday weekend/next week?
(21:50:03) dantheta: I've just had a drive failure on the server upstairs!
(21:50:08) graphiclunarkid: dantheta: If you happen to hacking, look out for me in here and give me a yell if you're on, maybe there's stuff we can work on together.
(21:50:21) dantheta: Sure, will do!
(21:51:15) stripe left the room (quit: Quit: http://www.kiwiirc.com/ - A hand crafted IRC client).
(21:52:09) dantheta: Righto - have a good weekend all!
(21:52:14) graphiclunarkid: Cheers dantheta!
(21:53:01) dantheta left the room.