ORG-tech-vols IRC meeting log 2014 05 12

(20:33:35) vasilis: Hi everyone
(20:34:43) vasilis: graphiclunarkid
(20:35:51) graphiclunarkid: Hey vasilis - how's it going?
(20:36:10) vasilis: graphiclunarkid: Fine, how are you?
(20:36:19) graphiclunarkid: Good thanks.
(20:36:25) graphiclunarkid: Not sure who else is around?
(20:37:57) graphiclunarkid: Perhaps I should start sending out reminders again - to myself also!
(20:39:27) dantheta [~daniel@dsl-217-155-42-217.zen.co.uk] entered the room.
(20:39:53) graphiclunarkid: Evening dantheta
(20:39:59) dantheta: hey - sorry I'm late!
(20:40:05) vasilis: Hi dantheta
(20:40:09) dantheta: Hello!
(20:40:13) graphiclunarkid: No problem - I was a bit late too.
(20:41:39) vasilis: dantheta: Did you manage to connect to the A&A VMs?
(20:41:57) dantheta: Yep, they're back up and running as of this morning - bit of routing wierdness, I think
(20:42:24) vasilis: Did you initiated any tests?
(20:43:29) dantheta: I haven't yet - I don't think that the filters are enabled on some of the lines, but I haven't checked that today
(20:43:46) dantheta: I did the alexa 10k on talktalk and A&A the week before last
(20:44:03) dantheta: We needed sudo to be able to do full rollout (although I managed to work around that for a while)
(20:44:39) vasilis: I guess it makes more sense to run alexa top 1m!!
(20:44:43) dantheta: It would be good to ask Paul @ A&A to switch on the filters (which probably requires ISP account access) on BT, Sky
(20:45:01) dantheta: Yep, I've clocked the python probe at 3,000 requests per hour
(20:45:30) dantheta: They can probably do more!
(20:45:34) vasilis: We can test both cases filter and unfiltered.
(20:45:42) vasilis: dantheta: Which probe?
(20:45:53) dantheta: The one based on the old android probe
(20:45:57) dantheta: (but in python)
(20:47:30) dantheta: I'm nearly ready to demo real-time probe results
(20:48:13) vasilis: Nice!
(20:49:48) vasilis: Do you need any help testing things?
(20:50:14) graphiclunarkid: dantheta: I think plett was going to dig out the login details for each ISP so we could check the filter settings.
(20:50:57) dantheta: Yep, and that would also allow us to co-ordinate changing the settings between us, so that we can get results of strict/moderate/light settings
(20:51:27) dantheta: vasilis: OK for the moment, but it will be good to get ooni results too and compare!
(20:51:32) graphiclunarkid: dantheta: Indeed. Though for the release at the end of the month I think the default settings would be best.
(20:52:27) vasilis: dantheta: OK I'll setup all boxes with ooniprobe
(20:52:35) dantheta: yes, I thinkmoderate is the default on BT, I'm not sure with the others
(20:56:36) dantheta: (sorry, just finishing a bit of food - late meeting at work)
(20:56:57) dantheta: website integration is also going pretty well with Matt. Results display next, I guess!
(20:57:32) graphiclunarkid: Yeah - it seems you to got a bunch of useful stuff working over the weekend :)
(20:58:46) dantheta: Yep - I thought I'd do a similar example client for fetching results
(20:59:08) dantheta: (or perhaps I already did). Seemed like a good approach, although Matt's got the idea
(21:00:39) graphiclunarkid: Coolio :)
(21:02:03) vasilis: Has anyone tested lepidopter?
(21:02:25) dantheta: I've had it running on Vodafone & T-Mobile, although vasilis was driving :)
(21:03:35) dantheta: Just looking over the issues list - if we can get the account details for the ISP lines, then we can switch on filtering and close #8, #9, #18 and #19
(21:03:59) dantheta: I'm still working on #6.
(21:04:56) ***graphiclunarkid looks up the issue numbers...
(21:05:12) dantheta: Sorry - #6 is batch feeding URLs to ooni
(21:05:20) graphiclunarkid: Ah yes, feeding URLs to ooniprobe.
(21:06:59) dantheta: I was also thinking that there are a couple of admin functions in the API that haven't been covered anywhere else yet: approving new probes, managing users etc. Should I open issues for these?
(21:08:58) dantheta: We should have UI for these, perhaps an admin control panel. Not essential for release, I don't think.
(21:09:24) graphiclunarkid: Yep - I think you're right.
(21:09:57) graphiclunarkid: Once we have the initial version up and running we can decide how to open up the system to new probes.
(21:12:09) dantheta: Yep, that sounds cool.
(21:12:21) dantheta: The website has got its credentials now, so that's one bit sorted.
(21:12:57) dantheta: How are things going with frontend?
(21:14:22) graphiclunarkid: Not much has changed but now Alexxx is back hopefully he'll find time to pick things up again.
(21:14:44) graphiclunarkid: I don't believe there's much left to do now though. It's largely adjusting and writing copy.
(21:15:03) graphiclunarkid: We also need to figure out how to transfer the site from the dev server to staging.
(21:15:22) dantheta: Is that just the modx site, or is the API code and database moving too?
(21:16:10) graphiclunarkid: Just the modx site I think.
(21:16:28) graphiclunarkid: We should have a care for the security of the server on which the database and API reside though.
(21:16:33) dantheta: That makes sense - the DNS and SSL cert already set up pointing to this host.
(21:17:01) graphiclunarkid: I believe we have a new SSL cert ready to go for dev-censor-1. I need to check with Lee what's happened to this.
(21:17:38) dantheta: I'm going to add authentication and ACL to AMQP
(21:17:52) graphiclunarkid: Cool.
(21:17:58) dantheta: I'd like to merge the AMQP enhancements into the main branch soon, if that's OK
(21:18:05) graphiclunarkid: I was also thinking of hardening the OS and underlying services.
(21:18:36) dantheta: The firewall's a little bit on the open side at the moment. I can help with SSH config, firewalls, MySQL security if you like
(21:18:50) dantheta: Plus there's always a few extra services to shut down with RHEL/CentOS
(21:19:35) graphiclunarkid: I'm not experienced with RHEL / CentOS - I'm familiar with Debian though.
(21:20:43) graphiclunarkid: I'll add a github issue for hardening the server and we can add comments to it with things that we think of.
(21:21:13) dantheta: Sure, that's cool. I've got an ansible playbook for deploying services and managing config for things that the API uses.
(21:21:56) dantheta: I was thinking that since the config repository will hold things that we don't want to share with the public (keys, etc), perhaps sticking a git repository on the dev host itself to contain the config would be useful?
(21:22:13) dantheta: Accessible over SSH only
(21:22:41) graphiclunarkid: Yep ok.
(21:23:35) dantheta: Anything else I can take a look at over the next week?
(21:23:43) vasilis: I have modified the "official" raspi-config script and added ooniprobe related things.. in case that you have any things to be added for ooniprobe or blocked reports.
(21:24:12) vasilis: https://github.com/anadahz/raspi-config_lepidopter
(21:25:46) dantheta: Have you been able to look at ORG's ooni-backend?
(21:26:00) graphiclunarkid: dantheta: Not for me - Just keep on doing what you're doing :)
(21:26:07) dantheta: :)
(21:27:23) ***graphiclunarkid goes to take a look at vasilis's repo.
(21:27:27) dantheta: vasilis: do you have any examples of ooni reports collected at the backend? Are they the same as the reports that get created and stored on the probe
(21:27:32) dantheta: ?
(21:28:07) vasilis: dantheta: well I 've asked hellais_ to make an official (ORG powered) ooni-backend..
(21:28:09) vasilis: I *
(21:28:38) vasilis: sorry it took longer than I thought..
(21:28:57) dantheta: Not at all, it's all good
(21:29:21) vasilis: dantheta: Not exactly the same.. I can give some examples.
(21:30:45) dantheta: I just wanted to make sure that I was looking at the right reports - a couple of weeks ago I had a script that tried to summarize ooni reports into yes/no block reports
(21:31:30) vasilis: dantheta: OK are you going to be online later today?
(21:32:05) dantheta: Yep, after 21:30 UTC
(21:32:06) graphiclunarkid: vasilis: Thanks for setting up the raspi-config stuff. Should we maybe raise issues for the changes and features we want to make?
(21:32:22) vasilis: graphiclunarkid: Definetely!
(21:32:48) dantheta: (back in a mo)
(21:33:02) vasilis: After months of running lepidopter (the custom images) it seems that the Raspberries are pretty stable!
(21:34:07) graphiclunarkid: vasilis: I think this raspi-config might also need some brief explanation as to our aim. How about we update README.md to indicate where it fits into the project?
(21:34:18) graphiclunarkid: vasilis: Nice to hear it's stable :)
(21:34:36) vasilis: There are some running for more than 2 1/2 months without issues ;)
(21:34:46) graphiclunarkid: Awesome :)
(21:35:02) graphiclunarkid: I'm lucky if I can get my debian server to do that ;-)
(21:35:17) graphiclunarkid: (Though it's usually me fiddling with it that breaks it.)
(21:35:17) vasilis: graphiclunarkid: I have some TODO as well but please raise requests and issues.
(21:35:29) graphiclunarkid: Will do.
(21:37:50) dantheta: Back again -
(21:37:57) vasilis: graphiclunarkid: Any news about the "spare" RasPis?
(21:38:40) dantheta: vasilis: I didn't have any luck running rasp with a vodafone dongle plugged straight in - power issues, but with a powered hub it should be fne
(21:39:17) graphiclunarkid: I spoke with ORG staff last Thursday. I think we are going to go ahead and order enough kit to cover the four MNOs in the UK.
(21:40:38) dantheta: I've got a recipe for running multiple mobile broadband connections from one server, if that helps (and can daisy chain raspis behind that using vlan)
(21:41:22) vasilis: dantheta: Did you need to load any modules for the dongle to run?
(21:41:53) dantheta: It did need to load modules, but they didn't need to be compiled or installed, they were already on the image
(21:42:02) dantheta: just usb-serial and one other thing
(21:42:13) dantheta: then install wvdial & pppd, and away it goes
(21:42:17) vasilis: dantheta: Nice please provide this recipe.
(21:42:23) graphiclunarkid: dantheta: I think that would be useful if we have a server already, or could repurpose one, but Raspis are so cheap plus we have lepidopter to run on it. I think we should go that way in the short term. When we start adding more probes we can definitely experiment though!
(21:42:50) vasilis: dantheta: This should be added on the raspi-config then, please open a request if you can.
(21:43:12) dantheta: OK, will do. The config for wvdial varies by mobile operator, but it's pretty straightforward
(21:44:11) dantheta: I was very pleased with my FireFox OS phone - debian stable picked up the USB tethered interface straightaway and it was ready to run immediately
(21:44:23) vasilis: graphiclunarkid: What about this company that could provide us with a bunch of RasPis?
(21:45:16) graphiclunarkid: If they end up sponsoring us it will probably be by way of reimbursement - ORG are happy to pay for the kit up front, I think.
(21:47:30) vasilis: Do you have some time to talk about the infrastructure?
(21:48:13) graphiclunarkid: vasilis: Sure
(21:49:40) graphiclunarkid: dantheta: I've only seen a couple of FireFox OS phones "in the wild" - one of them was at Mozilla HQ at the kick-off meeting ;-)
(21:50:51) vasilis: We need a private git repo as well, the need arised as dantheta wanted to share some server recipes examples.
(21:51:15) dantheta: I was thinking we could just have a bare repo on the dev server for the time being
(21:51:46) vasilis: OK I was thinking if it's better setting a Trac-like system.
(21:52:40) vasilis: To have tickets, wiki, and repos integrated in a way.
(21:52:41) graphiclunarkid: I understand there's a git plugin for trac.
(21:53:57) graphiclunarkid: dantheta: Yes - we can certainly do that without difficulty to meet the immediate need
(21:54:23) vasilis: If it's too much work for the time being we can still postpone it and set up some ad-hoc services.
(21:54:25) graphiclunarkid: vasilis: Maybe we should start a wiki page to flesh out ideas for doing a self-hosted infrastructure project?
(21:55:37) graphiclunarkid: It's a matter of priority rather than time I think. As long as we don't delay the CMP v2.0 we can be pursuing other things alongside.
(21:56:01) graphiclunarkid: I don't want to move the dev infrastructure for CMP until after v2.0 is live though - I think that would be a distraction.
(21:56:15) graphiclunarkid: But that doesn't mean we can't set up services etc. in advance.
(21:57:04) vasilis: You are right.. let's not distract the dev process at the moment.
(21:58:08) dantheta: BTW, I've installed git, screen and supervisor on the A&A hosts
(21:58:30) vasilis: dantheta: with ansible recipes?
(21:58:36) dantheta: Yes, hope that's OK
(21:58:45) graphiclunarkid: dantheta: supervisor?
(21:59:11) dantheta: It's a process manager - if you can't trust your processes to stay running from an init script, it makes sure they stay running
(21:59:30) graphiclunarkid: Ah right, fair enough!
(21:59:47) dantheta: It also takes care of output redirection and privilege dropping
(22:00:59) dantheta: I'll send round the repo with the ansible recipes - it will co-exist with fabric just fine, if you want to use that.
(22:02:23) vasilis: dantheta: I have experiment with ansible as well..
(22:03:30) vasilis: no need to use fabric
(22:03:37) dantheta: OK, cool!
(22:03:38) graphiclunarkid: That would be useful. TBH I could do with learning a spot of ansible for other reasons besides this project!
(22:03:55) mkillock [5c133e52@gateway/web/freenode/ip.92.19.62.82] entered the room.
(22:04:04) dantheta: I really like it - much lighter weight than puppet
(22:05:00) mkillock: I guess puppets are easier to carry than blow ups? ;)
(22:05:16) mkillock: umm hi
(22:05:28) mkillock: first thing I read was the puppet thing
(22:05:28) graphiclunarkid: :D
(22:05:39) graphiclunarkid: Hi mkillock - how's it going?
(22:05:50) dantheta: mkillock: lol!
(22:06:13) vasilis: mkillock: hi
(22:06:14) mkillock: hi, fine thanks, Hi dantheta. How are you both?
(22:06:18) mkillock: hi
(22:06:51) dantheta: All good thanks, how's it going
(22:07:13) mkillock: yeah, alright, I was wondering about whether I can close some of those github issues
(22:07:28) mkillock: the frontend integration
(22:08:24) mkillock: just wondering what your thoughts are with it all being PHP snippets outputting to placeholders
(22:08:47) mkillock: is there a problem with that?
(22:08:50) graphiclunarkid: mkillock: I think if you've implemented stuff against those issues you can close them.
(22:09:02) graphiclunarkid: And if we need to do more later we can raise new tickets for that.
(22:09:07) dantheta: The URL submit one looks fine to me
(22:09:15) graphiclunarkid: dantheta is probably best placed to comment on the technicalities.
(22:09:18) dantheta: Works lovely!
(22:09:36) mkillock: ok, thanks, yes well I was using your code really!
(22:09:39) dantheta: For the Modx parts, I'm happy if you're happy :) I don't know much about ModX at all
(22:10:46) mkillock: ah, well, modx is really based around writing PHP code outputting to MODx variables that put strings in amongst HTML
(22:11:32) mkillock: Shall I put the result of the API call into the formsave table?
(22:11:51) dantheta: The API response, you mean?
(22:11:55) mkillock: yeah
(22:12:08) mkillock: for syncing later, if necessary
(22:12:14) mkillock: i.e. search for failures
(22:12:19) dantheta: Can do - the uid that comes back from the API would be useful if we ever have to reconcile
(22:12:27) dantheta: Snap!
(22:12:41) mkillock: :)
(22:12:46) dantheta: I'm really sorry, I'm going to have to toddle off in a mo!
(22:12:48) mkillock: ok, I'll do that
(22:12:51) mkillock: oh, ok!
(22:12:56) dantheta: (emerging family situation, so sorry!)
(22:13:04) mkillock: umm well,
(22:13:07) dantheta: I'll be on later
(22:13:07) mkillock: no worries
(22:13:09) mkillock: ok
(22:13:19) graphiclunarkid: dantheta: Cheers - see you later.
(22:13:20) dantheta: Sorry everyone!
(22:13:22) dantheta left the room.
(22:14:10) mkillock: ah well, I might have some questions you can answer, graphiclunarkid
(22:14:23) mkillock: let me check github - one mo
(22:14:35) ***vasilis idling
(22:14:52) vasilis: if you need anything just ping me
(22:14:59) mkillock: sorry if I've butt-in a convo
(22:15:13) mkillock: didn't think
(22:15:46) graphiclunarkid: Yep, shoot, mkillock
(22:16:00) graphiclunarkid: We were just talking about ansible before you arrived.
(22:16:26) mkillock: I'd have to google that
(22:17:03) graphiclunarkid: It's a thing to automate configuring servers
(22:17:12) graphiclunarkid: installing software, adding users, etc.
(22:17:36) graphiclunarkid: We're using it to keep the config of all 6 A&A servers identical.
(22:17:41) graphiclunarkid: And to save us time.
(22:19:48) mkillock: ah, righto. was just reading about it.
(22:20:11) mkillock: ok - first question!
(22:21:03) mkillock: user submits URL, goes to results page - this is only if the URL has been checked already? https://github.com/openrightsgroup/blocked-org-uk/issues/29
(22:21:42) mkillock: ok
(22:21:47) mkillock: that's obvious from the issue
(22:21:52) mkillock: not a question
(22:22:09) graphiclunarkid: Yeah, I think that's right.
(22:23:13) graphiclunarkid: We may of course decide to change it once it's built - if we look at it and decide it should do something different - but that's the first guess anyone's made and I'm not going to second-guess it!
(22:23:45) mkillock: ok, so I think the question is: What are we doing when URL hasn't been submitted before? Emailing the person when the results come in?
(22:24:30) graphiclunarkid: I think we're doing that in both cases.
(22:24:39) graphiclunarkid: In the case where we already have results we'll do two things:
(22:24:44) graphiclunarkid: 1. Display the historical results
(22:24:52) graphiclunarkid: 2. Queue a new check
(22:24:58) graphiclunarkid: That lets people discover whether things have changed.
(22:25:38) mkillock: ah, ok. So that answers one of my other questions: What if the URL is no longer blocked
(22:26:04) mkillock: ok, so I need to study the api docs to check for a response that indicates previous submission
(22:26:20) mkillock: can you give a URL that has been previously tested?
(22:26:58) graphiclunarkid: Hmm. I'm not sure how the API works from this angle.
(22:27:47) graphiclunarkid: I think we'd like people to be able to check any URL regardless of whether it's been checked before.
(22:28:10) graphiclunarkid: The only difference will be that if we have previous results we'll display them.
(22:28:21) mkillock: sure, it's just that I wasn't sure if the probing is going on at the moment?
(22:28:23) graphiclunarkid: However I'm not even sure that's essential for the minimum viable product - just nice.
(22:28:48) graphiclunarkid: I believe vasilis and dantheta have been doing some test probing with the Alexa top 10k.
(22:28:56) graphiclunarkid: So there will be results in the database already when we launch.
(22:29:30) mkillock: right, so the url I have been using should have been tested as well?
(22:29:53) mkillock: I didn't receive any email with results
(22:30:01) graphiclunarkid: It makes sense to check popular websites ahead of time so we can show immediate results I guess.
(22:30:22) graphiclunarkid: I'm not sure how they've been doing it. Probably the whole thing isn't wired up end-to-end yet.
(22:30:35) graphiclunarkid: I don't think the "email out results" feature is implemented yet either.
(22:30:51) mkillock: ok! Let me query the api with an alexa top 10k site then
(22:31:29) graphiclunarkid: vasilis posted a 1-liner to the list that will grab the Alexa top 10k from their site for testing purposes.
(22:31:31) mkillock: ah right, all things like google, facebook etc
(22:31:36) graphiclunarkid: Yeah.
(22:32:09) mkillock: ok. cool, that's enough for me to get started!
(22:32:41) graphiclunarkid: Awesome :)
(22:32:47) dantheta [~daniel@dsl-217-155-42-217.zen.co.uk] entered the room.
(22:33:07) graphiclunarkid: dantheta: Everything OK?
(22:33:12) dantheta: All sorted
(22:33:19) mkillock: thumbs up
(22:33:21) graphiclunarkid: Cool.
(22:33:26) dantheta: Vulnerable relative not answering phone - scramble
(22:33:32) jimkillock [~jimkilloc@cpc3-hari15-2-0-cust66.20-2.cable.virginm.net] entered the room.
(22:34:00) mkillock: hello
(22:35:13) graphiclunarkid: dantheta: mkillock and I were just discussing the finer points of API integration with the front end.
(22:36:10) mkillock: I figure I'll set up a test page that will become the submit results page, and have a go at querying the API history
(22:37:00) graphiclunarkid: mkillock: That sounds like a good idea.
(22:37:06) dantheta: I think there's an example client page for querying url history
(22:37:51) mkillock: For GET /status/request/{url} ?
(22:38:23) dantheta: This one is for /status/url (example-url-query.php)
(22:38:42) mkillock: ah, right, yes. Just looking at wiki
(22:39:05) dantheta: /status/request (slightly confusing naming scheme) is an admin function for getting submitter info
(22:39:22) dantheta: That's the one that returns the $additional_info that you stash away from the user form
(22:39:33) dantheta: /status/url is for public consumption
(22:39:33) mkillock: ok
(22:39:41) mkillock: right
(22:39:59) dantheta: I do need to clean up the headings a bit, sorry!
(22:40:04) mkillock: looks like I can reuse the same signing function
(22:40:16) dantheta: Yep, it's fairly generic server-side as well
(22:40:29) dantheta: (or API-side), I should say
(22:41:15) mkillock: ok!
(22:41:19) dantheta: The /status/url results are pretty raw straight from the DB - if we want any further refinement, like a single result per ISP (preferable the most recent result) we can do that too. I'm happy to take suggestions!
(22:41:54) mkillock: ah, right. So there can be multiple sections for each blocking provider?
(22:42:40) dantheta: There can be, yes. Just a list of dictionaries containing (network_name, created, and status) in no particular order
(22:43:09) mkillock: mm ok, so if a URL has been submitted 20 times, will there be 20 sets of results?
(22:43:31) dantheta: If a URL has been tested 20 times, you'll get up to 20 results per ISP
(22:43:47) dantheta: It's probably something we'll need to get a handle on :)
(22:44:06) mkillock: :)
(22:44:25) mkillock: ok, well I guess the website user will only need the latest for each ISP
(22:44:33) mkillock: I can filter the results
(22:44:50) dantheta: It's fine - I'll do it my side, save bandwidth and browser memory
(22:45:14) mkillock: ok, that's cool
(22:45:17) mkillock: thanks
(22:45:26) mkillock: well, I can get started anyway
(22:45:43) dantheta: MySQL isn't great at these, but it's definitely doable.
(22:46:09) dantheta: Yep - I don't think you'll have to worry about a huge number of results for a while
(22:46:25) dantheta: but we are aiming to periodically re-test URLs in a background process
(22:46:49) mkillock: do we want to record the fact that a site was originally blocked, and for how long?
(22:47:00) mkillock: or rather, show the user?
(22:47:14) dantheta: We will have that in the database, but I don't know how or where we show that to the user.
(22:47:35) mkillock: Well, that's a step on from where we are anyway!
(22:47:55) mkillock: I can just show the results you give me from the API, and that can be the last result, right?
(22:48:52) dantheta: Yep, that's right. In a later version, we could allow a click-through on the ISP name to show the history of the URL on that ISP, but that's only an idea.
(22:49:20) mkillock: I was thinking on the submit page - like, showing Virgin blocked this on X for Y days, current status blocked/ok
(22:49:39) dantheta: Yep, we could do that too.
(22:50:02) dantheta: I can give you back the results in ISP ascending, date descending order
(22:50:10) dantheta: so that the latest result is at the top of each ISP block
(22:50:40) dantheta: if that would be useful?
(22:50:45) mkillock: ok, sure
(22:50:58) mkillock: I can filter the results
(22:51:07) mkillock: do a bit of analysis
(22:52:09) dantheta: I'm just wary of having to send relatively large amounts of JSON data if some if it is going to be discarded - but at the same time, we don't want to wind up building a full reporting system into a JSON API endpoint!
(22:52:13) mkillock: but if the results are in date order then it's easier searching through
(22:52:54) dantheta: OK, that sounds cool. I'll fix the ordering and we can see how it goes from there.
(22:53:25) mkillock: perhaps we could limit the results by latest user submission + periodic automatic checks?
(22:53:33) mkillock: + original submission
(22:53:54) mkillock: would that still be large amts of data?
(22:54:23) dantheta: It could be - also the results table doesn't differentiate between rechecks and checks in response to a user submission
(22:55:13) mkillock: ok, well I don't really know what to suggest, sorry! If sending all the data across is OK with you, then it's OK with me
(22:56:32) dantheta: The simplest starting point is latest result for each ISP, though I like the idea of having a bit of history in there as well.
(22:58:14) dantheta: I'm happy with either, really :) So many possibilities!
(22:58:26) mkillock: :) I'm thinking
(23:01:37) mkillock: I like the idea of showing that a URL was blocked by the broad brush blocking bollocks, rather than create the impression that all is well with their efforts, just becase it's ok now
(23:02:27) mkillock: so I reckon I want to show the last time a URL was blocked if it is not currently blocked
(23:03:24) graphiclunarkid: mkillock: Good idea - and also it's interesting to see if a particular URL is flip-flopping between blocked and otherwise.
(23:03:54) mkillock: So a count of number of times blocked-then-unblocked?
(23:03:56) dantheta: mkillock: yep, that's perfect - one entry per ISP, current status, timestamp for current status, and timestamp of last blocked result
(23:04:46) dantheta: mkillock: That one is beyond the abilities of SQL to do, it would require offline analysis. We can definitely do that, and store the results back to the database for retrieval, but it isn't something that we can get straight-off
(23:06:05) mkillock: ok, so are you proposing to make the API provide: 'one entry per ISP, current status, timestamp for current status, and timestamp of last blocked result '?
(23:06:51) mkillock: or should I do that?
(23:06:52) jimkillock left the room (quit: Quit: jimkillock).
(23:07:19) dantheta: mkillock - I'll do that, it'll become the standard behaviour of /status/url.
(23:07:38) mkillock: ok, cool! that solves the problem of sending too much data too!
(23:07:43) mkillock: nice one
(23:08:25) dantheta: Yep, I'm happy with that. Many thanks!
(23:10:15) mkillock: ok, sorry if that messes up your API spec!
(23:10:42) mkillock: will you put this into 1.2?
(23:11:31) dantheta: Not at all - 1.2 isn't quite frozen yet. I also know that there are no consumers for status/url yet :)
(23:11:50) mkillock: ah, good!
(23:12:20) dantheta: As we get closer to launch day I'm trying to avoid introducing incompatible endpoint signature changes, but some of the semantics are most definitely still up for discussion!
(23:12:46) mkillock: :)
(23:13:49) mkillock: ok then! Thanks for your help and accommodating my request!
(23:14:20) mkillock: It's time for me to quit for the night, now so, cheers all!
(23:14:38) mkillock: night!
(23:14:41) dantheta: 'Tis all good - it's a very good plan. It should be live on the API in the next day or so, but in the meantime the old behaviour is sufficiently similar for early developement.
(23:14:45) dantheta: Thanks again, and good night!
(23:15:10) mkillock left the room (quit: Quit: Page closed).
(23:15:18) graphiclunarkid: night all :)
(23:15:57) dantheta: Yep, I fear after the earlier excitement I too am pooped.
(23:16:04) dantheta: Vasilis: is it OK if we catch up in the next day or so?
(23:16:13) graphiclunarkid: It's been a long un - but productive I think.
(23:16:22) graphiclunarkid: I'll post the logs when I get a minute.
(23:16:26) dantheta: Yeah!
(23:16:37) dantheta: All good - that way I can catch up on the half-hour I missed.
(23:16:42) dantheta: Have a good evening all.
(23:19:16) dantheta left the room.