Google person finder integration

[jobinlawrance]

Any possuof utilizing Google's person finder to provide or extract information from google will be beneficial for both parties.

http://google.org/personfinder/2018-kerala-flooding

[biswaz]

Discussion needed

[jobinlawrance]

They have an API endpoint for it
https://github.com/google/personfinder/wiki/DataAPI

Check this out guys

[jobinlawrance]

They currently have around 4500 records last I checked

[bobinson]

Sharing private data to google even if in this crisis situation should be done ONLY after the checking the privacy implications.

@jobinlawrance - Can you confirm

1. This data will not be used for targetted ads ?
2. Can we PURGE this data - where is the APIs or method to PURGE the data ?
3. Which countries data protection laws are applicable ?

[jobinlawrance]

I doubt if Google will stoop that low, but given its Google I will get back on this.

[a-thomas-ny]

some of the Malayalee organizations are already pushing this... in other words , this is a problem too because now itself there are no central place for information gathering.

[jobinlawrance]

But given Google's reach it is a given that people have a lot of trust when it comes with Google. Let's use it to out benefit.

[kannankumar]

Which is the central place that government rescue officials are using now?? Is it keralarescue.in or google people finder or some other platform?
Won't it add to the chaos if there are multiple portals doing the same thing?

[vrthra]

Perhaps link it with #161? (The database of people at relief camps?)

[biswaz]

keralarescue.in is the most used. It is used by Kerala State IT Mission. But we've come to know some officials are forming their own spreadsheets

[jobinlawrance]

@bobinson check this out https://github.com/google/personfinder/wiki/TermsOfService

[a-thomas-ny]

The reason I am here because there is no central coordination... that is a big problem. My intention is to make a graphical reporting interface so that intensity of the issues in specific areas can be projected... share that info with all available whatsapp groups , coordinating MLAs, local teams etc,

[a-thomas-ny]

Now this message is spreading in all WhatsApp groups and organization groups...
"https://google.org/personfinder/2018-kerala-flooding
Google has launched in partnership with the government to locate those who need rescue , please share this link or fill it for someone you know needs rescue." If this is fake info, it should be reported to cyber crime

[a-thomas-ny]

if it is the right info, we should integrate

[kannankumar]

With the number of requests pouring in at keralarescue.in, we have to ensure they get the attention of rescue officials. [PRIMARY OBJECTIVE]
We need a single source of truth for Central coordination.

[a-thomas-ny]

last 48 hours I was struggling to dial the numbers circulating in WhatsApp to rescue my family members... all those numbers are not working or busy.

[kannankumar]

To integrate with Google Person Finder, what details does GPF have that helps rescue officials?
It looks like a crowdsourced platform to ask and add information about missing people and if they have been found. Not sure how it would help rescue officials.

[kannankumar]

@aswinmohanme and few others are working on a cluster view for the help requests that allows rescue officials to plan and get to severely hit areas.
https://github.com/aswinmohanme/kerala-flood-map/issues/6

[jobinlawrance]

At this point I think crowdsourcing is the only source, even officials are depending on that. It would be really helpful if we can channel it correctly instead of what the media is doing right now. @kannankumar

[a-thomas-ny]

in my opinion ongoing situation in Puthencavu, chengannur, Arattupuzha, mangalam, etc can be solved only with experienced fishermen/Army help and that should be coordinated through the government.
If the govt agreed to use person finder use it. people to people interaction is not helping much for last two days on the rescue operations... (not other aspects...)
There should be enough data to the government to request right help at the right time to the right place... I this rescuekerala should be able to provide that info with some advanced graphs and making an API end point to expose the info...

[arjun-pp]

Have shared an API key to 2018-kerala-flooding repo with @jobinlawrance. If this is necessary, how should we proceed this? Add their data to our database or update their db?

[jobinlawrance]

Yes thank you, we need to do both actually, both data are little different.

[arjun-pp]

So a lot of requests are being added in GPF(presently around 8400 records) A lot of requests with lat long being posted there with similar formats. An import of their data with basic deduplication would be beneficial IMO.

[galahad]

I think this is a great idea.

[jobinlawrance]

Guys I am almost done, need some help. Can you guys share the json response of request query as well as the request body while adding a new record.

[arjun-pp]

Its a CreateView Generic. So the request body should be same as the Request in serialized format.

{ 'dateadded': '2018-08-16T09:17:58.895Z', 'detailcloth': '', 'detailfood': '', 'detailkit_util': '', 'detailmed': '', 'detailrescue': '', 'detailtoilet': '', 'detailwater': '', 'district': 'ekm', 'id': 6765, 'is_request_for_others': False, 'latlng': '10.0021295,76.3230197', 'latlng_accuracy': '64.0989990234375 Meters', 'location': 'East kadangallur', 'needcloth': False, 'needfood': False, 'needkit_util': False, 'needmed': False, 'needothers': 'Rescue from water filled house', 'needrescue': False, 'needtoilet': False, 'needwater': False, 'requestee': 'Sreeram', 'requestee_phone': '094470581', 'status': 'new', 'supply_details': '' }

[biswaz]

Waiting for Dev from Google to join. License was added by their request.

[nworden]

Hi,
Does anyone know where the code in progress for this is? It doesn't look like Jobin's pushed commits up to GitHub yet.

[nworden]

All right, I don't know what the status of existing efforts here is, but I don't know where to find the code that's been written, so I'll start fresh. One obstacle is that Person Finder data is structured differently (has some fields Kerala Rescue doesn't and vice-versa). Two questions:
a) Would it be all right to add a free text field, so that we could at least include the data Person Finder from fields that KR doesn't have?
b) Do all the boolean fields (needwater, needfood, etc.) need to be populated? Person Finder doesn't have equivalents.

[binithb]

@nworden If I understand correctly you are looking to push GPF data to KR? If so how are you planning to do that? I guess there is no rest interface in KR that you can push to (or is there)? Would you like to have a REST interface at KR side so that you can POST GPF data easily?

[nworden]

My plan is to pull from KR, since Person Finder already has an API implemented that lets other apps read from it. Someone from KR has already requested a key for that API (and it was granted earlier today I believe), so we just need to add code to KR to use that API key to read from PF periodically.

[binithb]

OK. I think it will be a few more hours till the people who are working on this can join. I am afraid I cant give definitive information on this topic. But generally what I saw is people have been forking and submitting PRs. So if you need to make changes please fee free to fork and the admins can comment on how to take it forward. Thank you again :)

[nworden]

Ok, thanks a lot Binith. I'll give it a shot and hopefully send out a pull request before I go to sleep myself.

[nworden]

I have some code for importing from GPF here:
https://github.com/nworden/rescuekerala/tree/gpersonfinder

It's far from perfect and there's a few things to be aware of:

• GPF is for people, not exclusively rescues. A lot of records are people trying to connect with family/friends, posting messages saying they're ok, stuff like that. We're trying to figure out a way to have volunteers identify the rescue requests from among them.
• Related to the above: the data fields are pretty different. I tried to make up for it by putting a bunch of data in a new free text field. I didn't include all the fields we have; it might be a good idea to change it to include more.
• It's one-way (GPF->KR), and I haven't written any code to handle updates.
• There's around 20k records in Person Finder. I'm not sure about how many records you have right now or if it'd be a good idea to put them all in the DB at once. So, it's designed to do batches. You access the URL with start/end IDs, e.g., "start=0&end=10", "start=10&end=50", etc. so we can scale up cautiously.
• Data in Person Finder has privacy requirements around it; we can't include it in bulk downloads. So the GPF data is excluded from the /data page.
• There's a env variable for the API Key that needs to be filled in.

I'm about to head home; I'll be online for a little bit after that and then I'll be offline for a while (my timezone is GMT-4).

[nworden]

Oh, another very important note: I was not able to get SSL working locally, which meant I could not test the bulk data change (you need SSL for that page). It's a simple change, but obviously it needs to be tested before being brought into the master branch. I didn't even want to make a pull request without being able to test it.

[biswaz]

I'm worried about the differences in the data structure. Should look deeper into it

[nworden]

We estimate that over 10k of the records in our DB are rescue requests. We understand that it would be a problem to have a lot of duplicates in the KR database, but right now there's a lot of data stuck in the PF database and that's a problem too. How good does the deduplication need to be to be a good balance between solving those two problems? Could we just look for exact name matches? Are most of the records in KR self-requests (i.e., not requests for someone else)?

[arjun-pp]

@jobinlawrance isn't replying and I don't see any progress in his profile. If anyone needs the key for integrating/testing this. Ping me at slack with your fork. I have analyzed the data too. Will contribute to cleaning, integrating and normalising the data.

[nworden]

As discussed on Slack, I've written a new import script in a different branch (has a 2 at the end):
https://github.com/nworden/rescuekerala/tree/gpersonfinder2
It writes to a separate table. I'll add some basic filtering and dupe-checking later tonight (mid-morning IST).
There is still an open question of what is going to happen to this data once it's in these separate tables. @biswaz can you comment on the plan for that?

[nworden]

https://github.com/nworden/rescuekerala/tree/gpersonfinder2 now has basic rescue request filtering and untested dupe detection. As mentioned in Slack, the admin page doesn't do Request or Person records for some reason, and the user interface seems to be having some problems with phone numbers, so I can't load my local DB with sample KR data to test dupe detection with. That was the only reason I didn't think it was ready to open a pull request; besides that I think it's ready to go.

There's two parts to PF data (person records and notes). They're imported separately, person records first, and you can do it in stages by hitting these paths:
http:///gpersonfinder_import/?max_results=123&type=persons
http:///gpersonfinder_import/?max_results=45&type=notes
(there's tens of thousands total, so of course max_results should increase as long as you're able to import a few small batches without problems first)
That page is only accessible to superusers. Be aware it's kinda slow (maybe like 1k/minute), and the dupe detection will probably make it even slower.
It'd probably be good to set up a cron job once it's backfilled. I didn't design it to be thread-safe, so I wouldn't set up a cron job until you're done backfilling the bulk of it.

Again, please let me know if there's anything else I can help out with. We really want to get this data to people who can use it. I'll be back online in my morning, and Kevin can page me if there's anything urgent.

[jobinlawrance]

Guys sorry about the late reply. Check the endpoint out
http://keralarescue4917.cloudapp.net/search?name=jobin

[jobinlawrance]

Further work and discussion is required

[jobinlawrance]

@bobinson @arjun-pp do take a look and let me know how we can use this data. Should i start with analytics and deduplication.